NATIONAL RAILROAD PASSENGER CORPORATION 1 Massachusetts Avenue, NW, Washington. DC 20001 Tel 202.906.2486 Fax 202.906.2850 AMT RA Staphen J- Gardner Senior Executive Vice President. Commercial, Marketing, and Strategy March 29, 2019 The Honorable Daniel Lipinski United States House of Representatives 2346 Rayburn House Of?ce Building Washington, DC 20515 Dear Representative Lipinski: Thank you for your letter to Mr. Anderson of March 8, regarding the February 28 signal outage in Chicago Union Station (CUS). I am answering on Mr. Anderson?s behalf. All of us at Amtrak are mindful of how many people were affected by the delays that resulted from this disruption, and I assure you we are taking immediate, concrete steps to ensure the causes of this event are addressed. To ensure that you have a complete understanding of the circumstances that led to this disruption, let me explain the sequence of events of February 28 in detail. Amtrak maintains both an older pair of dispatching system servers at the Chicago Control Center and a newer pair of servers in the 14th Street backup control center. On the day of the incident, the communications and signals to control Chicago Union Station were operating from server in the 14th Street facility. The system was con?gured for the second, or server, at 14th Street to serve as the primary backup, and it was operationally ready to assume control should anything happen to the ?rst server. The, two older servers in the Chicago control center usually serve as additional backup to cover the dispatch functions should anything happen at 14th Street. However, as we were conducting work associated with positive train control (PTC) implementation on February 28, both of the older servers at the control center were being used to test Michigan Line PTC operation. From a policy and procedure standpoint, personnel are trained to understand that any signal?related maintenance or upgrades must not be performed during rush hours. Nonetheless, on the day of the incident, an inexperienced manager authorized an experienced senior technician to install new PTC hardware in the Chicago control room. This new hardware was to be installed in the same rack as equipment that handled communications between the Chicago control center and the 14th Street control center. The technician accidentally shorted the communications equipment, which disabled the primary and backup systems in place at 14th Street. The additional backup at Chicago control center was not available to serve as a backup as it was con?gured for PTC testing. Despite extensive efforts to ensure every resource was directed to the fastest possible restoration of service, it still took our employees over 12 hours to ?x the problem and restore complete automatic control over the switches. The Honorable Danie! March 29, 20M Page 2 Clearly, the disruption should not have taken place and the cause for the disruption should not have occurred. Nonetheless, the fact remains that Amtrak has performed signi?cant work over the past three years to prepare the tracks we control in the Chicago and Michigan area for PTC operation. We have done this while the system has continued to manage Amtrak?s 58 daily trains along with Metra?s 240. Many of the systems in place in Chicago were designed and originally installed long before PTC was conceived or mandated, and it has been a dif?cult process to integrate, test, and verify this vital new safety system. The underlying issue is not the reliability of our systems, but the dif?culty of adopting this new technology. Signi?cant planning and work have gone into doing so with an absolute minimum of disruption to service, and we deeply regret this mistake and its consequences for those who rely on us, either directly or indirectly. Regarding our communications, Amtrak noti?ed both Metra and BNSF of the issue as soon as it took place, in addition to the Metra representative who was on site. Within a few minutes of the outage just after 8:30 am, Metra informed its customers of the event and attributed it to the failure of the Amtrak dispatching system. Amtrak con?rmed that in its messaging to the news media. Coordination continued between Metra Media Relations and Amtrak through the day. By noon, Metra advised its customers to use alternate routes rather than those at Union Station, particularly the BNSF line. Amtrak had similar messaging in communications to news media and t0 the public. At 4:30 pm, Amtrak reached out to the media to acknowledge the repairs were not yet complete, to apologize, and to continue to accept responsibility. This was also designed to keep the news media from obstructing the North and South Concourse with camera crews during early evening news programs. Metra participated in that brie?ng and again attributed the outage to Amtrak. A plan was announced to update the media and the public that evening regarding the following morning?s rush-hour plans. Both carriers updated the public and the news media by 8:45 pm, informing the public and the news media that a normal morning rush was expected the next day. To address your list of questions speci?cally: I Planning and scheduling surrounding software and hardware system upgrades are being reevalu? ated as part of the work being done by a cross?disciplinary Amtrak team that was set up following February 28. This group has been conducting an investigation that will result in changes and improvements as to how Amtrak manages the systems that control train traf?c in and around CUS. These include a full review of the local infrastructure and data center operations environment, increased access controls, the adoption of a rigorous change management methodology, and a reevaluation of how to accomplish the PTC testing that we need to perform, while also safeguarding the systems needed to manage regular train movements. Going forward, we are reassigning organizational responsibility for some of these functions at CUS to ensure that IT best practices are applied and followed through these systems? full lifecycle. This will include The Honorabfe Danie! March 29, 20179 Page 3 an end-to-end architecture evaluation which will, where appropriate, rely on cloud?hosted services to ensure high availability. Given the security implications surrounding this sort of work, I cannot go into further detail here, but we will provide Metra with suf?cient detail for them to be assured about the changes that are being put into place. I As explained above, the work being done on February 28 related to PTC testing and was being performed as part of that multi?year effort to bring that important safety system online. I New management protocols are being developed by the group mentioned in the first bullet above. It is too early to list them here, but we do plan to share the outcome of their investigation with Metra in the near future. I As explained earlier, Metra and other railroad operators were notified within minutes of the communications failure. I To communicate with the traveling public, Amtrak relies on a combination of media advisories, social media channels, and direct contact with other stakeholders to ensure the timely dissemination of accurate, relevant information so that affected customers can revise their plans and pursue alternatives should that be necessary. These procedures were followed on February 28. We are reviewing the effectiveness of these plans and will adopt whatever changes we deem necessary as those evaluations are completed. I On February 28, Amtrak had two pairs of servers to provide redundancy. The electrical fault that disabled communications introduced a failure mode for which the systems were not prepared, as the requirements associated with the PTC testing meant that the backup servers were not able to function in that capacity. Since February 28, we have added a third backup system, and the team studying the incident will scrutinize the dependencies among these systems and de?ne whatever configuration changes are necessary to ensure the backups share no possible failure modes. I The summary of the incident at the start of this letter speaks to the speci?c circumstances of the February 28 failure, and again I refer to the team we have assembled which will establish new procedures and safeguards to ensure no repeat of this incident can occur. I Amtrak does not contemplate reimbursing those who paid out of pocket for alternate means of commuting on February 28. The Honorabie Daniel Lipinski March 29, 2019 Page 4 Metra ridership ?gures are collected by calendar year, and for 2018 Metra reported almost 33.4 million annual riders. Amtrak?s FY 2018 CUS ridership was 3.3 million, which equates to approximately 9,000 per day. Amtrak fully understands the central importance of CUS to both the city and our network and that the facility merits additional investment. To this end, over the last ?ve years, our capital budget at Chicago Union Station and the surrounding area has been approximately $100 million, including more than $60 million in direct investment in the station, more than $1 1 million in communications and signals, and more than $15 million in other facilities and support, among other critical investments in the area, including PTC. By comparison, by Metra?s own numbers, it has taken them almost 30 years for them to contribute an equivalent amount in CUS investments. As this has taken place, Amtrak has requested increased funding from Metra in both basic maintenance and capital replacement of and other railroad infrastructure. Under the expiring CUS Lease agreement, Metra operates approximately 75 percent of trains operating in the station with 90 percent of the passenger traf?c while contributing signi?cantly less of a proportionate share of rail and station maintenance and capital costs, resulting in insuf?cient investment in aging infrastructure. Comparatively, Metra runs 40 percent more train traf?c than they did when the lease was signed in 1984. Amtrak is seeking an increase in annual funding by Metra, as well as a joint effort to prioritize investment and apply for available federal grant funds. To that end, we have recently ?nalized and transmitted to Metra a comprehensive Asset Inventory List for the station, which includes replacement values and timeframes that will bring the assets to a State of Good Repair within the next ten years. Amtrak?s 20,000 employees work hard every day to be good stewards of the dollars you invest in our services and we value your support. Thank you again for your letter and the strong care you demonstrate for your constituents. Stephen J. Gardner Senior Executive Vice President, Marketing, and Strategy cc: Richard H. Anderson