,. ... ~ lw\ o .. v" "EOfJ~ ~o~ '0 . : i OFFICIAL USE ONLv-=-t)IG INVESTIGATION INFORMATION -----.j NUCLEAR WASHINGTON, D.C. 20555-0001 R~GULATORY COMMISSION UNITED STATES ~ "+., , );1.>>0 -:t .~ {I .QI OFFICE OF THE INSPECTOR GENERAL September 30,2010 MEMORANDUM TO: R. William Borchardt Executive Director for Operations Joseph A McMillan IRAJ Assistant Inspector General for Investigations FROM : SUBJECT: ESSENTIAL SERVICE WATER LEAK AT BYRON NUCLEAR STATION (OIG CASE NO. 08-12) This memorandum conveys the results of an Office of the Inspector General (OIG) investigation into a leak that occurred in the essential service water system (ESW), a significant safety system, at Byron Nuclear Station (Byron) on October 19, 2007. This leak necessitated a 12-day shutdown of both reactors located at that site. This was a significant event, and NRC initiated a Special Inspection Team (SIT) soon after the shutdown to evaluate licensee actions surrounding the ESW failure. The SIT found that the licensee had not taken timely corrective action to correct degradation in ESW piping, and that a licensee analysis of the degradation's impact was faulty. OIG's investigation focused on NRC's oversight and awareness of corrosion in the ESW in the months just prior to the leak. OIG found that: o The NRC oversight of licensee operability decision making was not successful in learning of the steady reduction in margin of wall thickness of the ESW piping over a 7-month period until just 2 days before pipe failure. Although the Byron resident inspectors carried out routine oversight responsibilities in accordance with agency requirements , the licensee's failure to analyze a problem correctly resulted in the resident inspectors' lack of awareness of a significant problem with the ESW. o I , TW ' < NT OR DISTRIBUTED OUTSIDE THE RECEIVING AGENC Y WITHOUT THE PERMISSION OF THE OFFICE OF THE INSPECTOR GENERAL v- OFFICIAL USE ONLY OIG INVESTIGATION INFORMATION I r==Q;FICIAL USE ONLY OIG INVESTIGATION INFORMATION o Although NRC inspection guidance conveys an expectation that resident inspectors should be aware of the status of safety systems, it was not specific enough to result in effective oversight of the degraded ESW by the resident inspectors. This report addresses circumstances typically experienced by resident inspectors, who are expected to be aware of plant conditions but are dependent, to a certain extent, on the licensee's ability to identify and correct plant problems. The Byron ESW event illustrates some of the challenges of this dependency and offers insights for meeting similar challenges in the future . I. Byron ESW Byron, located in northern Illinois, has two 1,200-megawatt output Westinghouse pressurized water reactors. The power station is owned and operated by Exelon. Like other pressurized water reactors , Byron has a safety related heat removal system that is used to remove decay heat and cool critical components during accident conditions . At Byron , the ESW fulfills these safety functions . The ESW is common to both units at Byron and is physically a very large system . It includes a dedicated cooling tower that serves as the ultimate heat sink for the Byron reactors. While much of the ESW piping is above ground, the portion of the piping that transports water to and from the ESW cooling tower is underground . The ESW cooling tower consists of eight separated cells. Each cooling tower cell contains a fan and spray nozzles. The pipes that convey the heated ESW water into each cooling tower cell are 24 inches in diameter, made of carbon steel, and have a nominal pipe wall thickness of 0.375 inches. The pipes emerge from the ground through a concrete floor, which is located inside a concrete valve vault. After penetrating the vault floor, each pipe connects to a bolted flange . The pipe connected to the other end of the bolted flange rises vertically and exits the vault. Each pipe section above the flange includes a single valve. The pipes direct the ESW water onward to the spray nozzles in the cooling tower. The vertical section of ESW piping inside the vault, above and below the flange, is called a pipe riser. At Byron, n these pipe risers are designated "OA through "OH" and these designations correspond to the eight cooling towers that they supply. The October 19, 2007, leak that resulted in Byron's shutdown occurred in the " ~C " pipe riser. 2 i on IEURIAL SSE ON LY I 7 OtG INVESTIGATION INFORMATION- ~ ~leiAE USE Ol4n ola INVESTIGATION INF@RMATI~ ) The eight valve vaults are made of concrete and each has a removable steel hatch plate that allows personnel entry into the vault, an area which is otherwise inaccessible. The hatch plates require a crane for removal. The valve vaults are designed to provide physical protection for their contents against the impact of high velocity debris associated with a tornado. II. Byron ESW Corrosion History Byron has a history of ESW piping corrosion in the cooling tower valve vaults. Byron staff as far back as 1990 had identified excessive corrosion on this piping. At that time, the licensee attributed excessive corrosion of the piping to a failure of the pipe coatings. The condition was described as extensive with loose scaly rust that originated on the outside surface of the piping. In 1994, the licensee initiated plans to replace the ESW carbon steel piping in the vaults with stainless steel piping because of the enhanced corrosion resistance of stainless steel. In 1997 and 1998, the ESW piping above the bolted flange was replaced ; however, the short section of pipe riser coming through the concrete floor and terminating below the bolted flange was not replaced. While the lower section of the pipe was originally within the scope of the planned changes, Byron later removed it from the scope and replacement never occurred. Between 1998 and 2007, the Byron staff continued periodic visual inspections as required by the American Society of Mechanical Engineering code. Until 2006, most of the inspections noted no degradation, although some instances of corrosion were identified. In 2006, two pipe risers that were inspected were documented as having some corrosion, but the corrosion was not characterized as severe. III. ESW Corrosion in 2007 Between March 2007 and October 2007, Byron staff conducted a series of inspections of the pipe riser, identified progressive thinning of the pipe wall had resulted in wall thickness that was less than the established standard, and four times reduced the minimum value for allowed riser pipe wall thickness: first from 0.375 to 0.153 inches, then from 0.153 to 0.121 inches, again from 0.121 to 0.06 inches, and finally from 0.06 inches to 0.03 inches. The progression of inspections and decisions to reduce the wall thickness standard was as follows: 3 ~CIAL USE ONLY OIG IUVESTIGATION-tNFORMATION;> G~FICIAL o USE ON LY OIG INVEcSTIGATION INFORMAII~ March 2007: During a periodic visual inspection, the Byron staff noted significant corrosion on the "~C '' pipe riser, including metal corrosion product flaking off the pipe. The staff determined that corrosion had worsened since the last inspection (May 2006) , and determined that additional inspection was needed of other pipe risers . In May 2007, Byron staff performed a visual inspection of the "OA" pipe riser and found significant corrosion . In response to the corrosion identified on the "OA" and "OC" pipe risers, the staff decided to schedule ultrasonic testing of the "DE" riser to determine the th ickness of that pipe. The Byron staff also established a new minimum pipe wall thickness of 0.153 inches in expectation that future measurements of the pipe wall thickness would be less than the nominal value of 0.375 inches and that the pipe wall would have adequate thickness as long as it was greater than 0.153 inches. June 4, 2007 : Byron staff visually inspected the "OH" pipe riser and found that the protective coating that was supposed to be on the pipe was degraded and the pipe's bare metal was exposed . June 14, 2007: Byron completed ultrasonic pipe wall thickness measurements for the ESW pipe riser in the "DE" valve vault. The testing identified localized areas below the allowed minimum wall thickness of 0.153 inches that had just been established in May. The two thinnest areas measured had wall thickness values of 0.122 and 0.124 inches, well below what had been established as the allowed minimum. July 11 , 2007: After the below-threshold thicknesses were identified in the "OE" pipe riser, Byron engineering staff completed a detailed evaluation and lowered the minimum acceptable wall thickness for continued operability to 0.121 inches. The calculation that determined the new minimum standard was documented in an "engineering change" document. It was not unusual at Byron at that time to document what was essentially an operability determination in an engineering change document. The document also stated that action had been taken to present the pipe riser thinning issue to the Plant Health Committee and recommend that the piping be restored to original design conditions. September 2007: Visual inspections were performed on the "OB" and "00" pipe risers and no degradation was noted , although degradation had been noted on both of these risers in previous inspections and no repairs had been made. o o o o 4 TRIs-tkJC UM EN I IS IHE PROPER I y OF tHE NRC. IF LOAN ED 10 ANa IHER AGENCy II AND liS CON I EN IS ARE Nor 10 BE REPROOUCEd """'ORLJTS I k lBu I ED 00 I SIDE I HE RECEiv iNG AGENC Y WI i H O ~ i i HE P!RMI SSIOI~ OF THE OPPle ! OF TME I N SP ECTO ~ GI!!N~ ~ (OFFICIAL USE ONty QfG"1NvESTIGATION INFORM ATIO~ C~ICIAL USE ONLy o OIG INvESTIGA liON INF()RMATION~ October 10, 2007: The "OH" pipe riser wall thickness was measured at inches. Byron staff lowered the allowed thickness from 0.121 to 0.06 inches through a formal operability evaluation dated October 12, 2007. The documentation of operability in a formal operability evaluation rather than an engineering change document indicated that the licensee believed the issue was significant. October 17, 2007: the "OB" pipe riser wall thickness was measured at 10 locations; all measured thicknesses were less than 0.1 inches, including one location measuring 0.047 inches. The licensee subsequently lowered the allowed thickness to 0.03 inches. October 19, 2007: While Byron staff were brushing corrosion product off the "OC" pipe riser to conduct further ultrasonic testing, the brush scraped through the pipe wall, the pipe began leaking, and the licensee was forced to shut down both Byron reactors to make repa irs . O~ o o IV. NRC Awareness of Corrosion Prior to Leak A. Corrective Action Program Documentation The degradation of the pipe risers as well as the licensee's actions to evaluate and correct the degradation was documented in Byron's Corrective Action Program (CAP). During 2007 , about 13,300 issues were entered into the CAP. This equates to an average of 37 issues per day, although the daily rate can vary greatly based on conditions or activities (e.g., problems with a particular system, plant outage) underway at the plant. Between March 2007 (the point at which corrosion was identified on the "OC" pipe riser) and October 19, 2007 (date of the leak), there were about 277 issues that concerned some aspect of the ESW, and about 25 that dealt specifically with pipe riser corrosion or degradation. This included action requests in June and July to perform ultrasonic testing analyses and repairs of the pipe risers . NRC Inspection Procedure 71152 , "Problem Identification and Resolution ," establishes the expectations for resident inspector review of the CAP. The procedure instructs resident inspectors to review each CAP input and follow up on selected items and be alert for adverse performance trends. 5 THIS OOCUMENT IS THE PROPERTY OF THE NRC . IF LOANEC TO ANOTHER AGENCY It AND liS CONtENtS ARE NOI to"BCREPRODOCEO -" E;;;;;;-;;;'--;;;;=";'~;;;;;;;;;:~ ~E"";l)NEY ."s,,,,,,,,,,,"';' """NY"""" '_ ) ~f'IGIAL USE ONLY OIG INVESTIGAtiON INFORM.o.TIOY The two NRC resident inspectors who worked at Byron during the months leading up to the leak both stated that they reviewed every CAP issue report that Byron generated. The senior resident inspector recalled thinking in September 2007 that there was an unusually high number of issue reports on the ESW piping degradation, perhaps as many as a dozen. However, he told OIG he did not follow up on this information prior to the leak because he believed the degradation was minor surface corrosion and that the licensee was addressing the issue appropriately. He also learned , through review of materials for a presentation at the September Byron Nuclear Safety Review Board meeting, the licensee had found the ESW piping was degraded and had performed operability determinations and decided to replace the pipe. B. Discussion During Licensee Meetings Commercial nuclear power plants have a structure of routine meetings to identify and resolve issues. For example, most nuclear power plants, including Byron, hold daily Plan of the Day meetings to coordinate work activities at the site each workday. Meetings are also held regularly to plan and perform maintenance, establish priorities for funding for repairing plant equipment, and provide management oversight of the plant activities and performance . During the summer of 2007, the Byron staff used this structure of meetings to address the problems that had been discovered with the ESW pipe riser degradation. For example, the corrosion found on the "OA" pipe riser in May was presented to the Management Review Committee, which reviews inputs to the CAP . The Management Review Committee assigned the issue to an engineering rapid response team to address the wall thinning . The Plant Health Committee, which helps establish the priority for plant repairs and modifications and meets weekly, identified the ESW riser piping as a new focus area on June 25 , 2007. On July 16, the riser piping degradation ultrasonic testing results were presented at the committee meeting , and on August 16, another presentation on pipe riser degradation was held to seek approval of modifications to replace the degraded riser piping . On September 5, another presentation was held seeking approval to fund modifications. That presentation stated that the riser piping degradation had the potential to cause the loss of the plant ultimate heat sink due to a leak that could not be isolated and could compromise nuclear safety. In addition to these formal presentations, the plant system engineer was required to report on the system at these meetings at least biweekly throughout the summer of 2007. Byron also holds a daily Risk Meeting that includes Byron managers and discusses 6 OFFICIAL USE ONLY OIG IN'IES IIGA liON INFORMATION) C!:;FICIAL USE ON LY OIG INVESTIGATION INFORMATION'\ current s and activities that could involve the site's after hours duty staff. while attending an October 17, 2007, Risk Meeting (2 days before the leak occurred) that a Byron resident inspector learned , for the first time, of the significance of the ESW piping degradation. Inspection Manual Chapter 2515, "Light -Water Reactor Inspection Program Operations Phase," Appendix D, provides guidance to NRC inspectors on methods to maintain awareness of plant status. This chapter states that inspectors should attend licensee meetings on a routine basis to provide them with knowledge of overall plant status, but leaves it to the inspectors' discretion to select which specific meetings to attend . The Byron resident inspectors told OIG that they did attend Byron plant meetings on a regular basis. The senior resident inspector said the resident inspectors attended the Plan of the Day meeting about three times per week, and the daily Risk Meeting , about once a week. The inspectors also attended the operations shift turnover meeting on a daily basis and the Plant Health Committee meeting once every 3 or 4 weeks. Byron staff confirmed that the resident inspectors attended the meetings on a regular basis. The senior resident inspector said that after he learned about the extent of the degradation at the October 17, 2007 , Risk Meeting, he e-mailed his regional manager to inform him about this issue. The regional manager recalled learning about the issue approximately 2 days before the leak. The regional manager said that when he learned of the pipe riser degradation, he and the regional engineering staff began engaging in telephone conversations with the licensee to understand the scope of the issue. C. Oversight of Operability Decisionmaking NRC Inspection Procedure Attachment 71111 .15,"Operability Evaluations," requires resident inspectors to inspect 19 to 25 operability evaluations each year as part of NRC's baseline inspection program. The procedure requires a proportion of the total sample to be reviewed each quarter, but contains no other timeliness requirements. The Byron senior resident inspector provided OIG with a list of 21 operability decisions the resident inspectors reviewed in 2007 . None of the decisions reviewed pertained to the ESW pipe wall thinning; however, the sample included a portion of the nine formal operability evaluations performed at Byron in 2007, and some of the less formal operability determinations. The senior resident inspector stated that the sample selection was based on his judgment about the safety significance of each issue. iI-~as The senior resident inspector also stated that while Byron completed a formal 7 THIS DOCUMENT IS THE PROPERTY OF THE NRC. IF LoANED to ANO t RER AG ENCy I f AND l I S CONIENIS ARE NO I l O SE REPRODUCED -0'" OISTftlSUTEO OUTSiDE nlE RECEiv iNG AGENC I WI THOtJT 1HE PEAMISSfO N OF TttE OFF iCE OF I HE 'NSPEC I OfrGENER;II;t: 1= \ ( O~ICIAL USE ON LY OtG INVESTIGATION INFORMA'ION~ operabl ' evaluation of the "OH" pipe riser degradation on October 12, 2007, review it before the leak, which occurred just 1 week later, on October 19, 2007 . ~~IClA[ USE ONLY OIG INVESTIGA liON IN FORMAflON \ h~id not OIG learned that the SIT reviewed Byron's July 11 , 2007, evaluation of the "OE" pipe riser that lowered the minimum acceptable wall thickness for continued operability from 0.153 inches to 0.121 inches. The SIT determined that the calculation that arrived at the minimum wall thickness did not include all the loads to which the pipe was subject. The SIT concluded that the minimum acceptable wall thickness would have been greater than 0.121 inches had those loads been included in the calculation . OIG noted that this would bring the operability decision made by the Byron staff at the time into question since the measured wall thickness of 0.122 inches allowed little margin for calculation error. The SIT also noted problems with Byron's calculations to reduce the minimum acceptable wall thickness to 0.06 inches and, later, to 0.03 inches. OIG noted that while the SIT challenged the calculations, it did not question the overall reduction in minimum acceptable wall thickness during this time period . D. Resident Inspector Communication with Byron Staff OIG interviewed members of the Byron staff familiar with the event to determine the level of importance the Byron staff placed on the ESW degradation and the communication of this issue to NRC. The senior managers at the site were aware of the degradation, but did not recall being aware of its severity or extent until shortly before the leak occurred. None of the senior managers interviewed recalled the basis for considering the ESW operable in July, given the 0.121-inch thickness standard in an actively corroding system with measured thickness as low as 0.122. An engineering manager familiar with the event and the root cause analysis performed by the licensee said that the Byron staff did not understand that the corrosion was active and the significance of that on operability until the leak occurred in October 2007. The Byron senior managers interviewed who maintained frequent communications with the NRC resident inspectors said they did not recall informing the resident inspectors about the problems during the months leading up to the ESW leak. Byron, like many commercial nuclear power plants, rates the overall condition of each plant system using a color-coded system, which is reported in a System Health Report and updated quarterly. The ESW system was rated as red (the worst condition rating for a system) from the 4th quarter in 2006 through the 3rd quarter in 2007 . The senior resident inspector stated he was aware that the ESW system health report was red, indicating that Byron personnel viewed the system as having problems. However, neither the senior resident inspector nor the Byron system engineer responsible for the 8 I I (OFFICIAL USE ONLY OIG INVESTIGATION INFORMATION~ \..:nJ;FICIAL USE ONLy r .r-- OIG INVESTIGATION INFORMATIOr Although the Byron resident inspectors reviewed CAP entries, attended plant meetings, and inspected Byron operability decisionmaking in accordance with agency guidance, this level of oversight did not lead to knowledge of ongoing and significant ESW degradation over a 7 -month period . The senior resident inspector only learned of the significance of the problem 2 days before the leak that caused the inoperability of a significant safety system and forced both reactors to be shut down. A major factor in the resident inspectors' lack of awareness of this issue was the licensee's poor handling of operability decision making and the licensee's apparent lack of recognition of the significance of the problem. Please respond to this office within 120 days on what, if any, action you intend to take in response to this report. If you have any questions, please contact me, at 301-415-5929, or Rossana Raspa, Senior Level Assistant for Investigative Operations, at 301-415-5925. 11 THIS OOCuWERTlS THE PROPERly OF IHE NRC . IF [ OANED to ANOt HER AGENe, II AND l iS CONTENTS ARE NOT TO BE ptl!Pftoot:teett OR Dls tRiBUTEb OUTSIDE THE RECEIVING AGeNCY Wi I HOU I iHE PERMtSStOI4 0,. THE OFFle!: 011 THE1NSpeeroR 6EI4ERAL ~IAL USe-oNlY OIG INVEStf(JATIONiNFORMATION)