FOR OFFICIAL USE ONLY ~Deliberative Process SAFETY TROUBLE REPORT PUGET SOUND NAVAL SHIPYARD AND INTERMEDIATE MAINTENANCE FACILITY DATE OF ISSUE: 6/03/2013 REPORT NO.: Relates) DATE OF EVENT: 4/5/13 LOCATION BUILDING 1109 TIME OF EVENT: NA UNPLANNED EVENT: On Multiple Occasions Chlorine Levels Exceeded Established Limits SUBSAFE DEEP SUEMERGENCE OP CERTIFICATION PRELIMINARY NAVAL NUCLEAR PROPULSION INFORMATION [3 FINAL 1. SUMMARY OF EVENT: Background: The Industrial Wastewater Pretreatment Facility (IWPF) Building (Bldg) 1109 was commissioned in 2006 and was designed to handle a variety of in?uents from the Metal Preparation Facility (MPF) 873 and other shipyard locations. Shop 99 operates the IWPP. The IWPF consists of dedicated and general batch treatment tanks of different sizes to handle cyanide, chrome, acid/alkaline (AA) as well as other wastewater in?uents. In addition to the batch treatment tanks, the IWPF has other process equipment that is used for the removal of heavy metals, some organics, suspended solids and miscellaneous pollutants from the wastewater in?uents. The IWPF was commissioned with Open top tanks, the Architecture and Engineering (AE) drawings Speci?es tank covers. It is unknown why no tank covers were installed at the IWPF commissioning. In 2010 Shop 99 was having difficulty treating Cyanide (CN) and switched from the normal procedure (procedure 2) to procedure I which requires additional levels of sodium hypochlorite (bleach) and the acidi?cation of the wastewater. Additionally, there were chronic equipment de?ciencies that required manual manipulation of the plant versus utilizing the automated features for treating cyanide. This timeframe also coincides with the plant operators noticing/experiencing stronger odors of chlorine in the plant. In response to Shop 99 Plant Operator concerns, Code 106.22 industrial hygiene staff monitored suspected chlorine?generating processes using detector tubes. This monitoring did not identify chlorine concentrations above Permissible Exposure Limits (PELs). Chlorine concerns prompted a work order to be submitted to install a chlorine alarm system. The chlorine alarm was installed in February 2012 and was commissioned inJuly2012. TM 3 -J . kwT?r?zta Ha.? es tau issued Trouble Report Unantz'cz?pated Exposure of Personnel to Hydrogen Chloride and Chlorine Vapor on 2/13/13. The unanticipated exposure occurred in the IWPF 1109. One of the interim corrective actions assigned to Code 106.22 Occupational Safety and Health (OSH) Of?ce was to request Naval HOSpital Bremerton Industrial Hygiene Branch to conduct an Industrial Hygiene Survey to verify adequacy of process controls for transfer and processing of wastewater in the IWPF. FOR OFFICIAL USE ONLY ?Deliberative Process NAVSEA Distribution Statement Distribution authorized to Components; other requests must be referred to COMNAVSEA or the cognizant NAVSEA code. FOR OFFICIAL USE ONLY ?Deliberative Process This included air monitoring of 1109 processes. This report documents air concentration during monitoring that exceeded the 1109 established 1% for chlorine iW?r four different nrocessgg the transfers of treated . OWE CN wastewater, untreated CN wastewater, untreated chrome wastewater; and the treatment 3 of CN wastewater. During these monitored events, the building's visual and audible it" chlorine alarms did not activate. The monitoring of breathing zone air concentrations during these process evaluations prevented exposure above the PELs to the personnel involved at that time. The PELs and other limits that apply to chlorine are: a. The 8-hour Time?Weighted Average (TWA) is 0.5 parts per million (ppm). The worker?s exposure is averaged over an 8?hour day and compared to the limit. b. The Short Term Exposure Limit (STEL) for chlorine is 1 ppm. The ST BL is a concentration that is not to be exceeded over a 15?minute period at any time during the work day. The worker?s exposure is averaged over a 15?minute period and compared to the limit. c. The immediately dangerous to life or health (IDLH) limit for chlorine is 10 ppm. The IDLH limit is intended to allow a worker to escape without injury or without irreversible health effects within 30 minutes in the event of respiratory protection equipment failure; at IDLH concentrations, every effort should be made to immediately exit the environment. d. The IWPF installed chlorine alarm system provides a visual alarm at 0.5 and a visual and audible alarm at 1.0 (alarm set points). 25K The following evolutions were monitored by Code 106.22 and the Naval Hospital Bremerton Industrial Hygienist (1H): On 4/5/2013 - Shop 99 Plant Operators were transferring treated cyanide wastewater from cyanide Tank 113 to Acid/Alkaline Tank 3A when the airborne chlorine alarm set point concentration was exceeded in the plant operator?s breathing zone (on the mezzanine) and 1109 chlorine alarm did not activate. [Problem 1] The chlorine air concentrations reached 2.1 in the breathing zone beside Tank 3A. [Event] At this time, personnel immediately evacuated the mezzanine level. The nearest installed chlorine sensor was approximately 3 feet away, between Tanks 3A and 3B. The most likely scenario for the chlorine release was the sodium hypochlorite present in the treated wastewater potentially released chlorine gas during the transfer. The tanks had open tops and the ventilation system was not adequate to remove the chlorine from the breathing zone. On 4/5/2013 - Shop 99 was transferring untreated cyanide wastewater from 873 Retention Tank to 1109 cyanide Tank 1B when the airborne chlorine alarm set point concentration was exceeded in the plant operator?s breathing zone (on the mezzanine) and the chlorine alarm in 1109 did not activate. [Problem 2] During the transfer the chlorine air concentrations reached 2.0 in the breathing zone beside Tank 18. [Event continued] At this time, personnel immediately evacuated the mezzanine level. The nearest chlorine sensor on the mezzanine was about 15 feet away from Tank 1B, between Tanks 3A and 3B. Cause unknown, most likely scenario for generating chlorine: Chlorine was present in the tank due to previous tank contents (treated cyanide wastewater containing sodium hypochlorite) and the tank was not rinsed out from the previous treatment of cyanide. FOR OFFICIAL USE ONLY ?Deliberative Process Page 2 of 12 FOR OFFICIAL USE ONLY ~Deliberative Process 200TR?l3?l3 On 4/8/13 Shop 99 Plant Operators were transferring untreated chrome wastewater from 873 Retention Tank to 1109 chrome Tank 2B when the airborne chlorine alarm set point concentration was exceeded in the plant operator?s breathing zone (on the mezzanine) and the chlorine alarm in 1109 did not activate. [Problem 3] During the transfer the chlorine air concentrations reached 4 in the breathing zone beside Tank 2B. [Event continued] At this time, personnel immediately evacuated the mezzanine level. The nearest chlorine sensor on the mezzanine was ~8 feet away from Tank 2B, between Tanks 3A and 3B. Subsequent investigation revealed the Waste Stream from the photo etching shop potentially contained household bleach (sodium hypochlorite, a source of chlorine). Additional chemicals were found in the waste stream that were potentially not compatible with processing at the IWPF. A separate problem noti?cation was generated and will be investigated under Critique Report 200CR-J 3?098 Process Products Disposed to Waste Stream without Designation. On 4/10/13 - Shop 99 was conducting?Cyanide Destruction Procedure?l Second Reaction when the airborne chlorine alarm set point concentration was exceeded in the plant operator?s breathing zone (on the mezzanine) and the chlorine alarm in 1109 did not activate. [Problem 4] During the treatment process, the chlorine air concentrations rapidly rose over a two minute period from 0.1 to 22.5 in the breathing zone beside Tank 1A. [Event continued] At this time, personnel immediately evacuated the mezzanine level. The multi-gas meter was left on the mezzanine next to Tank 1A and recorded a high of 90 chlorine. Shop 99 Plant Operators used Cyanide Destruction Procedure 1 versus Procedure 2 (the recommended procedure in manual) because Procedure 2 was not effective in cyanide destruction. [Problem 5] Procedure 2 does not require the acidi?cation of the cyanide wastewater which greatly reduces the chances of producing chlorine gas. Procedure 1 could not be performed as written as the plant?s oxidation reduction potential (ORP) and pH probes were not operational; rendering the IWPF automated cyanide treatment system inoperable. Instead, the Shep 99 Plant Operators used hand-held ORP and pH probes and manually operated the injection system to add sulfuric acid and/or sodium hypochlorite. This process greatly reduced the ability to effectively control the pH and ORP during the cyanide destruction process. Most likely scenario for this event was the Plant Operators added too much sulfuric acid dropping pH below 6.0 resulting in the generation of chlorine gas versus stopping at that the target pH of 8.5. Contributing causes were: The established procedures in the IWPF manual were not suf?cient for the level of training the plant operators received (On the Job Training). The design that was implemented at plant commissioning was not adequate for current operations. The material deficiency resolution and tracking for the IWPF was deficient. With these chronic equipment failures, the plant operators became accustomed to using workarounds to process the CN. There was no overall Cognizant Technical Authority for waste water processing in the IWPF 1109 for Senior management was not cognizant that plant operators were using the alternative cyanide destruction procedures and of the required workarounds to process wastewater based on the factors described above. The Building 1109 chlorine alarm was not periodically tested as required by the manufacturer speci?cations after commissioning. [Problem 7] The Chlorine alarm system?s Operation and Maintenance Manual was not initially incorporated into the service FOR OFFICIAL USE ONLY ~Deliberative Process Page 3 of 12 FOR OFFICIAL USE ONLY ?Deliberative Process provider?s maintenance program as the maintenance provider?s contract was undergoing new bid proposals at the time. Naval Facilities (NAVFAC) management decided to manage the chlorine alarm testing with the use of service tickets. There was no tickler to alert management of the periodicity of the chlorine aiarm required testing and subsequently the chlorine alarms were not tested after commissioning. Chlorine alarm was commissioned 7/2012. During the course of the investigation a non?contributing problem to the de?ned event was discovered: A Code 106.22 employee returned to the Building 1109 mezzanine level to retrieve a multi? gas meter when chlorine levels were unknown. [Problem 6] This was during the Cyanide Destruction procedure. 2. DESCRIPTION OF PROBLEM. GENERAL DESIGNATION. AND DISCUSSION OF S): (DESIGN MATERIAL PERSONNEL PROCEDURE a. DESCRIPTION OF EACH PROBLEM: Problem 1: Shop 99 Plant Operators were transferring treated cyanide wastewater from cyanide Tank 1B to Acid/Alkaline Tank 3A when the airborne chlorine alarm set point concentration was exceeded in the plant operator?s breathing zone (on the mezzanine) and 1109 chlorine alarm did not activate. Problem 2: Shop 99 was transferring untreated cyanide wastewater from 873 Retention Tank to 1109 cyanide Tank 1B when the airborne chlorine alarm set point concentration was exceeded in the plant operator?s breathing zone (on the mezzanine) and the chlorine alarm in 1109 did not activate. Problem 3: Shop 99 Plant Operators were transferring untreated chrome wastewater from 873 Retentibn Tank to 1109 chrome Tank 2B when the airborne chlorine alarm set point concentration was exceeded in the plant operator?s breathing zone (on the mezzanine) and the chlorine alarm in 1109 did not activate. Problem 4: Shop 99 was conducting Cyanide Destruction Procedure?1 Second Reaction when the airborne chlorine alarm set point concentration was exceeded in the plant operator?s breathing zone (on the mezzanine) and the chlorine alarm in 1109 did not activate. Problem 5: The IWPF Cyanide destruct procedure(s) were not effective in the destruction of cyanide. Problem 6: A Code 106.22 employee returned to the Building 1109 mezzanine level to without a multi-gas meter when chlorine levels were unknown. FOR OFFICIAL USE ONLY "Deliberative Process Page 4 of 12 FOR OFFICIAL USE ONLY ~Deliberative Process Problem 7: The Building 1109 chlorine alarm was not periodically tested as required by the manufacturer specifications after commissioning. b. DISCUSSION OF Problem 1: (Local Process) (1) Unknown most likely scenario for generating chlorine: The treatment of cyanide wastewater includes the addition of sodium hypochlorite. The residual sodium hypochlorite in the treated cyanide wastewater potentially released chlorine gas. (2) Plant Operators position themselves at the top of the tank during ?lling to prevent over?ow as the Tank Level Indicators (TLI) do not provide an accurate tank level during ?lling for the Plant Operators desired tank level. The Plant Operators want to have the tank as full as possible to facilitate maximum processing of wastewater and to facilitate the use of hand held probes. (3) The tops of the processing tanks are open allowing chlorine to escape to the breathing zone. (4) The chlorine alarm system sensors were not in a location to detect chlorine in the immediate breathing zone for the plant operators. (5) The installed ventilation system was not adequate to remove chlorine gas from the breathing zone. (6) Unknown if the chlorine alarm system was functional as the system was not tested per the manufacturer?s speci?cation after commissioning. (See Problem 7) Problem 2: (Local Process) (1) Unknown most likely scenario for generating chlorine: Chlorine was present in the tank due to previous tank contents (treated cyanide wastewater containing sodium hypochlorite) and the tank was not rinsed out from the previous treatment of cyanide. (2) Problem 1, Cause 2-5 apply. Problem 3 (Local Process) (1) Unknown most likely scenario for generating chlorine: A waste stream (WSN 211? 0003) from 857 (photo etch shop) which potentially contained household bleach (contains sodium hypochlorite?a source of chlorine) was previously mixed with the chrome wastewater in Tank 2B. (2) It was unknown to the Shop 99 plant operators that WSN 211-0003 from 857 contained bleach as the WSN did not re?ect bleach. (3) Problems 1, causes 2?5 apply. Problem 4 (Management) (1) Shop 99 plant operators used the alternative Cyanide Destruction Procedure 1 versus Procedure 2 (the recommended procedure in manual) because Procedure 2 was not effective in cyanide destruction. (See Problem 5) Due to chronic equipment de?ciencies, plant operators used manual means to adjust and monitor pH and ORP versus the installed automated system. This resulted in the lowering of pH below the neutral pH threshold and the generation of chlorine gas. FOR OFFICIAL USE ONLY ?Deliberative Process Page 5 of 12 FOR OFFICIAL USE ONLY ~Deliberative Process (2) Senior management was not aware plant operators were using the alternative cyanide destruction procedures and of the required workarounds to process wastewater. (3) Due to the chronic equipment failures, the plant operators became accustomed to using workarounds. (4) The established procedures in the IWPF manual were not suf?cient for the level of training the plant operators received (On the Job Training) (5) The original design for the plant was not adequate for current operation. (6) No overall Cognizant Technical Authority for waste water processing in the IWPF 1109. (7) Material de?ciency resolution and tracking was deficient. (8) Problems 1, causes 2-5 apply. Problem 5 (Unknown) . (1) Cyanide destruction procedures were ineffective (Unknown). Problem 6 (Local Process) . (1) Code 106.22 failed to recognize the potential hazard that still existed on the mezzanine level due to incorrect assumptions made regarding temporary ventilation and the multi-gas meter alarm. (2) Code 106.22 only had one multi-gas meter with a chlorine sensor and that meter was left at the tank opening when chlorine levels spiked, leaving no means for progressively monitoring the return to the mezzanine. (3) Code 106.22 did not have a codi?ed process for monitoring waste water treatment with established step points and response actions if those stop points were reached. Problem 7 (Management) (1) The Chlorine alarm systems Operation and Maintenance Manual was not initially incorporated into the service provider?s maintenance program as the maintenance provider?s contract was undergoing on new bid proposals at the time. NAVFAC management decided to manage the chlorine alarm testing with the use of service tickets. There was no tickler to alert management of the periodicity of the chlorine alarm required testing and subsequently the chlorine alarms were not tested after commissioning. Chlorine alarm was commissioned 7/2012. 3. CONDITION OF SHIP AND AT THE TIME OF EVENT: a. CONDITION OF FACILITY: Operational b. CONDITION OF SYSTEM COMPONENT: Various mechanical components, for the IWPF wastewater treatment equipment, were non-operational.- 4. IMMEDIATE CORRECTIVE ACTIONS) TAKEN AND RESULTS: a. Secure all wastewater treatment and transfer processes at 1109. Action Org: Shop 99 Complete FOR OFFICIAL USE ONLY ?Deliberative Process Page 6 of 12 FOR OFFICIAL USE ONLY ?Deliberative Process b. Establish a Management working group to: (1) Develop a wastewater treatment and transfer process matrix for the prioritization of each 1109 process. (2) Conduct a detailed review of each wastewater process to include the characterization, procedures, and work practices, frequency of treatment/transfer, applicable personal protective equipment/engineering controls and required monitoring to safely conduct each wastewater process. (3) Identify measures to resume each wastewater plant process and develop contingencies for disposal of wastewater if it cannot be treated. Include all necessary Operation and Maintenance Manual modi?cations to support the processes. Action Org: Code 106, Shop 99, Code 134, Shop 99, Shop 31 and Code 980 Complete 5. INTERIM CORRECTIVE a. [Problem I, 2, 3, 4] Conduct incremental wastewater processing based on the wastewater processes mitigations developed in Immediate Corrective Action 4.b. Action Org: Shop 99 Complete b. [Problem 1, 2, 3, 4] Gather questions and concerns about potential exposure issues and process problems from the Shop 99 Plant operators. Address those questions and concerns at a meeting with Code 106.2, Shop99 and Bremerton Metal Trades Council (BMTC) representative. Action Org: Shop 99 Complete 0. [Problem 1, 2, 3, 4] Institute a Job Hazard, Analysis in the evaluation of each transfer/treatment process in the Standard Operating Procedure. Action Org: Code 980 Due: 6/28/13 d. [Problem 3] Terminated receiving WSN 211?0003 from 857 pending further investigation. Action Org: Shop 99 Complete e. [Problem 3] Identify all waste streams processed by the IWPF. Action Org: Code 106.3 Due: 6/10/13 f. [Problem 4] Identify and document all material deficiencies in the IWPF 1109 and submit job orders for repair. Action Org: Code 980 Complete OFFICIAL USE ONLY HDeliberative Process Page 7 of 12 FOR OFFICIAL USE ONLY ?Deliberative Process g. [Problem 6] Brief all Code 106.22 employees on industrial hygiene monitoring procedures and applicable response actions. Action Org: Code 106.22 Complete h. [Problem 6] Obtain additional chlorine sensors for the multi-gas meter which will allow for two multi-gas meters to be set-up for chlorine monitoring providing a back?up meter for response actions. Action Org: Code 106.22 . Complete i. [Problem 6] Establish an interim procedure(s) for transfers of wastewater from Building 873 to Building 1109 and for other miscellaneous 1109 processes. Capture detailed data on waste water properties and the transfer process pH, conductivity, previous tank contents, etc.). Specify safety precautions to mitigate personnel exposure concerns engineering controls, prevent personnel from being on the mezzanine, perform remote monitoring, etc.). (0) Establish stop points for gas hazards being monitored and specify response actions if those stop points are reached additional monitoring, alarm activation, calling etc.). Action Org: Code 106.22 Date: 6/14/13 j. [Problem 7] Schedule a 1109 chlorine alarm test. Report results to management. Action Org: Code 980 Complete 6. PERMANENT CORRECTIVE a. [Problem 1, 2, 3, 4, 5] Contract for a complete Architecture and Engineering (AE) assessment of all wastewater processes in the Industrial Wastewater Pretreatment Facility (Bldg 1109). Action Org: Code 980 Due: 10/31/13 b. [Problem 1 2, 3, 4, 5] Based on the AE assessment, revise as required the Operations and Maintenance Manual and develop an Industrial Process Instruction for all waste water treatment processes in 1109, plant equipment de?ciency resolution and required training for 1109 Plant Operators. Assist from Code 134 and Shop 99. Action Org: Code 980 Due: 3/3 1/ 14 c. [Problem 1, 2, 3, 4, 5] Implement plant modi?cation based on the AE recommendations. Action Org: Code 980 Due: 2/28/14 FOR OFFICIAL USE ONLY ?Deliberative Process Page 8 of 12 FOR OFFICIAL USE ONLY ~Deliberative Process . d. [Problem 1, 2, 3, 4] Perform an Industrial Health Survey based on the post AE design and corrective action implementation. Action Org: Code 980 Due: 3/28/14 e. [Problem 1, 2, 3, 4] Coordinate with Naval Hospital Industrial Hygiene Department to accomplish worksite assessments for all processes accomplished by Shop at . Building 1109 once process controls have been established and stabilized. Note: Action from NSRO Agenda Item NSR 13-01. Action Org: Code 106.22 Due: 08/30/13 f. [Problem 1, 2, 3, 4, 5] Establishment a Cognizant Technical Code for all processes in IWPF 1109 involving wastewater transfer and treatment. Action Org: Code 980 Due: 6/28/13 g. [Problem 2] Include in the manual change the requirement to rinse tanks after transferring waste water from the tank. Action Org: Code 980 Due: 12/20/13 h. [Problem 1,2, 3, 4] Codify a Shep 99 management expectation policy speci?cally for the IWPF. Action Org: Shop 99 Due: 6/28/ 13 i. [Problem 3] Validate all waste streams processed by the IWPF for compatibility for treatment in 1109. Assign new waste streams for disposal if necessary. Assist from Code 134. Action Org: Code 106.3 Due: 9/27/13 j. [Problem Develop a process for documenting and tracking plant equipment de?ciencies which includes an equipment status list for review by management. Action Org: Code 980 Complete k. [Problem 5] Add statement of work to the AE contract for the evaluation of the cyanide treatment process 2. Action Org: Code 980 Due: 6/28/13 FOR OFFICIAL USE ONLY ?Deliberative Process Page 9 of 12 FOR OFFICIAL USE ONLY ~Deliberative Process 1. [Problem 6] Develop a Standard Operating Procedure (SOP) detailing response actions for Code 106.22 personnel responding to unusual events resulting in building or space evacuation. Action Org: Code 106.22 Due: 7/12/13 m. [Problem 7] Codify in the facilities maintenance program (MAXIMO) for or quarterly chlorine alarm testing as required by Action Org: Code 980 Due: 6/28/13 n. [Problem 7] Conduct a sample audit of small, medium and large contracts that require scheduled maintenance to verify whether maintenance requirements were entered into the MAXIMO program within the required 90 days from commissioning. Report results to management and take appropriate actions. Action Org: Code 980 Due: 12/20/13 . 7. CORRECTIVE ACTIONLS): a. [Problem 1] Code 980 report to the Chemical High Risk Advisory Committee and make recommendations based on the outcome of the AE study. Action Org: Code 980 Due: 11/14/13 b. [Problem 1] The Chemical High Risk Advisory Committee review the corrective actions outline in Action Org: Code 980 Due: 3/31/14 8. OF. AND RESPONSIBILITY FOR. FURTHER EVALUATION: a. [Problem 1] Create a core Chemical High Risk Advisory Committee. The primary purpose of the group is to identify and bring together responsible shops and codes to de?ne and review high risk chemical processes, to ensure Open communication and sharing of integrated processes and support ownership in the use of high risk chemicals. Core members are Code 106.2 chair, Code 106.22, Code 106.31, Code 106.33, Code .134 and Code 980. Note: Action from NAVSEA Representative Officer (NSRO) Agenda Item NSR 13-01. Action Org: Code 106.2 Due: 6/28/13 b. [Problem 1] Review, modify and/or develop training, quali?cation, and proficiency requirements for high risk chemical processes. Include proper reSponse actions for unusual or unexpected results. Action: Shop 31, Shop 64, Shop 99, Code 106.11, Code FOR OFFICIAL USE ONLY ?Deliberative Process Page 10 of 12 FOR OFFICIAL USE ONLY ~Deliberative Process 106.22, Code 106.31, Code 134, Code 980, and Code 1124 EMTR assist. Note: Action from NSRO Agenda Item NSR 13-01. Action Org: Code 900T.5 Due: 03/28/14 9. SIMILAR TROUBLE REPORTS (BY REPORT NUMBER): 21. Trouble Report Unanticipated Exposure of Personnel to Hydrogen Chloride and Chlorine Vapor issued on 2/13/13 FOR OFFICIAL USE ONLY ?Deliberative Process Page 11 of 12