CALIFORNIA PUBLIC UTILITIES COMMISSION Safety and Enforcement Division Utilities Safety and Reliability Branch Incident Investigation Report Report Date: 4/23/2018 Incident Number: Utility: E20170505-01 PG&E Date and Time of the Incident: 5/5/2017, 6:40:00 AM Location of the Incident: Bahia Substation Benicia, CA County: Solano Summary of Incident: A power outage occurred at Valero’s Benicia Refinery, which resulted in flaring and the release of smoke. My investigation found that a failed transformer at PG&E’s Bahia Substation in conjunction with a planned switching operation activated Valero’s protection scheme causing the refinery to disconnect from Bahia Substation and the subsequent outage. The system operators at the time of the incident failed to identify the transformer as part of Valero’s protection scheme. Fatality / Injury: None reported Property Damage: In excess of $50,000 Utility Facilities involved: Vaca-Bahia 230 kV Circuit, Bahia Substation 1 Witnesses: Name 1 Wilson Tsai 2 Charles Filmer 3 Mark Quinlan 4 Sheriff Gunther 5 Laxmi Terala Evidence: Source 1 PG&E 2 PG&E 3 PG&E 4 PG&E 5 PG&E 6 PG&E 7 PG&E 8 CPUC Title CPUC Investigator PG&E Compliance PG&E Grid Operations Director PG&E Transmission Maintenance Supervisor PG&E Compliance Description Final Electric Incident Report Bahia Station Inspection Meter Reads for April & May 2017 Exponent Report – Valero-Bahia Disconnection Incident Exponent Report Summary – Valero-Bahia Disconnection Data Request Response 1 Description of Operation Database and Procedures UO Guideline G13166, Revision: 9/18/02 PG&E Document TD-1400P-02 – Transmission Application for Work and Timelines Site Visit Photos Observations and Findings: On May 5, 2017 at 0640 hours, a power outage occurred at Valero Energy’s Benicia Refinery in the City of Benicia. The outage lasted for 18 minutes and resulted in flaring and the release of large plumes of smoke from the refinery. A failed coupling capacitor voltage transformer (CCVT) at PG&E’s Bahia Substation in conjunction with a planned maintenance operation at the substation caused the anti-islanding protection scheme at Valero Substation to operate. The operation of the protection scheme disconnected the refinery from PG&E’s Bahia Substation resulting in the subsequent loss of power. PG&E’s Bahia Substation is fed by two 230 kV circuits, the Vaca Dixon-Bahia and BahiaMoraga. Bahia circuit breakers 212 and 242 act as protective devices for the Vaca DixonBahia and Bahia-Moraga lines, respectively. Bahia Substation serves Valero’s Benicia Refinery through Valero’s own substation located within Bahia Substation. Valero Substation connects to Bahia via Valero circuit breakers 476, 486, and 496. Bahia also serves customers through Bahia substation banks #1 and #2. 2 Figure 1: One of Valero Substation’s breakers located within PG&E’s Bahia Substation Figure 2: Valero/Bahia Single-line Diagram including Valero’s Cogeneration plant 3 Valero’s Benicia Refinery refines and produces gasoline, jet fuel, crude oil, and asphalt. In 2002, Valero installed a 51 MW cogeneration power plant at the refinery and “shall be used solely to serve electrical loads connected to the electric service account that PG&E uses to interconnect Producer’s Generating Facility.” The purpose of the cogeneration plant is to support the refinery and not to provide generation to support other electrical loads on PG&E’s system. To prevent this from occurring, Valero developed an anti-islanding protection scheme to disconnect Valero Substation from Bahia Substation. Islanding is the condition in which a distributed generator continues to power a location even though electrical grid power is no longer present. Islanding is hazardous, due to it causing a circuit to remain energized without the utility workers knowledge and it may prevent automatic reconnection of devices to return power to customers. An anti-islanding protection scheme serves to detect islanding and immediately disconnect the distributed generator from the circuit. In the event that both 230 kV lines feeding Bahia Substation lost power, the Substation and PG&E customers off banks #1 and #2 could continue to be served via Valero’s cogeneration plant. However, the cogeneration plant would not be able to sustain the additional load. On May 10, 2017, ESRB conducted a site visit of Bahia Substation. Staff noted the substation’s configuration and that Valero’s substation was enclosed in a subsection of Bahia Substation. Valero’s third circuit breaker was still undergoing repairs at the time. Exponent, a third party engineering firm conducted an investigation into the cause of the incident. On October 5, 2017, Exponent concluded their investigation and presented their findings in the report, “Exponent Report – Valero-Bahia Disconnection Incident.” Exponent’s investigation determined the following sequence of events leading up to the incident: 1. May 1- May 4, 2017 – Bahia-Moraga 230 kV line de-energized for planned work on transmission towers. Valero notified of work on March 2. 2. Bahia-Moraga was scheduled to be re-energized by 1800 hours on May 4, but during switching to re-energize line, operator noticed near-zero potential (voltage) reading on the line. 3. Operator and station staff determined that a failed CCVT connected to phase C on the line side of circuit breaker 242 failed. Operator reviewed single-line meter and relay diagram and description document for Valero/Bahia anti-islanding protection scheme and determined no protection was jeopardized by the failed CCVT. 4. Operator notified substation maintenance and operations of the failed CCVT and reenergized Bahia-Moraga line. 4 5. On May 5, 2017 at 0549 hours, scheduled switching to de-energized Vaca Dixon-Bahia 230 kV line started. Valero was notified of work on March 6. 6. Previous operator on duty relived by another operator at approximately 0600 hours. PG&E provided the following timeline of events for the incident: 1. On the day of the incident, May 5, 2017, Vaca-Bahia 230kV line was to be removed from service for scheduled work. 2. At around 0640 hours, while performing routine switching to clear the Vaca-Bahia 230kV line, the Bahia Circuit Breaker, CB 212 was opened by PG&E. At this time, Valero’s 230kV Circuit Breakers 476, 486 and 496 opened automatically. 3. A sustained interruption to Valero Substation resulted, causing loss of power at the Valero Refinery. Valero Substation is owned and operated by Valero Refining. 4. At 0647 hours, Bahia Circuit Breaker, CB 212 was closed by PG&E. 5. At 0648 hours, PG&E gave permission to Valero to reconnect to PG&E’s transmission system. 6. At 0658 hours, Valero closed the first of the three 230kV circuit breakers. 7. At 0659 hours, Valero closed the second of the three 230kV circuit breakers. Any of the following conditions will cause the Valero anti-islanding protection scheme to activate: 1. Both Bahia circuit breakers 212 and 242 are opened. 2. There is an undervoltage condition on both the Bahia-Moraga and Vaca Dixon-Bahia 230 kV lines. 3. Bahia circuit breaker 212 is opened and there is an undervoltage condition on the Bahia-Moraga line. 4. Bahia circuit breaker 242 is opened and there is an undervoltage condition on the Vaca Dixon-Bahia line. Exponent describes each of the activation conditions: 1. “When Bahia circuit breakers 242 and 212 are opened, the Bahia substation has no connection to the rest of the 230 kV network and is therefore separated from the PG&E network. 2. When Bahia circuit breakers 242 and 212 are closed but the voltage on both lines is abnormally low, it indicates that the Bahia substation is disconnected from transmission paths at more remote locations than Bahia breakers 242 and 212. For example, if Moraga circuit breaker 222 and Vaca Dixon circuit breaker 462 are opened, Bahia substation loses both its transmission sources and is therefore separated from the PG&E network. 3. When Bahia circuit breaker 212 is opened, the Bahia substation has a single transmission source through the Vaca Dixon-Bahia line. In this case, if the Vaca Dixon5 Bahia line voltage drops due to being remotely disconnected, then the Bahia substation has lost both transmission sources and is therefore separated from the PG&E network. 4. An open circuit breaker 242 and a loss of the transmission path through the BahiaMoraga line is similar to the third scenario described above.” When the CCVT on circuit breaker 242 failed, it created an undervoltage condition on the Bahia-Moraga line. On the day of the incident, PG&E crew opened circuit breaker 212 to clear the Vaca Dixon-Bahia line. With this, both conditions were satisfied for condition 3, which activated the anti-islanding protection scheme. Additionally, it shows why Valero circuit breakers 476, 486, and 496 opened as soon as Bahia circuit breaker 212 opened. “The anti-islanding scheme description documents provide four descriptions of operations labeled A through D… Description of operations part A describes the activation requirements of the anti-islanding scheme. The system operator on duty on May 4, 2017 reviewed this section of one of the anti-islanding scheme description documents as well as the single-line meter and relay diagram... When the CCVT failure on the Bahia-Moraga line was discovered on May 4, the operator used this information to identify that opening Bahia breakers 242 and 212 would cause an anti-islanding scheme activation but did not recognize that the undervoltage part of the scheme depended on the failed CCVT. Description of operations part D describes the conditions for the anti-islanding scheme to be cut out. The description of operations states that the scheme can or should be cutout when a line potential device is cleared. However, the system operator stated he did not specifically clear the line potential device, which would have involved isolating the high-voltage jumper on the CCVT and opening the low side connections. Therefore, he did not believe it was necessary to cutout the anti-islanding scheme. Nevertheless, the line-side CCVT was not identified as being the source of the undervoltage relay device for the anti-islanding scheme.” On the tools and training on the anti-islanding scheme (including the description document), Exponent stated: “The anti-islanding scheme description document is perceived by operators to be unclear on conditions necessary to activate the scheme, the instrumentation equipment (such as CCVTs) that the scheme depends on, and the conditions under which the scheme should be cut out. The operator on duty when the CCVT failure was discovered reportedly reviewed the antiislanding scheme description document and single-line meter and relay diagram but did not identify that the failed CCVT provided a voltage signal input to the anti-islanding scheme. 6 The perception among system operators is that the level of formal training on anti-islanding schemes specific to the operator’s area is minimal. Training on anti-islanding schemes in each operator’s area is mostly on-the-job training.” Exponent made the following recommendations to prevent this incident from reoccurring: 1. “Consider adding EMS line-item alarms and/or SCADA screen indicators for activated anti-islanding protection scheme inputs.” 2. “Modify the existing PG&E procedure TD-1400P-02 “Transmission Application for Work and Timelines” to require a check of whether an anti-islanding protection scheme is present in the clearance area and identify the anti-islanding scheme activation conditions in the pre-outage requirements.” 3. “Consider performing regular reviews of anti-islanding scheme description documents to ensure that the conditions for scheme activation are clearly indicated and the instrumentation equipment that they rely on, such as CCVTs, is indicated. 4. “Modify the existing PG&E procedure TD-1400P-01 “Transmission Operating Procedures” to require contacting the System Protection group to verify the status of protection systems when instrumentation equipment such as CCVTs fails.” 5. “Consider developing regular required refresher training for operators on specific antiislanding protection schemes.” PG&E plans to implement all of Exponent’s recommendations. PG&E is currently in the process of implementation with the intent to complete all recommendations within 2018. GO 174, Rule 12: General, states in part: “Substations shall be designed, constructed and maintained for their intended use, regard being given to the conditions under which they are to be operated, to promote the safety of workers and the public and enable adequacy of service.” Valero’s anti-islanding protection scheme tripped as a result of the failed CCVT and a routine switching procedure to clear the Vaca Dixon-Bahia line. The CCVT failed a day prior to the incident but despite reviewing system diagrams and documents, operators of PG&E’s Bahia Substation did not believe the CCVT would affect operations. Operators on the day of the incident also failed to identify the CCVT as part of Valero’s anti-islanding protection scheme. Therefore, PG&E is in violation of General Order 174, Rule 12 for not maintaining the CCTV for its intended use. In addition, PG&E did not provide adequate training, documents, or diagrams for its operators to fully understand the anti-islanding protection scheme. If the Bahia substation operators had sufficient training and documentation on the anti-islanding protection scheme, they would have properly identified and resolved the failed CCVT and prevented the incident from occurring. 7 Preliminary Statement of Pertinent General Order, Public Utilities Code Requirements, and/or Federal Requirements: 1 General Order GO 174 GO Rule 12 Violation Yes Conclusion: My investigation found that PG&E violated General Order 174, Rule 12 because 1) PG&E failed to maintain the CCTV for its intended use and also 2) PG&E failed to provide adequate training, documents, or diagrams for its operators to identify the CCVT as part of the anti-islanding protection scheme since the documents, diagrams, and training on the protection scheme did not provide enough clarity on the activation conditions and how they relate to the failed CCVT. 8