STATE OF MAINE PUBLIC UTILITIES COMMISSION Docket No. 2019-00015 January 9, 2020 PUBLIC UTILITIES COMMISSION Investigation into Central Maine Power Company’s Metering and Billing Issues EXAMINERS’ REPORT Note: This Examiners’ Report is written in the form of a Commission order but is the Staff’s recommendation. Parties may file exceptions to this Examiners’ Report by close of business on Thursday, January 23, 2020, for the Commission’s consideration. Table of Contents I. INTRODUCTORY SUMMARY ............................................................................. 1 II. EXECUTIVE SUMMARY ...................................................................................... 2 A. Breadth of This Investigation ..................................................................... 3 B. No Systemic Problem Creating Erroneous Usage on Bills......................... 3 C. Consideration of SmartCare ...................................................................... 5 Imprudence ..................................................................................... 5 2. Added Testing with Third Party; Ongoing Oversight and Maintenance ................................................................................... 6 3. Remedies for Specific Defects ........................................................ 6 D. Establishment of Independent Electricity-Use Audit Program .................... 7 E. Termination of Interim Payment Policy ...................................................... 7 F. III. 1. 1. Policy Terminated Effective with this Order..................................... 8 2. Request for Comment on Follow-on Procedures ............................ 8 Conclusion ................................................................................................. 8 PROCEDURAL AND FACTUAL BACKGROUND ................................................ 9 A. Events Leading Up to Docket No. 2018-00052 .......................................... 9 B. Summary Commission Investigation in Docket No. 2018-00052 and the Liberty Audit ............................................................................................. 10 C. Establishment of Interim Payment Policy ................................................. 11 D. Opening of This Adjudicatory Investigation .............................................. 13 1. Bifurcation of Issues Between Metering and Billing (in This Docket) and Customer-Service-Related Issues (in the Rate Case Docket) 13 2. Processing of This Case ............................................................... 14 3. Public Witness Hearings in July .................................................... 15 Examiners’ Report IV. Docket No. 2019-00015 OVERVIEW OF POSITIONS OF THE PARTIES AND SUBMISSIONS ............. 16 A. B. C. D. V. ii Liberty Audit ............................................................................................. 16 1. Overview ....................................................................................... 16 2. Metering and Billing System Accuracy .......................................... 18 3. SmartCare Implementation ........................................................... 29 Central Maine Power Company ............................................................... 34 1. Metering and Billing....................................................................... 34 2. SmartCare Implementation ........................................................... 42 3. Remedies ...................................................................................... 45 Office of the Public Advocate ................................................................... 46 1. Metering and Billing....................................................................... 46 2. SmartCare Implementation ........................................................... 48 3. Remedies ...................................................................................... 53 Lauren Loomis and CMP Ratepayers Unite ............................................. 54 1. Metering and Billing....................................................................... 54 2. Remedies ...................................................................................... 55 E. Levesque Intervenors .............................................................................. 56 F. Commission Staff ..................................................................................... 56 G. Mary Fournier .......................................................................................... 59 OVERVIEW OF INDIVIDUAL COMPLAINANTS’ POSITIONS ........................... 59 A. B. C. Experience of the Consumer Assistance and Safety Division .................. 59 1. Adoption of Special Process for Managing Complaints of HighUsage ............................................................................................ 59 2. Number of Complaints Raised and Resolved ............................... 61 Public Witness Testimony ........................................................................ 62 1. High Bills ....................................................................................... 62 2. Delayed Bills or New Customers Not Receiving Bills .................... 64 3. Incorrect or Confusing Bills ........................................................... 66 4. Estimated Bills............................................................................... 67 CMP’s Response to Individual Complainants’ Statements ...................... 68 1. Response to Concerns About Usage ............................................ 69 2. Response to Concerns Over Net-Energy Billing ........................... 71 3. Response to Customers with Non-Standard Meters ..................... 71 4. Response to Claims That the Company Made an Error or Was Delayed in Processing a Transaction ............................................ 71 Examiners’ Report D. VI. VII. Docket No. 2019-00015 5. Response to Claims of Being Denied for a Low-Income Program 72 6. Response to Complaints About Frequency of Power Outages ..... 72 7. Response to Complaints About Service Options .......................... 72 Overview of Individual Complainants’ Rebuttal to CMP’s Case Studies .. 74 1. Claims That the Company’s Portrayal of Its Interaction with the Customer Was Misleading or Inaccurate....................................... 75 2. Claims That the Company’s Case Study Left Out Key Information77 3. Claims That the Case Study Failed to Address the Customer’s Primary Concern ........................................................................... 78 STATUTORY STANDARDS ............................................................................... 79 A. The Prudence Standard ........................................................................... 79 B. The Evidentiary Burden ........................................................................... 82 DISCUSSION AND DECISION .......................................................................... 82 A. Is CMP’s SmartCare System Accurately Recording Usage? ................... 82 B. How Were Customers Affected by CMP’s Anomalous Meters? ............... 85 C. Were CMP’s Customers Affected by Other Billing Issues? ...................... 91 D. VIII. iii 1. Liberty’s Bill Reconciliation and Proration/Truncation ................... 92 2. Defects and Errors ........................................................................ 94 3. Conclusion .................................................................................. 106 Was CMP’s Implementation of SmartCare Reasonable and Prudent? .. 107 1. Compression of Testing Schedule .............................................. 108 2. Relaxation of Standards for Testing and for Go-Live .................. 109 3. Insufficient Resources to Handle Post-Go-Live Errors and Exceptions .................................................................................. 109 4. Conclusion .................................................................................. 111 REMEDIES ....................................................................................................... 112 A. Additional Oversight and Testing of the SmartCare System .................. 112 B. Monitoring Future Performance ............................................................. 115 C. The OPA’s Proposal to Disallow SmartCare Investments ...................... 116 1. Positions Before the Commission ............................................... 116 2. Decision ...................................................................................... 118 D. Independent Electricity-Use Audit Program for Resolving Yet-Unresolved Usage Complaints ................................................................................. 120 E. Costs of the Liberty Audit ....................................................................... 127 Examiners’ Report IX. X. iv Docket No. 2019-00015 THE INTERIM PAYMENT POLICY .................................................................. 128 A. Background............................................................................................ 128 B. Termination of Policy ............................................................................. 128 C. Proposed Method for Closing Out Policy and Paying Amounts Set Aside; Request for Comments .......................................................................... 129 CONCLUSION AND ORDERS ......................................................................... 131 List of Appendices Appendix A: Detailed Procedural History Appendix B: Tabular Summary of SmartCare-Related Defects Examiners’ Report I. 1 Docket No. 2019-00015 INTRODUCTORY SUMMARY As detailed in this order, the Commission makes two principal findings. First, the Commission finds that the evidence in the record shows that there is no systemic problem within Central Maine Power Company’s (CMP or the Company) metering and billing systems that has caused erroneous high usage on customers’ bills. Instead, the evidence—including the detailed forensic audit conducted by an independent third-party auditor—demonstrates that CMP’s metering and billing systems have been, and continue to be, recording and transmitting customer usage data accurately, and, with the exception of discrete billing calculation and presentation issues, customers’ billed amounts have been accurate. The large number of customers who experienced high bills and high usage in late 2017 and early 2018 resulted from increases in electricity usage due to a record-breaking cold snap from December 2017 into January 2018 and a double-digit increase in the standard-offer electricity-supply price in January 2018. While billing defects in SmartCare (as well as human error) did, in many cases, lead to incorrect billed amounts or incorrect displays of billed usage, the evidence in the record demonstrates that errors in CMP’s metering and billing systems were not a root cause of the significant bill increases many customers experienced during the winter of 2017– 2018. Given this finding—that CMP’s metering system is accurately recording usage, is accurately transmitting it to the billing system, and, with the exception of discrete defects that have been or are being corrected, is accurately billing customers—the Commission orders that: (a) the interim payment policy is hereby terminated, and customers may no longer defer portions of their bill for future payment under that policy; and (b) CMP shall establish an Independent Electricity-Use Audit Program, in Examiners’ Report 2 Docket No. 2019-00015 partnership with a third party, to streamline the resolution of yet-unresolved customer complaints about high usage. Second, the Commission finds that some aspects of CMP’s management of the implementation of its billing software, known as SmartCare, were imprudent. Due to the flaws in the implementation, after CMP went live with SmartCare, defects or exceptions affected tens of thousands of customers who experienced delayed bills or bill errors. Most, though not all, of these defects have now been corrected. In response to this imprudence, the Commission requires CMP to do the following: (a) engage a third party to conduct targeted testing of SmartCare and remedy the few remaining defects under third-party oversight; (b) report to the Commission monthly on the status of closing out open defects in SmartCare; and (c) submit a comprehensive plan for managing the ongoing maintenance of the SmartCare system. The costs for these items will be borne by CMP’s shareholders, not its ratepayers. II. EXECUTIVE SUMMARY The Commission’s investigation of CMP’s metering and billing practices is, in many ways, without precedent. The Commission has not in recent history—and probably never before—seen complaints against a utility reach the numbers they have here, nor seen the kind of public skepticism of customers’ utility bills that has been raised against CMP in the last two years. The unusual circumstances that created this skepticism—record-high electricity usage and experience with an unfamiliar and errorprone software program—demanded a regulatory response. Here, the Commission provides a high-level description of this investigation and the conclusions in this order. Examiners’ Report A. 3 Docket No. 2019-00015 Breadth of This Investigation The Commission’s investigation has been nothing if not thorough. It began in the spring of 2018 and consisted of three major phases: the summary investigation in Docket No. 2018-00052, The Liberty Consulting Group’s forensic audit of CMP’s metering and billing systems, and the adjudicatory investigation in this docket. Liberty’s audit was wide-ranging and in-depth, examining billing data for the six-month period following the cutover to SmartCare, November 2017 through April 2018. Liberty examined data for every CMP customer—about 650,000 meters, including 7,400 manually read meters—for this six-month period. This involved an analysis of about 4 million billing records and about 2.3 billion data points to trace data from one end of CMP’s metering and billing system to the other. Liberty also conducted 52 interviews with CMP’s management and analyzed a statistically significant sample of CMP’s meters. Liberty published its detailed audit report in December 2018. In developing the record, CMP (and, to a lesser degree, the intervenors, Liberty, and Commission Staff) responded to well over 700 data requests with narrative answers and voluminous document production, and answered questions in six days of technical conferences and two days of hearings. Those responses to discovery, Liberty’s audit, the record transcripts, and several rounds of pre-filed testimony in this docket have produced countless pages of valuable information about CMP’s metering and billing practices. The Commission also heard impassioned testimony from nearly 80 public witnesses on their experiences and frustrations as customers of CMP. B. No Systemic Problem Creating Erroneous Usage on Bills In response to allegations that customers were billed for electricity they did not use, Liberty, the Staff, and the OPA took different approaches to analyzing CMP’s Examiners’ Report 4 Docket No. 2019-00015 metering and billing data. Based on the results of the forensic audit it conducted of the operations of CMP’s meters, and the accuracy of CMP’s meter-data storage, meter-data transmission, and billing system during the 2017–2018 winter period, Liberty concluded that CMP’s systems were accurately metering customer usage, accurately transmitting the metered usage to the billing system, and accurately billing for that usage. Given the results of its forensic audit and its corroborating analysis of CMP customers’ usage as a function of cold weather, Liberty attributed high bills to extreme cold weather during the winter of 2017–2018 that coincided with a large increase in the price for electricity supply. Commission Staff corroborated Liberty’s conclusion by showing that the increases in billed usage in CMP’s service territory after SmartCare went online were nearly identical to the increases in billed usage in Emera Maine’s adjacent service territory during the same period. For its part, despite analyzing data focused only on those customers who made complaints of high usage, the OPA could find no systemic cause of that high usage in CMP’s systems, and found no cases where customers were ultimately billed for electricity they did not use. Defects in SmartCare that affected customers immediately following go-live— which in many cases led to errors on customers’ bills or significantly delayed bills—were likely a contributing factor to customers’ perception of overbilling. But the large number of complaints customers lodged about high bills and high usage in late 2017 and early 2018 were chiefly due to increases in customers’ usage relating to a record-breaking cold snap from December 2017 into January 2018 and a double-digit increase in the standard-offer supply price in January 2018. Examiners’ Report 5 Docket No. 2019-00015 Overall, there is simply no evidence of a metering or billing problem that is leading to erroneously high billed usage. In exceedingly rare cases involving an unusual series of events, some meters in CMP’s service territory were susceptible to registering incorrect usage. Because of the way these “registration anomalies” occur and the way they self-correct, the likelihood of it having a meaningful or noticeable effect on a customer’s bill is even lower than the likelihood of it happening to any individual customer. As of November 5, 2019, CMP had already completed, for 94% of customers’ meters, firmware upgrades that will eliminate this problem going forward; CMP continues to perform those upgrades for the remaining meters. We are ordering CMP to compensate any customers found to be affected by this for the amount that they are calculated to have been overbilled, if any. C. Consideration of SmartCare 1. Imprudence Prior to going live with SmartCare, CMP compressed the schedule for critical testing of the software. Instead of running different types of tests in a serial fashion— one after the other, as they had originally planned—CMP opted to run critical testing in parallel, or concurrently with one another. CMP also contradicted its software integrator in finding that the overall system was essentially ready for go-live, where the software integrator had recently identified critical ongoing, open issues with the software that required attention, and gave the project a red-light. Also, close to go-live, CMP relaxed its standards for go-livereadiness. Liberty and the OPA’s consultant, BerryDunn, criticized these decisions. These facts, and others, lead the Commission to find that CMP’s implementation of SmartCare was imprudent. By “imprudent,” we mean that CMP did not act under a Examiners’ Report 6 Docket No. 2019-00015 “course of conduct that a capably managed utility would have followed in light of existing and reasonably knowable circumstances.” 2. Added Testing with Third Party; Ongoing Oversight and Maintenance As a remedy, the Commission orders CMP to arrange for additional testing of SmartCare, to be performed by a third party. The testing will not be a full top-to-bottom retest of the system, but will be designed to target gaps in the initial testing as well as any new testing that is needed. The independent third party will be selected based on a competitive-bidding process and will be subject to Commission approval. The Company will also need to provide the Commission, in short order, plans and reports that will directly require CMP to act and will provide critical information to allow the Commission to closely monitor ongoing defect resolution and system maintenance. 3. Remedies for Specific Defects After go-live, tens of thousands of customers experienced bill errors related to defects in SmartCare. Between go-live in November 2017 and November 2019, CMP had identified around 70 defects, many of which have been corrected. Many of these errors had no financial effect on customers, but instead were presentment errors—that is, errors in the way information was presented on the bill, but which did not affect the amount owed as stated on the bill. Some had a small financial effect, and in many cases CMP remedied these by correcting the error on the customers’ account. A few had a large financial effect, and one of the more important of these affected only lowincome customers. For certain defects, we prescribe additional remedies for affected customers. Examiners’ Report D. 7 Docket No. 2019-00015 Establishment of Independent Electricity-Use Audit Program The Commission is requiring CMP to establish an Independent Electricity-Use Audit Program for customers who have experienced significant increases in usage, the cause of which remains unexplained, since SmartCare was adopted. The purpose will be to obtain a resolution of these questions for the customer. The initial costs of this program will be borne by CMP’s shareholders. Under this program, eligible customers may obtain an electricity-usage audit conducted by an independent entity such as Efficiency Maine. 1 The audit would compare the expected amount of electricity usage by the customer with the amount for which the customer is metered and billed. If the audit shows that billed usage is consistent with expected usage, the auditor will provide the customer advice about options for reducing usage, including through programs offered by Efficiency Maine. If the audit shows a discrepancy between the customer’s electricity usage and the amounts being metered and billed by CMP, more analysis and review will be conducted to determine the cause and amount of the discrepancy, as well as to determine an appropriate remedy for the customer. E. Termination of Interim Payment Policy The interim payment policy has been in place since 2018. Under that policy, customers whose billed usage or total bill was at least 25% higher than that for the same month in the prior year were allowed to defer for possible future payment the difference in the bill, pending the outcome of this investigation. The amounts customers 1 This order uses the terms “Efficiency Maine Trust,” “Efficiency Maine,” and “Trust” interchangeably. Examiners’ Report 8 Docket No. 2019-00015 were not required to pay immediately under that policy have come to be known as the “set-aside amount” or as having been “set aside.” 1. Policy Terminated Effective with this Order The purpose of the interim payment policy was to give customers the benefit of the doubt about their high usage until the Commission could determine whether the billed amounts were valid. Through this investigation, the Commission has confirmed that billed amounts were, indeed, valid, with few exceptions, and there is no systemic problem in SmartCare or with CMP’s meters that would lead to incorrectly billed usage. The policy, thus, no longer serves its intended purpose. The Commission terminates the interim payment policy effective with this order. 2. Request for Comment on Follow-on Procedures There remain open questions about the treatment of amounts that have been set aside. The Commission proposes a process in which customers who availed themselves of interim payment policy would enter into payment arrangements to pay those set-aside amounts if they have not already been placed on a payment arrangement by the CASD or agreed to a payment arrangement with the Company. We ask the parties to comment on this proposal, or propose alternatives, within 30 days of this order. F. Conclusion In addition to the remedies we describe in this order, in the companion case (CMP’s rate investigation, in Docket No. 2018-00194) the Commission has ordered a downward adjustment to CMP’s return on equity of 75 basis points, which has the effect of lowering the approved revenue requirement in that case by approximately $5 million. This downward adjustment addresses CMP’s substandard customer service. Many of Examiners’ Report 9 Docket No. 2019-00015 the complaints customers have had about their bills have not been merely about mathematical errors—they have been closely related to the substandard customer service we describe in our order in Docket No. 2018-00194. In ordering remedies, the Commission’s goal is to improve outcomes for customers and, having determined what problems led to this situation, fix the problem going forward and ensure that CMP makes customers whole in their bills. III. PROCEDURAL AND FACTUAL BACKGROUND This background section lays out both the events leading up to this case and the most critical procedural steps of the case itself at a high level. Please see Appendix A for a detailed procedural history. A. Events Leading Up to Docket No. 2018-00052 In late October 2017, Central Maine Power Company (CMP or the Company) went “live,” or operational, with its new customer information system and billing program, known as SmartCare. The go-live of SmartCare coincided with an extraordinary windstorm, which created the largest power outage in the Company’s history. See Public Utilities Commission, Investigation into the Response by Public Utilities to the October 2017 Storm, Docket No. 2017-00324, Order at 11 (Oct. 4, 2018). Two months later, the standard-offer electric-supply price increased by 18% for CMP’s residential customers. Commission-Initiated Standard Offer Bidding Procedure for CMP and Emera Maine-BHD Small, Medium, and Large Non-Residential, Docket No. 201700184, Order Designating Standard Offer Providers and Directing Utility to Enter [Into] Entitlements Agreement (Nov. 8, 2017). Around that same time, CMP’s service territory experienced approximately two weeks of extremely low temperatures and, Examiners’ Report 10 Docket No. 2019-00015 consequently, record electricity usage among customers. See Liberty Report 2 at 57. By early 2018, the CASD had received approximately 380 complaints from CMP customers, most of which related to high bills and possible billing errors. A number of these customers also reported difficulty, or a complete inability, in reaching someone at CMP to discuss their billing issues. B. Summary Commission Investigation in Docket No. 2018-00052 and the Liberty Audit While the CASD had been able to resolve many of the complaints filed by early 2018, the Commission found that additional information was needed to determine and understand the existence and source of any metering, billing, and customer communication problems affecting CMP’s ability to provide service to its customers. Thus, due to the numerous customer complaints about high bills and other problems customers were experiencing with their bills, the Commission opened a summary investigation of CMP’s metering, billing, and customer communication issues on March 1, 2018, to determine whether CMP was meeting its obligation to provide reasonable, adequate, and reliable service to its customers. Pub. Utils. Comm’n, Investigation of Central Maine Power Co. Metering, Billing and Customer Communication Issues, Docket No. 2018-00052, Notice of Investigation (Mar. 1, 2018). 3 2 Liberty Report refers to the Final Report – Forensic Audit of CMP’s Metering and Billing Systems, prepared by The Liberty Consulting Group and presented to the Commission (and made public) on December 20, 2018. It was filed in this docket on February 22, 2019. 3 On July 10, 2018, the Commission expanded the scope of Liberty’s audit to include the customer-communication issues identified in the Commission’s March 1, 2018 Notice of Investigation. Docket No. 2018-00052, Order Modifying Scope of Audit at 1 (July 10, 2018). The issue of customer communication and customer service is addressed in Docket No. 201800194. Examiners’ Report 11 Docket No. 2019-00015 Shortly thereafter, given the complexity of the metering and billing issues, the Commission initiated a forensic audit of the Company’s customer billing system. Docket No. 2018-00052, Order Initiating Audit (Mar. 22, 2018); see also 35-A M.R.S. § 113. The Commission engaged The Liberty Consulting Group (Liberty) to complete the forensic audit. After completing its audit, on December 20, 2018, Liberty issued its final report on all aspects of the audit. 4 This became known as the Liberty Report. C. Establishment of Interim Payment Policy On March 12, 2018, the Office of the Public Advocate (OPA) filed a request that the Commission order CMP to take no action on disconnections after the “winter disconnect” period expired on April 15, 2018. The OPA also requested that the Commission order a stay on any new disconnection notices pending the outcome of the Commission’s summary investigation. In response to the OPA’s request, the Commission established a payment policy for residential customers who had experienced a significant increase in their electric bills and disputed the increase. Pub. Utils. Comm’n, Investigation of Central Maine Power Company Metering, Billing, and Customer Communication Issues, Docket No. 2018-00052, Order at 1 (Apr. 11, 2018) (April 11, 2018 order). This became known as the interim payment policy. The policy established a methodology for defining an undisputed amount of the customers’ bills which customers would be required to pay to avoid disconnection while the summary investigation was pending. The policy also defined a class of customers who were eligible to take advantage of this mechanism. Id. at 3–4. The policy defined “eligible customers” as “any residential customer who has 4 The Commission filed the Liberty Report in this docket on February 22, 2019. Examiners’ Report 12 Docket No. 2019-00015 received or will receive a bill issued on or after November 1, 2017 that reflects total CMP delivery charges that are 25% or more than delivery charges for which the customer was billed for the same month in the prior year and the customer has disputed the increase (Criteria One).” Id. at 3. For customers who were not receiving CMP service 12 months prior to the month of the bill at issue, the April 11, 2018 order defined an “eligible customer” as “any residential customer who has received, or will receive, a bill issued on or after November 1, 2017 that reflects total CMP delivery charges that are 25% higher than the average bill issued to that customer prior to November 1 of 2017 and the customer has disputed the increase (Criteria Two).” Id. at 3–4. Under the April 11, 2018 order, any customer that met the definition of “eligible customer” (including the requirement that the customer has disputed the increased bill) could not be threatened with disconnection or have their service disconnected for failure to pay the entire amount of the bill for any qualifying month 5 provided that the customer paid the following amounts for each qualifying month: (1) for eligibility Criteria One, an amount that is at least equal to the amount of the delivery service bill issued for the same month in the prior year plus, if applicable, standard-offer charges based on current standard-offer prices and the kWh usage billed for the same month in the prior year; or (2) for eligibility Criteria Two, an amount that is at least equal to the average delivery service bill issued to that customer prior to November 1, 2017, plus, if applicable, standard-offer charges based on current standard-offer prices and the average kWh usage reflected in the customer’s bills prior to November 1, 2017. If a 5 A “qualifying month” was any month in which a customer’s delivery charges met either Criteria One or Criteria Two. Charges associated with any month that did not meet Criteria One or Criteria Two could be collected by CMP under its normal credit and collection procedures. Examiners’ Report 13 Docket No. 2019-00015 customer failed to pay at least the amount specified in (1) or (2) above for a qualifying month, CMP could pursue disconnection under section 10 of Chapter 815 of the Commission’s rules. D. Opening of This Adjudicatory Investigation 1. Bifurcation of Issues Between Metering and Billing (in This Docket) and Customer-Service-Related Issues (in the Rate Case Docket) Following the release of the Liberty Report, the Commission opened a full investigation into CMP’s metering, billing, and customer-communication issues. The Commission decided that metering and billing issues would be addressed in this docket (Docket No. 2019-00015), and customer-service-related issues would be addressed in the ongoing investigation into CMP’s rates and revenue requirement (Docket No. 201800194). In bifurcating the investigation in this way, the Commission stated: In reaching the conclusion that the billing and metering investigation should be conducted as part of a separate adjudicatory proceeding while the investigation of the customer service and communication investigation be consolidated into the rate case, the Commission finds that the billing and metering issues and the customer service issues are distinct enough to have the follow-up investigations of these issues performed on separate tracks. The one area where there may be some overlap is the issue of bill errors or “exceptions.” This overlap area can be resolved by including bill error[s] or exception[s] that actually involved wrong or erroneous bills being issued to customers in the separate metering and billing investigation. Billing error issues involving customer service, communication, and delays in billing customers can be addressed in the rate case. Jan. 14, 2019 Order and Notice of Investigation at 9–10 (emphasis added). Examiners’ Report 2. 14 Docket No. 2019-00015 Processing of This Case To begin this investigation, the Staff granted the petitions to intervene of the OPA, several representatives or members of the Facebook group CMP Ratepayers Unite, 6 Dot Kelly, Ed Friedman, Mary Fournier, and a group of ratepayers known as the Levesque Intervenors. 7 CMP was automatically made a party to the case. Later, the Staff granted the petition to intervene of the Governor’s Energy Office. 8 In February through April, the parties conducted discovery on the Liberty Report, including a workshop, written discovery, and a technical conference. In May and June, CMP filed its testimony, and the parties and Staff conducted discovery on it. In June, the OPA requested approval to analyze the accounts of CMP customers who had made high-bill complaints after the Liberty Report’s study period—i.e., beginning May 1, 2018. The OPA expected to complete both its analysis and the drafting of its written testimony within eight weeks. The Staff granted the OPA’s request. On August 30, Lauren Loomis (for CMP Ratepayers Unite) and Dot Kelly submitted written testimony. On September 3, the Staff issued its Bench Analysis and the responsive reports of Liberty on (a) SmartCare implementation and (b) the reconciliation of CMP’s billing exceptions. On September 6, the OPA filed the testimonies of: (x) Laurel Arnold of BerryDunn on SmartCare implementation; (y) Julie 6 These ratepayers included Lauren Loomis, Judith Hopkins Hyde, Valery Harris, Kristy Pottle, and Kathleen Doucette. Ms. Loomis was the administrator of CMP Ratepayers Unite on Facebook. 7 The Levesque Intervenors were CMP ratepayers Mark Levesque, Christie Decker, Lisa Mcleod, Michael Platt, Katie Morin, Nicole and Phillip Riley, Brittney Russell, Marc and Jen Day, Carol Foss, and Julia Lawson. 8 Representative Seth Berry also filed a petition to intervene, which was granted, though Rep. Berry later withdrew as a party. Examiners’ Report 15 Docket No. 2019-00015 Keim of BerryDunn on CMP’s metering and billing issues; and (z) CMP ratepayer N. David Semon on issues he experienced with his bills. Discovery on these filings followed in September 2019. In October, CMP submitted its rebuttal testimony, which was followed by discovery. On November 5 and 6, the Commission presided over evidentiary hearings. On November 19, the OPA and CMP submitted post-hearing briefs; on November 26, the OPA, CMP, and CMP Ratepayers Unite submitted reply briefs. The Staff issued its Examiners’ Report on January 9, 2020. 3. Public Witness Hearings in July The Commission presided over three public-witness hearings in this case (in conjunction with the rate case, Docket No. 2018-00194). The first was held at the University of Southern Maine in Portland on July 16, 2019 (where 31 people testified); the second was held at the University of Farmington on July 18, 2019 (where 15 people testified); and the third was held at the Commission’s offices in Hallowell on July 22, 2019 (where 32 people testified). The Commission also accepted affidavits from public witnesses who could not attend one of the hearings, provided the affidavits: (1) were truthful, sworn, and notarized; (2) explained why the individual was unable to attend any of the public witness hearings; (3) provided testimony on the matters before the Commission in these dockets; and (4) no later than Monday, July 22, 2019, (a) received by the Commission or (b) postmarked and sent to the Commission. Once received, the Commission Staff reviewed the affidavits both to determine whether they meet the four conditions and to redact customer-specific information and graphs of usage history from Examiners’ Report 16 Docket No. 2019-00015 them. Twenty-three affidavits were determined to be admissible and uploaded into the case file. IV. OVERVIEW OF POSITIONS OF THE PARTIES AND SUBMISSIONS Most of the parties who actively participated in this case submitted testimony or a brief. The parties’ positions on the issues in this case, as well as the findings and conclusions in the Liberty Report, are summarized as follows. A. Liberty Audit 1. Overview In its audit, Liberty addressed the following six questions: (a) whether meters have produced and are producing accurate measurements of customer usage; (b) whether database communications systems have accurately, completely and timely transmitted meter data to meter data management and billing systems, and are currently doing so; (c) whether CMP’s billing system produced accurate and timely bills, and is it continuing to do so; (d) whether CMP’s bills conveyed accurate usage information to its customers, and do they continue to do so; (e) whether the implementation and cutover to the SmartCare billing system affected customer billing or customer complaints, and (f) whether customer communications around SmartCare billing and its implementation was timely and effective. Liberty Report at 1. To answer these questions, Liberty undertook the following factfinding activities leading up to the issuance of its report: Examiners’ Report • 17 Docket No. 2019-00015 52 interviews with CMP’s management, conducted in successive rounds as Liberty gained knowledge from the other work in progress; • 215 data requests, also conducted in successive rounds as Liberty learned more about the seven major components of the meter-to-bill process; • Random selection of a sample of meters to test in the field for accuracy, based on a statistically driven approach designed to corroborate the results of CMP management’s own meter testing; • Direct in-the-field observation of those tests and recording of their results; • Direct observation of the systems that maintain usage and billing records to understand their operation and to determine how to test their completeness and accuracy; • Extracted more than 4 million records from those systems to create a master database for use in testing their accuracy, completeness, and timely delivery of usage information registered by meters; • Designed and conducted in-person, statistically-driven tests to ensure that the extracted information matched records in the systems from which extracted; • Examined system-in/system-out matches of these millions of meter usage registration data points through all components of the meter-to-bill process; and Examiners’ Report • 18 Docket No. 2019-00015 Independently established formulas addressing all billing determinants required to calculate accurately the delivery and supply charges of customer bills. Liberty Report at 5–6. The issue of customer service is being considered in the Company’s pending rate case (Docket No. 2018-00194). Liberty’s work and findings on the remaining issues—all related to CMP’s metering and billing practices—are described below. 2. Metering and Billing System Accuracy In response to the numerous complaints from CMP customers about high bills in the winter months immediately following both the October 2017 windstorm (and resulting outages) and the Company’s transition to SmartCare, the Commission retained Liberty to conduct a forensic audit of CMP’s metering, billing, and related systems. 9 As part of the forensic audit, Liberty examined system billing data for the sixmonth period following the SmartCare cutover, from November 2017 through April 2018. Liberty examined data for every CMP customer account and meter—about 650,000 meters, including 7,400 manually read meters, for this six-month period. This examination involved review of about 4 million billing records and about 2.3 billion data points to trace data from one end of the CMP metering and billing system to the other. Liberty also tested a statistically significant sample of 60 meters in the field. Liberty Report at 8–9, 16–17. 9 Liberty’s audit also addressed CMP’s SmartCare implementation and customer-service performance. The issue of customer service is addressed in our companion order in CMP’s rate case, Docket No. 2018-00194. Examiners’ Report 19 Docket No. 2019-00015 As described in the Liberty Report, CMP’s metering and billing system consists of several components or groups of components. One of those components is CMP’s Advanced Metering Infrastructure (AMI), which comprises about 640,000 meters, of which about 360,000 are GE I-210+c meters and the remainder are Landis+Gyr meters. A wireless AMI network, provided by Trilliant, enables communication from the AMI meters to the Head End System (HES), which collects the meter information. The meter information is then conveyed from the HES to the Meter Data Management System (MDMS), which then uploads the information to the SmartCare system, where the bill calculation and preparation functions are performed. Liberty Report at ES-2. a. Meter Accuracy As part of its audit, Liberty examined CMP’s Augusta meter lab and connected warehouse to verify that it is a sound and effective operation. Liberty found that it was. Liberty also reviewed the calibration and meter test equipment used by CMP and, again, found no issues. Liberty noted that, in response to the customer complaints and under Chapter 32 of the Commission’s Rules, the Company tested 2,290 so-called complaint meters (that is, meters of customers who made complaints of high usage), and all but one meter met the Commission’s accuracy limits. The one meter was a non-AMI, analog meter. While Liberty was not able to validate the results of these 2,290 meter tests through direct observation, Liberty stated that if CMP used the same meter-testing equipment and techniques that it did in the tests that Liberty was able to observe, then Liberty was confident in the results of the complaint-meter tests. Liberty also did its own meter testing as part of its audit. Working with its statistical experts, Liberty sought to identify a number of randomly selected meters for a Examiners’ Report 20 Docket No. 2019-00015 test that would produce results having a 95% confidence interval, assuming an error rate of 2%. Liberty chose the 2% error rate based on the requirement that 98% of meters required to be tested under state standards must exhibit accuracy within the state’s +/-2% tolerance limit. Considering the numbers and types of meters employed by CMP and their dispersal across CMP’s service territory, Liberty determined that a sample of 60 meters would be statistically sufficient to corroborate the testing already performed by CMP. Liberty carried out these tests in September 2018. The test group of 60 randomly selected meters consisted of 24 GE I-210+c meters, 15 GE KV2c meters, and 21 Landis+Gyr AXR-SD meters. These included 46 residential customer meters and 14 non-residential customer meters located in the Portland, Farmington, Brunswick, Rockland, and Belfast areas. All of the 60 meters in Liberty’s sample group tested within the required +/-2% accuracy tolerance. The largest percentage error found was 0.41%, with the remaining meters testing at higher accuracy. The clocks of all 60 meters showed values within one minute of real time, save one that was 10 minutes behind. No meter clocks showed future time or indications that they were counting time lapse at too fast a rate. Given the results of the Company’s testing of the complaint meters and Liberty’s more limited but statistically significant testing, Liberty concluded that CMP’s meters produce accurate measurements of customer usage and have done so since November 1, 2017. Liberty did, however, find one source of meter registration error, referred to as “anomalous meters,” which is discussed below in Section IV.A.2.c. b. Meter Data Collection, Storage, and Transmission Liberty examined the systems used to transmit usage data from the meters to meter databases, and then to the customer information system (CIS), SmartCare, to Examiners’ Report 21 Docket No. 2019-00015 determine whether the data was being transmitted accurately, completely, and on a timely and secure basis. With respect to the communication from the meters to the HES, the examination was limited due the relatively short duration of time meters can store data. However, Liberty was able to directly test the communication of data from the 60 meters that were used for the field tests described above. Liberty found that the data from all 60 of these meters entered the HES accurately. Liberty Report at ES-5. Liberty also examined performance metrics for information sent but not received, AMI meter read rates, and network trouble reports for the November 2017–April 2018 audit period. According to Liberty, these metrics can be an indicator of the reliability of communications between the meters and the HES. This examination did not indicate any problems related to these communications during the audit period. Liberty concluded, thus, that the AMI network transmitted data accurately, completely, and in a timely manner. Id. With respect to the data transfer from the HES to SmartCare, Liberty noted that CMP retained data that allowed for testing of the data collection, storage, and transmission for nearly all meters during each of the audit period months. Based on this data, Liberty’s testing indicated a data point “matching” level of 99.8% between (1) the HES and the MDMS and (2) the MDMS and the usage used by SmartCare for customer billing. Liberty characterized this review as “extensive” and, based on the review, concluded that “meter-related databases and communications systems are accurately, completely, and timely transmitting data from meters to the meter data management and billing systems, and have been since November 1, 2017.” Id. at ES-6. Examiners’ Report 22 c. Docket No. 2019-00015 Meter Anomaly Liberty states that it found one source of error in CMP’s measurement of electricity usage: the anomalous meter. The anomalous-meter error occurs when a certain meter model (the GE I-210+c) resets improperly following a power outage. According to Liberty, CMP’s GE I-210+c meters can go into “anomalous mode” under a certain set of rare conditions. These conditions are: • An outage occurs which results in a meter resetting (initialize) when power is restored; • At about 3.36 seconds after restoration of power to the meter and while initialization is underway, voltage at the meter must sag to less than 90-volts on Phase A (one of the two 120-volt phases at the meter); and • The voltage sag must occur for at least 100 milliseconds and the sag must be maintained throughout a 66-millisecond window of vulnerability during which the metering processor resets, but the other control processor has not reset. Liberty Report at 23. When a meter goes into anomalous mode, two separate anomalies result: register anomaly and fast-clock anomaly. During anomalous mode, the meter’s clock registers the passage of time at 120% of the actual rate. Liberty noted that if a meter was in fast-clock operation for multiple days during the billing cycle, usage could appear to the customer to be high because the readings actually span a larger number of days. After the meter is reset, the following month’s bill will appear to the customer to be low for the indicated number of days because the readings actually span a smaller number of days. Examiners’ Report 23 Docket No. 2019-00015 Register anomaly occurs when a meter is in anomalous mode, and after power is restored, the meter reverts to 120-volt meter configuration (Phase A only) rather than 240-volt configuration split between Phase A and Phase C. This causes usage at the customer premises to register at twice the actual amount used on Phase A and no usage on Phase C. If the load at the customer’s premises was split equally between the two phases during the time of the registration anomaly, no error will occur. But if there was more load on Phase A than on Phase C, over-registration will occur, and if there was more load on Phase C than on Phase A, under-registration will occur. When in anomalous mode, a meter will continue to suffer from fast-clock operation and register anomaly until either a “special reset” or a “healing outage” occurs. A special reset is separate from the automatic reset that occurs after the initial power restoration; it requires intervention by a technician at the meter or at a central location. A healing outage is a second outage (after the initial outage and reset) that is not affected by the conditions causing anomalous operation. A special reset or healing outage repairs the fast-clock operation and register anomaly. Liberty Report at __. Liberty noted that CMP’s system had approximately 358,000 of the GE meters that were susceptible to going into anomalous mode. Given the confluence of events that must occur for a meter to go into anomalous mode, Liberty concluded that the number of occasions of register-anomaly were relatively small. In addition, Liberty noted that, since Phase A and Phase C loads are generally balanced at customer premises, register anomaly makes only a small degree of error likely. Liberty noted, though, that: For a small number of individual customers, the impacts (in either direction) could be substantial. For example, a customer away from the premises during a cold snap might shut down most sources of load, but mistakenly forget to turn off an inefficient space heater. If that heater is on Examiners’ Report 24 Docket No. 2019-00015 Phase A when anomaly mode occurs, the customer may see a properly alarming level of usage when the bill arrives. Alternatively, that same customer may see a bill with very low usage if that heater was on Phase C for the same period. Liberty Report at 24. Liberty reported that the first awareness of the registration-anomaly problem seemed to occur in 2012, when GE described the registration problem and the fix for the problem which could be made through a firmware upgrade. Liberty noted that it did not find substantial documentation of CMP management awareness of the GE meter issue until 2014. In September 2014, CMP emailed GE and reported the following: • “A number” of meters each day displaying an incorrect date and time, most of them following a power outage causing them to “lose their clocks”; • Multiple meters with clock times running fast by up to 120%; • 10 or more meters experiencing the issue on any given day; • Automatic clock resets through the AMI Network, after which some reset meters continue to show future and clock time, progressing as each day passes; and • An August 11, 2014 field investigation of six customer meters with disk emulators moving too rapidly to perform a traditional “stopwatch” check. GE responded in October 2014, recommending the same firmware upgrade that it had in 2012. This report noted that pending the firmware upgrade, resets over the AMI network would clear the condition but would not eliminate future vulnerability. After receiving GE’s 2014 report, CMP decided to await a then-pending AMI (Trilliant System) enhancement, which would enable a mass firmware upgrade to be done over the air. According to Liberty, the Company provided the following reasons as to why it followed this approach: Examiners’ Report • 25 Docket No. 2019-00015 Lack of belief that registration errors were at issue (noting the failure of GE’s October 2014 report to discuss them); • The belief that the anomalous GE meter operations were not widespread; • The belief that the AMI Network enhancement was imminent; and • The ability to address individual cases in the interim through special resets, as management discovered them. Liberty Report at 22–23. Liberty reported that as part of its review of high-bills in 2018, CMP pressed Aclara, the successor company to GE’s meter business, to analyze outage events causing anomalous operation, the consequences of anomalous operation on meter registration, and the actions required to prevent those conditions. Aclara responded to CMP in June 2018 with a clear explanation of the anomalous meter situation and how it caused over- or under-registration. While awaiting a more permanent solution, which the Trilliant system enhancement would enable, CMP began a more rigorous effort to identify meters in anomalous mode. Liberty Report at 23. Liberty concluded that, given the circumstances causing the anomalous operation of GE meters and the generally self-moderating nature of registration errors, the anomalous meters did not contribute materially to any customer usage issues recorded after SmartCare implementation. The numbers of meters that experienced erroneous registration while the problem was occurring was likely, at most, in the few thousands. Of those customers affected, it is likely that a large number of customers neither substantially benefitted nor were harmed as a result of incorrect usage registration. Nevertheless, clearly some customers overpaid and some customers Examiners’ Report 26 Docket No. 2019-00015 underpaid for service since 2010, due to the anomalous operation of the GE meters. Although not a cause of high bills, given CMP’s responsibility to ensure accurate billing, management should have, back in 2014, more aggressively pursued efforts to remedy the registration anomaly problem by either upgrading the firmware through an on-theground meter-upgrade program, or implementing the more aggressive protocol of resetting meters that was eventually adopted by the Company in 2018. Liberty Report at 30. d. SmartCare Billing Finally, to test the accuracy of the SmartCare billing system, Liberty independently calculated expected bills over the audit period—comprising more than 3.9 million bills—using customer billing determinants retained by CMP and the applicable rates for each customer, and compared these calculated bills to the actual amounts billed by CMP during the audit period. Liberty’s calculations were done for all of CMP’s major rate classes: Residential; Residential Time-of-Use (TOU); Small General Service; Small General Service TOU; Medium General Service; Intermediate General Service; and Large General Service. Liberty calculated expected charges for both delivery and supplier service for these bills and compared its calculated bills to the bills CMP issued during the audit period. Liberty Report at 47. In testing billing accuracy, Liberty engaged in an iterative process with the Company to explore any variances between the results produced by Liberty’s modelling and CMP’s SmartCare system. Liberty’s iterative matching process of all delivery and supply determinants of all bills processed during the period of November 2017 through April 2018 produced a match level of 99.9% for delivery and 99.8% for supply. In its report, Liberty noted 48 delivery variances with a cumulative over-billing amount of Examiners’ Report 27 Docket No. 2019-00015 $6,573, and 7,997 supply variances with a cumulative overbilling amount of $28,165. Liberty stated that its iterative process with management did reflect some billing errors that were corrected by the Company and that it was likely that continuing interaction with the Company would further reduce the variances. Liberty, therefore, requested that the Company complete the process of reconciling amounts CMP actually billed to Liberty’s calculations to ensure that any billing errors are appropriately addressed. Liberty Report at 49. CMP subsequently provided the results of its further reconciliation which did not identify any miscalculations within the billing system. 10 Liberty then requested that CMP provide PDF copies of customer billings for all 64 delivery exceptions and a randomly selected sample of 1,096 supplier exceptions. Based on its analysis, Liberty confirmed the calculations in all of the delivery charge bills, however, Liberty could not confirm 72% of CMP’s supplier charge reconciliations which all seemed to involve interim supplier price changes and how CMP was prorating the usage during that month. Liberty found that the results of its analysis cast doubt on the remaining supplier exceptions which were not reviewed and requested that CMP re-examine all of the supplier charge exceptions identified by Liberty and address such findings, including necessary corrective actions, as part of its rebuttal. Liberty Review of Delivery and Supply Change at 4. e. Relationship Between Customer Usage and Weather In addition to the testing and auditing described above, Liberty also examined the correlation between weather and the usage of CMP residential and general service 10 See EXM-002-004, EXM-002-005. Examiners’ Report 28 Docket No. 2019-00015 customers during the winter months. Liberty analyzed the relationship between usage and heating-degree days (HDD) for the audit period and for the same months (November through April) for the prior four years. Id. at 53. Liberty noted that HDDs are a major factor driving electricity use, particularly in cold regions where electricity may be a source of heat. Liberty obtained daily residential and general-service customer usage and daily HDD data for the months of November, December, January, February, March, and April for the period November 2013 through April 2018. Liberty Report at 53. With this data, Liberty calculated a set of daily ratios of usage-to-degree days (HDD Ratio) and compared these ratios for the audit period and the four prior years. Liberty’s HDD Ratios matched usage to days of the week, so as to not introduce differences in usage patterns between weekdays and weekends. Id. Liberty sought to compare usage during days in the audit period to the most weather-comparable day during one of the prior years. Liberty also compared usage during the audit period to the immediately prior period, for two reasons: (1) given the recent customer concerns about increases in bills during the audit period compared to the same month in the prior year; and (2) to capture any changes in overall usage patterns that may have occurred over time. Id. at 54. This analysis showed that the usage of CMP customers during the audit period was consistent with winter-period usage during the prior four-year period. Specifically, Liberty found that CMP customer usage during the November 2017 to April 2018 period, both in total and as a function of heating-degree days, was consistent with HDD ratios and usage volumes experienced in prior years, beginning in 2013. Id. at 59. Stated another way, the amount by which usage exceeded historic ratios matched the amount Examiners’ Report 29 Docket No. 2019-00015 by which weather during the period was colder. Id. Based on this weather and usage analysis, Liberty concluded that high bills during the audit period resulted from (1) cold weather and (2) increases in the price of electricity supply. Id. This analysis corroborated Liberty’s conclusions that CMP metering and billing systems did not create any systemic over-billing errors during the audit period. 3. SmartCare Implementation As part of its audit, Liberty also examined the implementation and early post-golive operation of the Company’s SmartCare billing system. According to Liberty, wellmanaged software implementation projects center around a customer-service delivery vision with clearly defined objectives and a full understanding of how the software solution will support that vision. Successful projects should: (a) implement and test the solution’s design; (b) conduct business-process assessment and re-engineering; (c) plan hardware and software configurations in detail; (d) convert existing data to enable its successful processing in the new system; (e) define and apply go-live acceptance criteria; (f) conduct pre-go-live testing to ensure satisfaction of those criteria; (g) train system users; and (h) create and execute post-go-live transition plans. Liberty Report at 60. Careful attention to each ensures a smooth transition with minimum adverse customer impacts. Id. On CMP’s implementation of SmartCare, Liberty found that CMP’s management held to a planned go-live date of October 30, 2017 by compressing its planned timelines and by overlapping critical testing phases. The approach changed from the more cautious intended approach of ensuring completion of planned tests, to one that allowed only such testing that fit into the ever-compressed time remaining before the end of October 2017 and the remaining budget permitted. Liberty found that management Examiners’ Report 30 Docket No. 2019-00015 addressed the threat to the October 30, 2017 go-live date through the high-risk approach of overlapping the phases of Unit Testing, Integration Testing, and Parallel Bill Testing. The original plan included 30 weeks of testing, but ultimately only 17 weeks of testing was conducted in overlapping test phases. 11 As a result, testing had to be executed on processes still under development. Liberty Report at 62. 12 Liberty noted that CMP’s compressed schedule eliminated User Acceptance Testing, under which business end-users confirm that intended business requirements have been met. While CMP claimed that User Acceptance Testing was incorporated into Integration Test Cycle 4 and Integration Test Cycle 5, testing in this manner was not in accordance with best practices. Best practices would have allowed for more time for User Acceptance Testing and would have conducted User Acceptance Testing later in the process, after greater integration with capabilities working operationally. Liberty noted that CMP’s SmartCare development process employed Parallel Bill Testing. Best practice typically excludes this type of testing, because it does not include Business Process Procedure documentation, which details the business steps to execute a required transaction, identifying all required and optional fields. The preferable approach (User Acceptance Testing) includes this documentation. Management’s decision to use Parallel Bill Testing instead, eliminated thorough testing of non-bill processes. Liberty also noted that CMP eliminated two tests from its testing protocol. Thus, according to Liberty, CMP both used an approach that did not test all functionality and ran out of time to do further needed testing. 11 Liberty excluded from this calculation the weeks for initial Integration Testing. 12 All of the relevant types of testing are listed and defined at pages 48–49 of this order. Examiners’ Report 31 Docket No. 2019-00015 Liberty also found that CMP did not adequately staff the implementation project, which had a substantial effect on the Company’s ability to deliver expected system functionality and quality and to deal effectively with the operational and technical challenges that arose after go-live. Liberty cited to project reports that showed resourcing challenges, including the following statements: • “Stretching resources very thin for project tasks in parallel with production support/other activities”; • “Stretching resources very thin for meeting project tasks and key deliverables”; and • “Spreading project resources too thin—overlapping efforts impacting quality of execution.” Liberty Report at 63. Also, according to Liberty, CMP’s quality-assurance reporting was not clear and consistent. Liberty cited an instance in February 2017, where the system integrator for the project assigned a green indicator to one area (meaning that there was no problem with the area), a yellow indicator to three areas (meaning that there were emerging issues affecting the project that required attention), and a red indicator to four areas (meaning active issues were affecting the project and required management’s attention). Overall, the system integrator assigned a red indicator to the project status. At the same time, Avangrid’s 13 team provided a very different view, giving green status to a majority of categories with a few activities rated as yellow. The last system- 13 Avangrid is CMP’s parent company, with Avangrid Services Company and Avangrid Management Company performing many types of services for CMP under Commissionapproved affiliate service agreements. Examiners’ Report 32 Docket No. 2019-00015 integrator review, dated July 17, 2017, showed an overall status of red. In contrast, the overall criteria ratings given by the Avangrid team at that point in time were either green or yellow. Liberty Report at 65–67. Liberty found that compressed timelines leading up to go-live, the volume of exceptions and defects, insufficient reporting, and insufficient monitoring of key performance indicators post-go-live all adversely affected the rollout of SmartCare. Also, the readiness sign-off for go-live had no signatures, and CMP reported that approval was oral, not written. The final sign-off included the following go-live goals: no critical open defects; no high open defects; number of open medium defects with agreed workarounds. The sign-off reported no critical or high defects, but this data was inconsistent with other reporting from CMP. Liberty Report at 69. Liberty stated that the data CMP supplied about defects at go-live was inconsistent. Liberty noted that the last status report it could find that had substantial detail was dated October 13, 2017, which indicated that much work remained to be done. Post go-live, CMP did not set a target supporting the measurement of success, based upon meeting key performance indicators. 14 Liberty Report at 69–70. Liberty concluded that: Lack of testing and the right type of testing caused post go-live defects that affected customer satisfaction and revenue. Management minimized or failed to recognize the severity of defects raised before launching the CIS. Conflicting information in status reports, readiness reports and the master list of defects demonstrates a lack of recognition of the magnitude of the issues . . . . ... 14 “Key performance indicator,” or KPI, is a term of art in the software-implementation industry. It refers to objectives and targets set for performance of the software and permits the implementer to track progress against those goals. Liberty Report at 69–70. Examiners’ Report 33 Docket No. 2019-00015 Lack of post go-live planning and management for defect resolution and staff to manage these defects created a large backlog, which still remains. SmartCare went live with known defects requiring later resolution. Defects discovered internally after go-live and in response to customer inquiries and complaints added substantially to an already substantial list. Liberty Report at 72. In response to CMP’s criticism of Liberty’s findings on the implementation, Liberty filed a summary report responding to CMP’s SmartCare testimony. Liberty’s summary report was prepared by Rose Minton. In that report, Ms. Minton stated that, contrary to the Company’s assertions that Liberty’s review of SmartCare implementation was unsubstantiated, Liberty examined thousands of pages of documents about the implementation, which Liberty secured through the issuance of 36 data requests. Also, in June 2018 and again in September 2018, Liberty interviewed and had discussions with CMP’s project director and SmartCare team leaders. Overall, Liberty found that CMP’s testimony on SmartCare testing did not provide any new information that would change its conclusions about the quality of CMP’s testing. Liberty stated that CMP’s testimony did, in fact, point out some inconsistencies between CMP’s testing methodology and its test execution. Specifically, although the documents CMP provided indicated that it intended to utilize the so-called waterfall approach to testing, with testing completed in sequence and not concurrently, CMP did not do so: CMP continued to do Unit Testing, which should have been completed before Integration Testing began. Unit Testing continued in September 2017, well into the “Testing and Validation” phase and well into the final phase of the project. CMP’s Project Plan, filed along with its metering and billing testimony, showed that CMP completed User Acceptance Testing on August 3, 2017. If User Acceptance Testing was conducted as part of Integration Testing, which was done on June 22, 2017 Examiners’ Report 34 Docket No. 2019-00015 and October 13, 2017, that should have been documented in the Project Plan. But the Project Plan contains no such documentation. In response to CMP’s claim that 74 weeks of testing were actually executed and not 17 weeks as reported by Liberty, Ms. Minton notes that Liberty and CMP used different methodologies to measure the duration of the testing effort. Liberty counted the total calendar duration of testing, while CMP calculated the number of test weeks, which in some cases overlapped in time. (For example, under CMP’s methodology three tests performed concurrently over the same two-week period would count as six weeks of testing.) B. Central Maine Power Company 1. Metering and Billing a. High Bills and Bill Errors CMP framed the “primary question driving this investigation” as whether CMP’s “metering and billing systems have accurately, completely and timely recorded customer usage, transmitted that usage to the billing system, and billed customers based on the recorded usage since” the transition to SmartCare. CMP Br. at 1. CMP asserts that based on the Liberty Report, the Staff’s analysis, the analysis of the OPA’s consultant, BerryDunn, and CMP’s “own audits and extensive analysis, the uncontroverted evidence demonstrates that since the SmartCare Go-Live, CMP’s metering and billing systems have been working properly. With minor exceptions,” which CMP discusses in its brief, “customers have been and continue to be billed accurately based on their actual usage.” CMP Br. at 1. Examiners’ Report 35 i. Docket No. 2019-00015 No Systemic Cause of High Bills In support of its position that there is no systemic problem with CMP’s billing system that is causing high bills, CMP describes in detail the analysis Liberty performed on its metering and billing practices and systems, which led Liberty to conclude that there was no systemic problem. CMP Br. at 26–29; see also Section IV.A above (describing Liberty’s audit). CMP also cites to BerryDunn’s analysis of CMP’s metering and billing of customers who made high-bill complaints, in which BerryDunn did not find any systemic root cause of high-bill complaints. CMP Br. at 29–30; see also Section IV.C.1 below (describing BerryDunn’s work). Finally, CMP points to the Staff’s analysis comparing billing data of CMP with that of Emera Maine’s Bangor Hydro District over the winter of 2017–2018, which led Staff to conclude that the cause of high bills during that period was correlated with weather, and not related to a systemic problem with CMP’s metering or billing system. CMP Br. at 29; see also Section IV.F below (describing the Staff’s analysis). Based on the conclusions of these third-party analyses, CMP states that “[n]o party has offered any credible evidence to refute these findings” and that the “record supports a clear and unequivocal finding that CMP’s bills are accurate,” CMP Br. at 29, 31. CMP urges the Commission to accept these third-party analyses and conclude that since the transition to SmartCare “CMP’s metering and billing systems have worked properly to produce accurate bills for customers reflecting their actual usage, except in very narrow circumstances have been corrected, and that there are no systemic problems with these systems that have caused CMP to overcharge customers based on their usage.” CMP Br. at 31. CMP asserts that by laying out the detail and transparency Examiners’ Report 36 Docket No. 2019-00015 of this investigation in an order, “the Commission will help restore customer confidence and trust in CMP and its bills and the Commission’s oversight of both.” CMP Br. at 31. ii. Bill Reconciliation CMP also discussed Liberty’s work to compare amounts on customers’ bills with Liberty’s own calculation of the amounts due based on the customers’ metered usage and to reconcile charges on customers’ bills. CMP Br. at 32–37. Liberty performed this work in its audit (as summarized in the Liberty Report at page 9 and described above in Section IV.A). After Liberty ultimately could not reconcile 8,300 accounts during its audit, CMP performed additional work to reconcile the remaining 8,300 accounts that Liberty could not. At that point, Liberty worked to verify CMP’s calculations (as summarized in Liberty’s Reconciliation Report, also described above at Section IV.A) and was able to confirm all but 789 of these. CMP asserts that the remaining then-unreconciled bills— which involved third-party supplier charges, not CMP’s delivery charges—were subject to special proration that Liberty was not aware of, and that these explain the discrepancies Liberty encountered. Liberty confirmed its agreement with this, thus verifying all of the bill reconciliation work except for bills affected by an interim-read defect (Defect #5885) and bills affected by truncation of billed amounts instead of rounding of those amounts (which has a net impact of $63.94). CMP Br. at 34–35 (citing Tr. at 178–79 (Nov. 5, 2019 Hr’g)), 36. CMP states that this reconciliation work “regarding essentially all bills issued by CMP during the audit period confirms the conclusion that there is no systemic flaw in CMP’s metering and billing systems that have caused the Company to overcharge customers based on metered usage.” CMP Br. at 37. Examiners’ Report 37 iii. Docket No. 2019-00015 Billing Defects CMP enumerates and describes the defects it identified after the rollout of SmartCare, and describes their status. CMP Br. at 48–62. (Some of these defects are described in Section VII.C.2, below, and they are detailed in several responses to data requests, including, for example, LOO-001-031, EXM-004-007, EXM-004-008.) CMP asserts that as it “has identified bill-related defects and issues, the Company has informed the Commission, and worked diligently to resolve them, inform impacted customers, and provide appropriate redress to those customers.” CMP Br. at 47–48. CMP explained how it prioritized defects as their existence became known, focusing (for instance) on those that caused errors in the amount billed to the customer over those that did not affect the amount billed.” CMP Br. at 50. CMP adds that it “is now very close to having resolved the bill presentment issues identified in the BerryDunn testimony and to providing appropriate relief to impacted customers . . . .” CMP Br. at 48. For customers it identified who were over-billed as a result of a defect, CMP explains that it “has applied the appropriate credits” plus a customer-service guarantee. CMP Br. at 50. For “most” customers who were under-billed as a result of a defect, CMP did not issue corrected bills but instead “applied a Customer Service Guarantee as a credit to the customers’ accounts.” CMP Br. at 50. CMP pointed to statements by witnesses from both Liberty and CMP’s system integrator to argue that the defects it has experienced are within the normal range for this type of software conversion. CMP Br. at 49. It argued that BerryDunn’s findings “for the most part restate defects previously known by CMP and disclosed by Company representatives first to the CASD during regular phone meetings and the Liberty auditors as part of Liberty’s forensic audit and then to the BerryDunn auditor.” CMP Br. Examiners’ Report 38 Docket No. 2019-00015 at 51. In relation to BerryDunn’s analysis, CMP explained that only one defect discussed in the testimony (#4711) was not previously known to CMP and only one other defect (#6621) remains unresolved, “but this defect does not affect the billed usage . . . and thus does not have any financial impact on customers.” CMP Br. at 51. A handful of other defects remained open: #4711 (which has been corrected but identification of affected customers is ongoing), #5302, #6723, #6655/6621, #5326, and #6691. CMP Br. at 51–52. iv. Effect on Customers Even though CMP asserted that its metering and billing systems are correctly charging customers, CMP acknowledged that many customers experienced bill errors or bill-presentment defects after the transition to SmartCare, and that this “created customer confusion and eroded customer confidence in the accuracy of their CMP bill.” CMP Br. at 1, 18, 57, 62. “CMP accepts that these shortcomings have contributed to customer dissatisfaction and complaints.” CMP Br. at 2. CMP explained the steps it has taken to address the harm to customers affected by defects after they were discovered. These steps have included correcting customers’ bills or applying Customer Service Guarantees to customers’ accounts, sending letters to customers to explain the defects they were experiencing, or otherwise reaching out to customers to discuss their concerns. CMP Br. at 50, 57, 87. b. Meter Anomaly CMP responded to Liberty’s findings regarding the meter anomaly issue in the Company’s initial testimony on Metering and Billing. The CMP witnesses noted that CMP first became aware of the anomalous meter issue in 2018 when it observed a large number of meters in a specific geographic area having “clock drift” issues. CMP Examiners’ Report 39 Docket No. 2019-00015 then entered into extensive discussions with Aclara, after which Aclara provided CMP a report titled “Fast Clock Overview and Corrective Action” for meter model I-210+c. That report described the condition that the I-210+c meter could enter into anomalous mode which could result in errors in the amount of usage being recorded. In response to Liberty’s statement that it did “not find substantial documentation of management’s awareness of the GE meter issues or efforts to address them until 2014,” Liberty Report at 22, the Company’s witnesses stated that it would have been more accurate to say that there was zero documentation of management’s awareness of the issue until 2014. The Company’s witnesses noted that when they did first become aware of the anomalous meter issue in October 2014, the GE report describing the issue was short on detail and only referred to its fact-clock aspect; there was no mention of register anomaly. That report recommended that the fast-clock issue be addressed either through a firmware upgrade, which was available at the time, or through a manual meter reset process. Because CMP was unable to do the firmware upgrade over the air, CMP opted for the manual-reset process. CMP Met. & Bill. Dir. at 17. CMP’s witnesses agreed with Liberty about the very specific conditions that must be present for a meter to enter anomalous mode. They also agreed with Liberty that there was no correlation between the customers who complained of high usage and the particular meter (the GE I-210+c) that could go into anomalous mode. CMP’s witnesses disagreed with Liberty that some CMP customers likely suffered material overregistration from the anomalous-meter problem; according to CMP’s witnesses, no evidence supported that conclusion. Examiners’ Report 40 Docket No. 2019-00015 In CMP’s rebuttal, it provided several updates on the anomalous meter issue. First, CMP’s witnesses described the Company’s efforts to eliminate the susceptibility of the GE meters going into anomalous mode by upgrading the meter firmware, as Aclara recommended. The first step in that process was to upgrade the subject meter’s network interface cards, or NICs. As of October 14, 2019, 99% of the NICs of the problem meters had been upgraded. The second step in the process was to upgrade the meter firmware. CMP’s witnesses reported that as of October 14, 2019, 90% of meters have successfully had their firmware upgraded. CMP Met. & Bill. Reb. at 14. CMP also provided an analysis of the likely impact of the meter-anomaly issue on individual customers. Using meter events, outage data and meter-read patterns, CMP identified 5,074 occasions where a meter was believed to have been in anomalous mode for some amount of time since October 2013. Less than 0.009% of meter days have been in anomalous mode since October 2013, according to CMP. CMP’s witnesses stated that Aclara confirmed its methodology for identifying fast-clock meters. CMP Met. & Bill. Reb. at 16–17, Exh. MB-5. The following figure from CMP’s Exhibit MB-5 summarizes the scope of the problem based on CMP’s analysis: Examiners’ Report 41 Docket No. 2019-00015 Figure 1: CMP Summary of Anomalous Events, 2013 to 2019 15 Year Anomalous Events Average Event Duration (Days) Minimum Duration of Anomalous Event (Days) Maximum Duration of Anomalous Events (Days) 2013 354 1,300 298 705 945 1,304 168 5,074 43 24 45 15 34 13 3 24 2 3 2 2 2 1 1 1 1,766 1,642 1,180 808 278 87 9 1,766 2014 2015 2016 2017 2018 2019 All Years Once these meters were identified, CMP then analyzed the usage pattern associated with each meter. The individual usage patterns were then used to quantify the expected usage registration for each affected meter during the anomalous period. The recorded usage by meter was then compared to the expected usage by meter to determine the likely impact of the anomalous mode on each affected meter. The Company’s meter-by-meter analysis of recorded usage versus expected usage during the anomalous period for each meter indicated that 3,425 meters likely under-registered usage by 334,559 kWh, while 1,649 meters likely over-registered usage by 319,457 kWh. CMP provided this example for context: if 319,457 kWh of the over-registered usage was valued at $0.157—an average rate for CMP’s residential customers in 2019—then the total value of over-registration would be $50,155. Based on this, the Company concurred with Liberty’s conclusion that it was likely that meter anomalies caused significant over- or under-registration for only a small number of customers. CMP Met. & Bill. Reb. at 18 (citing Liberty Report at 27). 15 CMP Met. & Bill. Reb. Exh. MB-5 at 3, Table 1. Examiners’ Report 42 Docket No. 2019-00015 Regardless of the monetary effect on customers, the Company’s witnesses testified that CMP was committed to correcting every error in which customers were over-charged as a result of a meter being in anomalous mode including potential errors based on historical customer usage. Therefore, CMP proposed to return to each customer the amount presumed to have been over-billed based on its historical analysis, along with payment of a customer-service guarantee. CMP Met. & Bill. Reb. at 18–19. Also, to mitigate the effect of the Company’s under-recovery of revenues from those customers who were likely under-billed, the Company proposed to provide credits to the affected rate classes as part of its revenue-decoupling reconciliation in the 2020 Annual Compliance Filing proceeding. CMP Met. & Bill. Reb. at 19. 2. SmartCare Implementation The Company argues that, with the assistance of its system integrator, Deloitte, it appropriately and successfully planned, tested, and implemented its SmartCare system. During the different implementation phases, Deloitte dedicated up to 50 consultants who worked nearly 150,000 hours on the project. CMP Br. at 64. In consultation with Deloitte, CMP developed its SmartCare implementation plan and project charter, which is in conformance with Deloitte’s Enterprise Value Delivery for SAP methodology and project-management best practices. CMP Br. at 64–65. CMP contends that, contrary to BerryDunn’s and Liberty’s claims, its pre-go-live testing was appropriate and complete so as to confirm that the system at go-live functioned in accordance with requirements. CMP Br. at 66. In total, the Company executed 79 weeks of testing which encompassed Unit Testing of all SAP transactions and development objects, Integration and User Acceptance Testing as well as Examiners’ Report 43 Docket No. 2019-00015 Performance, Stress, Regression, Batch Schedule, Parallel Bill, and Cutover Testing to validate that SmartCare functioned as required. CMP Br. at 66. The Company pointed out that it executed 41 weeks of additional testing and two additional rounds of Integration Testing than originally planned. CMP Br. at 67. CMP contended that its test scenarios and test cases met appropriate requirements and that the achievement of the requirements were appropriately traced through its Requirements Traceability Matrix. CMP Br. at 68. According to the Company, all defects were properly tracked and linked to the business requirements. At the start of the project, defects were tracked using an Excel spreadsheet. Once HP ALM 16 software was installed, that software was the only tool used to track defects. CMP Reply Br. at 21. CMP argues that, contrary to the OPA’s assertion, all business requirements can be traced to the proper test cases. In CMP’s view, this proves that the Company’s Requirements Traceability Matrix was adequate and that there were no gaps in its pre-go-live testing of requirements. CMP Reply Br. at 19–20. The Company also insists that the delivery of a quality product that met the business’s and customers’ requirements was an integral objective of the project management. CMP maintains that, contrary to BerryDunn’s and Liberty’s assertions, it adequately prioritized the quality of the project, as shown by the adjustments to budget and schedule parameters throughout the project, additional rounds of Integration 16 HP ALM (Application Lifecycle Management) is the software CMP used to manage SmartCare requirements, test cases, test scripts, test progress, and defects during implementation. Tr. at 124 (June 13, 2019). Examiners’ Report 44 Docket No. 2019-00015 Testing, and the satisfaction of all go-live-readiness criteria (none of which related to cost or schedule goals) prior to go-live. CMP Br. at 70–71. On project staffing, the Company asserted that it staffed the project sufficiently throughout all phases. CMP Br. at 71. As part of the project planning, CMP and Deloitte conducted a staffing analysis, which led to the project’s staffing requirements. A total of 241 full-time-equivalent personnel from CMP, Avangrid, and Deloitte committed over 360,000 hours throughout the project. The Company pointed out that “[t]he SmartCare system has been effectively supporting key business processes and its 600 users since its implementation two years ago” and had functioned largely as planned. CMP Br. at 79 (citing SmartCare Reb. at 29–31). CMP insisted that it had an appropriate post-go-live plan which addressed the resources available for post-go-live support, their responsibilities and the process for resolving identified issues. CMP Br. at 74. The Company committed over 35 people for post-go-live support. CMP Br. at 72. The Company argued that post-go-live defects are to be expected for any complex implementation such as SmartCare, but that it appropriately prioritized customer-facing defects and intended to have all defects completed by the end of 2019. CMP Br. at 74–75. The Company supported additional focused testing on the end-to-end meter to bill processes, but urged the Commission to reject the OPA’s recommendation for full validation testing. CMP Br. at 78–79, 80. The Company argued that fully re-testing the system will provide little value since the system has already been subject to extensive pre-go-live testing and two external audits, and successfully functions across many business processes as planned. CMP Br. at 79. Instead, full re-testing would divert Examiners’ Report 45 Docket No. 2019-00015 resources away from implementing on-going improvements for the purposes of testing and would cause any system enhancement or fix to be “frozen” while the testing was being completed. CMP Br. at 79–80. The Company was also “open to establishing a more formal and independent quality assurance position with oversight authority over the Company’s ongoing efforts to address the remaining identified defects in SmartCare and the installation of requested upgrades to the SmartCare system and reporting obligations to the Commission or another appropriate third party.” CMP Br. at 81. The Company recommended that any third-party oversight be for a defined period with the potential for extension based on project status. CMP Br. at 82. CMP suggested an initial oversight period of 12 months. 3. Remedies Rather than propose any new remedies, CMP points to the work it has done and continues to do to correct billing defects and restore customers’ confidence in their utility bills. CMP explains that, despite its shortcomings in customer service and in fixing defects, “[i]ntense work has been done to fix identified issues, including hiring additional customer service and field personnel and implementing a new Maine-based customer service leadership structure” to return CMP’s customer service “to the high levels experienced pre-SmartCare implementation.” CMP Br. at 2. CMP also explains that it has established a home-assessment pilot program with the Efficiency Maine Trust “to evaluate the effectiveness of such a program in addressing future unresolved high usage complaints.” CMP Br. at 2. CMP points to Docket No. 2018-00194, where it has proposed creating a $6 million customer-benefit fund to compensate those affected by CMP’s customer-service performance and the billing defects identified in SmartCare. CMP Br. at 2. Examiners’ Report C. 46 Docket No. 2019-00015 Office of the Public Advocate 1. Metering and Billing The OPA argued that CMP’s billing and metering system is flawed and that this has led to distrust or even mistrust in customers’ utility bills. In support of this position, the OPA pointed to the testimony of its consultant, Julie Keim of BerryDunn. Ms. Keim led an analysis of information about high-bill complaints raised after the Liberty Report was submitted. The analysis involved comparing the electricity-metering data received at CMP’s head-end system or its field-collection system for customers who made these complaints and comparing that to the amount of electricity ultimately billed to them. The OPA distinguished the work of BerryDunn from that of Liberty by pointing out that BerryDunn focused on customer-facing information (that is, the invoices customers received) and only customers who lodged complaints, while Liberty focused on invoice information stored in SAP and all CMP customers. OPA Br. at 13; OPA-010-001. The OPA asserted that “BerryDunn’s analysis of 1,370 customer accounts and more than 5,400 customer invoices revealed numerous ongoing errors in the billing process,” including errors in meter readings, incorrect meter-read dates, number of days on the invoice differing from the number of days usage was recorded, calculation errors in the “Your Meter Details” box on the bill, billed kWh stated differently on different parts of the bill, billing periods that are not sequential or that overlap, and separate invoices with the same invoice number showing different usage amounts. OPA Br. at 9–11. The OPA expressed concern that these problems “continue to occur . . . on the invoices of other CMP customers.” Id. at 10. The OPA explained that BerryDunn’s analysis also revealed a previously unidentified problem in CMP’s billing, in which customers whose usage was (under-) Examiners’ Report 47 Docket No. 2019-00015 estimated for successive months and later trued up failed to receive the full benefit of the 750-kWh-per-month sales-tax exemption for residential electricity delivery. OPA Br. at 14; Keim Dir. at 10, 13, 14; 36 M.R.S. § 1760(9-B)(A). The OPA asserted that the evidence shows several ways in which CMP’s postSmartCare billing defects violated the Commission’s consumer-protection rules. The OPA provided the following nonexhaustive list of ways in which these defects violated rules: (1) failing to “obtain an actual meter reading every month” (OPA Br. at 15; MPUC Rules, ch. 815, § 8(L)); (2) failing to correctly state the “beginning and ending dates of the period for which service was provided” (OPA Br. at 15; MPUC Rules, ch. 815, § 8(C)(1)); (3) failing to include a “clear and conspicuous marking of all estimates” (OPA Br. at 16; MPUC Rules, ch. 815, § 8(C)(10)); (4) failing to correctly apply state and federal taxes due to inaccurately estimated bills (OPA Br. at 16; MPUC Rules ch. 815, § 8(C)(5)); and (5) failing to provide a bill format that includes “comparative usage information for the prior 12-month period and for the equivalent 12 months ago” (OPA Br. at 16; MPUC Rules, ch. 815, § 8(D)(2)). The OPA suggests that the “failure of CMP’s SmartCare system to comply with these basic requirements undermines customer confidence regarding all aspects of its accuracy and functionality.” OPA Br. at 16. The OPA expressed deep concerns about customers’ ability to trust their CMP bills given ongoing defects that have yet to be resolved and CMP’s insufficient reaction to and communication about its billing problems. OPA Br. at 1, 8, 12–14, 22. The OPA asserted that the errors BerryDunn describes in its report “are serious problems for a utility that does not enjoy the trust of its customers that its bills are accurate.” OPA Br. at Examiners’ Report 48 Docket No. 2019-00015 14. While “CMP asks that customers trust that the dollar amount billed is correct, . . . other issues like bill presentment and excessive monthly estimating do not lead to an atmosphere where such trust can exist.” OPA Br. at 14. Although it observed various billing errors or bill-presentment errors, “[a]fter eight weeks of intense review and analysis, BerryDunn was unable to isolate a defect, set of defects or root cause for the numerous complaints relating to high bills.” OPA Br. at 9. 2. SmartCare Implementation As part of its direct case, the OPA submitted the testimony of Laurel Arnold on SmartCare implementation. In her testimony, Ms. Arnold agreed with the Liberty’s assessments and conclusions about the implementation of SmartCare. Arnold Dir. 17 at 4, 5–6. Ms. Arnold found that CMP’s pre-implementation testing did not align with best practices or even CMP’s approved strategy, which Ms. Arnold found to be inadequate. Arnold Dir. at 4. The Testing Strategy document set forth the following types of tests: 17 • Unit Testing. The point of Unit Testing is to verify that individual system components support the related functional, technical, and interface requirements, with a focus on the integrity and correct functioning of a series of transactions within a specific module. Unit Testing is highly iterative. OPA008-011, Att. 1 at 7. • String Testing. This type of testing bridges Unit Testing and Integration Testing, during which the teams begin to integrate related pieces of functionality into a single test without testing the overall end-to-end process. Id. at 7. • Integration Testing. The point of Integration Testing is to determine whether CMP’s operations can be performed in SAP. It includes the testing of integrated functionality between SAP and other external systems to validate that the system correctly accesses, updates, and transfers data appropriately. Id. at 8. Arnold Dir. refers to the BerryDunn refers to the BerryDunn Independent Assessment Report, led by witness Laurel Arnold of BerryDunn on behalf of the OPA, filed on September 6, 2019. Examiners’ Report 49 Docket No. 2019-00015 • User Acceptance Testing. Here, the users test the complete, end-to-end business processes to verify that the software “performs the intended functions and satisfies the business requirements.” User Acceptance Testing “is the last test cycle of a solution implementation and is an essential part of gaining end user acceptance of the system.” It “may be a separate test or could occur as part of the last cycle of Integration Test[ing].” Id. • Performance and Stress Testing. These tests “are designed to ensure the system will perform appropriately under production level processing requirements.” Id. at 8–9. • Regression Testing. This “is the process of selective retesting of systems and system components to verify that modifications have not caused unintended effects and that the system or component still complies with its specified requirements.” CMP Implem. Reb. at 23. • Parallel Bill Testing. This type of testing was meant to demonstrate that billing results from the legacy CSS production system could be replicated within SAP within accepted tolerances. The objective was to “validate the charges calculated by periodic billing for all types of customers and rate combinations at line item or individual charge level, to build confidence in the accuracy of the SAP CRM&B billing solution, as well as to find and resolve potential issues ahead of go-live.” OPA-008-011, Att. 1 at 9. • Converted Data Testing. During Converted Data Testing, “the migrated data [in the SmartCare system] was compared with the legacy system to validate the appropriate field mapping.” CMP Implem. Dir. at 14. Ms. Arnold testified that this list was incomplete and did not include all of the types of testing required by best practices for large software-development projects. Ms. Arnold points to System Testing, Security Testing, and Smoke Testing as tests that CMP failed to include. Arnold Dir. at 7. Ms. Arnold also noted these other deficiencies in CMP’s Testing Strategy: • The Testing Strategy did not require development of test cases to cover all code path scenarios. Industry standards and best practices provide that test cases should be created and categorized for all possible paths the user may follow when using the system. Examiners’ Report • 50 Docket No. 2019-00015 The Testing Strategy did not sufficiently detail how test data would be identified and selected to ensure that the test scenarios would exercise all code paths. With this deficiency, CMP risked the test data not providing a correct and complete sample of production data and that defects related to test data may remain. Arnold Dir. at 11–12. According to Ms. Arnold, without completing all of the types of necessary testing, CMP could not know all of the possible issues and defects that remained when the system was in use. Ms. Arnold also testified that the testing conducted did not follow the Company’s approved Test Strategy. Ms. Arnold pointed to the Company’s decision to allow for test cycles to run in parallel instead of serially, or in a waterfall method. In a waterfall method, each type of test must be completed before the next text type can begin. Testing occurs as a sequence of stages, without overlap. Under this model, System Testing and User Acceptance Testing must run one after another. In September 2017, CMP decided to conduct Integration Testing simultaneously with User Acceptance Testing. Ms. Arnold stated that CMP’s abandonment of its waterfall methodology, and its failure to replace it with any accepted testing methodology, prevented the Company from validating that the system met its business requirements. Arnold Dir. at 13–15. Allowing for User Acceptance Testing to be done as part of the last cycle of Integration Testing was not in accord with industry best practices and created the risk that a “buggy system” would be distributed to users of the system, which in turn would expand turnaround time and slow the deployment of responses. Arnold Dir. at 18. Examiners’ Report 51 Docket No. 2019-00015 Ms. Arnold noted that Stress Testing should be designed to root out defects by placing demands on the system that mimic real-life situations. An example of this is a situation where hundreds or thousands of users visit the application at once in real time. Arnold Dir. at 9–11. The OPA questioned whether CMP performed such testing at all. OPA Br. at 20. Ms. Arnold testified that the test results did not meet CMP’s approved go-live criteria. Ms. Arnold notes that CMP’s Testing Strategy specifically required “zero open high or medium priority defects[,] unless there is a workaround.” OPA-008-011, Att. 1 at 17. This criterion was changed to no “critical or high” defects. This lowered the bar for go-live, yet the change was not formally documented, nor was it analyzed or its risk assessed. OPA Br. at 22. Ms. Arnold was most critical of the Company’s failure to track testing and the defects identified during testing. Specifically, Ms. Arnold testified that CMP did not use a proper Requirements Traceability Matrix, which would have provided evidence of what testing was done and what was missing. In addition, a properly used and completed Requirements Traceability Matrix would provide evidence that all test cases necessary to adequately test each requirement were actually done. Under best practices, a Requirements Traceability Matrix should include system requirements, the test cases that cover those requirements, and the defects that turn up during testing. Although CMP recognized the need for traceability in its Testing Strategy, in practice CMP did not carry out this requirement. OPA Br. at 26. The OPA asked the Company to provide its Requirements Traceability Matrix. The OPA argues that the matrix ultimately produced for the project and approved by Examiners’ Report 52 Docket No. 2019-00015 CMP’s management included only requirement-specific information and did not provide traceability. OPA Br. at 26 (citing OPA-007-083, Att. 2). The linkages the Testing Strategy mandated to map defects to test cases were never developed. The Requirements Traceability Matrix was, thus, not completed as planned and failed to serve its intended purpose to verify that SmartCare would perform as expected at go-live. Rather than use the Requirements Traceability Matrix as specified in CMP’s Testing Strategy, CMP used a dashboard approach that relied upon the number and type of unresolved defects to measure readiness for go-live. Ms. Arnold testified that reliance on defect counts alone to measure the quality of the system and readiness for go-live is risky. OPA Br. at 27– 28; Arnold Dir. at 21. Ms. Arnold also stated that CMP failed to consistently manage defects in a single tool, employing shared Excel spreadsheets at some points and the HP QC 18 tool at others. When the HP QC tool was used, CMP’s testers did not use the tool properly and failed to enter defects detected in Unit and String Testing. As a result, testers and analysts could not easily compare new defects with previous defects to determine patterns or trends, thus making it more difficult to prioritize defect fixes and create appropriate Regression Testing cases. Arnold Dir. at 16–17. Ms. Arnold found that CMP’s management of risk during the implementation was also not consistent with industry best practices. Here, Ms. Arnold pointed to the Company’s failure to properly identify the project’s primary constraint (scope, timing or cost) and its failure to properly identify a single critical path at the project level. In addition, CMP failed to properly develop a contingency or roll-back plan that could be 18 HP QC and HP ALM are the same product; HP QC is a former name of the HP ALM product. Tr. at 123 (June 13, 2019) Examiners’ Report 53 Docket No. 2019-00015 used if the Company decided not to go live with SmartCare. Ms. Arnold noted that the danger of not having an executable contingency or roll-back plan is that managers will feel pressure to implement or continue with a project even when such a course is unwise. Arnold Dir. at 28–29. Based on the deficiencies Ms. Arnold identified, the OPA argued that CMP’s management of the SmartCare project was imprudent. As such, the OPA argued that the Commission should adopt the ROE adjustment recommended by the Staff in its Bench Analysis in Docket No. 2018-00194. The OPA also requested that the Commission order validation testing of the SmartCare system to be done by an independent third party. As part of this process, CMP and the third party would adopt a complete Requirements Traceability Matrix to provide evidence that: • • • • • all functional and non-functional requirements have been accounted for in testing; requirements were specific, measurable, attainable, relevant and time-bound; . . . test scripts written for each requirement covered not only positive and negative paths but also others such as alternate, boundary, edge, error and exception paths; . . . all planned and approved test scenarios and scripts were executed; [and] defects were traced to test cases and requirements, allowing visibility into requirements that are flawed or incomplete. OPA Br. at 35. 3. Remedies The OPA pointed out several ways in which CMP has violated Chapter 815 of the Commission’s rules. OPA Br. at 14–16. The OPA did not, however, propose that administrative penalties be imposed in this case. Instead, the OPA suggested that the “failure of CMP’s SmartCare system to comply with these basic requirements . . . Examiners’ Report 54 Docket No. 2019-00015 provides a separate basis [both] for imposing a cost of equity adjustment [in Docket No. 2018-00194] and for disallowing a portion of the costs incurred by CMP in implementing SmartCare” due to imprudence. OPA Br. at 16, 34. The OPA agreed with the Staff’s proposal (summarized below in Section IV.F) for independent electricity-use audits through a third party, such as the Efficiency Maine Trust, and a review process for customers with high-usage complaints that are not resolved by this docket. OPA Br. at 38. The OPA suggests that the Commission’s order should explicitly leave open the option for customers to seek remedies in court or through the CASD. OPA Br. at 39. D. Lauren Loomis and CMP Ratepayers Unite 1. Metering and Billing Ms. Loomis submitted direct testimony on behalf of CMP Ratepayers Unite on August 30, 2019. The testimony focused primarily on the limitations of SmartCare and different types of meter anomalies. At the hearing, Ms. Loomis testified that she was not trained or educated in engineering or electric-meter technology, and that the research she used in preparing her testimony was found by way of Internet searches using Google and asking questions of other members of CMP Ratepayers Unite. Tr. at 128– 29 (Nov. 6, 2019 Hr’g). In the testimony, Ms. Loomis made several claims about CMP’s metering and billing practices, including erroneous, manual, or miscoded entries in records of metered usage. She claimed that the fast-clock anomaly affects every meter that does not have a battery backup. Ms. Loomis expressed concern that customers were disputing their bills, yet their cases were being closed and they were being forced to pay their bills. Examiners’ Report 2. 55 Docket No. 2019-00015 Remedies Through Ms. Loomis’s testimony, CMP Ratepayers Unite proposed that the Commission adopt the following remedies: reimburse all customers affected by a billing defect, an inaccurate meter-read entry, or metering anomalies; require CMP to hire the necessary employees to upgrade the meter firmware, and monitor the upgrades; audit the AMI network and meters again; reopen all closed billing disputes and hire an independent third party to investigate them; monitor the accuracy of high-usagecomplaint meters with a shadow meter; question employees of GE, Aclara, and Landis+Gyr about meter anomalies; require that CMP rebill customers for any overbilling for up to six years before November 1, 2017; require that validating, estimating, and editing software be installed on CMP’s meter data management system; require that CMP ‘s shareholders pay for any underbilling as a “penalty for not understanding the importance of” using validating, estimating, and editing software on their meter data management system; and require CMP to open a temporary welcome center for customers to review their bills with a customer-service representative. Loomis Dir. at 6–7. CMP Ratepayers Unite also requested that any “compensation” or “damages from the metering anomalies and billing issues be handled outside of the [Commission] and [instead] in the courts. Compensation for any damage or harm done to a customer is not handled by the [Commission] and is outside of their jurisdiction. The class action lawsuit should be able to proceed” alongside the Commission’s investigation “as both would be dealing with separate issues affecting customers.” Loomis Dir. at 4. Examiners’ Report E. 56 Docket No. 2019-00015 Levesque Intervenors The Levesque Intervenors did not submit testimony or a post-hearing brief. The Levesque Intervenors did, however, cross-examine CMP’s metering-and-billing panel at the hearing. Through cross-examination, the Levesque Intervenors challenged CMP’s billing of one CMP ratepayer, Mr. Samuel Jones, over the summer of 2019. Tr. at 8–24, 43–48 (Nov. 5, 2019 Hr’g). F. Commission Staff On September 3, 2019, the Staff submitted its Bench Analysis, which: (1) summarized the factors that led to this investigation; (2) described the statistics related to high bills during the winter of 2017–2018; (3) summarized Liberty’s findings related to metering and billing; (4) presented Staff’s comparative analysis of customers’ billing patterns in Emera Maine’s Bangor Hydro District and CMP’s service territory; and (5) presented a proposal for an electricity-use audit process to address customers’ concerns about their high billed usage. BA at 1. Staff explained that, as part of the summary investigation in Docket No. 2018-00052, it obtained information from CMP about customers’ billed usage during the months of December 2017, January 2018, and February 2018 (Winter 2) and the same months in the prior year (Winter 1) for all residential customers whose bills in any of the months in Winter 2 showed a year-over-year increase of 50% or more. Id. at 2. According to the information CMP provided, 90,044 unique customer accounts experienced an increase in total monthly billed usage of 50% or more. Id. Staff then described Liberty’s conclusions relating to high bills in its audit (described above in Section IV.A.2). At a high level, those findings were that: (1) there were no systemic problems with CMP’s metering or billing systems causing billed usage to be erroneously Examiners’ Report 57 Docket No. 2019-00015 high; and (2) the increased usage was due to weather that was significantly colder than normal. Id. at 4–5. Staff explained that, if the complaints about high bills for CMP customers were only weather-related, it begged the question as to why customers of other Maine transmission-and-distribution utilities had not reported the same level of concern about high usage or bill increases. Staff noted that answering this question was important because it could inform the validity or accuracy of the high usage observed for CMP customers. Thus, to explore the matter, Staff obtained and analyzed data for residential customers of Emera Maine’s Bangor Hydro District (BHD) during the same months of Winter 1 and Winter 2. Id. at 6. Staff made certain adjustments to ensure that the CMP and BHD data were comparable. 19 Staff then presented the results of its analysis, which showed that increases in billed usage were experienced by a virtually identical proportion of customers in the two service territories. The comparison is shown below in Figure 2. 19 The figures were made comparable by screening out low-usage bills, including bills for which all usage was covered by the “minimum bill” charge. These low-usage bills showed usage of 100 kWh or less per month. BA at 3, 6. Examiners’ Report 58 Docket No. 2019-00015 Figure 2 20 Residential Accounts with Winter 2017/2018 kWh Usage Increase of 50% or More Compared to Same Month in Winter 2016/2017 ("High Use Customers") CMP Emera Maine BHD a. High Use Customers 75,439 14,349 b. Total Residential Customers 552,236 103,858 13.7% 13.8% c. Percent (a) / (b) Total residential customers reflects the average during Winter 2. Staff noted that this corroborated Liberty’s finding that the usage increases were driven by weather and not by any systemic problem with CMP’s systems. Id. at 7. Finally, Staff expressed its concern that, even though it believes weather was the cause of CMP’s high-usage issues in winter 2017–2018, a significant number of customers had high-use concerns that had not been adequately addressed or explained by CMP. Staff recommended that the Commission require CMP establish an independent review process for customers who experienced a significant increase in billed usage after SmartCare was implemented for which the cause remains unexplained. As part of this process, customers could opt for an electricity-usage audit by an independent third-party, such as the Efficiency Maine Trust. If the audit showed a discrepancy between the customer’s expected usage and the amount for which the customer was billed, additional analysis and review would be necessary. Staff suggested the following criteria for determining the customers for which the audit would be an option: (1) the customer’s concern must be related to unexplained high usage; 20 BA at 7, Fig. 1. Examiners’ Report 59 Docket No. 2019-00015 (2) the high usage occurred after the implementation of SmartCare; and (3) the customer had filed a high-usage complaint with the CASD. Id. at 9–10. G. Mary Fournier Intervenor Mary Fournier did not submit testimony or a post-hearing brief. Intervenor Fournier cross-examined CMP’s metering-and-billing panel, Liberty’s witnesses, the OPA’s consultants, and the Commission Staff at hearing with questions about several topics, including bill accuracy, an independent review of SmartCare, and how customers’ complaints were being handled. Tr. at 37–40, 127–53 (Nov. 5, 2019 Hr’g); Tr. at 13–17, 57–59, 163–73 (Nov. 6, 2019 Hr’g). V. OVERVIEW OF INDIVIDUAL COMPLAINANTS’ POSITIONS A. Experience of the Consumer Assistance and Safety Division The CASD began receiving a large number of complaints from customers relating to billing concerns in December 2017 and January 2018. These concerns were about customers receiving higher than usual bills, not receiving proper credits on their bills, having payment arrangements changed without notice, and not receiving bills at all. The two most prevalent billing complaints the CASD received were about highusage complaints and customers not receiving bills (referred to by CMP as “delayed bills”). 1. Adoption of Special Process for Managing Complaints of HighUsage Due to the large number of high-use complaints being filed with the CASD and the nature of those complaints, the CASD requested in early February 2018 that CMP establish a specialized group of individuals who were familiar with CMP’s billing and metering processes to investigate and resolve customer complaints of high usage. Examiners’ Report 60 Docket No. 2019-00015 Under this process, when the CASD receives a customer complaint of high usage, the CASD forwarded the complaint to the specialized team, 21 which reviews the customer’s account to ensure that bills are accurate and attempts to ascertain the cause of the high bill. If CMP’s team could determine the cause of the increased usage or bill amount and resolved the matter to the customer’s satisfaction, the CASD considered the matter closed. If CMP could not determine the cause of the increased usage or otherwise resolve the matter to the customer’s satisfaction, the matter was referred back to the CASD. 22 The customer was then notified that their complaint would be resolved based on the conclusions and findings in the Commission’s high-bill investigation. The intent of this process was to ensure that complaints that involved customer-specific issues or situations were resolved when possible, while incorporating into the Commission’s billing investigation any complaints of high usage that could not be resolved. 23 21 This specialized team is sometimes referred to as the High Bill Resolution Team or just CMP’s team. 22 When the CASD receives a billing complaint from a customer, it conducts a preliminary investigation to determine if a billing error or anomaly is involved. If a billing error or anomaly appears to be the cause of the customer’s billing concern, the CASD: (1) determines whether the customer has been billed accurately for the customer’s actual usage; and (2) orders the Company to adjust the customer’s bill if necessary. If a billing error or anomaly is not at issue and the customer’s billing concern relates only to high usage, the CASD refers the customer’s complaint to CMP’s high-use team for resolution. 23 When CMP’s High Bill Resolution Team was first established, the CASD and CMP agreed on a two-business-day period for the team to contact each customer. Due to the large number of customers being referred to the specialized team, the contact period was extended to two weeks at CMP’s request. Even with this extension, CMP was unable to consistently meet the contact goal. In September of 2018, the Director of CASD sent a letter to CMP expressing concern that at times it was taking CMP’s team over three months to contact customers. The letter also stated that, at that time, only 45% of the total number of customers referred to CMP’s team had been contacted by CMP. The letter further stated that this delay had resulted in a large backlog of outstanding customer complaints, as well as a significant number of calls to the CASD from referred customers concerned about the time it was taking to hear from CMP. In a subsequent letter sent February 6, 2019 to CMP, the CASD again notified CMP that the time it was taking the Company to address customer complaints or contact customers who have Examiners’ Report 61 Docket No. 2019-00015 Some customers were contacting the CASD stating that they had not received a bill from CMP for a number of months. Customers were concerned that they would end up receiving a large make-up bill at some point in the future that they would be unable to pay. According to CMP, it identified bills with a potential problem and held the bill (i.e., did not issue it) until the billing problem could be rectified. These customer complaints led the Commission to initiate its summary investigation into CMP’s metering, billing, and customer-communication issues. Public Utilities Commission, Investigation of Central Maine Power Co. Metering, Billing and Customer Communication Issues, Docket No. 2018-00052, Notice of Investigation (Mar. 1, 2018). 2. Number of Complaints Raised and Resolved From the implementation of SmartCare on November 1, 2017 through January 8, 2020, the CASD has taken 2,380 customer complaints against CMP. The CASD has been able to resolve 1,755, or 74%, of the complaints received. When those customers had concerns about high usage in addition to the billing error or anomaly addressed by the CASD, these same complaints were also referred to CMP’s High Bill Resolution Team. As of August 19, 2019, 3,628 customers with concerns about high usage have been referred by the CASD to CMP’s High Bill Resolution Team. Of the 3,628 customers referred, CMP was able to reach 2,455 customers and, of these customers, had resolved 1,208 to the customer’s satisfaction. That left 1,247 customers for whom CMP had not been able to resolve the high-usage complaint to the customer’s satisfaction. complained, continued to be insufficient and stated that the CASD viewed this failure to contact customers in a timely manner as a violation of the Commission’s rules. Examiners’ Report B. 62 Docket No. 2019-00015 Public Witness Testimony The Commission held three public-witness hearings in this case. At those hearings, the Commission accepted testimony about this docket and Docket No. 201800194, CMP’s rate case. That testimony has been incorporated into this record where it relates to the issues in this case (i.e., metering or billing issues); the testimony that applies to issues being considered in the rate case (i.e., customer service) will be considered in that case. The hearings were well attended and the people who testified provided impassioned testimony about their experiences with and opinions about CMP. Customers expressed great concern with the accuracy of the Company’s bills and the way that CMP responded to customers’ concerns about their bills. Many customers expressed frustration first with receiving what appeared to be inaccurate or confusing bills, and then either being unable to reach CMP to discuss their concerns or CMP’s being unable to reasonably explain the customer’s bill. The themes customers consistently expressed, as they related to the billing and metering issues being considered in this case, are summarized below by way of excerpts from sworn testimony of these public witnesses. 1. High Bills Customers testified about receiving significantly higher bills after the implementation of SmartCare. Examples of testimony on this theme are: • “Ever since the new billing took place, I’ve had a billing problem. Now, I was on SimplePay so I didn’t realize that I was going to get swamped with this bill. . . . May 7th of 2018 my SimplePay changed to $574 a month. I was on Examiners’ Report 63 Docket No. 2019-00015 SimplePay, it was 144. How is that possible? That’s $6,000 a year. That’s insane.” Tr. at 83 (July 16, 2019 Pub. Wit. Hr’g). • “So my problems started back in February of ‘18. I had gotten a bill in the mail, noticed the bill was doubled. I immediately called CMP. I actually thought—it was so high, I actually thought somebody had tapped into my electric. I was like, hey, you got to get down here and check this out, there’s something wrong. They totally, like, blew me off.” Tr. at 87 (July 16, 2019 Pub. Wit. Hr’g). • “Bill number one is the family camp, Evergreen Valley, Lovell, Maine. Before the new meters and billing system, the monthly bill when the camp was not in use was under $20, about 17, $18 a month. As soon as we had new meters and/or a new billing system, the bills jumped to over $30 a month to about 32 a month when the camp was not in use. Bill number two, our family home in Biddeford did the same thing as our camp bill. It went from about 75 to $85 a month to approximately 125 to 130. Basically both bills increased for no reason other than we got new meters and there was a new billing system. No change in usage. No change.” Tr. at 92 (July 16, 2019 Pub. Wit. Hr’g). • “My first bill after the storm was for 1,597 kilowatt hours, double my normal usage. My usage was doubled even after having no power for eight days. So my three-week usage was double the normal.” Tr. at 22 (July 22, 2019 Pub. Wit. Hr’g). Examiners’ Report 2. 64 Docket No. 2019-00015 Delayed Bills or New Customers Not Receiving Bills Customers testified about experiencing delayed bills after the implementation of SmartCare: • “By February I stopped receiving bills. I thought did I hit, like, you know, agree to pay this online or something, what has happened. So I got online, put my account number in, showed a balance, and I paid it. I continued to do this. Come June that summer, I logged on and my balance said I owed 34 cents. I knew we had a problem. So I said I don’t know where I would owe 34 cents. It’s got to be a typo error. I’ve got to owe more money than that. So I called CMP. Was on hold forever. Come to find out, long story short, I hadn’t been receiving bills. I’d been logging on and paying what I thought was my balance and found out that I wasn’t being charged for my usage.” Tr. at 18 (July 22, 2019 Pub. Wit. Hr’g). • “I live in Camden, and I’ve been a CMP customer since 2001. And I really didn’t have any serious problems until the fall of 2017. And what happened with my household, . . . we just had a period where we didn’t get any bills starting in about January of 2018. It was about a three-month stretch, no bills. I called them to see if there was a problem or if something needed to be done. Didn’t really get a response. Finally a bill came in about April of that year, and it . . . didn’t seem like it was accurate. It wasn’t wildly inflated, but it just seemed like a kind of odd number.” Tr. at 88 (July 22, 2019 Pub. Wit. Hr’g). Examiners’ Report • 65 Docket No. 2019-00015 “Our problem started in September of 2017, right before the alleged meter change. Our bills normally run about $11,000 to $12,000 a month. The bill for that month, excuse me, was $46,000. The month after that, which is during the time that we had the power outages, it was $24,000. I actually didn’t realize the problems with the billing until February when I was going over endof-year reports for our accountant, and I noticed the huge rate increase. . . . And what’s kind of weird is that after the . . . $46,000 bill, a $24,000 bill, starting in January of 2018, they were down to a thousand dollars a month which I explained to her that now it’s underbilling us. So when she finally did get back to me, excuse me, she said there’s been an error on the October bill which was the 26,000 but the September bill, there is no error. You owe $46,000 which we had already paid. And I told her that we have access to the MV website where I can pull historical data for the usage that we have and the data on that site said we only used 211,000 kilowatts for that entire billing period and CMP was billing us for 467,000. Her reply to me was that bill is right.” Tr. at 92 (July 22, 2019 Pub. Wit. Hr’g). New customers testified about not receiving bills. This included the following testimony: • “And when I first went on, I signed all the paperwork with Central Maine so that they would bill me. And being unused to what I was supposed to be receiving, I finally called them after four months because I hadn’t got a bill yet. And the girl just said, well, you know, I don’t know what we—so I gave them my account number again, and she said, well, I’m sorry. Well, that all turned Examiners’ Report 66 Docket No. 2019-00015 out to be nothing but everybody says here as lip music. That’s all I got was lip music. So when I finally got my bill, I had six months and it ran into December. . . .” Tr. at 32 (July 22, 2019 Pub. Wit. Hr’g). • “So I got my power hooked up. And that was first week in August, second week in August, something like that. And I call in October and I say, you know, it’s a new service and I haven’t gotten a bill yet, much like the other gentleman who spoke tonight. And they said, well, it just takes us a couple months. We’ve had a big backlog of new construction. It’s going to take a little while. I said, okay, I just don’t want to get billed for late charges. I’m just calling to find out. Call again in December. Well, the same thing, you know, we’ll take care of it. So I get the first bill that I receive is in late March, and I . . . didn’t use much electricity so I don’t have an issue with that.” Tr. at 99 (July 16, 2019 Pub. Wit. Hr’g). 3. Incorrect or Confusing Bills Customers testified about receiving incorrect or confusing bills, as follows: • “When they started the billing, the new billing system, I can tell you exactly when they started it because I paid them $109.39. It disappeared off the face of the Earth. I had a confirmation number that I had paid it. I had it come out of my checking account. And I never got credited. I dealt with customer service till it escalated to a supervisor named Mary. She was a wonderful person. It took me till the second week in January before it got rectified. Let’s see, that was the second week in January. So then I thought everything was straightened out. She had credited me my $109.39 and explained it was a Examiners’ Report 67 Docket No. 2019-00015 new billing system, shouldn’t have happened.” Tr. at 18 (July 22, 2019 Pub. Wit. Hr’g). • “In going through all this, I started updating, and I extrapolated some information of the utility data from the CMP website. And I realized that on my last bill—I looked at the previous month’s reading and the current month’s reading, and I did the simple math to see what my energy usage should be. It is not what I am being billed for. And that’s a simple calculation. This isn’t based on time of use. This isn’t based on overall usage. This is a simple mathematical what was it previously, what it is now, the difference, that’s your kilowatt usage.” Tr. at 74–75 (July 16, 2019 Pub. Wit. Hr’g). • “At one point, my balance, automatic pay, remember, $4,395.36. That was on 11/27/18. The next one, 12/21, $1,815.35. 12/26, $2,305.35. My statement, as of 6/26/19 is a credit of 241. I have 15 pages that CMP cannot explain. I am out thousands of dollars. I am at my wit’s end.” Tr. at 46 (July 22, 2019 Pub. Wit. Hr’g). 4. Estimated Bills Customers with non-standard meters provided the following testimony about receiving estimated bills, such as the following: • “And with the analog meter they’re supposedly coming out every other month to read it, and we read it and call it in the other months. In the past year and a half they’ve been out three times. My wife is home during the day time, and we have a dog that likes to let us know when someone’s in the yard. So we know that she says they’ve been out three times in the past year and a half. Examiners’ Report 68 Docket No. 2019-00015 And the other thing is if they’re charging . . . our delivery charge, we pay for the maintenance of equipment, maintenance of the lines, and all of that. With the analog meter, we’re also paying an additional charge for maintenance of the equipment which we’re already being charged. So basically they’re double billing us for . . . having an analog meter.” Tr. at 41–42 (July 16, 2019 Pub. Wit. Hr’g). • “After that, they started reading the meters every other month. And bring it up to your attention, last year, the month of April I got a bill, but the month of May, June, and July were all three estimated. So I didn’t get a bill until the month of August. So that way the meter is way up here because they estimated them. And now this year they’re . . . giving me estimated bills based on last year’s proceedings, which was way up when, in fact—I do something different than most people do.” Tr. at 15 (July 22, 2019 Pub. Wit. Hr’g). • “And I was a person that kept the old meter when smart meters were installed, and it was $12 a month and then it went to $15 a month and that was to get you guys to—I was told at first it was to get . . . them to read them, to pay for someone to read them. And at the end, I was told it was to maintain them. But nobody was reading them. They’re estimating them. And my bill, every month, is estimated.” Tr. at 20–21 (July 22, 2019 Pub. Wit. Hr’g). C. CMP’s Response to Individual Complainants’ Statements CMP stated that it is committed to providing excellent customer service, that it takes customer complaints seriously, and is committed to identifying the root cause and remediating problems. CMP Met. & Bill. Reb. at 40. CMP added that it reviewed the 279 Examiners’ Report 69 Docket No. 2019-00015 pages of transcripts from the three public-witness hearings and conducted an in-depth review of each customer’s experience back as far as 2015. Id. at 41. The Company created a case study for each customer situation. 24 Each of the Company’s case studies includes four sections: (a) a summary of the customer’s concern(s) expressed at the public witness hearing (the Company referred to this as the Customer Claim); (b) the customer’s Account Details, which included billing/usage information, payment behavior, and CMP/customer interactions; (c) a Summary section of the customer’s interactions with the Company; and (d) a Follow-up section that described the Company’s follow-up with the customer after the public-witness hearing. In CMP’s assessment, the experiences customers testified about at the publicwitness hearings generally fit into one of seven categories: 1. Response to Concerns About Usage CMP stated that approximately two-thirds of the customers expressed concerns about usage. The Company added that usage questions are not unique to CMP or to the post-SmartCare timeframe, and that the Liberty Report and the Bench Analysis concluded that severe weather from late 2017 to early 2018 did affect customer usage. Id. at 42–43. CMP said that its review of the 80 customer case studies did not show any sustained increase in usage since the implementation SmartCare. Id. at 43–44. The Company concluded that the usage depicted in the case studies showed no systemic problem with its metering or billing systems that would have caused customers’ usage 24 See Confidential Exhibit MB-8 for CMP’s compilation of the 80 case studies. Several of these case studies were made public by procedural order; CMP filed the public versions in the case file on December 11, 2019. See Dec. 6, 2019 Procedural Order at 2–3. Examiners’ Report 70 Docket No. 2019-00015 to be overstated, but instead generally tracked expected seasonal-usage patterns. Id. at 44. The Company also stated that its usage analysis of the case studies revealed a category of customers who reported installing heat pumps. In these cases, the customers’ interval usage data is consistent with the use of heat pumps. Also, conversations with these customers showed a misunderstanding of the effects of heatpump use on electric bills. According to CMP, its case studies also showed that: • Four complaints were from customers who experienced an increase in usage that began weeks or months before the implementation of SmartCare. • Two complaints were from customers who installed energy-efficient appliances or took steps to stop using certain appliances and expected their usage to go down. The Company stated that it can be difficult for a customer to detect evidence of improved efficiency because not all factors affecting usage remain constant before and after installing the new appliance. Id. at 44. • Five cases involved customers who moved into a new house or apartment and complained of high usage with no pre-SmartCare usage for comparison. Id. at 45. The Company stated that one-third of the customers who testified were dissatisfied with an experience that was not related to electricity usage (the remaining six categories). The Company identified these six situations as: Examiners’ Report 2. 71 Docket No. 2019-00015 Response to Concerns Over Net-Energy Billing The Company stated that with the implementation of SmartCare, the billing process for net-energy billing customers was automated and that the billing is complicated and some SmartCare issues resulted in delayed billing for these customers. Id. The Company added that some net-energy-billing customers complained that their banked usage was not accurate and were surprised that their net generation was much lower in winter months. Id. at 45–46. 3. Response to Customers with Non-Standard Meters The Company stated that some customers experienced billing errors associated with non-standard (analog) meters due to the unique requirements for the smart-meter opt-out program. A specific defect (#4711) affected customers who switched from a smart meter to a non-standard meter after the implementation of SmartCare. This defect caused bills to be estimated with zero usage. The Company added that this defect has been corrected, but it may have caused customer confusion prior to its correction. Id. at 46. 4. Response to Claims That the Company Made an Error or Was Delayed in Processing a Transaction The Company stated that some customers were affected by errors or delays in the Company’s processing of a transaction, such as changing a service or mailing address, reverting service to a landlord, mixed-meter situations, or a problem with tenants taking service at the wrong location. Id. at 46–47. The Company attributes several of these cases to human error and states that in each case, it has taken steps to apologize to the customer and remedy the situation. The Company also stated that, Examiners’ Report 72 Docket No. 2019-00015 when human error is found, employees are coached and trained to avoid the error reoccurring. 5. Response to Claims of Being Denied for a Low-Income Program The Company stated that its case study showed that two customers raised concerns about their denial of benefits under the Company’s Electric Lifeline Program. The Company added that its analysis showed that, in each case, the Company’s eligibility determination was accurate. 6. Response to Complaints About Frequency of Power Outages The Company states that two customers complained about frequent power outages. The Company added that it investigated each of these complaints and the causes of the outages. Id. at 47. 7. Response to Complaints About Service Options The Company summarized customers’ complaints on other service areas, such as: payments scheduled on the due date posting after the due date, so-called greenbutton options for downloaded data are not helpful, dissatisfaction with assigned rate class, dissatisfaction with offered payment arrangements, dissatisfaction with being assigned a new account number without notice, inaccurately estimated bills for new tenants, delays in CMP’s energizing new service, dissatisfaction with some meters being estimated on a bill, and frustrating customer-service experiences. Id. at 47–48. The Company stated that it is committed to resolving all customer complaints and that since January 2018 it has reached out to over 3,500 customers with usage concerns through a special process agreed to with the CASD in an attempt to help customers diagnose, understand, and resolve their usage concerns. Id. at 48. The Company also stated that it recognizes the numerous complaints from customers Examiners’ Report 73 Docket No. 2019-00015 alleging that bills sent after the implementation of SmartCare were too high and not reflective of their actual usage. The Company added that it agrees with the findings of the Liberty Audit and the Bench Analysis that the higher bills experienced during the winter of 2017–2018 were caused by colder-than-normal temperatures and an 18% increase in the standard-offer supply price for residential customers. Id. at 50. Nonetheless, the Company added that it generally agreed with Staff’s recommendation that an independent dispute-resolution process be established that would be conducted by an independent entity, such as the Efficiency Maine Trust, as an additional step in the usage-dispute-resolution process. Under such a program, qualifying customers with unresolved high-usage disputes would be referred to the independent entity and offered a more in-depth audit or assessment of their electric usage. In addition to a more detailed review of metered usage and discussions with the customer, this independent assessment would provide an in-home evaluation of the customer’s premises and electricity-consuming devices. Id. at 53. The Company pointed out that it has collaborated with the Trust to develop a pilot program that will test the effectiveness of the concepts described in the Bench Analysis. The Company has entered into a memorandum of understanding (MOU) with the Trust to implement such a program. Under the MOU, the Company will fund a pilot program that is consistent with the process described in the Bench Analysis, using qualified technical professionals to perform an in-home energy assessment for qualifying customers. A qualifying “Usage Concern Customer” under the MOU must be a residential customer of CMP with an unresolved billing dispute that has progressed through the process managed by the Company’s High Bill Resolution Team and has Examiners’ Report 74 Docket No. 2019-00015 experienced an increase in usage of at least 25% over a historical baseline level. For purposes of this pilot program, qualified customers will be limited to new customers who have recently progressed through the process managed by the High Bill Resolution Team. The Company added that CMP and the Trust will evaluate the effectiveness of the pilot program in addressing its stated objectives after 50 customers have advanced through the program or six months have elapsed, whichever occurs first. If after this evaluation CMP and the Trust agree that the pilot has achieved its objectives and should be continued, the program could be extended or made permanent with any modifications agreed upon by the parties. If there is ultimately a consensus that the inhome assessment program is a useful addition to the high-bill dispute-resolution processes, the Company suggested that the program be adopted permanently and become customer-funded, either directly through utility rates or indirectly through the Trust’s own funding assessment. Id. at 55. D. Overview of Individual Complainants’ Rebuttal to CMP’s Case Studies On November 7, 2019, the Examiners issued a procedural order requiring that CMP mail a copy of each case study from Exhibit MB-8 to the customer whom the case study is about. In that mailing, customers were informed of, among other things: (a) their right to respond to the case study by December 3, 2019; and (b) their right to consent to making the information in the case study public. If any customer were to seek the right to respond, the Examiners would “determine what additional process (e.g., supplemental briefing) is necessary.” Nov. 7, 2019 Procedural Order Ruling on Motion to Disclose at 3. The Commission received responses from 20 CMP customers. Examiners’ Report 75 Docket No. 2019-00015 In most of the 20 responses, the customer consented to public disclosure of their account information, waiving the privacy protections of Chapter 815, § 4 of the Commission’s rules. The customers did this through either (a) a written or oral waiver of consent to confidentiality or (b) by submitting the information as a public comment in Docket No. 2019-00015 (and thus consent was provided implicitly through the customers’ public submittal). Where the customer did not consent to public disclosure of their account information, the responses remained confidential, though they are still part of the evidentiary record. The Commission and the parties to the case have access to the confidential information under Protective Order No. 1 (Customer-Specific Information), but the public does not. 25 In general, the 20 customers who responded to Company’s case studies stated that the case study did not accurately portray the customer’s concern(s), contained factual inaccuracies, left out key information, failed to address the customer’s primary concern, or failed to accurately describe the customer’s experience dealing with CMP. Examples of these concerns are provided below. 1. Claims That the Company’s Portrayal of Its Interaction with the Customer Was Misleading or Inaccurate In one case study, the Company stated that the customer complained that she had experienced high usage, that Energy Manager 26 did not match the meter readings that she had photographed, and that after she changed to a non-standard meter that CMP did not read her meter. The Company also stated that the customer’s average 25 26 That protective order was issued in this docket on February 22, 2019. Energy Manager is a feature of CMP’s online customer portal that allows a customer to view electricity-usage interval data. The usage shown in Energy Manager is not necessarily the same usage that is billed to customers. See Tr. at 132 (Nov. 6, 2019 Hr’g). Examiners’ Report 76 Docket No. 2019-00015 usage had decreased, noting that before SmartCare implementation the customer had used an average of 1,341 kWh per month and after SmartCare implementation had used an average of 1,141 kWh per month. The Company added that the customer had provided a picture of a meter reading, her bill, and a usage alert. The exact timing of the customer’s photograph, usage alert, and bill are not the same and therefore rendered different usage amounts. Finally, the Company stated that the customer’s meter had been read in person on every alternating month from August 2018 to date. CMP Met. & Bill. Reb. Exh. MB-8, Case Study #9; see also Cust. Exh. T. 27 The customer responded to the Company’s case study by saying that CMP’s claim to having read his non-standard meter every other month since installation was inaccurate and deceiving. The customer went on to say that they (CMP) “obviously haven’t read my meter every other month, as my meter was changed on January 25, 2018 at 1:30 pm, leaving the months of February, April, June and August 2018 and October of 2019 . . . . l have found discrepancies between the readings that CMP claims their meter readers recorded and the readings submitted (on my bills); the usages stated on my bills reflect what I have submitted through CMP’s automated 1-800 number (option 6 for reporting meter readings) for these months. What is more baffling is why meter reads were not done on the dates set by them per their schedules and posted on their website as well as their letter dated April 17, 2018?” Cust. Exh. T (customer response to Case Study #9). 27 Staff labeled customers’ responses to CMP’s case studies as Customer Exhibits A through T and, where needed, redacted those exhibits for confidential information. The lettered exhibits in this section refer to those exhibits, which the Staff issued on December 6, 2019. Examiners’ Report 77 Docket No. 2019-00015 In another case study, the Company stated that the customer complained of higher-than-expected usage in November 2017 and January 2018, that her bills doubled and tripled from previous years, and that she had never received a bill prior to this period above $80. The Company stated that, prior to 2017, the customer’s winter bills were often over $100 and that the customer has consistently carried forward past due amounts on her bill. CMP Met. & Bill. Reb. Exh. MB-8, Case Study #61. The customer responded to CMP’s case study by stating that “[it] seems that CMP is trying to shift attention from their errors in attempting to discredit me by focusing on late payments and the four disconnect notices on my account since the investigation started in November 2017. There are also multiple mistakes, misinterpretations and omissions in their case study, which I have noted directly on the study.” The customer went on to say: “It is unethical and dishonest to have customer service representatives blaming customers for high bills, especially when they had been receiving multiple calls from multiple customers with the same issue. It is deceptive and/or just plain incompetent to make such egregious errors like they did when reviewing only 2 months out of a time frame of 9 months, and then having the nerve to follow that up with a disconnect notice.” Cust. Exh. Q (customer response to Case Study #61). 2. Claims That the Company’s Case Study Left Out Key Information In one case study, the Company stated that the customer “complained that he has had higher winter usage and problems using SimplePay. After the implementation of SmartCare [he] experienced delayed bills and SimplePay recalculations that he did not understand. [He] called on a number of occasions looking for delayed bills or to discuss SimplePay. [He] requested a supervisor call back in June of 2019 and did not receive a call back. [He] called in July of 2019 and the representative told him that the Examiners’ Report 78 Docket No. 2019-00015 Company would research his concerns and get back to him, which had not happened before he filed a MPUC complaint on 7/26/19. [He] has questioned his bill calculations because when he multiplies the rates by the usage on his bill he comes up with different amounts than are stated on the bill.” CMP Met. & Bill. Reb. Exh. MB-8, Case Study #66. In response to the Company’s case study, the customer stated that the Company’s response did not address the strange meter readings shown on his bills for April and May 2018. He disputed his account balance and criticized the idea that CMP’s Super Saver bill rates are calculated using three decimal points but the meter readings do not include decimal points. He claimed that meters are not capable of reads higher than 99,999, so a six-digit meter-read is impossible. The customer also stated that there was no mention of what happened to the interest on account balances that CMP owes the customer. Cust. Exh. C (response to Case Study #66). 3. Claims That the Case Study Failed to Address the Customer’s Primary Concern In one case study, the Company stated that the customer complained that her bills were confusing, that a meter worker asked the customer which meter was for generation, and that when the customer called CMP, she once had to wait an hour on hold and another time had to wait 30 minutes on hold. The Company added that the customer had installed solar generation in August 2017 and CMP estimated the bills without net generation in August and September. In October 2017, CMP rebilled the customer for August and September to include the solar generation. The Company went on to say that it had no record of a meter reader asking the customer which meter was used to measure generation and that there are no records of the customer calling the Company. CMP Met. & Bill. Reb. Exh. MB-8, Case Study #2. Examiners’ Report 79 Docket No. 2019-00015 In response to the Company’s case study, the customer stated that “nowhere in their document do they address my major complaint that they should have accompanied their corrections of my bills with some sort of letter of explanation and perhaps also an apology!” Cust. Exh. O (customer response to Case Study #2). In the response to Case Study #9 described in Section IV.D.1 above, the customer stated: “I received a response which did not adequately address the 234% increase in usage from the following month. They [CMP personnel] laid blame on the weather, faulty appliances, and alleged that we were using electric heaters (which we were not). They attempted to pin the increase in cost of my bill ($499) solely on my supplier, and while it’s true that my [then] supplier’s cost had increased, it was, in fact, CMP that provided the inflated meter readings to the supplier which was charged for. l know that there is no possibility that I could have used the amount of electricity that they claim.” Cust. Exh. T (customer response to Case Study #9). VI. STATUTORY STANDARDS A. The Prudence Standard The question of whether CMP acted prudently in the management of its metering and billing system has been raised in several areas of this investigation. Here, the Commission describes the law of prudence generally; later, we discuss how the standard applies to the particular areas of investigation. The law of prudence governs the Commission’s regulation of all of the state’s utilities. The Commission first gave clear guidance on the prudence standards in Maine Examiners’ Report 80 Docket No. 2019-00015 in a 1985 decision known as Seabrook. 28 In Seabrook, the Commission defined prudence as the “course of conduct that a capably managed utility would have followed in light of existing and reasonably knowable circumstances.”29 In examining prudence, the Commission must consider whether a utility is operating as efficiently as possible and using sound management practices. 30 The Commission in Seabrook laid out the following factors to consider in determining whether the utility has acted prudently: 1. Senior utility executives are expected to possess a high degree of financial and technical expertise. 2. While the prevailing practice of the utility industry is relevant, it is not determinative. The decisions of utility executives must also be reasonable when viewed against the decisions and courses of conduct of other corporations that make investment decisions of a comparable size and complexity. . . . 3. The size and nature of the undertaking being reviewed must also be considered. . . . 4. Review of utility decisions should consider the utility’s legal obligation to provide the safe, reasonable, and adequate service at the lowest possible cost over time throughout its service territory and to operate “as efficiently as possible” using “sound management practices.” 35-A M.R.S. § 51 [now section 301 of Title 35-A]. A utility is not free to tailor its decisions to profit maximization to the degree that an unregulated company would. . . . 5. A review of prudency requires an examination not only of the initial investment decision but also of the continuing action of the utilities in response to changing circumstances. 28 Public Utilities Commission, Investigation of Seabrook Involvements by Maine Utilities, Docket No. 84-113 (Phase II) Order (May 28, 1985) (“Seabrook”). 29 30 Id. at 12. 35-A M.R.S. § 301(2), (4); Maine Public Utilities Commission, Investigation into the Annual Reconciliation of CMP’s Stranded Cost Revenue Requirements and Costs, Docket No. 2006200, Order at 9 (Mar. 24, 2008). Examiners’ Report 81 Docket No. 2019-00015 6. If a utility has selected from among several reasonable courses of action[,] one [of] which turns out badly, the utility’s decision was not imprudent. 7. The utility’s course of conduct must be reviewed in light of existing facts and circumstances that either were known or knowable through an effort consistent with the size of the risk at the time decisions were made . . . [and] cannot be defined by hindsight. 31 If imprudence is found, the Commission must determine whether the imprudent action harmed ratepayers. If it did, then the injury or damage from that action needs to be quantified. 32 The Commission has recognized that while it is often difficult to determine the effect of imprudent conduct, that fact does not excuse the conduct. 33 The Commission has reasoned: It is true that a precise measurement of the rates that would have resulted had the Company followed a prudent and reasonable course of action is difficult to estimate at this time. This is typically a problem when measuring the harm of a course of conduct not taken. Still, the regulatory response must never be to ignore unreasonable and imprudent utility action because measuring precisely the costs of the reasonable and prudent course of action are difficult or even impossible. In fact, . . . our statutory duty to set just and reasonable rates would be violated if we set rates that reflect costs that are clearly unreasonable. 34 31 Seabrook, supra note 27, at 12–13. 32 Id. at 13–14. 33 Central Maine Power Company, Application for Fuel Cost Adjustment Pursuant to Chapter 34 and Establishment of Short-Term Energy Only Rates for Small Power Producers Less Than 1 MW Pursuant to Chapter 36 (Investigation of QF Contracts), Docket No. 92-102, Order at 55 (Oct. 28, 1993). 34 Id. (citing 35-A M.R.S. § 301(1), (4) (1988)). Examiners’ Report B. 82 Docket No. 2019-00015 The Evidentiary Burden As the party “adverse to the Commission,” CMP bears the general burden of proof in this matter, including on the question of prudence. 35 “[I]n the absence of a showing of inefficiency or improvidence” a court will not substitute its judgment for that of utility managers. 36 Yet there is no presumption of prudency when the issue arises in a proceeding. 37 Once a party or the Commission Staff has raised the issue of inefficiency or imprudence “in a sufficiently specific way,” the party or Staff has met its burden of production, and the burden then shifts back to the utility to demonstrate that its actions were prudent. 38 VII. DISCUSSION AND DECISION A. Is CMP’s SmartCare System Accurately Recording Usage? As described in Section IV.A above, in its audit Liberty examined every CMP customer account and meter, of which there were about 650,000, during the six-month audit period. Liberty reviewed about 4 million billing records and 2.3 billion data points to trace the integrity of data from end-to-end of CMP’s metering and billing systems. The 35 35-A M.R.S. § 1314(1), (2). 36 Cent. Me. Power Co. v. Pub. Utils. Comm’n, 153 Me. 228, 243 (1957); see also Seabrook, supra note 27, at 9. 37 Central Maine Power Company, Application for Fuel Cost Adjustment Pursuant to Chapter 34 and Establishment of Short-Term Energy Only Rates for Small Power Producers Less Than 1 MW Pursuant to Chapter 36 (Investigation of QF Contracts), Docket No. 92-102, Order at 12 (Oct. 28, 1993) (rejecting the argument that a presumption of prudency exists, and clarifying that only “[i]n the absence of any challenge [will] the utility’s actions . . . be presumed to be reasonable” (emphasis in original)). 38 Central Maine Power Co., Annual Price Change Pursuant to Alternative Rate Plan (ARP 2008), Docket No. 2011-77, Order at 19 (July 27, 2012); Central Maine Power Company, Application for Fuel Cost Adjustment Pursuant to Chapter 34 and Establishment of Short-Term Energy Only Rates for Small Power Producers Less Than 1 MW Pursuant to Chapter 36 (Investigation of QF Contracts), Docket No. 92-102, Order at 13 (Oct. 28, 1993). Examiners’ Report 83 Docket No. 2019-00015 audit period (November 2017–April 2018) was chosen because it was during this period—which immediately followed a major windstorm, system outages, and CMP’s cutover to its SmartCare billing system—that significant numbers of CMP customers experienced higher-than-usual bills or increases in electricity usage. Liberty also analyzed the correlation between usage and weather during the audit period and during the prior four years. Based on its in-depth audit, Liberty concluded that there was no inaccuracy in CMP’s metering and billing systems that would have resulted in any systemic overstatement of usage. Liberty’s usage-to-weather analysis showed an extraordinarily strong correlation between coldness of weather and amount of electricity consumed, one that even surprised Liberty. Tr. at 63 (Feb. 19, 2019 Tech. Conf.). This usage-to-weather correlation analysis further supported Liberty’s conclusion by showing that customer usage during the audit period had the same correlation with weather as usage in prior years. Liberty Report at 58–59. In addition to the information and analyses provided in the Liberty Report, the Staff’s separate and independent analysis demonstrated comparability between the usage increases of CMP customers and customers in an adjacent utility service territory, the Bangor Hydro District of Emera Maine. This supports a finding that the high usage and usage increases reported by and observed for CMP customers were not due to systemic problems unique to CMP’s metering and billing systems; rather, the strong similarity between usage increases of CMP’s and Emera Maine’s customers strongly indicates that the increases resulted from a factor that was common to both utilities: weather. Examiners’ Report 84 Docket No. 2019-00015 No party has provided evidence in the record to support a finding contrary to that of Liberty and the Staff. Indeed, no party has challenged the factual and analytical bases upon which these findings are based. Most notably, after conducting its own “eight weeks of intense review and analysis,” the OPA’s expert, BerryDunn, did not find “a defect, set of defects, or root cause for the numerous complaints related to high usage.” Keim Dir. at 16. The OPA appears to suggest that the lack of such a finding by BerryDunn is not relevant because BerryDunn’s scope of work did not include examining the accuracy of CMP’s metering and billing systems. OPA Reply Br. at 2. The OPA’s statement is surprising given that, in its June 24, 2019 “OPA Proposal for Testing of Central Maine Power Accounts,” the work to be done was described as “head-end to bill testing of specific accounts for the period of time following the close of the audit period – May 1 2018 to the present.” This was presumably the scope of work for which BerryDunn was retained. As described above, witnesses for CMP Ratepayers Unite have provided testimony describing several problems and errors with CMP’s metering and billing systems, including some related to meter anomalies, manual entry of meter data into SmartCare, and the accuracy of CMP’s Energy Manager system. In addition, the Levesque Intervenors provided testimony about bill errors experienced by an individual customer. The Commission finds that, although this testimony may describe discrete errors or flaws, it does not counter a finding that no systemic problem exists within CMP’s metering and billing systems that is, or has been, causing erroneously high customer bills. Examiners’ Report 85 Docket No. 2019-00015 Thus, based on the evidence before it in this proceeding, the Commission finds that CMP’s metering and billing systems are accurately measuring and billing customer usage and have been doing so since the go-live of SmartCare at the end of October 2017. The Liberty Report described Liberty’s extensive examination of the integrity of the meter and billing data as it was measured and transmitted from end-to-end of CMP’s metering and billing systems, concluding that data was being transferred accurately and in a timely manner, and that SmartCare was providing customer bills that reflected accurate usage levels and charges. The additional comparative weather and usage analysis provided by both Liberty and Staff provides further confirmation that the high usage and usage increases observed for CMP customers were driven by cold weather, and not by any systemic errors unique to CMP’s systems. By no means does this finding imply that there have been no CMP billing issues, nor that CMP management has reasonably managed the SmartCare implementation and its customer service during the post-go-live period. To the contrary, numerous billing defects and errors affected CMP’s customers. These issues are discussed in the following sections. B. How Were Customers Affected by CMP’s Anomalous Meters? As discussed above, for one of CMP’s susceptible meters (the GE I-210+c) to go into anomalous mode, the following set of conditions must occur: • An outage, followed by a meter resetting (initializing) when power is restored; • At about 3.36 seconds after restoration of power to the meter and while initialization is underway, voltage at the meter must sag to less than 90 volts on Phase A (one of the two 120-volt phases at the meter); and Examiners’ Report • 86 Docket No. 2019-00015 The voltage sag must occur for at least 100 milliseconds and then must be maintained throughout a 66-millisecond window of vulnerability during which the metering processor resets but the other control processor has not reset. Given the very specific circumstances and unusual events that must occur in the extremely tight time frames in which those events must occur for anomalous mode to happen, the Commission concludes that anomalous meter occurrences are, as Liberty and CMP have testified to, rare. In addition, given how loads are balanced between the two phases of service to a customer’s premises (assuming correct wiring and customer usage in accordance with expected usage), even when a meter is in anomalous mode and registering abnormally (twice the usage on Phase A and zero usage on Phase C), register anomalies generally will have only a small effect on the customer’s actual registered usage since the load that is being counted incorrectly by a factor of two is fully offset by the load that is not being counted at all. Liberty has provided testimony that although both the occurrence of a meter anomaly and the possibility that it would materially affect billed usage were rare, it was, nonetheless, likely that a small group of customers were affected and that the effect could have, in individual cases, been both real and significant. Mr. Antonuk estimated that several thousand customers over the years were probably affected by the problem. While CMP initially contested Mr. Antonuk’s opinion on this point, CMP’s analysis performed as part of its metering and billing rebuttal testimony, confirmed Liberty’s conclusions here. Examiners’ Report 87 Docket No. 2019-00015 In its rebuttal, CMP identified meters that might have been in anomalous mode since 2013 by looking first at meters that were in fast-clock mode, then comparing usage on those meters while in fast-clock mode to historical usage on the meters. Based on this analysis, CMP determined that there were 1,649 meters that likely over-registered usage since 2013. In total, over-registered usage was calculated to be by 319,457 kWh, with an approximate value of over-registration of $50,155 based on the average rate for CMP’s residential customers in 2019 of $0.157 per kWh. As Mr. Antonuk testified, given the fact that some of the anomalous meters may have selfhealed through a further outage and reset before CMP detected them, and that the customers meter was operating abnormally, exact actual usage from the anomalous meter period cannot be precisely determined retroactively. Mr. Antonuk went on to testify that while CMP’s methodology was inherently inexact, given the circumstances, it would likely not be possible to be more precise than CMP’s analysis was. Tr. at 180–87 (Nov. 6, 2019 Hr’g). CMP has offered to refund the over-registered amounts identified in its analysis to the affected customers. CMP has also offered to adjust rates for the general body of ratepayers for the amounts that were under-billed as a result of the registration anomalies, and thus recovered from other ratepayers as part of CMP’s revenuedecoupling mechanism (RDM) approved by the Commission. The Commission finds that while CMP’s methodology is inexact, it is reasonable, and is likely the best quantification that can be made of the harm caused by the anomalous meters at this time. The Commission thus orders that CMP refund the amount calculated to have been over-collected from customers be refunded to those customers under 35-A M.R.S. Examiners’ Report 88 Docket No. 2019-00015 § 1308. The Commission also concludes that CMP’s proposed RDM adjustment is appropriate and should be made at CMP’s next RDM reconciliation, which is expected to occur on July 1, 2020. The Company’s witnesses testified at the hearing that CMP has completed its firmware upgrade on 94% of the GE I-210+c with the intent of fully completing the upgrade by the end of the year 2019. This means that, at this point, the anomalous meter issue is now nearly behind the Company. An issue remains, however, as to whether the Company knew, or should have known, about the issue earlier than it now claims it did, and, if so, whether it should have acted upon that knowledge earlier. In its audit, Liberty found that GE first reported the anomalous meter issue by way of a firmware release in 2012 and that it was highly unlikely that GE would not have contacted CMP of this fact. Liberty noted, however, that it did not find substantial evidence of management’s awareness of the GE meter issues or efforts to address them, until 2014. Liberty Report at 22. CMP responded to Liberty’s statement by saying that it would have been more accurate to say that Liberty “found zero documentation of management’s awareness” of the GE meter issues back in 2012. At the hearing, Mr. Morrissette, on behalf of the Company, testified that the Company did not receive the 2012 firmware upgrade release until 2018, when it requested that Aclara provide all release notes about the GE meters. Tr. at 76 (Nov. 5, 2019 Hr’g). Based on the evidence of record in this proceeding, the Commission cannot find that CMP received the 2012 report or had knowledge of the issue in 2012. The evidence does show, however, that in 2014 CMP should have done more investigation of the issue of anomalous meter clocks and their possible effect on registered usage. Examiners’ Report 89 Docket No. 2019-00015 In 2014, CMP’s field personnel reported that while investigating fast clocks on six GE I-210+c meters, they observed that the disk emulator, which reflects usage being metered, was moving way too fast. When the meter was power-cycled (turned off and then repowered), the clock was fixed and the “fast disk emulator” slowed down and displayed the correct kWh load when checked with a stop watch. Staff Hr’g Exh. 3. Afterward, GE sent an Engineering Analysis which described the anomalous meter issue, the conditions which would cause a meter to go anomalous, and the fix for the issue. In a report dated October 29, 2014, CMP’s engineers noted: GE has a new meter firmware version 2.5.9.0 that addresses this issue. CMP’s GE I-210+c meters currently have meter firmware version 2.5.2.0. The 2.5.9.0 meter firmware patch also includes the “fix” for the “busy” mode field testing issue (which results in wildly inaccurate meter test results of around 200% registration) and it also includes the new “battery less” operation feature which auto fills interval data after a power outage event. The firmware “fix” is a permanent solution which prevents the “fast clock” issue from occurring in the GE I-210+c meters. TLCG-001-022, Att. 1 at 22. As noted in the Liberty Report, following the receipt of the 2014 GE report and the issuance of the Company’s October 2014 report, the Company did not pursue the firmware upgrade, hoping instead that soon enough Trilliant could provide the over-theair capability to perform the upgrade. CMP argued that, based on GE’s 2014 report, it had no reason to believe that the meter anomaly caused anything but a fast-clock issue that did not affect registration. But CMP’s own analysis of this issue, as reflected in the August 2014 emails and the October 2014 report, contradicts this position. At a minimum, CMP should have pushed GE for an answer to the question of why the fastclock situation was also causing the “fast disk emulator” registration issue. The Company did not do so, and we find that such a failure was not reasonable or prudent. Examiners’ Report 90 Docket No. 2019-00015 Even if, as CMP claims, in 2014 CMP only suspected that the anomalous GE meters caused fast-clock anomalies—since fast-clock anomalies, if not corrected before a bill issuance, might result in inaccurate average daily consumption by using the wrong number of days in performing the calculation—CMP had clear reason to address the issue in 2014. In 2014, it was clear to the Company that it had two options available to it to address the anomalous meter issue: an over-the-air meter-firmware upgrade using the Trilliant System, or doing special resets of the meters. CMP Met. & Bill. Dir. Exh. 3. Liberty suggested that the Company had a third option: cycling a set of meters out of the field to the shop for upgrading and then back to the field to replace other meters which would then be upgraded, and so on. Liberty Report at 28. It is difficult to say now that the delay in Trilliant’s over-the-air upgrade capability could have been predicted or that CMP’s decision to wait for this capability was unreasonable. But while waiting for the Trilliant upgrade, CMP should have managed and mitigated the anomalous-meter situation by aggressively doing the resets as suggested by GE and by its engineers in the Company’s October 2014 report. Yet it did not do so. In 2016 and 2017, CMP did not identify or track any meters in anomalous mode. Given the facts known or knowable to the Company at the time, CMP’s inaction here was not reasonable or prudent. During the hearing, CMP’s witnesses have explained that the over-the-air software upgrades are well under way and nearing completion. The update of the socalled network interface cards, or NICS, was over 99.2% completed by early November. The upgrade of the NICS is a threshold step to upgrading the firmware of the meters Examiners’ Report 91 Docket No. 2019-00015 themselves. By early November, CMP had completed firmware upgrades for 94% of susceptible meters. Meters that, for whatever reason, are unable to be upgraded by this method will be replaced. Tr. at 123–24 (Nov. 5, 2019 Hr’g). While CMP has made a good-faith attempt to address the effect of the registration errors caused by the anomalous meters on individual customers, CMP’s failure to address the anomalous meter situation until its customers were already questioning the accuracy of CMP’s metering and billing system in early 2018 likely contributed to customers’ distrust and lack of confidence in the Company’s ability to accurately meter and bill them for their usage. We address both the need for and how such confidence can be restored in Section VIII below, as well as in our companion order in the rate case (Docket No. 2018-00194). C. Were CMP’s Customers Affected by Other Billing Issues? Numerous defects and errors in CMP’s billing system affected tens of thousands of CMP’s customers in the months after the transition to SmartCare, and some of these defects and errors continue to be unresolved today. Since SmartCare was implemented, CMP has reported dozens of types of bill errors that have affected customers. EXM004-007, Att. 1 at 2–6. Some of these defects and errors affected the amounts customers were invoiced, while others did not. Some have been corrected in SmartCare, while others remain unresolved. Some were corrected in SmartCare, but later re-appeared. Some were SmartCare-related, while others resulted from human error. This section describes the Commission’s findings on several of the defects, how many people have been affected by them, their financial effect, and their current status. To be efficient, this section focuses primarily on the most egregious billing errors or defects and those the parties addressed in their briefs. (By focusing on the worst errors, Examiners’ Report 92 Docket No. 2019-00015 the Commission by no means excuses or ignores all of the other errors that have been discovered. For the sake of completeness, all billing errors or defects are described in a tabular summary in Appendix B.) 1. Liberty’s Bill Reconciliation and Proration/Truncation First, the Commission addresses one issue with SmartCare that was not a defect but is a characteristic of the software. As part of this investigation, Liberty worked through multiple rounds of bill analysis to reconcile accounts that it was initially unable to reconcile. Although in its audit Liberty determined that SmartCare correctly billed in 99.998% of cases for delivery and in 99.7% of cases for supply, Liberty identified 8,300 accounts it could not reconcile through its analysis; nearly all of these were supply exceptions, and 64 were delivery exceptions. Liberty Report at 9. Through its bill-reconciliation process and with more information, Liberty was able to reconcile all of the 64 delivery exceptions. Sept. 3, 2019 Liberty Exception Analysis at 1–2. Through a sample of bills (1,096) affected by supplier exceptions that Liberty had not been able to reconcile, Liberty was able to confirm that 307 had been billed correctly. Through additional process, Liberty was able to confirm that CMP’s explanations that the remaining bills had special proration treatment, depending on who the electric supplier, were valid. Tr. at 178 (Nov. 6, 2019 Hr’g). The one area that Liberty had lingering concern about was SmartCare’s tendency to use truncation instead of rounding for billing purposes when bills are being prorated. Id. at 179. Examiners’ Report 93 Docket No. 2019-00015 According to CMP, when prorating, SmartCare is designed to truncate usage instead of round it. 39 Although the effect of truncation as compared to rounding is very small for nearly all customers, the small number of CMP’s customers whose bills are calculated using meter multipliers are more likely to be adversely affected by truncation. A meter multiplier is used when the electricity a customer consumes is too large to be read by the meter. The meter reflects a fraction of the actual usage, and the meter multiplier is the inverse of that fraction. The meter multiplier is multiplied by the metered usage to calculate the actual usage that should be billed. About 8,000 customers of CMP are billed using meter multipliers, and some of these customers have multipliers in the thousands, as high as 24,000. Given the potential effect on these customers of truncating their usage, the Commission cannot conclude that truncation of usage is a best practice. Whether SmartCare should be modified to round instead of truncate is a question of cost and benefit. The Commission orders CMP to submit a filing, within 45 days of this order, explaining what would be required to change SmartCare to allow for rounding instead of truncation. The Company should report on the cost of the change, the benefit of the change for customers with meter multipliers, how long it would take to implement, and the resources necessary to complete it. Once the Commission has a chance to consider this information, we will decide whether it is appropriate to order that SmartCare be modified to round instead of truncate. 39 With rounding, decimal amounts of 4 and smaller are rounded down, and of 5 and greater are rounded up. With truncation, the number used for billing is simply the whole integer; the decimal numbers are eliminated. For example, usage calculated as 5.5 would be billed as 5 (truncated) instead of as 6 (rounded up). Usage calculated as 6.1 would be recorded as 6 through either truncating or rounding down. Examiners’ Report 2. 94 Docket No. 2019-00015 Defects and Errors Moving on to the defects in SmartCare (and related human error), the Commission summarizes a select list of defects. This section addresses defects in the following categories: (a) errors with a financial effect on customers; (b) presentment errors with no financial effect on customers; (c) errors relating to estimated usage; (d) errors that resulted in customers only being underbilled. a. Errors With a Financial Effect on Customers i. Defect #5667/#5844 (Case No. 2): Credits Erroneously Applied to Bills of Participants in the Electricity Lifeline Program This SmartCare defect affected low-income customers, and caused two major sets of errors: (1) miscalculation of benefits for CMP’s Electricity Lifeline Program, known as ELP (some bill credits too high, others too low); and (2) incorrect decisions on ELP applicants (erroneous approvals or denials of eligibility). EXM-004-007, Att. 1 at 8. The defect affected 1,850 participants in ELP (for context, 8,000 customers participated in ELP as of September 2017). Id. During the seven weeks the error was in effect, 1,177 ELP enrollments were processed correctly, while 1,850 enrollments were processed incorrectly. Id. This error was discovered in December 2017, and occurred from November 1, 2017 to December 20, 2017, when CMP shut off the automated processing of ELP accounts in SmartCare to avoid more problems. Id. The defect was corrected in SmartCare by January 19, 2018. Id. The 1,850 errors can be itemized as follows: (a) 717 customers were correctly approved for ELP but received excessive bill credits (totaling $181,414); (b) 335 customers were correctly approved for ELP but received insufficient bill credits (totaling Examiners’ Report 95 Docket No. 2019-00015 $68,977); (c) 706 customers were incorrectly approved for ELP and received excessive bill credits (totaling $177,185); and (d) 92 customers were incorrectly denied for ELP and thus did not receive the bill credits they should have (totaling $40,234). Id. By January 25, 2018, all affected “accounts were manually corrected to ensure proper credits.” Id. CMP also (1) mailed affected customers to explain the defect, (2) applied a “standard” customer-service guarantee of $10 to each affected account, (3) applied an “enhanced” customer-service guarantee of $40 to $90 to customers who incorrectly received (and which CMP then reversed) excess ELP benefits above $250, and (4) suppressed credit/collections and late-payment activity on the affected accounts through April 2018. Id. For those customers who received ELP benefits when they should not have, after discovering the error CMP removed the funds from the customer’s account. The corrections and customer-service guarantees for each category of customers are summarized in Figure 3. Examiners’ Report 96 Docket No. 2019-00015 Figure 3: ELP Errors and Adjustments and Application of Customer Service Guarantees by Category of Customer 40 Number of affected customers Total incorrect benefits Total adjustment to benefits Total customer service guarantees applied to affected accounts Correctly approved for ELP but received excessive bill credits Correctly approved for ELP but received insufficient bill credits Incorrectly approved for ELP and received excessive bill credits Incorrectly denied for ELP and thus did not receive the bill credits they should have Total 717 335 706 92 1,850 +$181,414 ($68,977) +$177,185 ($40,234) $249,388 ($181,414) +$68,977 ($177,185) +$40,234 ($249,388) +$23,430 +$3,350 +$23,690 +$920 $51,390 At the hearing, CMP claimed that “the Company’s decision in early 2018 . . . to remove the incorrect ELP credits and apply correct credits for these customers was made in the interest of fairness to all of the Company’s customers who might apply for ELP benefits.” Tr. at 157–58 (Nov. 5, 2019 Hr’g). In its brief, CMP stated that it would be open to “restoring to the customers whose credits were reduced or eliminated as a result of this correction the difference in their credits . . . .” CMP Br. at 61. a. Discussion and Decision The Commission is puzzled as to why CMP decided to revert the credits CMP erroneously applied to some low-income accounts. CMP’s explanation for its actions was that it was required to follow the ELP’s eligibility rules, and could not simply ignore them, since other ELP customers would end up receiving less money due to the error. 40 ODR-004-007. Examiners’ Report 97 Docket No. 2019-00015 Here, CMP overlooks the fact that it determines the eligibility rules for its own Electricity Lifeline Program. Those rules are not imposed by Commission statute or rule, but by CMP’s own terms and conditions. Cent. Me. Power Co. Terms & Conditions § 33. Instead of taking erroneously applied credits away from CMP’s low-income customers, CMP could have sought a limited waiver of its terms and conditions to allow it to maintain status quo, and avoid creating consternation among a vulnerable population. Even more confounding is that CMP treated an error on low-income customers’ accounts differently from all other errors it has discovered. Some other bill errors have resulted in customers being under-billed (for example, Defect #6621, discussed below). But in those cases, CMP did not back-bill the customers for the amount they were under-billed; instead, CMP simply wrote off the error. For some reason, CMP treated low-income customers differently. CMP’s witness admitted that reverting the credit would not have had any long-term implications. Tr. at 162 (Nov. 5, 2019 Hr’g). Why CMP would correct an error affecting low-income customers in a way that results in them unexpectedly owing more money, and not correct errors affecting customers at large the same way, is difficult to understand. CMP seems to have been oblivious to the harm its self-help remedy may have caused to low-income customers, many of whom are significantly affected by small changes in funds on accounts. As a remedy for the harm to customers who were paid a benefit and then lost that benefit with this defect, the Commission orders that CMP return that benefit to those customers by way of a credit to their utility bill. Any waiver of CMP’s terms and conditions required to allow this repayment is hereby granted. Examiners’ Report 98 ii. Docket No. 2019-00015 Defect #6103 (Case No. 13): Deposit Interest Rate Annual Change This defect affected customers who had deposits with CMP. CMP is supposed to pay interest on those deposits. The annual update to interest rates that occurred at the beginning of 2018 was changed in SmartCare 10 days late. The defect was identified on January 12, 2018, and appears to have been corrected the same day. But bills issued from January 2, 2018 to January 11, 2018, received the incorrect interest amount. It affected 11,845 customer accounts (out of a total of 25,600 with deposits) and led to an overall amount of $914 in interest being underpaid to customers. To remedy this error, CMP issued a bill credit for the correct interest amount (doubled) to the affected customers, adding up to $1,828.54 being returned to customers. For some reason, though, this was not completed until December 13, 2018. Other corrective adjustments were made to customers’ accounts in December 2018 and January 2019. EXM-004-007, Att. 1 at 17. iii. Defect #5500 (Case No. 29): CEP Supplier Receivables – Conversion and Partial Payments This defect affected the so-called payment waterfall (the way partial payments are prioritized, depending on to whom money is owed) for customers with past-due amounts owed to competitive electricity providers (CEPs). SmartCare erroneously applied partial payments to past-due amounts on the CEP portion of the bill before applying them to the past-due amounts on the delivery (CMP) portion of the bill. This defect was discovered on December 4, 2017, and corrected on December 7, 2017. EXM-004-007, Att. 1 at 31. It affected about 5,000 customers (out of about 107,000 customers taking supply from CEPs). The total dollar value of the effect on Examiners’ Report 99 Docket No. 2019-00015 customers was $428,611; the Company manually corrected the payments for the affected accounts by December 21, 2018, fully remedying the financial effect. In addition to having a short-term financial effect on customers, according to CMP some customers may have erroneously received a disconnect notice from CMP due to this defect. Id. iv. Defect #5174 (Case No. 1): SimplePay Conversion into SmartCare This defect caused a SimplePay customer with a debit or credit on their account and an unpaid outstanding bill from the legacy CSS billing system to be billed an amount other than their established SimplePay amount. SimplePay converted the total account balance from the legacy CSS billing system into an amount due and a SimplePay Difference, where the amount due was not equal to the established SimplePay monthly amount. The total account balance was, nevertheless, accurate. EXM-004-007, Att. 1 at 7. This defect was discovered shortly after the cutover to SmartCare—on November 6, 2017—and corrected by January 3, 2018. Out of about 23,000 total SimplePay accounts, this defect affected 9,800 accounts. The net financial effect was never fully quantified; some customers paid less for a particular month than they should have while others paid more for a particular month than they should have. CMP resolved the defect by manually correcting several accounts, while others were essentially self-correcting by being trued up the following month. EXM-004-007, Att. 1 at 7. Examiners’ Report 100 v. Docket No. 2019-00015 Defect #6655/#6621 (Case No. 64): Move In/Out Field Read Should Prorate to Requested Date This defect arose when meter-read dates occurred on days other than the date that was scheduled. In some cases, the meter-read date printed on the bill may not have matched the date the meter-read was obtained. According to CMP, this is not a SmartCare defect because the process worked the same way in CMP’s legacy CSS billing system. EXM-004-007, Att. 1 at 59. Because this defect was related to prorating of customers moving into and out of dwellings, this defect billed the wrong person for the correct usage; the usage shown on the bill was accurate for the period, but the wrong person was billed for it. Tr. at 26 (Oct. 31, 2019 Tech. Conf.). This defect also caused errors in the graphical portion of the bill showing the average number of days billed. The defect was discovered in March 2019 and a correction was targeted for December 2019. EXM-004-007, Att. 1 at 59. Until then, CMP is using a manual workaround to correctly prorate bills affected by move-in/move-out dates. Tr. at 26 (Oct. 31, 2019 Tech. Conf.). b. Presentment Errors with No Financial Effect on Customers i. Defect #5876 (Case No. 24): Usage Information This defect caused the monthly usage graph on customers’ bills to present erroneously. In many cases the graph displayed no bars for months when electricity was, in fact, consumed. This defect was discovered on January 30, 2018, and corrected by March 6, 2018. EXM-004-007, Att. 1 at 27. The defect had no financial effect, but it inevitably Examiners’ Report 101 Docket No. 2019-00015 created confusion among customers. CMP’s summary of this defect did not show how many customers were affected by it, but that number was likely in the many thousands. ii. Defect #5326 (Case No. 65): Incorrect Meter Multiplier 41 or Meter Location Adjustment for Accounts with Multiple Dwelling Units This defect affects only accounts with multiple dwelling units. EXM-004-007, Att. 1 at 60. For those accounts, this defect causes the meter multiplier or meter location adjustment to be incorrectly displayed on the bill. Id. CMP admits that this error “may have caused confusion as the multiple dwelling units are combined in the same box as the meter multiplier or [meter location adjustment] without a label for the multiple dwelling units.” Id. This defect is a presentment error. This defect was identified November 15, 2017. Id. It may have affected 135 accounts. Id. Well over two years later, it remains an open defect, though it does not affect the amounts customers owe (and thus has no direct financial effect), id., and no customers have contacted CMP about this defect. Tr. at 29 (Oct. 31, 2019 Tech. Conf.). iii. Defect #6224 (Case No. 41): On Accounts with Multiple Meters, Total Usage Data Erroneously Presents as Individual Meter Data on Every Meter This SmartCare defect affected accounts with multiple meters being billed on a single invoice. EXM-004-007, Att. 1 at 41. With this defect, the total data for all meters was displayed for each meter, rather than individual meter data. EXM-004-007, Att. 1 at 41. The defect occurred only on accounts that were billed on June 1, 2018. 41 As explained above in Section VII.C.1, a meter multiplier is used when the electricity a customer consumes is too large to be read by the meter. The meter reflects a fraction of the actual usage, and the meter multiplier is the inverse of that fraction. The meter multiplier is multiplied by the metered usage to calculate the actual usage that should be billed. Examiners’ Report 102 Docket No. 2019-00015 This defect affected 619 customers who were billed June 1, 2018. EXM-004-007, Att. 1 at 41. It was corrected in SmartCare three days later, on June 4, 2018. EXM-004007, Att. 1 at 41. This defect is a presentment error. The correct number of kWh were charged on the bills, so the defect had no direct financial effect on customers. EXM-004-007, Att. 1 at 41. By June 8, 2018, CMP had canceled and rebilled the affected accounts to provide the correct meter display, and sent these to customers with a letter explaining the correction. EXM-004-007, Att. 1 at 41. By October 28, 2019, CMP had applied the standard $10 customer-service guarantees to the affected accounts (totaling $6,190). EXM-004-007, Att. 1 at 41. iv. Defect #5885 (Case No. 32): Bill Erroneously Reflected Interim Meter Read or Zero as Prior Billed Read, Instead of Actual Meter Read Defect #5885 caused a bill-presentment error when interim meter reads were taken at times other than the periodic monthly meter reading. CMP Br. at 52. Due to a coding error, instead of showing the actual meter read, the bill erroneously showed that the prior billed read was the interim meter read or zero. CMP Br. at 52 (citing EXM-004007 Att. 1 at 34; EXM-004-004; ODR-007-001). Despite the error, the amount due on the bill was correct, so this error had no direct financial effect. EXM-004-007 Att. 1 at 34; ODR-007-001. The defect affected customers for about six months; it was discovered in January 2018 and is now closed, having been corrected in SmartCare by June 2018. EXM-004007, Att. 1 at 34; ODR-007-001. Examiners’ Report 103 v. Docket No. 2019-00015 Defect #6579 (Case No. 70): Bill Erroneously Showed Wrong Meter-Read Dates when Interim Meter Reads Were Taken This SmartCare defect caused the meter-read “from” and “to” dates on customers’ invoices to be prorated in error when meter reads were taken on dates other than the periodic monthly meter-read date. CMP Br. at 53; EXM-004-007, Att. 1 at 64. The actual meter read had, nevertheless, been properly captured and used to calculate the bill amount, EXM-004-007, Att. 1 at 64, though this would not have been obvious on the face of the bill. Despite the error, the amount due on the bill was correct, so this error had no direct financial effect. Id. This defect was identified on March 11, 2019 and was corrected in SmartCare on October 15, 2019. Id. vi. Defect #6621 (Case No. 58): Scheduled Meter-Read Date Presented on Bill / Not Actual Meter-Read Date for Manual Meter-Reads This defect caused the meter-read date presented on the bill to not match the actual meter-read date. This affected the graphical portion of the bill, not the billed amounts. EXM-004-007, Att. 1 at 53. This defect was discovered on March 29, 2019 and was not yet corrected; CMP stated that it was expected to be corrected by December 2019. Tr. at 22 (Oct. 31, 2019 Tech. Conf.) The number of customers affected was not known, but it had no financial effect, and the billed usage was accurate. EXM-004-007, Att. 1 at 53. vii. Defect #6723 (Case No. 62): Security Deposit Interest Doesn’t Reflect Allocation between Distribution and Supply on Bill This defect caused security-deposit interest to be incorrectly displayed as being applied to only supply or delivery, instead of to both evenly. Examiners’ Report 104 Docket No. 2019-00015 This defect was discovered in May 2019. CMP completed a manual review of accounts by August 2019, and awaited a correction in SmartCare. This defect had no financial effect; it was a presentment error. CMP adopted a manual workaround for this error pending a correction in SmartCare. EXM-004-007, Att. 1 at 57. viii. Defect #6691 (Case No. 66): Presentment Issue for Net Energy Bills For customers who moved from gross metering to net metering, this defect caused the historical information on the bill to display erroneous reporting of generation details after customers moved to net metering (e.g., skipping months, showing zeros). This defect was discovered in June 2019; CMP awaits a correction in SmartCare. This defect had no financial effect; it was a presentment error. EXM-004-007, Att. 1 at 61. c. Error Relating to Estimated Usage i. Defect #4711/#6183/#6767 (Case No. 42): Incorrect Estimates of Zero Usage Defect #4711 caused SmartCare to estimate zero kWh of usage for the first month after a customer’s meter was exchanged and no meter read was obtained at the time of the exchange. CMP Br. at 19–20; EXM-004-007, Att. 1 at 42. Subsequent estimated bills were estimated based on the usage in the prior month or year, instead of on ongoing periodic consumption. EXM-004-007, Att. 1 at 42. In these cases, the customer erroneously received one or more bills showing zero usage. The total actual usage was billed to the customer once an actual meter read was obtained, CMP Br. at 20—meaning that a customer would receive a bill for the next month with far higher usage than normal because the usage in the first correct bill was making up for (i.e., correcting for or reconciling for) the previous bill’s (or bills’) erroneously low usage. Examiners’ Report 105 Docket No. 2019-00015 Defects #6183 and #6767 caused the same bill error (and thus the same effect on the customer) but did not necessarily result from a meter exchange; instead, they involved the meter read not being correctly uploaded “due to a register group change [which] resulted in estimates of zero usage.” CMP Br. at 54. The OPA’s consultant, BerryDunn, pointed out that this error harmed customers not only by presenting erroneous and confusing information in their bills, but by preventing the customer from benefiting from both the sales-tax exemption for the first 750 kWh/mo of a residential customer’s bill and the first 50 kWh/mo included in the customer charge. CMP Br. at 54; Keim Dir. 42 at 16. These defects were corrected in SmartCare by September 4, 2019. CMP Br. at 54 (citing EXM-004-007 Att. 1 at 42). It remains an open defect, though (EXM-004-007 Att. 1 at 4), because CMP has not yet determined how many customers were affected by the error. Once it does, it intends to make the appropriate corrections and apply a customer-service guarantee to their account. CMP Br. at 51–52, 55. CMP continues to work to identify the customers affected by this defect. CMP Br. at 20 (citing Tr. at 86 (Nov. 5, 2019 Hr’g)). d. Error That Caused Customers to Be Underbilled i. Defect #5302 (Case No. 60): Time-of-Use Customers Not Changed to or from Their TOU Rate Defect #5302 is a human error, not a SmartCare error, where the billing rates for certain customers were not compatible with the type of meter required to bill the assigned rate. CMP Br. at 58 (citing EXM-004-007, Att. 1 at 55). Bills issued after the 42 Keim Dir. refers to the BerryDunn Analysis of Customer Accounts, led by witness Julie Keim of BerryDunn on behalf of the OPA, filed on September 6, 2019. Examiners’ Report 106 Docket No. 2019-00015 meter change erroneously included only the customer charge, and no usage charges. EXM-004-007, Att. 1 at 55. Although this was a human error, it remains an open defect because CMP’s intended remedies are not yet in place. EXM-004-007, Att. 1 at 5. To remedy the problem, CMP plans to ensure customers are on the most advantageous rate or complete meter changes if needed to support the customers’ chosen billing rate. Id. CMP also plans an enhancement to SmartCare to prevent this kind of error in the future. CMP Br. at 59 (citing EXM-004-007, Att. 1 at 55). Both of these remedies are expected to be completed by the end of December 2019. EXM-004-007, Att. 1 at 55. This human error affected customers’ accounts at least from November 2017 through April 2019—a 17-month period. EXM-004-007, Att. 1 at 55. In April 2019, CMP determined that this error affected 860 customer accounts. CMP Br. at 58 (citing EXM004-007, Att. 1 at 55). Its financial effect was to underbill customers, but the amount of underbilling was not known when the record closed. 3. Conclusion These defects cause the Commission great concern. Even the defects with no financial effect—where the only problem was information displaying incorrectly on the bill, a so-called presentment error—inevitably led to many customers being confused about their bill and questioning whether it was calculated accurately. The length of time it has taken for CMP to correct some of these defects in SmartCare only makes matters worse. In many or most cases, though, CMP has corrected the financial effect of the defect where there was one, either by correcting the specific error, in some cases with customer-service guarantee payments in addition, or by paying twice the correction Examiners’ Report 107 Docket No. 2019-00015 back to the customer. In cases where CMP has not yet quantified the financial effect but has some ability to do so (as described above), we hereby order it to do so. As described above, the remedy for the ELP-related defect—the first defect we discussed above—will be a payment back to those customers who had an erroneous credit removed from their bill. This repayment will help to address the understandable frustrations of those low-income customers. As discussed in Section VIII.B, the Commission expects CMP to correct all outstanding defects in a timely manner and will monitor CMP’s ongoing maintenance of the SmartCare system. D. Was CMP’s Implementation of SmartCare Reasonable and Prudent? A fundamental question before us is whether CMP’s implementation of SmartCare was prudently managed. As detailed below, we find that it was not. We address the remedies for this in Section VIII. From a review of the testimonies, it is evident that the OPA’s and Liberty’s assessments of the quality of CMP’s SmartCare implementation are vastly different from CMP’s assessment. Some of the disagreements are about basic facts. In those instances, in general our determination will turn on the credibility of the testifying witnesses. Among the various expert witnesses who have presented testimony in this case, we find Liberty’s auditor, Ms. Minton—based on her experience and independence, as proven by her frankness about both the positive and negative aspects of CMP’s SmartCare implementation—to be the most credible of the softwareimplementation witnesses before us. Examiners’ Report 1. 108 Docket No. 2019-00015 Compression of Testing Schedule The biggest area of controversy, and probably the issue that has the most significance in terms of whether or not CMP was ready to go live with the SmartCare system when it did, was whether the testing protocols in the weeks leading up to the rollout were reasonable and in line with standard utility practice. All witnesses appear to agree that User Acceptance Testing (the test phase in which business users validate that the requirements set out for the software project have been met) should not be done prematurely, but instead should occur as close to go-live as possible and on the finished product. Getting this timing right ensures that software updates made after User Acceptance Testing do not occur that, unwittingly, create more problems that could come to light only after go-live. CMP claimed that User Acceptance Testing was in fact performed as part of Integration Testing Cycle 5—the last cycle of Integration Testing. If User Acceptance Testing was indeed completed as part of the last cycle of Integration Testing, then that should be evident in CMP’s Microsoft Project Plan. But it is not; CMP’s project documentation is silent on whether User Acceptance Testing occurred then. See CMP Met. & Bill. Dir. at Exh. 6; Liberty SmartCare Resp. at 6. The failure to document this key phase of testing in a project-tracking tool corroborates Ms. Arnold’s testimony that CMP’s project tracking was inadequate. Ms. McNally of CMP recognized during the technical conference that no single report or document was presented to management to show that each requirement had been fully tested, that all test cases for each requirement were executed, and which requirements had passed and which had outstanding defects. Tr. at 156–57 (June 13, 2019 Tech. Conf.). Examiners’ Report 109 Docket No. 2019-00015 Even if User Acceptance Testing was performed as part of the last cycle of Integration Testing, conducting this testing in parallel with other tests (as CMP did) was contrary to the waterfall-testing methodology (defined above at pages 33 and 50) contained in CMP’s SmartCare Implementation Charter. As both Ms. Minton and Ms. Arnold discussed, overlapping testing in different environments leads to identification and correction of bugs that can be recognized in the other test environments. They both testified that a good software-test approach runs the Unit Testing, Integration Testing, and User Acceptance Testing one after another and never in parallel. Liberty SmartCare Resp. at 5; Arnold Dir. at 14. The Company’s decision to abandon the waterfall approach, without a clear rationale as to why, unreasonably increased the risk of defects at go-live. 2. Relaxation of Standards for Testing and for Go-Live The Company also appears to have relaxed, without a clear reason why, the standard of what the acceptable level of defects at the time of go-live would be. In its Testing Strategy, CMP’s standard was that prior to go-live there would be no open highor medium-priority defects without a workaround. CMP lowered the criteria to no criticalor high-level defects at go-live. The effect of this decision, which had the potential to increase the number of defects present in the system at go-live, was not analyzed or documented. 3. Insufficient Resources to Handle Post-Go-Live Errors and Exceptions According to Liberty’s auditors, the overall number of defects identified after golive was at the high end of the normal range for a utility customer-information-system implementation: Examiners’ Report 110 Docket No. 2019-00015 [T]here are always problems with the introduction of new systems. There were problems here. The problems were substantial. They were high by comparison to others we’ve seen, but they weren’t out of the box anomalous. . . . They would be what we would describe kind of at the high end, the very high end of normal range. Tr. at 25 (Feb. 19, 2019 Tech. Conf.). But it was not necessarily the number of defects that was problematic. Instead, the problem was how the Company responded to them. Over the two-year period since SmartCare was implemented, and especially at the time of the implementation, the Company’s reaction time responding to identified defects has been excessive. The relaxation of standards and CMP’s failure to properly track testing and defects may have resulted in a number of defects that caught CMP off guard, leaving CMP without the resources needed to address defects found after go-live, and to communicate them to the public in an effective way. As part of its response to the Liberty Report, CMP presented testimony on the adequacy of its staffing and claimed that the staffing resources it devoted to the implementation were consistent with industry standards for a project of this type and complexity. CMP. Implem. Dir. at 22–23. CMP nevertheless appears to have recognized the insufficient resources as an issue as far back as March 2017. In March 2017, a CMP SmartCare Project Report noted as an area of concern: “stretching resources very thin for meeting project tasks and key deliverables.” And in August 2017: “spreading project resources too thin” was again recognized a problem. TLCG-002-021, Att. 9 at 4, Att. 34 at 5. CMP’s Lessons Learned document, which was prepared after implementation, noted that an area for future improvement was to “ensure proper staffing of project to meet schedule and scope.” Following go-live, in February 2018 CMP completed an evaluation of the system and found that, even though SmartCare was producing a stable number of exception cases, the Company was challenged in maintaining the Examiners’ Report 111 Docket No. 2019-00015 pace required to reduce the backlog of open exceptions and, at the current rate, the backlog will not be reduced but might continue to grow. The analysis concluded that current volume of backlogged billing exceptions was beyond current staffing ability and affecting customer satisfaction, specifically noting the following issues: • Delayed billing to customers; • High call volumes and long wait times for the Call Center; • Increased complaints to regulators; • Increased AutoPay payment issues; and • Unresolved exceptions resulting in additional exceptions. Docket No. 2018-00052, EXM-001-011, Att. 1 at 4, 12. A series of emails between Company personnel about issues with net-energy billing and supplier billing also showed internal frustration with the level of resources available to address billing errors and defects. Docket No. 2018-00052, EXM-001-011, Att.1 at 141, 177, 226, 474, 482. 4. Conclusion In its brief, CMP argues that its handling of the implementation of SmartCare was prudent and in accordance with best practices. The implication of this is that if CMP could do the implementation over again, it would have done nothing differently. Considering the billing delays and errors that affected its customers after go-live, CMP’s position is implausible. The Commission concludes that the combination of CMP’s relaxation of testing standards, deviation from standard testing methodologies and implementation-tracking practices, and insufficient resources leading up to go-live, especially at implementation, was not reasonable and must be considered imprudent. This imprudence contributed to Examiners’ Report 112 Docket No. 2019-00015 delays in addressing defects and caused customer confusion and customer distrust of the Company and its billing system which have been clearly manifested in complaints customers have brought to the Commission’s CASD and in testimony customers presented at the three public-witness hearings. The remedies for the Company’s imprudence are addressed below. VIII. REMEDIES A. Additional Oversight and Testing of the SmartCare System To address the defects she found surrounding CMP’s SmartCare implementation, and to restore stakeholder confidence, Ms. Arnold recommended that the Commission require CMP to test the system as it should have been tested prior to go-live in accordance with applicable best practices. Ms. Arnold also recommended that, given the lack of public confidence in CMP at this point, it would be best to have an independent third party, selected by the Commission, validate the new testing plan and verify the results. At the hearing, Ms. Arnold seemed to back away from her initial recommendation of a full retesting of the SmartCare system. She explained in her testimony that the key to determining what additional testing needed to be done is the completion of the Requirements Traceability Matrix, which was part of the testing strategy but was never completed, or at least never provided. The purpose of the Requirements Traceability Matrix is discussed above at pages 46–48.) If, after proper completion, the Requirements Traceability Matrix showed that testing had already been substantially completed, then little further testing would need to be done. On the other hand, if the evidence showed significant gaps in testing, then a greater effort would be required. Tr. at 120 (Nov. 6, 2019 Hr’g). Examiners’ Report 113 Docket No. 2019-00015 In its SmartCare Implementation rebuttal, the Company’s witnesses testified about why the top-to-bottom testing approach proposed by Ms. Arnold in her direct testimony should not be adopted. As discussed there, in order to ensure consistency between the production and pre-production environments, any system enhancement or fix in the production would need to be “frozen.” This freeze would delay the implementation of both system enhancements and system fixes until such time as the testing was completed. In addition, Company resources would be diverted from customer service and system support functions. This position was corroborated by Ms. Minton, Liberty’s software system implementation expert. Ms. Minton testified that she did not believe that a full top-tobottom regimen of retesting at this point would be helpful since the SmartCare has been up and running for two years which essentially has provided two years of real-time testing. The Commission finds these positions to be persuasive and agree with them. Ms. Minton testified that it would be helpful to establish an independent qualityassurance function to oversee further testing and that, while this quality-assurance tester could be an employee of the current system integrator or even of CMP, it was imperative that the tester be independent of CMP’s management and its CIS business function. In addition, Ms. Minton testified that the third-party tester would need to be trained in testing processes and procedures and understand the particular system being tested. Ms. Minton testified that she envisioned an initial period of three to five years for such a program, with an evaluation done at the end of the initial period to see if the role needs to continue based on the results of the process. Tr. at 12 (Nov. 6, 2019 Hr’g). Examiners’ Report 114 Docket No. 2019-00015 In its brief, the Company stated that it would support more focused testing on the end-to-end meter to bill process. CMP suggested that the testing could use test scenarios and test cases developed through consultation between CMP and a qualified third-party consultant. CMP Br. at 80. CMP was also open to establishing a more formal and independent quality assurance position with oversight over the Company’s on-going efforts to address remaining defects which have been identified and testing of SmartCare upgrades. The Company envisioned that the third-party oversight be in place initially for a period of 12 to 18 months with an evaluation then done on the program’s continued value. 43 The Commission concludes that, given the deficiencies in CMP’s initial testing, the manner in which CMP responded to and addressed such defects, and the loss of customer confidence that has resulted, additional testing of the SmartCare system is required. For the reasons discussed in CMP’s testimony and in the testimony of Ms. Minton, the Commission finds that this testing need not be a full top-to-bottom retest of the system. Instead, the testing should be designed to fill in any gaps in the initial testing protocol identified during the quality-review process, as well as any other testing that now needs to be done based on the operations of the system. This additional testing regimen shall be designed by an independent third party in consultation with the Company. The third-party inspector should be familiar with SAP and have experience in software-system testing processes. The inspector will be selected based on a 43 In its reply brief, without apparent explanation, the Company seems to have retreated from its position on the benefits and acceptability of targeted testing as put forth in its earlier testimony and initial brief. CMP Reply Br. at 29. Examiners’ Report 115 Docket No. 2019-00015 competitive-bid process administered by CMP. CMP would then recommend a candidate, whose engagement will be subject to Commission review and approval. B. Monitoring Future Performance The Commission also believes that it is necessary and appropriate to monitor CMP’s ongoing performance in the operation and maintenance of SmartCare. With respect to open defects, the Commission directs CMP to file within 30 days its plan to close out all customer-facing defects. Even though the Company claimed that it was working to close the outstanding defects by the end of the year, the plan should be specific about the timeline for resolving each defect and the personnel needed to do so. Each month thereafter, the Company shall file a status report detailing the progress made toward resolution of each defect and compliance with the resource plan. The monthly reporting requirement shall continue until all customer-facing defects are resolved. The Company is also directed to file with the Commission, within 30 days of the final order in this docket, a comprehensive maintenance plan for resolving defects that are not customer-facing or are otherwise unknown at this time, and for managing the ongoing maintenance of the system, including development and deployment of all upgrades and enhancements. The plan should include the timeline and the personnel requirements necessary for the ongoing defect resolution and system maintenance. The Company shall file quarterly reports with the Commission demonstrating compliance with its maintenance plan, including any steps the Company intends to follow to remediate any noncompliance with the established plan. Examiners’ Report 116 Docket No. 2019-00015 The Commission expects the Company to maintain its performance at a level that meets or exceeds a reasonable bill-error benchmark. That benchmark is being established concurrently in Docket No. 2018-00194. C. The OPA’s Proposal to Disallow SmartCare Investments 1. Positions Before the Commission a. OPA In her testimony on behalf of the OPA, Ms. Arnold of BerryDunn recommended that the costs of the validation testing that she proposed be done following the case be disallowed or paid for by the utility’s shareholders. Until the testing can be done, Ms. Arnold also proposed that an amount equivalent to the costs of pre-go-live test planning, test execution and defect management be taken out of rates since these amounts were imprudently incurred. The outcome of the validation testing then could be used to calculate the final portion of costs associated with these areas that should be disallowed. The OPA argued that because of the many shortcomings of SmartCare, a portion of SmartCare costs should be removed from rate base. The OPA went on to argue that by looking at the gap between test coverage and test execution the independent thirdparty auditors have uncovered, the Commission will have an objective basis for calculating the amount of SmartCare costs that should be disallowed. The OPA requested that the Commission provide for further process to conduct this review. OPA Br. at 37. In the brief it filed in the rate case on September 11, 2019, the OPA argued that CMP’s imprudence in managing the implementation of SmartCare also supports Examiners’ Report 117 Docket No. 2019-00015 disallowing any recovery of expenses for continuing to maintain its legacy CSS billing system. b. CMP CMP argued that, contrary to the OPA’s assertions, CMP’s go-live testing and defect management were designed and executed in consultation with Deloitte, were thorough, and were done in conformity with best practices. In addition, the system had functioned as planned and had successfully completed hundreds of thousands of transactions per month over the last two years. CMP Br. at 83. The Company argued that even if imprudence were found here, to disallow costs the Commission must determine whether the imprudent conduct caused harm to ratepayers. According to CMP, the OPA offered no evidence that customers have suffered any specific harm as a result of SmartCare’s alleged shortcomings. CMP Reply Br. at 26. The Company went on to argue that the OPA’s assertion of ratepayer harm was supported only by the speculation that if CMP prepared a Requirements Traceability Matrix to Ms. Arnold’s satisfaction, or tracked defects pre-go-live more thoroughly, or paid more attention to quality during project implementation, then there would have been fewer post-go-live defects and bill-presentment issues and customers would have been at less risk of confusion. The OPA and its witnesses did not, however, point to any specific defect that would have been avoided if CMP had taken those actions or quantify how customers at large were harmed by the defects and presentment issues that affected certain customers. Id. at 27. The Company pointed to the fact that the objective of this investigation was to determine if the SmartCare billing system was overcharging customers and the Examiners’ Report 118 Docket No. 2019-00015 evidence in the case, in particular Liberty’s forensic audit of 3.9 million bills issued to customers, definitively establishes that systemic overcharges have not occurred. Id. 2. Decision As set forth in Section VII.D, the Commission has found that various decisions of CMP in its implementation of SmartCare were imprudent. The question then becomes what remedy is appropriate to address that imprudence. In Seabrook, the Commission held that in analyzing prudence, the Commission must determine whether the utility’s deficient behavior caused any harm to customers and, if so, what the quantification of that harm is. Id. at 14. The Commission has recognized that the costs of imprudence are often difficult to measure, especially when trying to assess the harm from a course of conduct not taken. Emera Maine, Request for Approval of a Proposed Rate Increase, Docket No. 2015-00360, Order Part II at 70 (Dec. 22, 2016). Here, it is not possible to tie the Company’s SmartCare implementation to any particular defect. Nevertheless, the Commission finds that, had the implementation been prudently managed, it is highly likely that far fewer defects would have resulted and far fewer customers would have been harmed. In Section VII.A, we found that, in fact, that CMP’s billing system was not systematically incorrectly billing customers for usage. But this finding is not a finding that all was well with the SmartCare implementation or that all is well today. To the contrary, as discussed in Sections VII.C and VII.D, there were deficiencies with the implementation and some defects persist today. Overall, the post-go-live deficiencies—the defects, bill errors, and the time taken to address these defects and errors—have led to a level of distrust of the Company’s Examiners’ Report 119 Docket No. 2019-00015 billing system, which is not misplaced. The Company has argued that what is needed to restore customers’ confidence in CMP’s billing system is an unequivocal finding that CMP’s billing system is accurately billing customers for usage. We have indeed found that there is no systemic issue in SmartCare of incorrect usage being billed. Yet the Company’s argument here both oversimplifies the issues surrounding the distrust of the Company’s billing system, and also misplaces where the responsibility lies for restoring such trust. It is CMP’s responsibility—not the Commission’s—to prove to its customers that the Company is acting in accordance with sound utility-management practices, and that it has fully resolved its SmartCare billing issues. The additional testing and monitoring that we have ordered are key components of that restoration of that trust. The costs of these programs were the result of the Company’s imprudence and thus should be borne by the Company. The Commission does not find, however, that the SmartCare implementation costs that are currently embedded in CMP’s current rate base should be removed. The problems with CMP’s SmartCare implementation were that the Company did inadequate testing—not that it did too much. The remedy that the Commission orders here, of having the utility bear the costs for the third-party testing process, captures any concern about that issue. In contrast, a broad disallowance of all SmartCare implementation testing costs would go beyond the particular prudence issues identified. The Commission also notes that the OPA does not point to any specific costs that Examiners’ Report 120 Docket No. 2019-00015 should be disallowed in the Company’s revenue requirement. Without that information, the Commission cannot make the requested disallowance. 44 D. Independent Electricity-Use Audit Program for Resolving Yet-Unresolved Usage Complaints Both the OPA and CMP Ratepayers Unite urged the Commission to make no findings on individual high-bill complaints so that customers could pursue any related claims in other forums. OPA Br. at 39; CMP Ratepayers Unite Reply Br. at 9. In response to this recommendation, the Commission points out that individual customer complaints are currently being resolved through the standard CASD complaint process, which is a voluntary process. A customer may pursue remedies through court, instead of through the CASD, if a customer so chooses. Our decision does not change that. In the Bench Analysis, Staff recommended that if this investigation found no systemic problems with the Company’s billing and metering systems that resulted in high billed usage amounts, and if the cause of a customers’ significant increase in usage remains unexplained, the Commission should direct the Company to establish an independent review process for those customers. BA at 8–9. Under that process, customers would have the option of being referred to an independent entity, such as Efficiency Maine, who would conduct an audit of each customer’s electricity usage. During this audit, the independent entity would assess the amount of electricity being consumed at the customer’s home compared to the customer’s metered and billed kWh usage. If the audit shows that the customer’s 44 Nevertheless, due to this imprudence, in the companion rate case order the Commission disallows the costs to maintain CMP’s legacy CSS billing system, which has been maintained only due to this investigation. Examiners’ Report 121 Docket No. 2019-00015 electricity usage is consistent with the amount of usage being metered and billed, the auditor could advise the customer about options for reducing usage, including energyefficiency programs offered by Efficiency Maine. If the audit indicates a discrepancy between the customer’s electricity usage and the amounts being metered and billed by CMP, an additional analysis and review will be necessary to determine the cause and amount of the discrepancy, as well as to determine an appropriate remedy for the customer. Staff further recommended that the initial population of customers eligible for the independent review be customers who have filed a complaint of high usage with the CASD, and for whom neither the CASD nor the Company has been able to determine the cause of the high usage. 45 Id. at 9–10. Staff also recommended that criteria be established to determine which customer complaints from this initial population should be referred for independent review and which complaints should be resolved by the CASD. The Staff noted that possible criteria for the program might include: (1) a customer’s concern must be related to unexplained high usage, (2) the high usage must have occurred after the implementation of SmartCare, and (3) the customer has filed a complaint of high usage with the CASD. Id. at 10. Staff also noted in the Bench Analysis that, depending on the findings made while the audit program was in place, the Commission could decide to alter or expand this population. Id. In its post-hearing brief, the Company stated that it supports the Commission’s establishing a mechanism through the CASD to expeditiously resolve any open 45 Approximately 1,247 customers have filed a complaint of high usage with the CASD and CMP’s high-use team where the cause of the high usage could not be determined to the customer’s satisfaction. Examiners’ Report 122 Docket No. 2019-00015 complaints on an individualized basis, and that the Company is committed to working with CASD and the particular customers to expeditiously do so. The Company also stated that the CASD can address, as necessary and appropriate, any customer complaints related to billing errors or anomalies not based on high usage, as recommended by Staff in the Bench Analysis. CMP Br. at 94. CMP added that it recognizes that some high-usage complaints likely will remain even after the Commission’s order in this proceeding and that the Company agrees with Staff that the high-usage dispute-resolution process should be augmented, in targeted situations, with such an in-home review/audit, where such an assessment “could provide additional insights into the customer’s usage that cannot be ascertained through telephone discussions and the customer’s review of their historical hourly, daily and monthly usage data.” Id. at 95–96. The Company also requested that the Commission recognize its pilot program already established with the Trust, as described earlier in this report, as the first step in the adoption of Staff’s recommended third-party in-home review/audit proposed in the Bench Analysis. The Company requested that the Commission review the effectiveness of the program, including appropriate cost recovery mechanisms, after CMP and the Trust complete their initial evaluation. Id. at 97. In its brief, the OPA stated that the Commission should adopt the Staff’s proposal for independent audits and review process for customers with high-usage complaints as a way of helping customers reach closure on these concerns and complaints. OPA Br. at 38. The OPA expressed concern, however, about Staff’s recommendation that customers would qualify for this program only if they had complained to the CASD and had found no explanation or other relief. The OPA stated its understanding from Examiners’ Report 123 Docket No. 2019-00015 interactions with many customers that some are assuming that their high bill concerns will be resolved in this docket and that many customers with high-bill concerns have not filed complaints with the CASD. Id. at 38–39. The OPA further stated that they believe these customers would be wrongly excluded from the program under Staff’s approach and suggested that the Commission require a public information campaign at the end of this proceeding whereby customers who seek to enter the program could be instructed to file a high-usage complaint if they have not already done so. Id. at 39. It is clear from customers’ responses that there is significant disagreement between the customers and the Company about specific events described in each of the case studies, as well as the Company’s portrayal of its interactions with the customers. While we find the customers’ responses to the Company’s case studies compelling and believe that individual customer concerns with the Company should be resolved, the customers’ responses do not change our overall finding that, systemwide, the Company’s billing and metering system is accurately measuring and billing customer usage. We encourage these customers to try and resolve any outstanding disagreements with the Company and to contact the CASD for assistance if they are unsuccessful in their resolution attempts. Customers’ concerns about CMP’s customer service—the time customers waited on hold or were provided unhelpful responses by the Company—are addressed in our companion order in the Company’s rate case, Docket No. 2018-00194. Based on the record and the parties’ arguments, we find that the independent audit process recommended by Staff in the Bench Analysis is the appropriate remedy for unresolved high-use complaints. We refer to this process as the Independent Examiners’ Report 124 Docket No. 2019-00015 Electricity-Use Audit Program. Further, complaints involving possible billing errors and billing anomalies should be resolved by the CASD through its standard complaint resolution process. We also find that the initial population of customer complaints eligible for the Independent Electricity-Use Audit Program is limited to those complaints that are: (1) related to unexplained high usage where the usage meets one of the criteria of the interim payment policy (discussed above in Section III.C); (2) where the disputed high usage has occurred after the implementation of SmartCare; and (3) where the customer has contacted the CASD with concerns about high usage. This population may be altered by the Commission in the event that the independent analysis reveals facts that indicate either an underlying problem exists that may affect additional customers or other reasons exist to expand or modify the initial population. To effectuate this remedy, we direct the Company to send a letter to each of the customers the CASD referred to its High Bill Resolution Team and with whom the Company made contact and was unable to resolve the customers’ high-usage concern. The letter should notify the customer of the opportunity to participate in the Independent Electricity-Use Audit Program and list the criteria for participation. When a customer contacts the Company and expresses interest in participating in the Independent Electricity-Use Audit Program, CMP will determine whether the customer meets the participation criteria. If the customer does meet the participation criteria, the Company must enroll the customer in the Independent Electricity-Use Audit Program and notify the customer. If the customer does not meet the participation criteria, the Company must notify the customer and explain the reason why the customer failed to meet the criteria. The Company must also notify the customer that they may contact the CASD to Examiners’ Report 125 Docket No. 2019-00015 challenge the Company’s decision to exclude them from the program. The Company should work with the OPA and the CASD on content of the notifications required by this order. In light of the above, the Commission rejects CMP’s recommendation that the Independent Electricity-Use Audit Program be treated as a “pilot” and initially limited to 50 customers. The record in this case shows that, where the Company was able to reach the complainant, CMP was not able to resolve to the customers’ satisfaction 1,247 high-use complaints. Thus, clearly more than 50 customers will meet the above criteria and there is no fair way of limiting the customers who meet the criteria to 50. We also disagree with the OPA’s recommendation that there be no initial limitation on the customers who may seek remedy through the Independent Electricity-Use Audit Program. We expect that an independent entity such as the Efficiency Maine Trust could have limited resources and thus may be limited in the number of independent audits that can be completed. As a clarification, only where there is evidence of a significant increase in a customer’s usage post-go-live of SmartCare should the customer be referred to the Independent Electricity-Use Audit Program. While some customers may have a concern in general about usage, it there is no evidence to indicate that a customer’s usage increased significantly after the implementation of SmartCare, there’s no baseline of “expected” usage upon which to identify “higher than expected” usage. Thus, the Independent Electricity-Use Audit Program would not be helpful in these situations. Rather, these situations, along with situations that involve billing errors or anomalies and not high usage, should be resolved by the CASD through its normal complaint-resolution process. Examiners’ Report 126 Docket No. 2019-00015 On the Company’s request that the Commission review the effectiveness of the program, including appropriate cost-recovery mechanisms, after CMP and the Trust complete their initial evaluation, the Commission finds that such a review is appropriate, at least for the effectiveness of the Independent Electricity-Use Audit Program. Thus, we direct the Company to report the findings of the program for individual customers to the CASD, consistent with the process CMP uses to report findings/activities in typical CASD complaint cases. The Company is also directed to file a report bimonthly (every other month) with the Commission that: (1) summarizes the findings of customer complaints reviewed for that time period through the Independent Electricity-Use Audit Program; (2) identifies and explains any findings of the Independent Electricity-Use Audit Program for that time period relating to usage on the customers’ side of the meter that is inconsistent with usage being registered on the Company’s side of the meter; (3) quantifies the individual and total credits applied to a customer accounts as a result of the Independent Electricity-Use Audit Program for that time period; and (4) describes all billing errors or anomalies identified through the program. The first report on the Independent Electricity-Use Audit Program is due two months from the date of this order. On the Company’s request that the Commission also review appropriate costrecovery mechanisms for the Independent Electricity-Use Audit Program, the Commission finds that, at least at this time, this review is unnecessary. Because CMP bears significant blame in creating confusion about usage through its handling of SmartCare, the initial costs of this program will be borne by CMP. As time passes, we Examiners’ Report 127 Docket No. 2019-00015 will review the costs of the program and determine whether a change in cost responsibility is appropriate. E. Costs of the Liberty Audit There remains the question of how the costs of Liberty’s audit will be recovered. Under 35-A M.R.S. § 113(3), [t]he full cost of the management audit must be recovered from ratepayers, except that if the audit of an investor-owned public utility contributes to a commission finding of imprudence that results in a cost disallowance, the commission shall determine how to fairly allocate the cost of the management audit to ratepayers or the shareholders of the investor-owned public utility. Here, Liberty’s audit contributed to the Commission’s finding of imprudence with respect to the Company’s customer-service function (as determined in Docket No. 2018-00194) and its implementation of SmartCare and handling of a meter anomaly (as determined in this docket). Those findings resulted in a downward adjustment to CMP’s return on equity in the rate case (Docket No. 2018-00194) and several postinvestigation audits and programs the cost of which will be borne by CMP. The Commission is, thus, in a position to decide the appropriate allocation of costs between ratepayers and CMP’s shareholders. Before making this determination, the Commission hereby directs the Staff to provide the parties an opportunity to file comments and reply comments on the recoverability of the audit costs. The Commission will make its decision on responsibility for the costs of Liberty’s audit after considering the parties’ comments. Examiners’ Report IX. 128 Docket No. 2019-00015 THE INTERIM PAYMENT POLICY A. Background The interim payment policy, and what led to its being established, is described above in section II.C. As we’ve described, the Commission created the policy in an April 11, 2018 order in Docket No. 2018-00052, and modified it in a March 11, 2019 order in this proceeding. To summarize, the interim payment policy allows customers whose billed usage or total bill is at least 25% higher than that for the same month in the prior year to defer for possible future payment the difference in the bill, pending the outcome of this investigation. The amounts customers are not required to pay immediately under that policy have come to be known as the “set-aside amount” or as having been “set aside.” In its brief, the Company requested that the Commission provide specific guidance on how the set-aside amounts established under the interim payment policy should be addressed for customers who (i) choose to withdraw their high-usage complaints as a result of the Commission’s order, (ii) continue to pursue their highusage complaints through the dispute resolution mechanism established by the Commission in this proceeding, and (iii) do not respond in any fashion to outreach by the Company and the CASD about their high-usage complaints. CMP Br. at 97. B. Termination of Policy The purpose of the interim payment policy was to give customers a reprieve on paying higher-than-typical usage amounts on their bill and avoid the potential disconnection of the customer’s service pending a determination on whether the billed amounts were valid. Through this investigation, the Commission has confirmed that (1) there is no systemic problem with CMP’s billing system that would bill for anything Examiners’ Report 129 Docket No. 2019-00015 other than actual usage, and (2) billing-system defects that affect the amount of a customer’s bill have been or are being resolved and any customer overcharges as a result of those defects have been or will be credited to affected customers. Based on these conclusions, the Commission finds that the interim payment policy no longer serves its intended purpose. Thus, the Commission hereby terminates the interim payment policy effective with this order. C. Proposed Method for Closing Out Policy and Paying Amounts Set Aside; Request for Comments Our termination of the interim payment policy raises the question of how to handle unresolved high-bill complaints and how to handle the amounts that have been set aside over the past two years. Below, the Commission proposes a method of handling these cases. We request that parties file comments on the proposed methodology within 30 days of this date of this order. First, for any customers who availed themselves of the interim payment policy, the Commission proposes that Company notify these customers of the termination of the policy. For customers who have availed themselves of the interim payment policy and who have an open complaint with the CASD, the Company shall follow the Commission’s rules while the complaint is pending (i.e., it may not threaten disconnection or disconnect service to a customer who disputes liability for the bill, a utility’s deposit request, or the terms of a payment arrangement required by a utility to Examiners’ Report 130 Docket No. 2019-00015 avoid disconnection and has filed a complaint with the CASD until the complaint is resolved). 46 For customers who have availed themselves of the interim payment policy and do not have an open complaint with the CASD, the Commission proposes that the Company notify these customers of their obligation to pay the set-aside amount, and offer the customers a payment arrangement on the set-aside amount. (This notification can be in the same letter that notifies the customer of the termination of the policy.) We also propose that CMP offer these customers payment arrangements for the set-aside amount. The payment arrangement should give the customer at least the same amount of time to pay the set-aside amount as the amount of time that has passed since the customer first qualified for the interim payment policy. For example, if a customer first availed him- or herself of the interim payment policy in March 2018, the Company must give the customer at least 21 months to pay the set-aside amount (April 2018 through December 2019). A customer may agree to a shorter term if the customer chooses, and the Company may agree to a longer term if it chooses. We also propose that CMP be prohibited from assessing any late-payment charges on set-aside amounts for customers who enter into a payment arrangement for the set-aside amount in response to the Company’s notification and who meet the terms of the payment arrangement for the duration of the payment arrangement. CMP requested guidance on situations in which customers have availed themselves of the interim payment policy and who: (i) chose to withdraw their highusage complaints as a result of the Commission’s order, (ii) continued to pursue their 46 See MPUC Rules, ch. 815 § 13(G)(1). Examiners’ Report 131 Docket No. 2019-00015 high-usage complaints through the dispute-resolution mechanism established by the Commission in this proceeding, or (iii) did not respond in any fashion to outreach by the Company and the CASD about their high-usage complaints. The Commission proposes that the Company follow the process outlined in this section and notify all customers in each of these three categories of the termination of the interim payment policy and offer each of these customers a payment arrangement on the set-aside amount. It would be up customers to avail themselves of the payment arrangement or to simply pay the setaside amount. If the customer does not respond to the notification and does not pay the set aside amount, the Company may proceed with its normal collection policies for the set-aside amount. The Commission requests parties’ comments on this proposed methodology, or proposed alternatives, within 30 days of this order. X. CONCLUSION AND ORDERS The Commission expects that the remedies, oversight, and monitoring we have imposed on CMP in this order will remediate the problems we have identified with CMP’s systems and, with enough time, foster a healthier relationship between CMP and its customers. Accordingly, the Commission ORDERS 1. That CMP shall, within 30 days of this order, initiate a competitive-bid process to select an independent inspector to conduct additional, targeted testing of SmartCare, and within 90 days of this order, submit its recommended independent inspector to the Commission for review and approval; Examiners’ Report 132 Docket No. 2019-00015 2. That the work of the independent inspector shall be paid for by CMP and not recovered from ratepayers due to our finding of imprudence; 3. That CMP shall, within 30 days of this order, file with the Commission its plan for resolving all outstanding customer-facing defects, and file with the Commission a status report monthly thereafter until all such defects are resolved; 4. That CMP shall, within 30 days of this order, submit its comprehensive SmartCare maintenance plan (as described in the body of this order), and submit a status report quarterly thereafter until further notice from the Commission; 5. That CMP shall, within 30 days of this order, file a detailed proposal to establish and implement the Independent Electricity-Use Audit Program described in section VIII.D of this order and submit the bimonthly (every two months) report (described in the body of this order) thereafter. Under 35-A M.R.S. § 107(4), the Commission delegates approval of this program to the Director of the Consumer Assistance and Safety Division; 6. That CMP shall, within 45 days of this order, file a report (as detailed in the body of this order) explaining what would be required to change SmartCare to allow for rounding in lieu of truncation when prorating; 7. That CMP shall, within 30 days of this order, return to customers negatively affected by Defect #5667/5884 when incorrectly applied funds were removed from their account, through no fault of their own, the dollar amount that was Examiners’ Report 133 Docket No. 2019-00015 removed from their account by way of a credit to their utility bill. Any waiver of CMP’s terms and conditions required to permit this refund is hereby granted; 8. That CMP shall, within 60 days of this order, ensure that customers affected by Defect #5302 are on the most advantageous rate or have meter changeouts as needed to support the customers’ chosen billing rate; 9. That CMP shall, within 90 days of this order, refund to customers affected by meter over-registration anomaly amounts overbilled as identified in CMP’s analysis; 10. That, in CMP’s next Annual Compliance Filing, CMP’s rates shall be adjusted so that the general body of ratepayers do not pay for amounts (as determined in CMP’s analysis) that were under-billed as a result of meter-registration anomalies; 11. That, in conjunction with Docket No. 2018-00194, the Hearing Examiner shall provide an opportunity for parties to file comments and reply comments on the apportionment of costs for Liberty’s audit prior to the Commission’s deciding that question; 12. That the interim payment policy, which the Commission established in an April 11, 2018 order in Docket No. 2018-00052, and modified in a March 11, 2019 order in this docket, is hereby terminated effective with this order; 13. That parties may file comments, within 30 days of this order, on the Commission’s proposal for closing out the interim payment policy and resolving amounts set aside under that policy. Examiners’ Report Dated: January 9, 2020 134 Docket No. 2019-00015 Respectfully submitted, ______________________ Charles Cohen Former Hearing Examiner 47 ______________________ Katie M. Gray Hearing Examiner And on behalf of Advisory Staff Faith Huntington Derek Davidson Sally Zeh 47 Hearing Examiner Cohen retired from the Commission at midnight on December 31, 2019. Although he is no longer an employee of the Commission or the state, his signature and name are included here in recognition of his significant contribution to this investigation.