United States Government Accountability Office Report to Congressional Committees April 2019 COLUMBIA CLASS SUBMARINE Overly Optimistic Cost Estimate Will Likely Lead to Budget Increases GAO-19-497 April 2019 COLUMBIA CLASS SUBMARINE Overly Optimistic Cost Estimate Will Likely Lead to Budget Increases Highlights of GAO-19-497, a report to congressional committees Why GAO Did This Study What GAO Found The Navy has identified the Columbia class submarine program as its top acquisition priority. It plans to invest over $100 billion to develop and purchase 12 nuclear-powered ballistic missile submarines to replace aging Ohio class submarines by 2031. The Navy’s goal is to complete a significant amount of the Columbia class submarine’s design—83 percent—before lead submarine construction begins in October 2020. The Navy established this goal based on lessons learned from another submarine program in an effort to help mitigate its aggressive construction schedule. Achieving this goal may prove to be challenging as the shipbuilder has to use a new design tool to complete an increasingly higher volume of complex design products (see figure). The shipbuilder has hired additional designers to improve its design progress. The Navy also plans to start advance construction of components in each major section of the submarine, beginning in fiscal year 2019, when less of the design will be complete. The National Defense Authorization Act for Fiscal Year 2018 and House Report 115-200 included provisions that GAO review the status of the program. This report examines (1) the Navy’s progress Shipbuilder’s Workload Increases as Design Efforts Continue and challenges, if any, in meeting design goals and preparing for lead submarine construction; (2) the reliability of the Navy’s cost estimate; and (3) how the Navy is implementing a special fund and associated authorities to construct Columbia class submarines. GAO reviewed Navy and shipbuilder progress reports, program schedules, and construction plans. GAO assessed the Navy’s cost estimate and compared it to best practices for cost estimating. GAO also reviewed certain Navy funding and acquisition authorities and interviewed program officials. This is a public version of a sensitive report that GAO issued in March 2019. Information that the Department of Defense (DOD) deemed sensitive has been omitted. What GAO Recommends GAO is making three recommendations: that the Navy update the lead submarine cost estimate with cost risk analysis using current cost data, develop a realistic estimate of savings from use of the Fund’s authorities, and use this updated cost estimate to inform its budget request for lead submarine construction. DOD concurred with GAO’s recommendations. View GAO-19-497. For more information, contact Shelby S. Oakley at (202) 512-4841 or oakleys@gao.gov. The Navy’s $115 billion procurement cost estimate is not reliable partly because it is based on overly optimistic assumptions about the labor hours needed to construct the submarines. While the Navy analyzed cost risks, it did not include margin in its estimate for likely cost overruns. The Navy told us it will continue to update its lead submarine cost estimate, but an independent assessment of the estimate may not be complete in time to inform the Navy’s 2021 budget request to Congress to purchase the lead submarine. Without these reviews, the cost estimate—and, consequently, the budget—may be unrealistic. A reliable cost estimate is especially important for a program of this size and complexity to help ensure that its budget is sufficient to execute the program as planned. The Navy is using the congressionally-authorized National Sea-Based Deterrence Fund to construct the Columbia class. The Fund allows the Navy to purchase material and start construction early on multiple submarines prior to receiving congressional authorization and funding for submarine construction. The Navy anticipates achieving savings through use of the Fund, such as buying certain components early and in bulk, but did not include the savings in its cost estimate. The Navy may have overestimated its savings as higher than those historically achieved by other such programs. Without an updated cost estimate and cost risk analysis, including a realistic estimate of savings, the fiscal year 2021 budget request may not reflect funding needed to construct the submarine. ______________________________________ United States Government Accountability Office Contents Letter 1 Background Navy Is Managing an Aggressive Build Schedule, but Early Design and Construction Challenges Signal Schedule Risk Columbia Class Cost Estimate Is Not Reliable and Does Not Reflect Program Risks Navy Is Using National Sea-Based Deterrence Fund and Associated Authorities, but Anticipated Savings May Be Overestimated Conclusions Recommendations for Executive Action Agency Comments and Our Evaluation 3 36 41 42 43 Appendix I Objectives, Scope, and Methodology 46 Appendix II Technology Readiness Levels 49 Appendix III GAO’s Assessment of the Reliability of the Navy’s Cost Estimate for the Columbia Class Submarine Program 50 Appendix IV Comments from the Department of Defense 52 Appendix V GAO Contact and Staff Acknowledgments 58 12 23 Tables Table 1: Columbia Class Critical Technologies (as Identified by GAO in 2017) Table 2: Characteristics of a High-Quality, Reliable Cost Estimate Table 3: Cost Estimates and Independent Cost Assessments for the Columbia Class Submarine Program (in billions of dollars) Page i 6 9 33 GAO-19-497 Columbia Class Submarine Table 4: Navy’s Plans to Implement Acquisition Authorities Associated with the National Sea-Based Deterrence Fund and Navy’s Estimated Savings Table 5: Technology Readiness Level (TRL) 40 49 Figures Figure 1: Columbia Technology Development, Design, and Construction Schedules Figure 2: Columbia Class Technology Development Efforts Figure 3: Design and Construction Phases for the Columbia Class Submarine Program Figure 4: Super Modules and Their Respective Functions for the Columbia Class Submarine Program Figure 5: Cost Estimating Process for the Columbia Class Program’s Milestone B Review Figure 6: Shipbuilder’s Workload Increases as Design Efforts on the Columbia Class Submarine Program Continue Figure 7: Current Monthly Design Schedule Performance for the Columbia Class Submarine Program Figure 8: Cumulative Design Cost Performance for the Columbia Class Submarine Program Figure 9: Start of Component and Equipment Advance Construction for Each Super Module for the Columbia Class Submarine Figure 10: Navy’s Design Disclosures and Construction Plans for the Columbia Class Submarine Program Figure 11: Touch Labor Hour Calculation for Lead Columbia Submarine, Based on Lead Virginia Class Submarine (millions of hours) Figure 12: Learning Curve Effect for Estimated Hours of Columbia Class Follow-on Submarines Based on Virginia Class Submarines Figure 13: Estimated Timeline of Future Key Events Leading to Columbia Class Submarine Lead Submarine Authorization and Funding Figure 14: Timeline of National Sea-Based Deterrence Fund Legislation Figure 15: National Sea-Based Deterrence Fund Deposit Process Figure 16: Summary of Assessment of Columbia Program’s Cost Estimate Compared to GAO’s Best Practices Page ii 4 5 8 9 11 13 14 15 17 18 25 29 34 37 39 51 GAO-19-497 Columbia Class Submarine Abbreviations CAPE CBO DOD the Fund NAVSEA NCCA TRL Office of Cost Assessment and Program Evaluation Congressional Budget Office Department of Defense National Sea-Based Deterrence Fund Naval Sea Systems Command Naval Center for Cost Analysis Technology Readiness Level This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. Page iii GAO-19-497 Columbia Class Submarine Letter 441 G St. N.W. Washington, DC 20548 April 8, 2019 Congressional Committees The Navy plans to invest approximately $128 billion to research, develop, and purchase the replacement for 14 Ohio class nuclear-powered ballistic missile submarines—the current sea-based leg of the nation’s strategic nuclear deterrent. 1 According to the Navy, the lead Columbia class submarine will need to make its first patrol in fiscal year 2031 in order to avoid a deterrence gap; the Ohio class submarines begin to retire in 2027. The Navy has identified the 12-submarine Columbia class program as its top acquisition priority and has set an aggressive schedule to deliver the lead submarine in fiscal year 2027, followed by a period of testing before the first patrol occurs. The Navy is continuing its design efforts and plans to begin advance construction of some of the submarine’s components in fiscal year 2019. In 2014, Congress created the National Sea-Based Deterrence Fund (the Fund), the use of which provides the Navy with several acquisition authorities. 2 One of the purposes of these authorities is to reduce material and equipment costs for Columbia class submarines. In light of the criticality of the deterrence mission and the cost and schedule pressures facing the Columbia class program, the House Armed Services Committee report accompanying the National Defense Authorization Act for Fiscal Year 2018 included a provision that we examine the program to include, among other things, technology development, design progress, and program cost estimates. 3 The act also included a requirement for the Navy to prepare and submit matrices on the Columbia program’s design and construction goals and progress and included a provision that we assess these matrices. 4 This report, which 1 This $128 billion represents in then-year dollars the total acquisition cost, including test and evaluation, and military construction costs. The Navy estimates that an additional approximately $140 billion in then-year dollars will be needed to operate, sustain, and dispose of the submarines over their life cycle. We use then-year dollars throughout this report, unless otherwise specified. Then-year dollars reflect the effects of inflation, including escalation up to and during the year of the appropriation, and throughout the period during which dollars are expended from the Treasury. 2 10 U.S.C. § 2218a: National Sea-Based Deterrence Fund. 3 H.R. Rep. No. 115-200, at 21 (Jul. 6, 2017). 4 Pub L. No. 115-91, § 231. Page 1 GAO-19-497 Columbia Class Submarine addresses both provisions, examines (1) the Navy’s progress and challenges, if any, associated with meeting design goals and preparing for lead submarine construction; (2) the reliability of the Navy’s cost estimate for the Columbia class submarine program; and (3) how the Navy is implementing the Fund and associated authorities to construct Columbia class submarines. This is our second public report examining the Columbia class program. 5 This report is a public version of a sensitive report that we issued in March 2019. The Department of Defense (DOD) deemed some of the information in our March report to be sensitive, which must be protected from public disclosure. Therefore, this report omits sensitive information about the Navy’s development of critical technologies for the Columbia class program, including specific details about the technologies. Although the information provided in this report is more limited, the report addresses the same objectives as the sensitive report and uses the same methodology. To assess the Navy’s progress and any challenges associated with meeting design goals and preparing for lead submarine construction, we reviewed Navy and General Dynamics Electric Boat’s (the lead shipbuilder) documents, including program briefings, schedules, and contract status reports to assess the program’s progress against its planned schedule. We reviewed the Navy’s and the shipbuilder’s plans for design management and completion and compared the plans with progress reports to identify any delays. We also reviewed ongoing development efforts and schedules for the Columbia program’s critical technologies to determine risks to their development and integration. We also reviewed the matrices submitted by the Navy to Congress in February and October 2018, to determine the status of the program and identify any changes to the Navy’s design, construction, and cost goals for the program since our December 2017 report. To assess the reliability of the Navy’s cost estimate, we determined the extent to which the estimate was consistent with cost estimating best practices as identified in our Cost Estimating and Assessment Guide. 6 5 GAO, Columbia Class Submarine: Immature Technologies Present Risks to Achieving Cost, Schedule, and Performance Goals, GAO-18-158 (Washington, D.C.: Dec. 21, 2017). 6 GAO, Cost Estimating and Assessment Guide: Best Practices for Developing and Managing Capital Program Costs, GAO-09-3SP (Washington D.C.: Mar. 2, 2009). Page 2 GAO-19-497 Columbia Class Submarine We also reviewed supporting documents, such as the program life-cycle cost estimate, briefs, memoranda, and relevant DOD and Navy policies. We compared the program’s cost estimate against independent estimates and assessments from the Office of Cost Assessment and Program Evaluation (CAPE); the Naval Center for Cost Analysis (NCCA); and the Congressional Budget Office (CBO). To assess how the Navy is implementing the Fund and associated authorities to construct Columbia class submarines, we reviewed the legislation establishing and modifying the Fund, as well as budget documents and DOD reprogramming approvals. To corroborate information for each of these objectives, we interviewed DOD and Navy officials and shipbuilder representatives responsible for the Columbia class program. Appendix I provides additional information on the scope and methodology of our review. The performance audit upon which this report is based was conducted from December 2017 to March 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. We subsequently worked with DOD from February 2019 to April 2019 to prepare this unclassified version of the original sensitive report for public release. This public version was also prepared in accordance with these standards. Background Over the next 10 years, the Navy plans to continue developing critical technologies, complete detail design, and begin construction of the lead Columbia class submarine. In December 2017, we found that the schedule to deliver the lead submarine was aggressive, with extensive overlap—or concurrency—between development, design, and construction, as shown in figure 1. Page 3 GAO-19-497 Columbia Class Submarine Figure 1: Columbia Technology Development, Design, and Construction Schedules Note: The Navy has stated that it will complete technology development in 2022, when one of its technologies completes testing in a representative environment. However, based on our work on best practices in weapon system acquisitions, we have previously recommended that mature technologies are those that have been developed into prototypes that represent the full form, fit, and function of the actual system and have been tested in a realistic environment such as in an aircraft, vehicle, or space. Examples include testing the prototype in a test bed aircraft. We have previously identified that demonstrating technologies in a realistic environment provides a higher level of technology understanding and reduces risk prior to starting product development. The Department of Defense has historically disagreed with this recommended practice. Our prior work reviewing shipbuilding programs has shown that the programs with the greatest amount of overlap between shipbuilding phases often have the highest cost and schedule growth, as well as quality and performance issues. 7 The National Defense Authorization Act for Fiscal Year 2018 included reporting requirements for the Columbia class program. 8 As part of these annual reporting requirements, the Navy must submit to Congress matrices that identify (1) key milestones, events, and performance goals for the design and construction of the Columbia class program; and (2) costs associated with the design and construction period of the Columbia class program. The Navy submitted its initial matrices to Congress in February 2018 and an update to the matrices in October 2018. The next matrices update is due in March 2019 and annually, thereafter, until the lead Columbia submarine is delivered. 9 7 GAO, Navy Shipbuilding: Past Performance Provides Valuable Lessons for Future Investments, GAO-18-238SP (Washington D.C.: June 6, 2018). 8 Pub L. No. 115-91, § 231. 9 Columbia class program officials told us that the Navy has submitted the matrices to Congress. As of the issuance of this report, we have not received a copy of the March 2019 matrices. Page 4 GAO-19-497 Columbia Class Submarine Columbia Class Critical Technologies The Navy is developing a number of new technologies related to submarine propulsion, missile tubes, and survivability that are planned to ensure that the Columbia class will remain operationally relevant throughout its planned 42.5-year service life, as shown in figure 2. Figure 2: Columbia Class Technology Development Efforts In 2015, as part of its technology readiness assessment, the Navy identified two technologies—the advanced carbon dioxide removal unit and the stern area system—as critical technology elements. However, as we found in 2017, several Columbia class technologies that met GAO’s definition of a critical technology element were not identified by the Navy as critical technologies. 10 In addition, several of these were immature, with technology readiness levels (TRL)—used to describe the maturity of critical technologies—of less than 7. 11 See appendix II for a description of TRLs. 10 In our guide, we identify criteria for a critical technology element, namely that it is a technology that is “new or novel, and needed for a system to meet its anticipated operational performance requirements; or that poses major cost, schedule, or performance risk during design or demonstration.” GAO, Technology Readiness Assessment Guide: Best Practices for Evaluating the Readiness of Technology for Use in Acquisition Programs and Projects, GAO-16-410G (Washington, D.C.: August 2016). 11 GAO-18-158. Page 5 GAO-19-497 Columbia Class Submarine As part of its matrices to Congress, the Navy is required to report on the TRLs of the integrated power system, nuclear reactor, propulsor, coordinated stern features, stern area system, and common missile compartment—which are the critical technologies we identified in our prior report. Table 1 lists each GAO-identified critical technology and its TRL as of October 2018, as reported by the Navy. Table 1: Columbia Class Critical Technologies (as Identified by GAO in 2017) Technology a Technology Readiness Level (TRL) as of October 2018 Expected or achieved date of b technology maturity (TRL 7) Advanced carbon dioxide removal unit 6 April-June 2019 Advanced propulsor bearing 6 2027 or later Common missile compartment 6 January-March 2019 Integrated power system 6 October-December 2019 Nuclear reactor 7 July-September 2018 Propulsor 7 prior to January 2018 d Propulsor shaft 7 prior to January 2018 d Stern area system 4 April-June 2022 X-stern planes 7 prior to January 2018 c d Source: GAO analysis of Navy data. GAO-19-497 a The Department of Defense (DOD) deemed specific details about the critical technologies to be sensitive; therefore the description of each technology was omitted from this table. For additional information on the technologies identified by GAO, see Columbia Class Submarine: Immature Technologies Present Risks to Achieving Cost, Schedule, and Performance Goals, GAO-18-158 (Washington, D.C.: Dec. 21, 2017). b Based on our work on best practices in weapon system acquisitions, we have previously recommended that programs fully mature technologies to a TRL 7 prior to passing Milestone B and entering the engineering and manufacturing development phase of the Defense Acquisition System. Under current law and DOD policy, DOD generally only needs to mature technologies to a TRL 6 by Milestone B. For shipbuilding programs, this is often aligned with the start of detail design. TRL 7 represents a major step up from TRL 6, requiring demonstration of an actual system prototype in a realistic environment such as in an aircraft, vehicle, or space. Examples include testing the prototype in a test bed aircraft. We have previously identified that demonstrating technologies in a realistic environment provides a higher level of technology understanding and reduces risk prior to starting product development. DOD has historically disagreed with this recommended practice. c The Navy does not plan to achieve a TRL 7 for the advanced propulsor bearing until after the lead submarine is delivered in 2027. d The propulsor, X-stern planes, and propulsor shaft reached a TRL 7 prior to the Navy’s first matrix submission in January 2018. Page 6 GAO-19-497 Columbia Class Submarine Columbia Design and Construction Approach Two shipbuilders—General Dynamics Electric Boat (Electric Boat) and Huntington Ingalls Industries Newport News (Newport News)—design and build nuclear submarines. 12 Electric Boat is the prime contractor for both design and construction of the Columbia class program, with Newport News serving as a subcontractor. Similar to the Virginia class program, each shipbuilder will construct segments of the submarine, but Electric Boat will complete final outfitting and deliver the submarines to the Navy. The Navy awarded a detail design contract in September 2017 to Electric Boat for work including completion of the submarine’s design, component and technology development, and prototyping efforts. The detail design process for the Columbia class program encompasses three activities, which began after the Navy set the technical requirements for the submarine in 2016: • Arrangements outline the steel structure and routes distributive systems—such as electrical or piping systems—throughout the submarine. At this time, the shipbuilder generates a three-dimensional computer-aided design model for the area. • Disclosures complete the design work for even the lowest-level items of the submarine, including material information. After these are completed, the shipbuilder can begin ordering material and long lead items for the submarine. • Work instructions are three-dimensional electronic products that shipyard workers use to construct the submarine. Figure 3 illustrates the design phases for the Columbia class program. 12 Electric Boat and Newport News both construct Virginia class attack submarines. Page 7 GAO-19-497 Columbia Class Submarine Figure 3: Design and Construction Phases for the Columbia Class Submarine Program The shipbuilder will design and construct Columbia class submarines in six large hull segments, referred to as super modules, a method also used to construct most of the Virginia class submarines. During construction, the modules will largely be outfitted with systems and connections prior to being attached together during final assembly. According to the shipbuilder, this method is more efficient than outfitting the hull after it is constructed because more workspace is available to install equipment. Figure 4 illustrates the super modules within the submarine. 13 13 DOD identified specific information about critical technologies as sensitive. As such, this information was omitted from this report. Page 8 GAO-19-497 Columbia Class Submarine Figure 4: Super Modules and Their Respective Functions for the Columbia Class Submarine Program Cost Estimating A reliable cost estimate is critical to program success. It provides the basis for informed investment decision making, realistic budget formulation and program funding, meaningful progress measurement, proactive course correction when warranted, and accountability for results. GAO’s Cost Estimating and Assessment Guide states that reliable cost estimates reflect four characteristics, which encompass 19 best practices. 14 These characteristics—comprehensive, well documented, accurate, and credible—are shown in table 2. Table 2: Characteristics of a High-Quality, Reliable Cost Estimate Comprehensive All costs of a program over its full life cycle, from inception through design, development, deployment, and operation and maintenance to retirement are included. A program should be completely defined, reflect the current schedule, and be technically reasonable. All cost-influencing ground rules and assumptions are documented. Well documented Detailed documentation captures the source data used, the calculations performed and their results, and the estimating methodology. Information should allow for easy replication and updating. Accurate Estimate is based on an assessment of most likely costs. Results are unbiased and not overly conservative or optimistic. Estimate should be updated regularly to reflect significant changes in the program and actual costs. Credible Cost drivers are cross-checked and the estimate is compared with an independent cost estimate to validate that different methodologies produce similar results. Sensitivity, risk and uncertainty analyses are performed. Limitations are discussed. Source: GAO. GAO-19-497 14 GAO-09-3SP. Page 9 GAO-19-497 Columbia Class Submarine For Navy shipbuilding programs, including the Columbia class, several different entities are involved in cost estimating: • The Naval Sea Systems Command (NAVSEA) Cost Engineering and Industrial Analysis Group develops the program life-cycle cost estimate, which is an estimate accounting for the total cost to the government of acquisition and ownership of a system over its full life. • NCCA develops an independent cost assessment for certain Navy programs, such as the Columbia class program, at milestone events in the defense acquisition system. This assessment is not a separate estimate, but rather a review of the NAVSEA program life-cycle cost estimate. • A cost review board, comprised of multiple Navy offices, establishes a service cost position based on their review of the program life-cycle cost estimate and the independent cost assessment. 15 • The Office of the Secretary of Defense’s CAPE conducts or approves independent cost estimates for major defense acquisition programs. Independent cost estimates are statutorily required for major defense acquisition programs at milestone events. 16 • The milestone decision authority, which in the case of the Columbia class program is the Under Secretary of Defense for Acquisition and Sustainment, reviews the service cost position and independent cost estimate and selects the cost estimate to baseline and fund the program. 17 The most recent milestone event for the Columbia program was the Milestone B decision in January 2017, where the program received approval to proceed to the next acquisition phase—engineering and manufacturing development, which includes detail design of the lead 15 Navy documents also refer to the service cost position as the component cost position. 16 10 U.S.C. § 2334. Independent cost estimates are conducted in advance of Milestone A or Milestone B certification, and in advance of the decision to enter low-rate initial production or full-rate production. 17 The milestone decision authority is the sole and final decision authority for a major defense acquisition program. Department of Defense Instruction 5000.02, Operation of the Defense Acquisition System para. 5.a(4)(a) (Jan. 7, 2015) (incorporating change 4, Aug. 31, 2018). At the time of the Milestone B decision in January 2017, the Under Secretary of Defense for Acquisition, Technology, and Logistics served as the milestone decision authority. As a result of recent acquisition reform, the Under Secretary of Defense for Acquisition and Sustainment is now the program milestone decision authority. Page 10 GAO-19-497 Columbia Class Submarine submarine. In a memo documenting that decision, the milestone decision authority noted that significant development risks remain for the Columbia program and cost control must remain a priority. To limit program cost growth, the milestone decision authority established an affordability cap: the average submarine procurement cost should not exceed $8.0 billion in constant year 2017 dollars. 18 Figure 5 summarizes the cost estimating process for the Columbia class program’s Milestone B review. Figure 5: Cost Estimating Process for the Columbia Class Program’s Milestone B Review 18 Constant-year dollars removes the effects of economic inflation and outlay rates. Page 11 GAO-19-497 Columbia Class Submarine Navy Is Managing an Aggressive Build Schedule, but Early Design and Construction Challenges Signal Schedule Risk The Navy is attempting to mitigate an aggressive schedule for lead submarine construction by (1) setting a goal to mature a significant amount of the submarine’s design prior to the start of construction and (2) beginning advance construction of submarine modules prior to October 2020. The shipbuilder is working to improve design performance and would have to maintain this increased pace to achieve its design goal, which is necessary to mitigate schedule risk associated with constructing the lead submarine. This may prove challenging as it must complete an increasingly higher volume and complexity of design products. At the same time, the Navy is continuing to develop several critical technologies and recent manufacturing defects with the integrated power system and missile tubes are among the challenges that the Navy is facing in ensuring timely delivery of critical components to the shipyard. Finally, to achieve Columbia’s aggressive construction schedule, while simultaneously building Virginia class submarines, the shipbuilder is working to ensure that it has sufficient shipyard capacity—including new facilities, additional suppliers, and an increased workforce. Shipbuilder Would Have to Maintain Its Increased Pace to Meet Its Design Maturity Goal and Reduce Schedule Risk The shipbuilder has failed to achieve its planned rates for completing design arrangements and disclosures to meet its design maturity goal in recent months—hampered by implementation of a new design software tool and an insufficient number of designers to meet monthly design completion rates. As we reported in December 2017, the Navy’s priority is to complete a high level of design—specifically, 100 percent of design arrangements and 83 percent of design disclosures—by the start of lead submarine construction in October 2020. By maturing the design before beginning construction on the lead submarine, the Navy is attempting to mitigate the risk of costly rework from design changes and subsequent delays to the Columbia class program’s 84-month construction schedule, which the Navy has acknowledged is aggressive. The Navy established the design maturity goal for Columbia based on lessons learned from the Virginia class program, when the shipbuilder began constructing the lead submarine with only 76 percent of arrangements and 43 percent of disclosures completed and, subsequently, realized 21 percent cost growth. Since the shipbuilder began work on the detail design, it has generally met its overall goal of completing the arrangements on schedule. As detail design continues, however, the shipbuilder is transitioning from relatively simple designs for the hull to the more complex designs for the submarine’s internal systems, increasing the pace needed to complete the remaining designs, as shown in figure 6. Page 12 GAO-19-497 Columbia Class Submarine Figure 6: Shipbuilder’s Workload Increases as Design Efforts on the Columbia Class Submarine Program Continue Navy officials stated that design disclosures are generally considered the most challenging phase of design work, where the shipbuilder specifies the lowest-level items and defines all aspects of the submarine. The shipbuilder has to maintain this increased pace in order to achieve the design maturity goal by the start of lead submarine construction. However, the shipbuilder’s design progress in completing disclosure products has fallen short of its plan in recent months as the planned pace and complexity of the design has increased. Using data from the program’s cost performance reports, we analyzed the shipbuilder’s monthly design progress according to a schedule performance index that measures the value of the work completed against the work scheduled. For example, if the schedule performance index is less than 1.00, then the shipbuilder has completed less than a dollar’s worth of work for each dollar that was scheduled. As shown in figure 7, since January 2018, schedule performance has consistently fallen below 1.00. Page 13 GAO-19-497 Columbia Class Submarine Figure 7: Current Monthly Design Schedule Performance for the Columbia Class Submarine Program Note: This schedule performance index reflects the actual cost of work completed in each month compared to the originally budgeted cost for that work. For example, if the schedule performance index is greater than 1.00, then the shipbuilder has completed more than a dollar’s worth of work for each dollar that was scheduled. If the schedule performance index is less than 1.00, then the shipbuilder has completed less than a dollar’s worth of work for each dollar that was scheduled. Both DOD and Navy officials attributed the shipbuilder’s design delays to challenges adapting to a new design software tool. Beginning with the Columbia class program, the shipbuilder transitioned to a new customized software tool for design and construction because its prior software was no longer supported by the original developer. However, the shipbuilder has experienced problems developing the tool, which has resulted in slower progress to complete both design arrangements and disclosures, as certain aspects of the software’s functionality were delayed. Navy officials stated that, as of June 2018, they believe that design software functionality was performing at a level that no longer impeded design progress. While the designers have gained proficiency with the new design tool to complete arrangements and disclosures, according to Navy officials, the shipbuilder is now facing similar challenges using the tool to generate work instructions. Navy program officials also stated that the shipbuilder has not delivered some of the software functionality needed to produce work instructions as scheduled. Further, Navy officials noted that Page 14 GAO-19-497 Columbia Class Submarine the process to create work instructions from completed disclosures takes longer with the new design software so the shipbuilder has begun generating work instructions earlier. According to Navy officials and shipbuilder representatives, the shipbuilder hired 150 additional designers in an effort to recover its design schedule and meet future monthly design goals. However, adding designers to recover and maintain the shipbuilder’s design schedule ultimately increases the program’s design costs. Similar to the schedule analysis above, we used data from cost performance reports to analyze the shipbuilder’s monthly design progress according to a cost performance index that measures the budgeted value of the work completed against what it actually costs to complete it. For example, if the cost performance index is less than 1.00, then less than a dollar’s worth of work has been completed for each dollar spent. As shown in figure 8, the shipbuilder’s cost performance has consistently fallen below 1.00 since December 2017. Figure 8: Cumulative Design Cost Performance for the Columbia Class Submarine Program Note: This cost performance index reflects the actual cost of work completed in each month compared to the originally budgeted cost for that work. For example, if the cost performance index is greater than 1.00, then more than a dollar’s worth of work has been completed for each dollar spent. If the cost performance index is less than 1.00, then less than a dollar’s worth of work has been completed for each dollar spent. Page 15 GAO-19-497 Columbia Class Submarine If the shipbuilder cannot address challenges associated with using the software tool to generate work instructions discussed above, it will likely need additional design hours in the future, resulting in higher costs in order to mature the design on schedule. Navy’s Use of Advance Construction to Mitigate Aggressive Schedule Is Not without Risk Navy officials and shipbuilder representatives expect to mitigate risks associated with the Columbia construction schedule by accelerating the building of certain components more than a year in advance of the formal start of construction. They anticipate that this advance construction strategy will allow them to gain 2 months of schedule margin for final assembly and testing prior to delivery of the lead submarine. Starting in December 2018, the shipbuilder will begin constructing modules of the submarine as part of its advance construction effort. In 2017, we reported that the Navy had planned to begin advance construction for four of the submarine’s six super modules, but since our report was issued, it now plans to begin construction on all six super modules including building components like the stabilizers, impulse tanks, and others. 19 Figure 9 shows the start of advance construction for each super module. 19 GAO-18-158. Page 16 GAO-19-497 Columbia Class Submarine Figure 9: Start of Component and Equipment Advance Construction for Each Super Module for the Columbia Class Submarine Note: Construction of missile tubes and other components for the common missile compartment began in 2016 to support the United Kingdom’s Dreadnought class submarine program, which are scheduled to deliver in 2028, and the lead submarine for the Columbia class program. Navy officials estimate that the current advance construction efforts will require approximately 631,000 labor hours. In addition, advance construction efforts would require that the Navy accelerate delivery of equipment provided to the shipbuilder for installation on the submarine, such as pumps and valves. Shipbuilder representatives stated that a lesson learned from the Virginia class program was that construction of certain complex components should begin as early as possible if capability requirements and designs are stable. However, based on its plan, the shipbuilder will begin advance construction having completed less than 40 percent of the total design disclosures for the Columbia class submarine, as shown in figure 10. Page 17 GAO-19-497 Columbia Class Submarine Figure 10: Navy’s Design Disclosures and Construction Plans for the Columbia Class Submarine Program The number of disclosures completed at the start of advance construction is less than half of those the shipbuilder plans to complete by the start of lead submarine construction in October 2020. Navy officials stated that they believe the risk associated with beginning construction with a less mature overall design is mitigated because the program selected components for advance construction that are well understood and unlikely to be affected by design changes, like ballast tanks, decking, and hull segments. In addition, Navy officials stated that they will not begin construction on the component or hull unless the arrangements associated with the structure of that area of the submarine are complete. However, based on the shipbuilder’s design plans, the arrangements and disclosures of adjoining areas of the super module may not be complete, which could negatively affect construction. Specifically, the shipbuilder’s design plans indicate that it will have completed 100 percent of disclosures for only one super module at the start of advance construction. As we have found in our prior work, proceeding with construction despite having completed fewer designs than planned increases the likelihood of design changes later that may, in turn, require costly and time-intensive re-work to change components that have Page 18 GAO-19-497 Columbia Class Submarine already been built. 20 Shipbuilder representatives acknowledged that there is risk in starting construction of some components prior to completing the design for individual super modules or the entire submarine. However, shipbuilder representatives stated that they believe this risk is reduced by only starting construction on components for which the disclosures are complete. Recent Challenges with Critical Technologies Have Reduced Available Schedule Margin While ship design is underway, the Navy is continuing to develop and mature the critical technologies related to the Columbia class program. 21 While these critical technologies are not required at the shipyard for several years, recent challenges have eroded available schedule margin, as illustrated below: • Integrated Power System: In 2017, we reported that the Navy experienced manufacturing problems associated with the integrated power system. 22 We found that the Navy continues to experience problems with the electric drive of the integrated power system that could potentially affect construction of the lead submarine. A manufacturing defect that affected the system’s first productionrepresentative propulsion motor required extensive repair that consumed 9 months of schedule margin at the land-based test facility. The Navy now plans to test the motor at the same time it had originally scheduled to make any final design changes before starting production. This could constrain opportunities to implement timely, corrective actions if problems are discovered during testing. • Common Missile Compartment: Navy officials stated that, in July 2018, the shipbuilder identified substantial weld defects in missile tubes from one of three tube suppliers and resulted in investigations of the missile tubes from all suppliers. These defects were discovered after seven tubes in various stages of outfitting had already been delivered to the shipyard and five additional tubes under production have been affected. Navy program officials stated defects occurred because inexperienced welders performed the complex work and inspectors at the supplier’s facility subsequently failed to identify the 20 GAO-18-238SP. 21 DOD deemed that a description of the development status of each technology is sensitive. As such, specific information about the critical technologies was omitted from this report. 22 GAO-18-158. Page 19 GAO-19-497 Columbia Class Submarine defects. While the Navy and shipbuilder are still determining the cost and schedule impacts of the weld defects, program officials estimated that addressing this issue will consume up to 15 of the 23-month schedule margin for these components. In addition, program officials stated that the Navy likely will be responsible for some of the cost associated with investigating the root cause of the defects and risk mitigation efforts going forward. Given the erosion of available schedule margin, there is less time available to address issues without resulting in schedule delays. For example, the shipbuilder’s construction plans for two super modules do not include schedule margin to accommodate any delays that may occur as the technologies are matured and detail design is completed. One of these, the stern super module contains three technologies that are not fully mature—the integrated power system, stern area system, and advanced propulsor bearing. The integrated power system is not expected to reach full maturity until October 2019 and the remaining two technologies will not be mature until after the shipbuilder begins construction on the lead submarine, not including those components that begin advance construction years earlier. Without schedule margin to accommodate any changes or issues, any delays in delivering equipment to the shipyard on time could disrupt the shipbuilder’s construction sequence for the lead submarine. Shipbuilder Is Facing Oversight and Capacity Challenges in Preparation for Columbia Construction To meet the Navy’s aggressive construction schedule for the lead submarine, the shipbuilder has to ensure that it has the capacity to meet a substantially higher workload and effectively balance Columbia and Virginia class construction. At the same time as construction on Columbia begins in 2020, the shipbuilder will also have begun constructing two modified Virginia class attack submarines per year. 23 To accommodate the construction of both submarine classes, the shipbuilder is planning an extensive expansion of its facilities, including new buildings, a pier, an ocean transport barge, and a floating dry dock. The anticipated increases in workload at the shipyard will also require the shipbuilder to manage a higher volume of build materials and an expansion of its workforce. While construction of new facilities is progressing on schedule, according to 23 The Virginia class will include a new payload module that increases its tubes for additional cruise missiles or other systems and adds an additional 84 feet to the submarine’s overall length. Page 20 GAO-19-497 Columbia Class Submarine shipbuilder representatives, it faces other challenges preparing for Columbia class construction. Ensuring Supplier Oversight Achieving the planned construction schedule will require the Navy and shipbuilder to ensure that materials arrive on time and meet quality expectations, but according to Navy officials, supplier oversight has been a challenge for this shipbuilder in the past. Both Navy officials and shipbuilder representatives stated that they are concerned about the capacity of its suppliers to meet the demand for high-quality components given an industrial base that has diminished significantly since previous major submarine construction efforts in the 1980s. Many of the parts and equipment on Columbia class are common with those used on Virginia class submarines but, in other instances, suppliers are producing components for the first time after a considerable break, such as missile tubes that have not been produced since the early 1990s. Navy program officials and shipbuilder representatives stated that they monitor supplier capacity and quality—among other areas—and they have several methods to intervene if a supplier is not able to perform as needed. The shipbuilder and the Navy have formed a group to assess the three primary areas of supplier performance: 24 • Capability: includes the uniqueness of the supplier’s product on the market, challenges in shifting to a different supplier due to intellectual property rights or technical knowledge, and the ability for the supplier to sustain their own supply base. • Capacity: includes the supplier’s ability to increase production without decreasing quality, maintain that capacity over the program’s production, their financial dependence on Navy programs for revenue, lead time needed to meet new orders, and the capacity of their own suppliers. • Cost: includes the costs of increasing production spread out across demand from Navy programs. In 2017, the shipbuilder assessed its supplier base using these areas, identified the criticality and risk of each supplier based on their potential impact to the program and potential alternate suppliers, and conducted a gap analysis comparing the supplier’s current performance to the 24 Navy officials stated that cyber security of suppliers is also included as part of their oversight. Page 21 GAO-19-497 Columbia Class Submarine program’s desired performance. Based on the results of the analysis, the shipbuilder identified and is monitoring at-risk suppliers in coordination with the Navy to determine if immediate intervention is needed, such as investing in new facilities for the supplier, improving manufacturing workflow, or finding new sources of material from that supplier. Despite these efforts, supplier oversight remains an issue, because—in the instance of the missile tube welds mentioned above—the shipbuilder focused on managing certain anticipated risks, as opposed to actively managing the supplier’s quality and performance with on-site independent inspections, according to Navy officials. In response to the missile tube issues, the shipbuilder has proposed additional supplier oversight by assessing the need for on-site inspection teams depending on the risk each supplier poses to the program. Navy officials stated that they have begun some assessments but, as of March 2019, had yet to determine who will pay for this additional oversight. We plan to more fully assess the Navy and shipbuilder’s oversight of its suppliers for the Columbia class program in future work. Building Workforce Capacity and Capability According to shipbuilder representatives, the start of lead submarine construction for the Columbia class, combined with expanding Virginia class construction, increases the demand for hiring and retaining skilled workers at levels not seen at this shipyard since the 1980s. Navy officials expressed concerns about the risk of adding large numbers of new workers, including an influx of inexperienced welders and inspectors— issues that also contributed to the defects in missile tubes discussed above. To support growing workload from both the Columbia and Virginia submarine programs, the shipbuilder plans to increase workforce at its two facilities over the next decade: by 66 percent at Quonset Point, Rhode Island—where the components and individual submarine modules will be constructed—and 174 percent at Groton, Connecticut—where the super modules will undergo final outfitting and assembly. To meet this increased demand in a skilled workforce, the shipbuilder assessed future demographic trends in the area surrounding its facilities and found that, while sufficient labor will likely be available, more training will be necessary. Consequently, the shipbuilder established internal and external training programs and partnerships with educational institutions in the area to grow the qualified workforce in time to begin lead submarine construction in October 2020. The influx of inexperienced workers can temporarily decrease construction efficiency as compared to a current, more experienced workforce. For example, when the Virginia class program expanded its Page 22 GAO-19-497 Columbia Class Submarine workforce to build a second submarine each year, the addition of new staff contributed to an 8 percent decrease in cost efficiency for the program. Shipbuilder representatives at one production facility have already reported reduced efficiency following increased hiring of new workers. The shipbuilder’s goal is to maintain an average of 8 years of experience for workers in core trades, such as welding. However, the shipbuilder’s projections show that the new workforce ramp-up at the Groton facility will reduce workers’ average experience from 13.1 years to a low of 5.6 years in 2028—just after the shipbuilder plans to deliver the lead Columbia class submarine. If workforce growth or efficiency assumptions are not met, the shipbuilder may resort to scheduling overtime work or outsourcing some activities to meet the program’s construction schedule, which would have cost impacts for the program. Columbia Class Cost Estimate Is Not Reliable and Does Not Reflect Program Risks The Navy’s procurement cost estimate of $115 billion to construct Columbia class submarines is not reliable because it does not reflect likely program costs and risks. 25 We assessed the Columbia class cost estimate by comparing it with the best practices identified in GAO’s Cost Estimating and Assessment Guide. 26 We found that it substantially met the criteria for the comprehensive characteristic of a reliable cost estimate, and partially met the criteria for the remaining characteristics, including accurate and credible. In particular, we found that the cost estimate • does not accurately reflect program costs because it is based on overly optimistic labor hour assumptions, and • is not fully credible because while the Navy conducted risk and sensitivity analyses to test the likelihood of achieving its assumptions, it selected a specific cost estimate that informs the program’s budget which does not include any margin in case those assumptions are not achieved. In addition, the cost estimates and assessments conducted by other entities produced a range of results, indicating that there is a high degree of uncertainty regarding program costs. See appendix III for the full results of our assessment of the Navy’s cost estimate. 25 The $115 billion procurement cost estimate does not include the $13.0 billion research, development, test, and evaluation costs or $0.2 billion military construction costs. 26 GAO-09-3SP. We consider a cost estimate to be reliable if the overall assessment rating for each of the characteristics is substantially or fully met. Page 23 GAO-19-497 Columbia Class Submarine Navy officials stated they plan to update the Columbia class cost estimate in support of DOD’s decision to authorize construction of the lead submarine and this decision is expected to occur in summer 2020. Navy officials also stated that they expect that the cost estimate will be complete by the end of fiscal year 2019, followed by an independent cost assessment to support the authorization decision. However, this timeframe does not provide assurance that both the update and the independent assessment will be complete before the Navy requests funding from Congress for lead submarine construction, as part of its fiscal year 2021 budget request, which could be submitted as early as February 2020. If so, decision makers may be basing their decisions on outdated or incomplete information. Columbia’s Cost Estimate Is Not Accurate Because It Relies on Overly Optimistic Labor Hour Assumptions The Columbia class cost estimate relies on optimistic program assumptions and does not reflect the likely labor hour costs that the Navy will incur to construct the submarines. As part of our assessment of the Columbia program cost estimate, we found that it did not fully meet the best practices for an accurate estimate. A cost estimate is considered accurate when it is based on an assessment of the most likely costs—that is, it is neither overly conservative nor overly optimistic. The Navy estimates that it will need $115 billion to design and construct 12 submarines and NAVSEA cost estimators identified labor costs as a primary source of cost risk. 27 As discussed below, if the program’s optimistic assumptions are not realized, the program may require more funding than originally planned to construct the Columbia class. The Navy anticipates that it will need 12 million labor hours to directly construct the lead submarine—referred to as touch labor. 28 This represents 17 percent fewer labor hours than what was needed for the lead Virginia class submarine, when adjusted for weight differences. To develop this estimate, NAVSEA estimators relied heavily on historical touch labor hour data from the construction of the lead Virginia class submarine and cost data from the Ohio class submarine program for unique ballistic submarine components, such as missiles. NAVSEA estimators took the following steps to develop the Columbia lead submarine estimate: 27 The $115 billion estimate reflects total procurement costs and does not include research, development, test, and evaluation costs. 28 The Navy estimates that in total the lead submarine will require 25.3 million labor hours. Page 24 GAO-19-497 Columbia Class Submarine • In general, heavier ships cost more to construct, so NAVSEA cost estimators calculated a weight-adjusted estimate based on Virginia class labor hours to account for the heavier weight of the Columbia class. This resulted in an initial estimate of 14.5 million touch labor hours for the lead submarine. • NAVSEA cost estimators then made numerous adjustments in the cost estimate that reduced the expected number of labor hours based on multiple assumptions that differences in the design and construction process would lead to more efficient construction of Columbia class submarines than previous submarine classes. These adjustments subsequently decreased the estimate to 12 million touch labor hours for the lead submarine. • NAVSEA cost estimators then used the lead Columbia submarine estimate as the basis to calculate labor hours for follow-on submarines, estimating an average of 8.9 million touch labor hours. Figure 11 illustrates NAVSEA’s touch labor hour calculation for the lead submarine. Figure 11: Touch Labor Hour Calculation for Lead Columbia Submarine, Based on Lead Virginia Class Submarine (millions of hours) Note: The weight-adjusted estimate accounts for the differences between the Virginia attack submarine and the Columbia ballistic submarine. The final estimate accounts for anticipated improvements in the shipbuilder construction process. Page 25 GAO-19-497 Columbia Class Submarine However, the touch labor hour estimate is overly optimistic—with assumptions on construction efficiencies that are either unsubstantiated or unprecedented compared to Virginia class and other shipbuilding historical data. Compared to the Navy’s estimate, Columbia’s estimated touch labor hours, as calculated by other organizations, are more conservative. For example, CBO questioned the Navy’s assumption that ballistic submarines are less expensive to build than attack submarines, after accounting for weight differences and estimated that for the overall class, including the lead and follow-on submarines, the Navy would more likely realize an 8 percent reduction rather than the 19 percent reduction estimated by the Navy. 29 While the shipbuilder will likely realize some efficiencies from initiatives to improve design and construction processes, our analysis of the Navy’s assumptions used to develop the cost estimate indicates that they likely overstate the labor hour reduction the shipbuilder can realistically achieve. These assumptions include that the program (1) achieves its design goals at the start of construction; (2) is constructed more efficiently than Virginia class submarines; and (3) successively reduces the number of hours needed to construct follow-on submarines. If these assumptions are not realized, overall program costs could be higher than the Navy’s procurement estimate of $115 billion. Navy officials stated that they believe that these assumptions are valid and that the cost estimate is achievable. However, our assessment indicates that the assumptions for the cost estimate are overly optimistic, as discussed below. Risk of Unrealized Design Goals The Navy’s cost estimate does not reflect the risk that the shipbuilder may not achieve its planned design completion goals. As we reported above, design performance to date has slowed and the shipbuilder has had to hire additional designers in an effort to mature its design on schedule. NAVSEA cost estimators stated that they recognize that an incomplete design at the start of ship construction was a significant driver of cost growth on other shipbuilding programs. For the Columbia class, NAVSEA cost estimators assumed that achieving the design maturity goal would eliminate 2 million labor hours by reducing costs associated with rework and out of sequence work. In October 2018, NCCA officials stated that they recently reviewed shipbuilder data and the expected design completion at construction start continues to range between 55 and 75 29 Congressional Budget Office, An Analysis of the Navy’s Fiscal Year 2017 Shipbuilding Plan, (February 2017). Page 26 GAO-19-497 Columbia Class Submarine percent complete—the same range that they estimated in their independent assessment. While this lower rate would be an improvement over the Virginia class program, it would still fall short of the 84 percent assumption built into the cost estimate. 30 If the shipbuilder does not complete the design at its planned rate and begins construction with a less mature design, it may need additional labor hours to construct the ship, resulting in increased program costs. Overly Optimistic Estimate of Efficiencies The Navy’s cost estimate includes assumptions that reduce Columbia’s estimated touch labor hours due to efficiencies from constructing Columbia and Virginia class submarines concurrently, an assumption with which the shipbuilder does not agree. NAVSEA cost estimators calculated a 1.1 million-labor hour reduction, attributing the decrease to efficiencies gained from constructing multiple submarines at the same time, basing their assessment on shipbuilder estimates of the Virginia class. However, it is unclear how increased shipyard production would result in fewer labor hours to construct each submarine. Shipbuilder representatives stated that rather than a reduction in touch labor hours, they expect to realize efficiencies from increased production primarily from reduced overhead rates and material costs. Further, the Navy’s independent assessment analyzed labor hour data for Virginia class construction and found that there was no correlation between the number of submarines constructed at a time and the total number of labor hours. However, increasing shipyard production to include both Virginia and Columbia class construction may increase schedule risk for the shipbuilder, which could result in additional costs if the shipbuilder does not achieve planned increases in its workforce and facility upgrades. When the number of Virginia class submarines under construction increased, both shipyards experienced inefficiencies due to poorly planned ramp-up requirements. In addition, DOD officials stated that problems encountered on one program could affect the other as the shipbuilder is relying on the same workforce and vendor base for both programs. The Navy’s cost estimate also assumed construction efficiencies— because the Columbia class submarine will be less dense than the Virginia class submarine—another assumption with which the shipbuilder does not agree. Navy officials stated that less dense submarines are less 30 The design completion goal has since been updated to 83 percent. Page 27 GAO-19-497 Columbia Class Submarine costly to construct as the additional space within the hull allows for faster and more efficient work. However, the shipbuilder conducted analysis to compare the density of various areas of the Virginia class and Columbia class submarines and found that areas had very similar density. Specifically, there was only a 1 percent and 3 percent difference, between the forward compartments and aft compartments, respectively—some of the more complex areas of the submarine. If the shipbuilder does not realize these construction efficiencies, more total labor hours would be required to construct the submarine, resulting in increased cost. Learning Curve Assumption The Navy’s cost estimate assumes that the costs for follow-on Columbia class submarines will decrease at a rate that may overstate the improvements the shipbuilder can realistically achieve. The Navy expects the number of labor hours to construct Columbia class follow-on submarines to decrease based on an assumed learning curve rate. Learning occurs when construction is consistent and continuous and the shipbuilder learns how to do repetitive tasks more efficiently. The decrease in the number of expected labor hours is expressed as a learning curve rate, where a lower percentage indicates that less labor is required for follow-on units. NAVSEA cost estimators calculated a learning curve of 88.9 percent for Columbia class submarines. A learning curve indicates that as the number of units doubles, unit cost decreases by a constant percentage. In this case, the cost estimate assumed that the fourth submarine would require only 88.9 percent the amount of labor to build the second submarine. NAVSEA’s assumption may overstate the potential learning rate that Columbia can expect to achieve. The first four Virginia class submarines, hull numbers SSN 774 through 777, incorporated modular construction techniques where submarines were built in 10 modules. The next six Virginia class submarines, hull numbers SSN 778 through 783, were constructed using four modules. As a result of the improvements in the modular construction process, construction across the first ten submarines was not consistent, which is a condition that is necessary to determine the learning curve rate. Therefore, there is no way to determine what share of the labor hour reduction on later submarines was due to learning or process improvements. Rather, SSN 778, the first Virginia class submarine to use the four modular construction approach is a better starting point to determine the shipbuilder’s capacity for learning. The Navy’s independent assessment included a separate learning curve analysis for Virginia class submarine hulls SSN 778 through 791 and calculated a potential learning curve rate of 93.9 percent. A learning curve assumption applies to all follow-on submarines and has a cumulative Page 28 GAO-19-497 Columbia Class Submarine effect on the number of labor hours and, ultimately, the cost of these submarines. In the case of the Columbia program, the rate will apply to the second through twelfth submarines. Figure 12 shows how the difference in the learning curve rate can affect the estimated labor hours for follow-on submarines. Figure 12: Learning Curve Effect for Estimated Hours of Columbia Class Follow-on Submarines Based on Virginia Class Submarines Therefore, a small change in the assumed learning curve rate can have a significant effect on the cost estimate for follow-on submarines. For example, the Navy’s independent assessment of the cost estimate calculated that production costs could increase by $3.59 billion in constant year 2010 dollars if a learning curve of 93.9 percent was realized, rather than the 88.9 percent rate estimate. Our previous work on Navy shipbuilding performance has shown that the Navy has consistently Page 29 GAO-19-497 Columbia Class Submarine underestimated the costs for follow-on ships, with costs for Virginia class submarines underestimated by close to 40 percent. 31 Columbia Cost Estimate Is Not Credible Because It Does Not Adequately Account for Program Risks The Columbia program cost estimate did not fully meet the best practice criteria to be considered credible because, in part, Navy program management did not sufficiently account for program risks when it selected the final estimate. 32 To determine the estimate’s credibility, we examined the extent to which • NAVSEA cost estimators tested, among other things, the sensitivity of key cost elements such as labor hours and conducted uncertainty analyses to quantify risks; and • an independent cost estimate and assessment were conducted by groups outside the acquiring organization (specifically, CAPE and NCCA) to determine whether other estimating methods produced similar results. We found that while the Navy program management’s $115 billion procurement cost estimate for the Columbia class is overly optimistic in some of its assumptions, the estimate does not reflect any contingency to offset the likely effects of not meeting the assumptions, which is a best practice. In addition, the independent cost estimates and assessments conducted by other organizations had varying results, indicating the high level of uncertainty regarding Columbia program costs. We further address these issues below. Sensitivity and Risk Analysis Indicate Insufficient Cost Risk Coverage Navy leadership’s decision to select $115 billion as the program cost estimate means that there is no margin in the program budget to cover likely program costs if risks are realized. The best practices identified in GAO’s cost estimating guide state that the results of a risk analysis should be used to select a cost estimate that is sufficient to manage program risks. NAVSEA cost estimators conducted a risk analysis to 31 GAO-18-238SP. The percent cost difference of follow-on ships is calculated by comparing the Navy’s initial average procurement unit cost at program start in its Selected Acquisition Report against the average initial cost estimate in the Navy’s budget request for all follow-on ships in the class. Increases in follow-on ship costs could be due to adding upgraded capabilities, but data account for increases in quantities. 32 See app. III for a list of cost estimating best practices as identified in GAO’s Cost Estimating and Assessment Guide, GAO-09-3SP. Page 30 GAO-19-497 Columbia Class Submarine identify and quantify program risks, and determined the effects of changing key cost driver assumptions and factors—important steps in creating a high quality estimate. 33 However, while NAVSEA cost estimators identified 54 risk parameters for construction costs, we found that some of the inputs for these ranges resulted in a cost estimate that understates the potential impact of program cost risks. For example, the risk ranges do not sufficiently account for the issues we identified above, including that • increased shipyard construction could result in similar inefficiencies that occurred in the production of the Virginia class, requiring more labor hours than estimated; and • shipbuilder workforce ramp-up could result in decreased efficiency and quality due to the influx of new workers even greater than the issues observed on the Virginia class when shipyard construction increased. For other risk parameters, such as cost of material provided by the shipbuilder, the cost estimate documentation was not sufficient for us to analyze whether the risk ranges included in the estimate were reasonable (i.e., not overly optimistic or pessimistic). As a result, we could not determine whether the risk analysis sufficiently captures the risk of program cost growth, or what the probability is of achieving the $115 billion procurement cost estimate. Further, Columbia’s program management and the milestone decision authority selected $115 billion as the program’s procurement cost estimate, without adjusting for the likelihood of cost growth in the design or construction of Columbia class submarines identified in the risk analysis. As we reported in December 2017, the risk analysis developed by NAVSEA indicated that there is only a 45 percent probability that the overall program cost estimate will be sufficient to cover program costs. 34 The cost estimating best practices identified in our cost estimating guide state that a risk-adjusted cost estimate helps ensure that sufficient funding will be available for the expected program costs. 35 Additionally, a 33 As part of the risk analysis, NAVSEA cost estimators identified risk parameters and determined the potential range of outcomes for each parameter. From these ranges, NAVSEA cost estimators conducted a sensitivity analysis which calculated the potential cost impact to the Columbia program if identified risks are realized. 34 GAO-18-158. 35 GAO-09-3SP. Page 31 GAO-19-497 Columbia Class Submarine risk-adjusted cost estimate is consistent with federal internal control standards, which indicate that risk mitigation efforts should be selected to sufficiently respond to risks. 36 However, Columbia program officials stated that they believe program risks can be managed within the current cost estimate—which they consider to be conservative—as it does not account for all of the program’s potential cost savings. Specifically, the Navy anticipates that the program will realize up to $1.9 billion in additional cost savings from use of authorities associated with the National Sea-Based Deterrence Fund (the Fund), such as the authority to purchase components for multiple submarines—which we discuss later in this report. As a result, the program office estimate represents the program manager’s cost goal for the Columbia program, rather than the riskadjusted estimate. Even if the Navy were to achieve the full anticipated $1.9 billion savings, these savings represent only 1.5 percent of program costs. Such cost savings are unlikely to cover program cost overruns for a high-risk program, such as Columbia, given that historically shipbuilding programs experience 27 percent cost growth. As the current estimate does not include any reserves for cost overruns, program management is relying on these potential savings to help mitigate likely cost growth. Independent Cost Reviews Indicate Varying Results Several entities have conducted independent reviews of the Columbia program cost estimate, with varying results. CAPE conducted an independent cost estimate and NCCA conducted an independent cost assessment of the program estimate in support of the Columbia class program’s Milestone B review. 37 CAPE’s independent cost estimate was 3 percent lower than the Navy’s service cost position, which it stated was due to CAPE’s use of lower shipyard labor rates. However, NCCA’s assessment did not produce similar results as the program cost estimate and concluded that the program is at risk of up to $6.14 billion in cost growth. The program manager reviewed the recommendations in the independent cost assessment and determined that the program office estimate appropriately weighs program risks. Navy leadership selected the program office estimate to serve as the Navy’s service cost position because program officials stated that they believe program risks can be managed within the program cost estimate. CBO also conducted a cost estimate and projected that procurement of 12 submarines would be 6 36 GAO, Standards for Internal Control in the Federal Government, GAO-14-704G (Washington, D.C.: September 2014). 37 An independent cost assessment reviews the program life cycle cost estimate methodology and assumptions; it is not a separate cost estimate. Page 32 GAO-19-497 Columbia Class Submarine percent higher than the program estimated. The results of these cost estimates and NCCA’s assessment are summarized in table 3. Table 3: Cost Estimates and Independent Cost Assessments for the Columbia Class Submarine Program (in billions of dollars) Program acquisition a cost estimate Independent cost b estimate 13.0 13.3 115.0 110.9 Research, development, test and evaluation Procurement Military construction Total 0.2 0.2 128.2 124.4 Independent cost c assessment An Analysis of the Navy’s Fiscal Year 2017 d Shipbuilding Plan Cost impact not e calculated 13.5 to 17.9 Up to 121.1 126.7 Not assessed Not assessed Up to 134.3 Up to 144.8 Source: GAO presentation of Department of Defense, Navy, and Congressional Budget Office data. GAO-19-497 Note: An independent cost assessment reviews the program life cycle cost estimate methodology and assumptions; it is not a separate cost estimate. a The Naval Sea Systems Command conducted this estimate. b The Office of Cost Assessment and Program Evaluation conducted this estimate. c The Naval Center for Cost Analysis conducted this assessment. d The Congressional Budget Office conducted this estimate and converted the estimate to then-year dollars. e The Naval Center for Cost Analysis assessed design agent hours and percent design completion but did not report the potential cost impact. As part of the Milestone B review, the milestone decision authority reviewed the service cost position and CAPE’s independent cost estimate. The independent cost assessment was reviewed by Navy leadership as part of the service cost position process and, therefore, was not briefed as part of the milestone review. The milestone decision authority accepted the Navy service cost position and directed the Navy to use this estimate as the basis of its fiscal year 2018 budget request. It also established an $8 billion affordability cap for the average procurement cost of all 12 submarines to control future program costs. Congress May Not Have Up-to-Date Cost Information When Considering Columbia Class Budget Request for Lead Submarine Funding Navy officials stated that they plan to update the cost estimate for the lead submarine in support of a planned Defense Acquisition Board review, in the third quarter of fiscal year 2020. At that point, the Navy will be seeking approval from the milestone decision authority to award the contract for construction of the lead submarine. However, the Navy and DOD’s general timeframes do not provide assurance that the planned update of the cost estimate would be completed prior to the fiscal year Page 33 GAO-19-497 Columbia Class Submarine 2021 budget request, which will include funding for lead submarine construction, as shown in figure 13 below. Figure 13: Estimated Timeline of Future Key Events Leading to Columbia Class Submarine Lead Submarine Authorization and Funding The milestone decision authority has directed CAPE, with assistance from NCCA, to assess the lead submarine cost estimate to support the decision to authorize the Navy to award the contract for lead submarine construction. Since this assessment will occur after the Navy has updated Page 34 GAO-19-497 Columbia Class Submarine the lead submarine cost estimate, it is even less likely that the program budget request will reflect the results from the independent cost assessment. Additionally, the current program cost estimate the Navy developed for the Milestone B review does not reflect the program’s current strategy to use authorities associated with the Fund to achieve cost savings, as discussed further below. The best practices identified in GAO’s cost estimating guide state that cost estimates should be regularly updated and reflect the program acquisition baseline. 38 Updating the cost estimate and risk analysis to include these anticipated savings and current program data would improve its reliability and help ensure that budget requests are sufficient to execute the Columbia program as planned. After we provided our draft report to DOD for comment, Navy officials briefed us on the changes they had made to the program’s estimate to date, stating that they updated the cost risk analysis as part of an internal program review. While the Navy plans to update the lead submarine cost estimate again by the end of fiscal year 2019 to support the Defense Acquisition Board review in the summer of 2020, it has yet to provide specific details on the steps it will take to update this estimate to ensure that it would include likely program costs and risks, such as the cost data it plans to include or the assumptions it may reassess. Further, since the Navy will likely submit its budget request to Congress as early as February 2020, Congress may be asked to authorize and fund lead submarine construction without the benefit of any changes to the estimate that may occur as a result of recommendations stemming from an independent review of the update. Further, although the Navy reports Columbia program cost information to Congress through annual matrices submissions, updates to the program cost estimate will not be reflected in these reports. For example, the Navy plans to report program manager and contractor cost estimates for individual submarines in the matrices once the submarines are under construction. Since these estimates are based on shipbuilder contract performance, they are initially calculated only after construction of each submarine is 15 percent complete, when sufficient data are available to show performance trends. While the Navy plans to award the contract for the lead submarine in October 2020, limited contractor performance data will be available in time for the February 2021 matrix submission. As a result, the earliest opportunity to report on the cost of the lead submarine would be the Navy’s next submission in February 2022, at which point the 38 GAO-09-3SP. Page 35 GAO-19-497 Columbia Class Submarine Navy will have already requested funding for the second and third Columbia submarine. Navy Is Using National Sea-Based Deterrence Fund and Associated Authorities, but Anticipated Savings May Be Overestimated In 2014, Congress created a National Sea-Based Deterrence Fund (the Fund) that provides DOD with greater discretion to fund the design, construction, purchase, alteration, and conversion of the Columbia class. 39 Since then, Congress has provided the Navy with enhanced acquisition authorities to buy and construct submarines and certain key components early, in bulk, and continuously, when using these funds. The Navy anticipates saving over $1.9 billion through use of these authorities, but these savings, which were not included in the Columbia class program’s cost estimate, may be overestimated. Navy Executes Columbia Program through the Fund and Its Associated Authorities Since its inception in 2014, Congress has expanded the special acquisition authorities under the Fund, in part, to allow the Columbia class program to gain economic efficiencies and realize cost savings. The timeline of the establishment of the Fund and legislative changes are shown in figure 14. 39 10 U.S.C. § 2218a. Page 36 GAO-19-497 Columbia Class Submarine Figure 14: Timeline of National Sea-Based Deterrence Fund Legislation The following authorities have been included as part of the use of the Fund: • Economic order quantity: Permits awarding of contracts that provide a quantity of supplies that will result in a total cost and unit cost most advantageous to the government by achieving economic efficiencies based on production economies. • Advance construction: Allows for manufacturing and fabrication efforts prior to ship authorization. • Multiyear procurement authority: Permits a single contract for more than one year of critical components. • Incremental funding authority: Facilitates the purchase of long lead items through partial funding of a contract with the expectation that full funding will be provided later. Using the Fund’s associated authorities, the Navy is able to purchase significant components and start advance construction prior to receiving Congress’s authorization of and funding to purchase each submarine. In total, the Navy will have requested and received $8.6 billion in funding, including 33 percent of funding for the lead submarine, before it receives authorization and funding to begin construction of the lead submarine in October 2020. At that point, the Navy will also have already requested Page 37 GAO-19-497 Columbia Class Submarine funding for the propulsor and advance construction for the second submarine. Under law, the Navy is required to deposit all appropriations for the Columbia class construction and design into the Fund. To date, the Navy has made three deposits from the Shipbuilding and Conversion, Navy account into the Fund, totaling over $1.6 billion. The Navy is using initial deposits of $773 million in fiscal year 2017 and $862 million in fiscal year 2018 for detail design and continuous production of missile tube components. The Navy Comptroller initiates all deposits into the Fund, which are approved by the DOD Comptroller as internal reprogramming actions, as shown in figure 15. Page 38 GAO-19-497 Columbia Class Submarine Figure 15: National Sea-Based Deterrence Fund Deposit Process Anticipated Savings from Use of Fund’s Associated Authorities May Be Overestimated The Navy anticipates achieving over $1.9 billion in savings through the use of the Fund’s associated authorities, but the Navy did not evaluate these savings when it developed the program office cost estimate. Table 4 provides a description of each authority and the Navy’s plans and estimated potential savings resulting from use of the authorities. Page 39 GAO-19-497 Columbia Class Submarine Table 4: Navy’s Plans to Implement Acquisition Authorities Associated with the National Sea-Based Deterrence Fund and Navy’s Estimated Savings Navy’s estimated savings for Columbia (in millions of dollars) Authority Navy’s implementation plans Economic Order Quantity Purchase bulk quantities of items for the Columbia program that are common to other nuclear-powered vessel programs, such as the Virginia class submarine. Appropriations in the Fund will only be used to pay for the purchase of equipment for Columbia submarines. 325 Advance Construction Begin work on some elements of the ship, including the bow, stern and common missile compartment prior to the start of construction. The Navy has used this authority to start portions of Columbia construction work early. 680 Multiyear Procurement Authority Purchase multiple years of material needed for common missile compartment. 560 Purchase multiple years of material needed for critical components: Launcher tubes Propulsors Hull valves Spherical air flasks Other components 273 37 26 10 37 Incremental Funding Authority Award incrementally funded contracts for long lead time items. No separate savings estimated Source: GAO presentation of Navy and GAO documentation. I GAO-19-497 Overall, while we were unable to fully assess the methodology and assumptions the Navy used to estimate anticipated savings, the information we reviewed indicated that the Navy may have overestimated some of the savings the program can realistically achieve through use of the Fund’s associated authorities. While the Navy provided some documentation of the cost estimate methodologies, we could not fully validate that the estimated savings were realistic because, in general, the documentation provided by the Navy did not include a detailed description of how the estimates were calculated or how historical data were used to develop the estimate—a best practice identified in GAO’s cost estimating guide. 40 In some cases, such as for individual critical components, the total value of the component costs was not documented. For other savings, such as advance construction, the Navy could not provide documentation of the calculations or a rationale for the estimated savings. 40 A well-documented estimate includes documentation that describes in sufficient detail the calculations performed and the estimating methodology used to derive the cost. See GAO-09-3SP. Page 40 GAO-19-497 Columbia Class Submarine In addition, the Navy assumes a higher rate for Columbia multiyear procurement savings than what has been typically achieved for other programs. The Navy has generally used multiyear procurement contracts after production has begun and some units have already been purchased. For example, according to the Navy, it did not receive multiyear procurement authority for the DDG 51 Arleigh Burke-class destroyer program—until 1998—more than 10 years after the contract for the lead ship was awarded and 38 ships had been purchased. We have reported that DOD typically overestimates savings from multiyear procurement authority. 41 Further, in a 2017 presentation to Congress, the Navy stated that multiyear procurement savings are historically 10 to 12 percent. When the Navy requested multiyear procurement authority for the DDG 51 program in fiscal year 2013, it estimated achieving a savings of 8.7 percent. Similarly, when planning material purchases for the Virginia class submarine, the shipbuilder estimated that it would achieve 10 to 15 percent savings through the use of multiyear procurement authority. However, the Navy estimates that the Columbia class program will realize savings of 15 to 20 percent using multiyear procurement authority. A realistic estimate of savings is essential because program management is essentially relying on these savings as the only cost reserve to address any issues that arise during design and construction of the submarines. Updating the cost estimate to reflect these savings will provide program management with a more realistic assessment of the margin available and resources needed to achieve their costs. Conclusions The Columbia class program is driven by the continued and pressing need to meet the Navy’s nuclear deterrent requirements as the legacy submarine fleet cannot extend its life any longer. From the outset this has translated into an aggressive and concurrent schedule for lead submarine construction. To counterbalance this schedule risk, the program plans to complete a substantial amount of the design before starting construction, which may prove challenging as the shipbuilder must complete an increasingly higher volume and complexity of disclosures. This, coupled with failures in missile tubes already delivered to the shipyard, highlight the potential for management challenges ahead. This is not to suggest that in a program of this size and complexity that some issues are not to be expected. Rather, the challenge for the Columbia class program is that 41 GAO, Defense Acquisitions: DOD’s Practices and Processes for Multiyear Procurement Should Be Improved, GAO-08-298 (Washington, D.C.: Feb. 7, 2008). Page 41 GAO-19-497 Columbia Class Submarine the Navy has a limited ability to slow the pace of the program given the mission imperatives. At present, the need for additional resources appears likely because the Navy’s margin to mitigate any cost growth from issues that develop during design and construction relies on overestimated savings from use of the Fund’s associated authorities. The steps that the Navy takes between now and the fiscal year 2021 budget request to understand and plan for likely program costs will determine whether sufficient funding is in place to cover potential cost growth. The Navy plans to update the lead submarine cost estimate to reflect its current acquisition strategy and, in doing so, the Navy has the opportunity to incorporate more realistic information into the risk analysis and lead submarine cost estimate. In addition, a realistic and well-documented estimate of savings from use of the Fund’s associated authorities would help ensure that the Navy has allocated the necessary resources to address any issues that emerge during design or construction of the lead submarine. Such steps will likely improve the reliability of the lead submarine cost estimate and would position the Navy to better align its fiscal year 2021 budget request with funding it will likely need to construct the lead submarine—the next key decision point in the Columbia class program. Without an updated cost estimate with more realistic assumptions, Congress will be asked to commit billions of dollars for the lead submarine without knowing the full potential cost of construction and the possible effect on other shipbuilding programs. Recommendations for Executive Action We are making three recommendations to the Secretary of the Navy: • The Secretary of the Navy should direct NAVSEA to incorporate current cost and program data and an updated cost risk analysis in its planned update of the Columbia class lead submarine cost estimate. (Recommendation 1) • The Secretary of the Navy should direct NAVSEA to develop a realistic and well-documented estimate of savings from use of the authorities associated with the Fund and incorporate the savings associated with the lead submarine into the Columbia lead submarine cost estimate. (Recommendation 2) • The Secretary of the Navy should direct the Columbia class program office to update the lead submarine cost estimate and cost risk analysis prior to requesting funds for lead submarine construction. (Recommendation 3) Page 42 GAO-19-497 Columbia Class Submarine Agency Comments and Our Evaluation We provided a draft of the sensitive report to DOD for comment. DOD’s written comments on the sensitive report are reprinted in appendix IV and summarized below. DOD concurred and described the actions they have taken or plan to take in response to all three of our recommendations. Regarding our recommendations to update its cost estimate update prior to requesting funds for lead submarine construction, the Navy has stated that it incorporated current cost and program data and an updated risk analysis into its cost estimate for the lead submarine in 2018, as part of an annual review. The Navy also stated that it will continue to update the lead submarine cost estimate with current data prior to requesting funding for lead submarine construction in fiscal year 2021. Until the updated estimate is independently validated—an essential cost estimating step— we cannot determine that the updated estimate is credible. Further, in response to our recommendation regarding the development of a realistic and well-documented estimate of savings from use of the Fund’s associated authorities, the Navy stated that it incorporated savings in its updated cost estimate. However, it has not provided any additional evidence to demonstrate that estimated savings from use of the Fund’s associated authorities are realistic and well-documented. Based on documentation that the Navy provided to us, it did not include a detailed description of how the estimates were calculated or how historical data were used to develop the estimate. Until these estimates are independently validated, the Navy cannot be confident that the program will achieve the planned amount of savings. The Navy also provided technical comments, which we incorporated as appropriate. DOD also raised a number of issues related to our assessment of the cost estimate, advance construction, and technology development, which we address in appendix IV. We are sending copies of this report to the appropriate congressional committees, the Acting Secretary of Defense, the Secretary of the Navy, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. Should you or your staff have questions, please contact me at (202) 5124841 or oakleys@gao.gov. Contact points for our Offices of Page 43 GAO-19-497 Columbia Class Submarine Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix V. Shelby S. Oakley Director, Contracting and National Security Acquisitions Page 44 GAO-19-497 Columbia Class Submarine List of Committees The Honorable James M. Inhofe Chairman The Honorable Jack Reed Ranking Member Committee on Armed Services United States Senate The Honorable Richard C. Shelby Chairman The Honorable Dick Durbin Ranking Member Subcommittee on Defense Committee on Appropriations United States Senate The Honorable Adam Smith Chairman The Honorable Mac Thornberry Ranking Member Committee on Armed Services House of Representatives The Honorable Peter J. Visclosky Chairman The Honorable Ken Calvert Ranking Member Subcommittee on Defense Committee on Appropriations House of Representatives Page 45 GAO-19-497 Columbia Class Submarine Appendix I: Objectives, Scope, and Methodology Appendix I: Objectives, Scope, and Methodology This report evaluates the Navy’s Columbia class submarine program. Specifically, we assessed (1) the Navy’s progress and challenges, if any, associated with meeting design goals and preparing for lead submarine construction; (2) the reliability of the Navy’s cost estimate for the Columbia class submarine program; and (3) how the Navy is implementing the National Sea-Based Deterrence Fund (the Fund) and associated authorities to construct Columbia class submarines. This report is a public version of a sensitive report that we issued in March 2019. The Department of Defense (DOD) deemed some of the information in our March report to be sensitive, which must be protected from public disclosure. Therefore, this report omits sensitive information about the Navy’s development of critical technologies for the Columbia class program, including specific details about the technologies. Although the information provided in this report is more limited, the report addresses the same objectives as the sensitive report and uses the same methodology. To assess the Navy’s progress and what challenges, if any, are associated with meeting design goals and preparing for lead submarine construction, we reviewed Navy and shipbuilder documents, including program briefings, schedules, and contract status reports to assess the schedule and performance risks of the Columbia class program. To evaluate the shipbuilder’s progress in maturing the Columbia class design, we reviewed the Navy’s plans for design management and completion, evaluated the shipbuilder’s design schedule, and compared them against design progress reports to identify any delays. To evaluate the Navy’s plans for advance construction, we analyzed metrics reported in Navy and shipbuilder documents, briefing slides, and other documentation including key dates and estimated construction plans. We compared design knowledge on the Columbia class program to our prior work on shipbuilding best practices. 1 We reviewed ongoing development efforts and schedules for the Columbia class program’s critical technologies to determine remaining risks to their development and integration. We also reviewed the matrices submitted by the Navy to Congress in February and October 2018, to determine the status of the program and identify any changes to the Navy’s design and construction 1 GAO, Best Practices: High Levels of Knowledge at Key Points Differentiate Commercial Shipbuilding from Navy Shipbuilding, GAO-09-322 (Washington, D.C.: May 13, 2009). Page 46 GAO-19-497 Columbia Class Submarine Appendix I: Objectives, Scope, and Methodology goals for the program since our last report in December 2017. 2 We also analyzed available documentation related to the status of the nuclear reactor and integrated power system. We reviewed the shipbuilder’s construction plans for its new facilities and its workforce hiring plans. We also reviewed the shipbuilder’s and Navy’s process for evaluating its suppliers. To corroborate documentary evidence and gather additional information in support of our review, we met with officials from the Navy’s Columbia class submarine program office; Naval Nuclear Propulsion Directorate; Naval Surface Warfare Center Philadelphia; Office of the Chief of Naval Operations; Supervisor of Shipbuilding, Groton; the Office of the Deputy Assistant Secretary of Defense for Systems Engineering; and the Office of Undersecretary of Defense for Acquisition and Sustainment. Additionally, we met with shipbuilding representatives from General Dynamics Electric Boat—the prime contractor—as well as their main subcontractor, Huntington Ingalls Industries Newport News Shipbuilding to understand their role in Columbia class design and construction. To assess the reliability of the Navy’s cost estimate for the Columbia class submarine program, we determined the extent to which the estimate met best practices as identified in GAO’s Cost Estimating and Assessment Guide. 3 We examined cost estimate documentation, such as the Columbia class program life-cycle cost estimate, briefs, memoranda, and other documents that contain cost, schedule, and risk information. We also examined the independent cost estimate conducted by the Office of the Secretary of Defense’s Office of Cost Assessment and Program Evaluation (CAPE), the independent cost assessment conducted by the Naval Center for Cost Analysis (NCCA), and the cost estimate conducted by the Congressional Budget Office, to determine what methodologies and assumptions differed from the program cost estimate. We met with Navy officials who were responsible for developing the cost estimate to understand the processes used by the cost estimators, to clarify information, and to allow the Navy to provide additional documentation on the data and methodologies used in the estimate. We also observed portions of the Columbia class program’s cost model during a 2 GAO, Columbia Class Submarine: Immature Technologies Present Risks to Achieving Cost, Schedule, and Performance Goals, GAO-18-158 (Washington, D.C.: Dec. 21, 2017). 3 GAO, GAO Cost Estimating and Assessment Guide: Best Practices for Developing and Managing Capital Program Costs, GAO-09-3SP (Washington, D.C.: Mar. 2, 2009). Page 47 GAO-19-497 Columbia Class Submarine Appendix I: Objectives, Scope, and Methodology presentation and discussion with Navy cost estimators. We also reviewed the matrices submitted by the Navy to Congress to identify any changes to the Navy’s cost goals and reported information. To further corroborate documentary evidence and gather additional information in support of our review, we conducted interviews with relevant DOD and Navy officials responsible for developing, updating, and assessing the Columbia class program cost estimate, including CAPE; NCCA; the Naval Sea Systems Command’s (NAVSEA) Cost Engineering and Industrial Analysis Group; and the Columbia class program office. To evaluate how the Navy is implementing the Fund and associated authorities to construct Columbia class submarines, we reviewed the legislation establishing and modifying the Fund, program budget request documents, and DOD reprogramming approvals. We also reviewed the Navy’s basis of estimate for the savings it plans to achieve from these authorities. To further corroborate documentary evidence and gather additional information in support of our review, we met with officials from the Office of the Assistant Secretary of the Navy for Financial Management and Comptroller; Office of the Under Secretary of Defense (Comptroller); and the Columbia class program office to discuss the Navy’s plans to use and execute the Fund and DOD’s role in approving transfers into the Fund. The performance audit upon which this report is based was conducted from December 2017 to March 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. We subsequently worked with DOD from February 2019 to April 2019 to prepare this unclassified version of the original sensitive report for public release. This public version was also prepared in accordance with these standards. Page 48 GAO-19-497 Columbia Class Submarine Appendix II: Technology Readiness Levels Appendix II: Technology Readiness Levels Table 5: Technology Readiness Level (TRL) TRL Definition Description 1. Basic principles observed Lowest level of technology readiness. Scientific research begins to be translated into applied and reported research and development. Examples might include paper studies of a technology’s basic properties. 2. Technology concept and/or application formulated 3. Analytical and Active research and development is initiated. This includes analytical studies and laboratory experimental function studies to physically validate analytical predictions of separate elements of the technology. and/or characteristic proof Examples include components that are not yet integrated or representative. of concept 4. Component and/or Basic technological components are integrated to establish that the pieces will work together. breadboard validation in a This is relatively “low fidelity” compared to the eventual system. Examples include integration laboratory environment of “ad hoc” hardware in a laboratory. 5. Component and/or Fidelity of breadboard technology increases significantly. The basic technological components breadboard validation in a are integrated with reasonably realistic supporting elements so that the technology can be relevant environment tested in a simulated environment. Examples include “high-fidelity” laboratory integration of components. 6. System/subsystem model or prototype demonstration in a relevant environment Representative model or prototype system, which is well beyond the breadboard tested for TRL 5, is tested in a relevant environment. Represents a major step up in a technology’s demonstrated readiness. Examples include testing a prototype in a high-fidelity laboratory environment or in a simulated realistic environment. 7. System prototype demonstration in an operational environment Prototype near or at planned operational system. Represents a major step up from TRL 6, requiring the demonstration of an actual system prototype in a realistic environment, such as an aircraft, vehicle, or space. Examples include testing the prototype in a test bed aircraft. 8. Actual system completed and qualified through test and demonstration Technology has been proven to work in its final form and under expected conditions. In almost all cases, this TRL represents the end of the true system development. Examples include developmental test and evaluation of the system in its intended weapon system to determine if it meets design specifications. 9. Actual system proven through successful mission operations Actual application of the technology in its final form and under mission conditions, such as those encountered in operational test and evaluations. In almost all cases, this is the end of the last “bug fixing” aspects of true system development. Examples include using the system under operational mission conditions. Invention begins. Once basic principles are observed, practical applications can be invented. The application is speculative and there is no proof or detailed analysis to support the assumption. Examples are still limited to paper studies. Source: GAO. I GAO-19-497 Page 49 GAO-19-497 Columbia Class Submarine Appendix III: GAO’s Assessment of the Reliability of the Navy’s Cost Estimate for the Columbia Class Submarine Program Appendix III: GAO’s Assessment of the Reliability of the Navy’s Cost Estimate for the Columbia Class Submarine Program To assess the reliability of the Navy’s cost estimate, we determined the extent to which the estimate was consistent with cost estimating best practices as identified in GAO’s Cost Estimating and Assessment Guide. 1 This guide groups the best practices into four general characteristics: well documented, comprehensive, accurate, and credible. 2 We reviewed documentation the Navy submitted for its cost estimate including limited portions of the Navy’s cost model, conducted numerous interviews, and reviewed relevant sources. We determined that the Columbia class cost estimate substantially met one, and partially met three of the four characteristics of a reliable cost estimate, shown in figure 16. We determined the overall assessment rating by assigning each individual rating a number: Not Met = 1, Minimally Met = 2, Partially Met = 3, Substantially Met = 4, and Met = 5. Then, we calculated the average of the individual assessment ratings to determine the overall rating for each of the four characteristics as follows: Not Met = 1.0 to 1.4, Minimally Met = 1.5 to 2.4, Partially Met = 2.5 to 3.4, Substantially Met = 3.5 to 4.4, and Met = 4.5 to 5.0. We consider a cost estimate to be reliable if the overall assessment ratings for each of the four characteristics are substantially or fully met. If any of the characteristics are not met, minimally met, or partially met, then the cost estimate does not fully reflect the characteristics of a high-quality estimate and is not considered reliable. 1 GAO, GAO Cost Estimating and Assessment Guide: Best Practices for Developing and Managing Capital Program Costs, GAO-09-3SP (Washington D.C.: Mar. 2, 2009). 2 Our assessment of the Navy’s cost estimate was based on cost estimate documentation and interviews with Navy officials who were responsible for developing the estimate. For additional information on the methodology for this assessment, see Appendix I. Page 50 GAO-19-497 Columbia Class Submarine Appendix III: GAO’s Assessment of the Reliability of the Navy’s Cost Estimate for the Columbia Class Submarine Program Figure 16: Summary of Assessment of Columbia Program’s Cost Estimate Compared to GAO’s Best Practices Note: The best practice in the Accurate category for documenting variance between planned and actual costs was removed from analysis because the availability of actual cost data is limited based on the stage of the Columbia class program. Page 51 GAO-19-497 Columbia Class Submarine Appendix IV: Comments from the Department of Defense Appendix IV: Comments from the Department of Defense Page 52 GAO-19-497 Columbia Class Submarine Appendix IV: Comments from the Department of Defense DOD included a copy of its response to GAO’s December 2017 report. For our response to that letter, see GAO-18-158. Page 53 GAO-19-497 Columbia Class Submarine Appendix IV: Comments from the Department of Defense Page 54 GAO-19-497 Columbia Class Submarine Appendix IV: Comments from the Department of Defense The following are our comments on the Department of Defense (DOD) letter dated February 1, 2019. GAO Comments Assessment of Columbia Class Program’s Cost Estimate In addition to responding to our recommendations, DOD also provided observations on a number of issues related to our assessment of the cost estimate, advance construction, and technology development. Our response to DOD’s observations is as follows. • In paragraph 4, page 1 of the letter above, the Navy did not agree with our assessment of the accuracy of the cost estimate and stated that the life cycle cost estimate includes accurate calculations, proper inflation tables, and updates to requirements. DOD also stated that GAO or other stakeholders did not identify any errors. This is incorrect. While the Navy allowed us to observe the model, we did not independently check the accuracy of the calculations because Navy officials stated that the cost model, which contains the cost calculations, could not be released. We informed the Navy that this would affect parts of our assessment. After we provided a draft of the report, the Navy provided a briefing summarizing the results of a program office cost checkpoint conducted in September 2018. At the briefing, we received information on updates that the Navy made to the program cost estimate. As a result, we updated our assessment to reflect that the Navy substantially met the best practice to regularly update the cost estimate to reflect significant changes. However, the additional information provided by the Navy did not change our assessment of the accuracy and, therefore, our overall assessment of the Columbia cost estimate remains valid. • In paragraph 1, page 2, the Navy did not agree with our assessment of the credibility of the cost estimate and stated that the life cycle cost estimate includes analyses that address sensitivity, risks, and uncertainty within the estimate. As we point out in the report, the estimate is based, in part, on optimistic assumptions regarding the number of labor hours needed to construct Columbia class submarines. The Navy has made updates to the program cost estimate based on a 2018 checkpoint review and stated that the cost risk analysis has been updated and program costs are less than originally estimated. The Navy provided us with a high-level brief of these updates. However, due to the timing of this report, we were not able to fully assess the update to the cost model. Given the size and complexity of the Columbia class program, we continue to believe that Page 55 GAO-19-497 Columbia Class Submarine Appendix IV: Comments from the Department of Defense the program’s cost estimate does not adequately account for program risks. • In paragraph 3, page 1, DOD stated that our findings were largely informed by an assessment conducted by the Naval Center for Cost Analysis (NCCA). However, our process for assessing program cost estimates is based on the extent to which the estimate met best practices outlined in GAO’s Cost Estimating and Assessment Guide. In conducting our assessment, we examined multiple sources of information, including the Columbia class program life cycle cost estimate, NCCA’s independent cost assessment, DOD’s Office for Cost Assessment and Program Evaluation’s (CAPE) independent cost estimate, and the cost estimate conducted by the Congressional Budget Office (CBO), to determine what methodologies and assumptions differed from the program cost estimate. We also relied on prior experience examining and reporting on the cost performance of Navy shipbuilding programs, issuing 26 reports over the past 10 years. We found, for example, that the cost estimate is based on optimistic labor assumptions which, while in agreement with NCCA’s assessment and CBO’s estimate, results from our independent assessment of the evidence we reviewed and on our prior work. Advance Construction • In paragraph 2, page 2, the Navy stated that it identified super modules and selected components where acceleration would reduce construction schedule risk. We acknowledge in the report that the design for these components will be complete prior to starting construction. However, we continue to believe that starting construction for components of the lead submarine before the arrangements for the submarine are complete increases design and construction risk. Even if the components included in advance construction are fully designed, risk remains for the adjoining and interfacing components within the module that may have ongoing design work, potentially requiring costly and time-intensive rework. Technology Development • In paragraph 4, page 2, the Navy notes that fully maturing all of the key technologies identified in our 2017 report—such as the advanced propulsor bearing—would require substantial investments in money and time. However, we continue to reinforce that a tenet of achieving design maturity is based on demonstrating a prototype in its final form, fit, and function in a realistic environment—which requires a design resembling the final configuration. Page 56 GAO-19-497 Columbia Class Submarine Appendix IV: Comments from the Department of Defense Integrated Power System Motor Manufacturing Delays • In paragraph 6, page 2, the Navy stated that it does not agree with our characterization that the Navy is continuing to experience manufacturing problems with the electric drive of the integrated power system. DOD stated that while the vendor experienced delays in manufacturing the prototype motor, it has taken proactive measures to deliver the motor to the shipyard, as scheduled. However, the Navy’s plan to concurrently test and finalize the design increases risk that any issues identified in testing could delay the delivery of the system to the shipyard. As a result, we continue to identify this as a key risk to the program. Additional details on this system are classified. Page 57 GAO-19-497 Columbia Class Submarine Appendix V: GAO Contact and Staff Acknowledgments Appendix V: GAO Contact and Staff Acknowledgments GAO Contact Shelby S. Oakley, (202) 512-4841 or oakleys@gao.gov. Staff Acknowledgments In addition to the contact above, the following staff members made key contributions to this report: Diana Moldafsky, Assistant Director; Laura Jezewski; Jessica Karnis; and Nathaniel Vaught. Other contributions were made by Brian Bothwell; Daniel Glickstein; Kurt Gurka; Stephanie Gustafson; and Robin Wilson. (103455) Page 58 GAO-19-497 Columbia Class Submarine GAO’s Mission The Government Accountability Office, the audit, evaluation, and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO’s commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony The fastest and easiest way to obtain copies of GAO documents at no cost is through GAO’s website (https://www.gao.gov). Each weekday afternoon, GAO posts on its website newly released reports, testimony, and correspondence. To have GAO e-mail you a list of newly posted products, go to https://www.gao.gov and select “E-mail Updates.” Order by Phone The price of each GAO publication reflects GAO’s actual cost of production and distribution and depends on the number of pages in the publication and whether the publication is printed in color or black and white. Pricing and ordering information is posted on GAO’s website, https://www.gao.gov/ordering.htm. Place orders by calling (202) 512-6000, toll free (866) 801-7077, or TDD (202) 512-2537. Orders may be paid for using American Express, Discover Card, MasterCard, Visa, check, or money order. Call for additional information. Connect with GAO Connect with GAO on Facebook, Flickr, Twitter, and YouTube. Subscribe to our RSS Feeds or E-mail Updates. Listen to our Podcasts. Visit GAO on the web at https://www.gao.gov. To Report Fraud, Waste, and Abuse in Federal Programs Contact FraudNet: Congressional Relations Orice Williams Brown, Managing Director, WilliamsO@gao.gov, (202) 512-4400, U.S. Government Accountability Office, 441 G Street NW, Room 7125, Washington, DC 20548 Public Affairs Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800 U.S. Government Accountability Office, 441 G Street NW, Room 7149 Washington, DC 20548 Strategic Planning and External Liaison James-Christian Blockwood, Managing Director, spel@gao.gov, (202) 512-4707 U.S. Government Accountability Office, 441 G Street NW, Room 7814, Washington, DC 20548 Website: https://www.gao.gov/fraudnet/fraudnet.htm Automated answering system: (800) 424-5454 or (202) 512-7700 Please Print on Recycled Paper.