United States Government Accountability Office Report to Congressional Committees May 2019 WEAPON SYSTEMS ANNUAL ASSESSMENT Limited Use of Knowledge-Based Practices Continues to Undercut DOD’s Investments GAO-19-336SP A Report to Congressional Committees May 2019 Weapon Systems Annual Assessment Limited Use of Knowledge-Based Practices Continues to Undercut DOD’s Investments Highlights of GAO-19-336SP. This year’s assessment comes at a time of significant change to how DOD manages and oversees its major weapons programs. At the direction of Congress, DOD restructured the Office of the Secretary of Defense to focus more on capability development and less on program oversight, which now resides primarily with the three military departments. Amid these changes, this year GAO found that cost performance has slipped. Further, unlike prior years, programs initiated after major acquisition reforms were implemented in 2010 are now showing cost growth. This report provides observations on: 1. cost and schedule performance and contract awards for DOD’s 2018 portfolio of 82 programs that provide acquisition reports to Congress and 2. knowledge that 51 selected programs attained at key points in the acquisition process. Further, GAO presents individual assessments of 51 programs. DOD’s 2018 Portfolio Is Smaller, yet Older and More Expensive than Last Year; Contracting Has Often Been Characterized by Ineffective Competition This special report is GAO’s 17th annual assessment of the Department of Defense’s (DOD) $1.69 trillion portfolio of 82 major weapon systems acquisition programs. This report (1) examines changes in the portfolio since last year, (2) evaluates DOD’s actions in areas related to recent, selected acquisition reforms, and (3) offers a quick look at cost and schedule performance of 51 individual weapon programs based on DOD documentation and questionnaire responses from their respective program offices. For years, GAO has highlighted the importance of applying knowledge-based acquisition practices as a way to improve DOD’s program outcomes. When programs enter development with insufficient knowledge, negative effects often cascade throughout the acquisition cycle. The figure below shows how three acquisition phases—technology development, system development, and production—align with three key points for demonstrating knowledge. Department of Defense (DOD) Acquisition Process DOD’s 2018 portfolio of major weapon programs has grown in cost by $8 billion, but contains four fewer systems than last year. GAO’s analysis suggests that one of the primary drivers of this cost growth is that, since 2012, the average age of programs has increased—indicative of DOD decisions to introduce new capabilities through additions to existing programs rather than by starting new programs. On average, programs in the current portfolio are about 4 months older than last year and nearly 3 years older than in 2012. (See figure.) DOD's Portfolio Increased in Cost and Average Program Age View GAO-19-336SP. For more information, contact Shelby S. Oakley at (202) 512-4841 or oakleys@gao.gov. Portfolio-wide cost growth has occurred in an environment where awards are often made without full and open competition. Specifically, GAO found that DOD United States Government Accountability Office did not compete 67 percent of 183 major contracts currently reported for its 82 major programs. GAO also observed that DOD awarded 47 percent of these 183 contracts to five corporations and entities connected with them. (See figure.) DOD Programs Competed One-Third of Currently Reported Major Contracts with Nearly Half of Awards Concentrated within Five Companies Although Knowledge-Based Acquisition Practices Can Lead to Better Cost and Schedule Outcomes, Programs Continue Not to Fully Implement Them As we have reported in previous assessments, DOD programs continue not to fully implement knowledge-based acquisition practices. GAO observed that most of the 45 current programs proceeded into system development, through critical design reviews, and into production without completing the key knowledge-based practices associated with each of these three points. (See table.) DOD Major Defense Acquisition Programs Continue Not to Fully Implement Key Knowledge-Based Acquisition Practices Programs that completed the KP before this assessment period Programs that completed the KP during this assessment period KP 1 practices Demonstrate all critical technologies are very close to final form, fit, and function within a relevant environment Demonstrate all critical technologies are in form, fit, and, function within a realistic environment Completed preliminary design review before system development start 38 programs Four programs ◐ ○ ◐ ○ ◐ ○ KP 2 practices Release at least 90 percent of design drawings to manufacturing Test a system-level integrated prototype 33 programs Two programs ○ ○ KP 3 practices Demonstrate critical manufacturing processes are in statistical control Demonstrate critical processes on a pilot production line 15 programs Three programs ◐ ○ ● ● Practices associated with the three key knowledge points (KP) Test a production-representative prototype in its intended environment ○ ○ ● ○ Programs completing each best practice: ● 75 - 100 percent; ◐ 50 - 74 percent; ◯ 0 - 49 percent Source: GAO analysis of Department of Defense data. │GAO-19-336SP This lack of knowledge and the effects it can have throughout a program’s acquisition life cycle can increase the risk of undesirable cost and schedule outcomes. In 2018, GAO conducted an exploratory statistical analysis of 15 programs in production to evaluate their completion of the eight key knowledgebased acquisition practices. In 2019, GAO expanded this analysis to include 17 programs. Over the past two years, GAO found that the major DOD acquisition programs that completed one or more of three specific practices had significantly lower cost and schedule growth than those that did not. These three practices were (1) demonstration that all critical technologies were very close to final form, fit, and function, within a relevant environment, before starting development; (2) completion of a preliminary design review prior to starting development; and (3) release of at least 90 percent of design drawings by critical design review. Contents Foreword 1 Letter 4 Background 6 Nine Observations on the Performance of DOD’s 2018 MDAP Portfolio 12 Our Observation on DOD’s Application of Key Acquisition Reforms 42 Eight Observations from Our Assessment of Knowledge Attained at Key Decision Points and Related Implications for Testing and Software Development 46 Assessments of Individual Programs 67 Army Program Assessments 72 Armored Multi-Purpose Vehicle (AMPV) 75 CH-47F Modernized Cargo Helicopter (CH-47F Block II) 77 Common Infrared Countermeasure (CIRCM) 79 Handheld, Manpack, and Small Form Fit Radios (HMS) 81 Integrated Air and Missile Defense (IAMD) 83 Improved Turbine Engine Program (ITEP) 85 Joint Air-to-Ground Missile (JAGM) 87 Joint Light Tactical Vehicle (JLTV) 89 Indirect Fire Protection Capability Increment 2-Intercept Block 1 (IFPC Inc 2-I Block 1) 91 Precision Strike Missile (PrSM) 92 Navy and Marine Corps Program Assessments 94 Amphibious Combat Vehicle Increment 1.1 (ACV 1.1) 97 Air and Missile Defense Radar (AMDR) 99 CH-53K Heavy Lift Replacement Helicopter (CH-53K) 101 CVN 78 Gerald R. Ford Class Nuclear Aircraft Carrier (CVN 78) 103 DDG 1000 Zumwalt Class Destroyer (DDG 1000) 105 Ground/Air Task Oriented Radar (G/ATOR) 107 F/A-18E/F Infrared Search and Track (IRST) 109 Joint Precision Approach and Landing System (JPALS) 111 Littoral Combat Ship-Mission Modules (LCS Packages) 113 MQ-4C Triton Unmanned Aircraft System (MQ-4C Triton) 115 MQ-25 Unmanned Aircraft System (MQ-25 Stingray) 117 Next Generation Jammer Mid-Band (NGJ Mid-Band) 119 Offensive Anti-Surface Warfare Increment 1 (OASuW Inc 1) 121 Page i GAO-19-336SP Weapon Systems Annual Assessment SSBN 826 Columbia Class Ballistic Missile Submarine (SSBN 826) 123 Ship to Shore Connector Amphibious Craft (SSC) 125 John Lewis Class Fleet Replenishment Oiler (T-AO 205) 127 VH-92A Presidential Helicopter Replacement Program (VH-92A) 129 DDG 51 Arleigh Burke Class Destroyer, Flight III (DDG 51 Flight III) 131 Guided Missile Frigate (FFG(X)) 132 LHA 8 Amphibious Assault Ship (LHA 8) 133 LPD 17 San Antonio Class Amphibious Transport Dock, Flight II (LPD 17 Flight II) [formerly LX(R)] 134 Next Generation Jammer – Low Band (NGJ-LB) 135 P-8A Poseidon, Increment 3 (P-8A Increment 3) 136 SSN 774 Virginia Class Submarine Block V (SSN 774 Block V) 137 Air Force Program Assessments 138 Advanced Pilot Training (APT) 141 B-2 Defensive Management System Modernization (B-2 DMS-M) 143 Combat Rescue Helicopter (CRH) 145 F-15 Eagle Passive Active Warning Survivability System (F-15 EPAWSS) 147 Family of Advanced Beyond Line-of-Sight Terminals Command Post Terminals (FAB-T CPT) 149 Global Positioning System III (GPS III) 151 Global Positioning System III Follow-On (GPS IIIF) 153 KC-46 Tanker Modernization Program (KC-46A) 155 Military Global Positioning System (GPS) User Equipment (MGUE) Increment 1 157 Next Generation Operational Control System (OCX) 159 Small Diameter Bomb Increment II (SDB II) 161 Space Fence Ground-Based Radar System Increment 1 (Space Fence Inc 1) 163 Utility Helicopter (UH-1N) Replacement 165 B-52 Radar Modernization Program (B-52 RMP) 167 VC-25B Presidential Aircraft Recapitalization (VC-25B) 168 Weather System Follow-On (WSF) 169 Joint DOD Program Assessment 171 F-35 Lightning II (F-35) 173 Agency Comments 175 Page ii GAO-19-336SP Weapon Systems Annual Assessment Appendix I Individual Subcontracting Reports in the Electronic Subcontracting Reporting System 177 Appendix II Objectives, Scope, and Methodology 181 Appendix III Knowledge-Based Acquisition Practices 196 Appendix IV Selected Acquisition Reforms Contained in the National Defense Authorization Acts for 2016, 2017, and 2018 198 Total Acquisition Cost Changes for Programs in the Department of Defense’s 2018 Portfolio 202 Performance against GAO, DOD, and the Office of Management Budget Cost Metrics 206 Program Rankings based on Total Acquisition Cost Change, Not Due to Quantity Changes, since First Full Estimates 207 Appendix VIII Technology Readiness Levels 209 Appendix IX Comments from the Department of Defense 211 Appendix X GAO Contacts and Staff Acknowledgments 212 Appendix V Appendix VI Appendix VII Page iii GAO-19-336SP Weapon Systems Annual Assessment Related GAO Products 215 Tables Table 1: Cost and Schedule Changes to DOD’s 2018 Portfolio of 82 Major Defense Acquisition Programs over the Past Year (fiscal year 2019 dollars in billions) Table 2: Portfolio Cost and Schedule Changes since Major Defense Acquisition Programs’ First Full Estimates (fiscal year 2019 dollars in billions) Table 3: Ten Major Defense Acquisition Programs with the Lowest and Highest Total Acquisition Cost Change since First Full Estimates (fiscal year 2019 dollars in millions) Table 4: Ten Major Defense Acquisition Programs with the Lowest and Highest Total Acquisition Cost Percentage Change since First Full Estimates Table 5: Programs’ Total Acquisition Cost and Cycle Time Changes, 2017 to 2018 (fiscal year 2019 dollars in billions) Table 6: Buying Power Changes Since 2017 (fiscal year 2019 dollars in millions) Table 7: Overview of Selected, Recent Acquisition Reforms Table 8: Actions in Areas Related to Recent Acquisition Reforms as Reported by 51 Current and Future Major Defense Acquisition Programs (MDAP) Table 9: Major Defense Programs Have Not Consistently Implemented Knowledge-Based Acquisition Practices Table 10: Technology Readiness Levels (TRL) Reported by 42 Current Major Defense Acquisition Programs Table 11: Number of 42 Selected, Current Programs that Met Key Knowledge Practices Prior to Beginning Development Table 12: Number of Programs Demonstrating Sufficient Design Knowledge Prior to Knowledge Point 2 Table 13: Program Completion of Key Production-Related Knowledge Practices Table 14: Planned Implementation of Selected Knowledge-Based Acquisition Practices at Development Start among Six Future Major Defense Acquisition Programs Table 15: Software Development Models Employed by 51 Selected Current and Future Major Defense Acquisition Programs Page iv 19 24 27 28 30 34 43 45 47 50 52 53 55 56 59 GAO-19-336SP Weapon Systems Annual Assessment Table 16: Software Risk Identification and Development Practices among 45 Selected Current Major Defense Acquisition Programs Table 17: Department of Defense’s (DOD) Six Acquisition Models and Their Basic Characteristics Table 18: Major Defense Acquisition Programs’ Individual Subcontracting Reports in the Electronic Subcontracting Reporting System (eSRS) Table 19: Best Practices for Knowledge-Based Acquisitions Table 20: December 2017 Cost Estimates and First Full Estimates for the Department of Defense’s 2018 Portfolio of Major Defense Acquisition Programs Table 21: Five Programs with the Lowest and Highest Total Acquisition Cost Changes by Total Dollars since First Full Estimates (fiscal year 2019 dollars in millions) Table 22: Five Programs with the Lowest and Highest Total Acquisition Cost Changes by Percentage since First Full Estimates Table 23: Technology Readiness Levels and Descriptions Table 24: GAO Staff Responsible for Individual Program Assessments 61 64 177 196 202 207 208 209 212 Figures Figure 1: DOD Acquisition Process and GAO-identified Knowledge Points Figure 2: Milestone Decision Authority in DOD’s Portfolio of Major Defense Acquisition Programs, 2012-2019 Figure 3: Historical DOD Portfolio Comparison Figure 4: Programs that Entered and Exited the 2018 Portfolio Figure 5: Annual Change in Average Program Age in DOD’s Portfolio of Major Defense Acquisition Programs, 20122018 Figure 6: Major Defense Acquisition Program Investments among Military Departments and DOD-wide within the 2018 Portfolio Figure 7: Total Acquisition Cost Increases and Decreases in Major Defense Acquisition Programs, 2017-2018 Figure 8: Major Defense Acquisition Programs’ Performance against Selected Cost Growth Metrics Figure 9: Portfolio Cost Growth by Acquisition Phase (fiscal year 2019 dollars in billions) Page v 8 11 13 14 15 18 21 23 25 GAO-19-336SP Weapon Systems Annual Assessment Figure 10: Programs’ Total Acquisition Cost Changes since First Full Estimate by Acquisition Phases Figure 11: Distribution of Total Acquisition Costs among Various Groups of Programs Figure 12: Competition in Currently Reported Major Contracts and Number of Offers Received Figure 13: Competition Rates for 183 Currently Reported Major Contract Awards within DOD’s 2018 Portfolio of Major Defense Acquisition Programs Figure 14: Dollars Associated with Competition Results for 180 Currently Reported Major Development and Procurement Contract Awards within DOD’s 2018 Portfolio of Major Defense Acquisition Programs Figure 15: Distribution of Contract Awards among Prime Contractors for 183 Currently Reported Major Contracts within DOD’s 2018 Portfolio of Major Defense Acquisition Programs Figure 16: Distribution of Dollars among Prime Contractors for 183 Currently Reported Major Contracts within DOD’s 2018 Portfolio of Major Defense Acquisition Programs Figure 17: Distribution of Software Development Models among 51 Selected Current and Future Major Defense Acquisition Programs Figure 18: Plans for Completing Software Development among 45 Selected Current Major Defense Acquisition Programs Figure 19: Plans for Utilizing System Development Models among 45 Current Major Defense Acquisition Programs Figure 20: Plans for Declaring Initial Operational Capability among 45 Selected Current Major Defense Acquisition Programs Figure 21: Illustration of Program Two-Page Assessment Figure 22: Illustration of Program One-Page Assessment Figure 23: Examples of Knowledge Scorecards on Two-Page Assessments Figure 24: Example of Knowledge Scorecard on One-Page Assessments Figure 25: Percentage of Programs in Department of Defense’s (DOD) Last Six Portfolios, the Cost Growth of Which Remained Under Acceptable Limits over the Last Year, 5 Years, and Since First Full Estimate Page vi 31 32 37 39 40 41 42 60 63 65 66 68 69 71 71 206 GAO-19-336SP Weapon Systems Annual Assessment Abbreviations AEHF CDR CPAF CPFF CPIF CVN 78 Page vii Advanced Extremely High Frequency Satellite Critical Design Review cost-plus-award-fee cost-plus-fixed-fee cost-plus-incentive fee Gerald R. Ford Class Nuclear Aircraft Carrier GAO-19-336SP Weapon Systems Annual Assessment DAMIR DDG 51 DOD eSRS F-35 FFP FPDS-NG FPI IDIQ MDAP MH-60R MRL NA NDAA OMB P-8A Increment 3 SAR SBIRS High SSN 774 TBD TRL USAF USMC USN WIN-T Inc. 2 WIN-T Inc. 3 WSARA Defense Acquisition Management Information Retrieval DDG 51 Arleigh Burke Class Guided Missile Destroyer Department of Defense Electronic Subcontracting Reporting System F-35 Lightning II Joint Strike Fighter firm-fixed-price Federal Procurement Data System-Next Generation fixed-price incentive indefinite-delivery/indefinite-quantity Major Defense Acquisition Program MH-60R Multi-Mission Helicopter manufacturing readiness level not applicable National Defense Authorization Act Office of Management and Budget P-8A Poseidon, Increment 3 Selected Acquisition Report Space Based Infrared System High SSN 774 Virginia Class Submarine to be determined technology readiness level United States Air Force United States Marine Corps United States Navy Warfighter Information Network-Tactical Increment 2 Warfighter Information Network-Tactical Increment 3 Weapon Systems Acquisition Reform Act This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. Page viii GAO-19-336SP Weapon Systems Annual Assessment Letter 441 G St. N.W. Washington, DC 20548 Comptroller General of the United States May 7, 2019 Congressional Committees I am pleased to present our 17th annual assessment of the Department of Defense’s (DOD) major weapon system acquisition programs—an area on GAO’s high-risk list since 1990. 1 This year’s report offers observations on the performance of DOD’s 2018 portfolio of 82 major programs, which the department expects to cost $1.69 trillion in total. 2 This significant financial investment demands keen oversight and continued implementation of legislative reforms and policies aimed at improving DOD’s stewardship. These include use of knowledge-based acquisition practices, which we have previously recommended to DOD, but which continue to lack consistent application within the department. This year’s assessment comes at a time of significant change within the department. In response to congressional direction, in February 2018, DOD dissolved the Office of the Under Secretary of Defense for Acquisition, Technology and Logistics. DOD has developed a new organizational structure that refocuses the Office of the Secretary of Defense’s (OSD) principal acquisition function to one focused on increasing the speed of capability development rather than conducting program oversight. As part of this restructure, overall responsibilities for program oversight, which DOD calls “milestone decision authority,” now generally reside within the three military departments. We recognize the magnitude of these changes and, through our ongoing work, know that OSD and the military departments have made some 1 GAO, High-Risk Series: Substantial Efforts Needed to Achieve Greater Progress on High-Risk Areas, GAO-19-157SP (Washington, D.C.: Mar. 6, 2019). 2 Our assessment of DOD’s portfolio does not include the cost of the Ballistic Missile Defense System (BMDS) as the program and its elements lack acquisition program baselines needed to support our assessment of cost and schedule change. Although 10 U.S.C. § 225 requires the Missile Defense Agency (MDA) to establish and maintain an acquisition baseline for certain elements of the BMDS, these baselines are not the same as the acquisition program baselines developed pursuant to 10 U.S.C. § 2435 and DOD acquisition policies, and they do not provide all the data we need to assess cost and schedule changes. For more information on BMDS and its elements, see GAO, Missile Defense: The Warfighter and Decision Makers Would Benefit from Better Communication about the System’s Capabilities and Limitations, GAO-18-324 (Washington, D.C.: May 30, 2018). Page 1 GAO-19-336SP Weapon Systems Annual Assessment progress implementing them. At the same time, we observed this year that cost performance in major programs continues to slip. Most troubling is that DOD’s newer programs—those initiated after implementation of major acquisition reforms in 2010—now show aggregate cost growth. In previous assessments, we observed that this group of programs generated cost decreases that, in part, offset cost increases within the rest of the portfolio. We attribute the deteriorating performance of newer programs to the inconsistent implementation of knowledge-based acquisition practices—a condition we highlighted in our 2018 assessment. Our work has found that when programs enter development with insufficient knowledge, negative effects can cascade throughout the acquisition cycle. These knowledge shortfalls, or gaps, often begin with program decisions to accept immature technologies at the start of system development, but then later manifest in other forms as the program approaches production. In this environment, decision makers are confronted with the choice of increasing program investments, despite lacking visibility on whether the program’s cost and schedule estimates are achievable, or truncating the program and subsequently depriving warfighters of a needed capability. We have made numerous recommendations over the years to address these knowledge gaps in DOD’s programs. While DOD has often agreed with these recommendations, and incorporated them into its acquisition policy, it has inconsistently applied the policy to its acquisition programs. Further, for the second year in a row, we completed an exploratory statistical analysis that continues to validate a linkage between the attainment of knowledge and the real-life cost and schedule outcomes that programs deliver. This year we expanded our data set by two programs to a total of 17, each of which has entered production. Over the past two years, our analyses show that, consistent with our best practices, programs that attained certain knowledge at key points had lower cost and schedule growth than other programs. As evidence of the importance of knowledge-based acquisition practices continues to mount, we are troubled by DOD’s continued reluctance to fully adopt them. Service acquisition executives, now assigned as the milestone decision authority for most major programs within their military departments, are positioned to insist that programs under their purview implement knowledge-based practices. Such leadership, if displayed, could help reverse the cost growth we observed and position the military Page 2 GAO-19-336SP Weapon Systems Annual Assessment departments to obtain persistent efficiencies in the acquisition programs they manage. Gene L. Dodaro Comptroller General of the United States Page 3 GAO-19-336SP Weapon Systems Annual Assessment 441 G St. N.W. Washington, DC 20548 May 7, 2019 Congressional Committees In response to the joint explanatory statement accompanying the Department of Defense Appropriations Act, 2009, this report provides insight into the department’s $1.69 trillion portfolio of major weapon programs. 1 It includes observations on (1) the cost and schedule performance of DOD’s 2018 portfolio of 82 major weapon programs, (2) the extent to which 51 current and future programs (i.e., those that have not yet begun development) are implementing recent key acquisition reform initiatives, and (3) the knowledge attained by 51 current and future programs attained at key decision points in the acquisition process. This report also includes information related to small business participation, pursuant to a provision in a report to the National Defense Authorization Act (NDAA) for Fiscal Year 2013. 2 Specifically, we determined whether individual subcontracting reports from a program’s prime contractor or contractors were accepted within the Electronic Subcontracting Reporting System (eSRS). 3 Results from this analysis can be found in appendix I. Our observations in this report are based on three sets of programs: 1 See Explanatory Statement, 154 Cong. Rec. H 9427, 9526 (daily ed., Sept. 24, 2008), to the Department of Defense Appropriations Act, 2009, contained in Division C of the Consolidated Security, Disaster Assistance, and Continuing Appropriations Act, 2009, Pub. L. No. 110-329 (2008). 2 H.R. Rep. No. 112-479, at 284 (2012). The National Defense Authorization Act for Fiscal Year 2013, Pub. L. No.112-239. 3 The government uses individual subcontracting reports on eSRS as one method of monitoring small business participation, as the report includes goals for small business subcontracting. Page 4 GAO-19-336SP Weapon Systems Annual Assessment • • We assessed 82 major defense acquisition programs (MDAP) in DOD’s 2018 portfolio for cost and schedule performance and their use of competition in contracting. 4 • First, we obtained cost, schedule, and quantity data from DOD’s December 2017 Selected Acquisition Reports (SAR)—which detail initial cost, schedule, and performance baselines and changes over the past year—and from the Defense Acquisition Management Information Retrieval (DAMIR) system, a DOD repository for program data. We assessed data reliability by comparing the SAR data we entered into our weapon system database and the DAMIR data and determined the data were sufficiently reliable for the purposes of our report. • Next, we identified active major development and procurement contracts that programs reported in their December 2017 SARs and entered those contract numbers into the Federal Procurement Data System—Next Generation (FPDS-NG) to obtain data specific to those contracts. 5 We assessed data reliability by obtaining and reviewing a federal-wide summary of the system’s completeness and accuracy and the data validation rules and determined that the data were sufficiently reliable for the purposes of our report. We also assessed 45 MDAPs currently between the start of development and the early stages of production. We developed a questionnaire to obtain information on the extent to which these programs are following knowledge-based acquisition practices for technology maturity, design stability, and production readiness. The questionnaire asked program officials to provide information about systems engineering, design drawings, manufacturing planning and execution, and the implementation of specific acquisition reforms. In addition, the questionnaire requested that program officials provide 4 Major defense acquisition programs (MDAP) are those identified by DOD or that have a dollar value for all increments estimated to require eventual total expenditure for research, development, test, and evaluation of more than $480 million, or for procurement of more than $2.79 billion, in fiscal year 2014 constant dollars. DOD maintains a list of programs designated as future MDAPs. These programs have not formally been designated as MDAPs; however, DOD plans for these programs to enter system development, or bypass development and begin production, at which point DOD will likely designate them as MDAPs. We refer to these programs as future or pre-MDAPs throughout this report. Recent amendments to the statutory definition of an MDAP expressly exclude those acquisitions using the rapid fielding or rapid prototyping acquisition pathway described in section 804 of the National Defense Authorization Act for Fiscal Year 2016. 5 FPDS-NG is an automated system used to collect and report on federal procurement spending and the authoritative source for government-wide contract award data. Page 5 GAO-19-336SP Weapon Systems Annual Assessment details on scheduling, critical technology levels, major development and early procurement contract data, and other information. We received questionnaire responses from all 45 current MDAPs from October 2018 through January 2019. • We also assessed six future MDAPs not yet in the portfolio to gain additional insights into knowledge they plan to attain before starting development and their plans for implementing recent key acquisition reforms. We provided a questionnaire to program offices to obtain information on schedule events, costs, and acquisition reforms, and received responses from all six future programs from October 2018 through December 2018. In addition, we present individual assessments of 51 MDAPs, which include the 45 MDAPs currently in development or early production, as well as the six future programs. We used December 2017 SAR data as a starting point for reporting program cost information in the individual assessments. For most assessments, we were able to include more current cost information that we obtained through our web-based questionnaires. Appendix II provides additional information on our objectives, scope, and methodology. We conducted this performance audit from May 2018 to May 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Background DOD acquires new weapons for its warfighters through a management process known as the Defense Acquisition System. 6 This system includes three phases that defense acquisition programs generally proceed through, which are (1) technology maturation and risk reduction, (2) engineering and manufacturing development, and (3) production and deployment. In this report, we refer to these three phases more simply as 6 Department of Defense Directive 5000.01, The Defense Acquisition System (May 2003 [incorporating change 2 (Aug. 2018)]); Department of Defense Instruction 5000.02, Operation of the Defense Acquisition System (Jan. 2015) [incorporating change 4 (Aug. 2018)] (“DOD Instruction 5000.02”). Page 6 GAO-19-336SP Weapon Systems Annual Assessment technology development, system development, and production. Programs typically complete a series of milestone reviews and other key decision points that authorize entry into a new acquisition phase. Our body of work has shown that attaining high levels of knowledge before programs make significant commitments during product development drives positive acquisition outcomes. 7 We have found that in order to reduce risk there are three key points where programs should demonstrate critical levels of knowledge before proceeding to the next acquisition phase: development start, system-level critical design review, and production start. Figure 1 aligns the acquisition milestones described in DOD Instruction 5000.02, which establishes policy for the management of acquisition programs, with these three key decision points. 7 GAO, Best Practices: DOD Can Achieve Better Outcomes by Standardizing the Way Manufacturing Risks Are Managed, GAO-10-439 (Washington, D.C.: Apr. 22, 2010); Best Practices: High Levels of Knowledge at Key Points Differentiate Commercial Shipbuilding from Navy Shipbuilding, GAO-09-322 (Washington, D.C.: May 13, 2009); Defense Acquisitions: A Knowledge-Based Funding Approach Could Improve Major Weapon System Program Outcomes, GAO-08-619 (Washington, D.C.: July 2, 2008); Best Practices: Capturing Design and Manufacturing Knowledge Early Improves Acquisition Outcomes, GAO-02-701 (Washington, D.C.: July 15, 2002); Best Practices: Better Matching of Needs and Resources Will Lead to Better Weapon System Outcomes, GAO-01-288 (Washington, D.C.: Mar. 8, 2001); and Best Practices: Better Management of Technology Development Can Improve Weapon System Outcomes, GAO/NSIAD-99-162 (Washington, D.C.: July 30, 1999). Page 7 GAO-19-336SP Weapon Systems Annual Assessment Figure 1: DOD Acquisition Process and GAO-identified Knowledge Points Knowledge associated with these three points builds over time. Our prior work on knowledge-based approaches shows that a knowledge deficit early in a program can cascade through design and production, leaving decision makers with less knowledge to support decisions about when and how to move into subsequent acquisition phases that require more budgetary resources. Under a knowledge-based approach, demonstrating technology maturity is a prerequisite for moving forward into system development, during which time the focus should be on design and integration. Similarly, a stable and mature design is a prerequisite for moving into production, where the focus should be on efficient Page 8 GAO-19-336SP Weapon Systems Annual Assessment manufacturing. Appendix III provides additional details about key practices at each of the knowledge points. To accommodate shipbuilding programs in this report, we correlated detail design contract awards, fabrication starts, and lead ship deliveries with development start, critical design review, and production start, respectively. Our work has led to multiple recommendations that DOD has generally or partially agreed with and took steps to implement some of them in its acquisition policy. For example, our previous work recommended DOD ensure programs conduct a preliminary design review prior to starting development, and DOD’s policy now reflects this. Further, our work has influenced efforts within DOD and Congress to address some of the challenges in the defense acquisition system—primarily, that it takes longer and costs more to develop and produce the systems required to perform DOD’s various missions and operations. Notably, the Weapon Systems Acquisition Reform Act of 2009 (WSARA) sought to improve the way DOD acquires major weapon systems and incorporated many of our related recommendations. 8 Our work has shown that DOD, through its Instruction 5000.02, has incorporated WSARA and other initiatives to address sound management practices, such as realistic cost estimates, use of prototypes, and systems engineering. More recently, the Congress has passed legislation that introduced numerous additional acquisition reforms related to improving the performance of weapon acquisition programs. For this assessment, we reviewed MDAP implementation of several of these reforms. We focused on the reforms that involve oversight and acquisition principles consistent with our knowledge-based practices and provide mechanisms for more rapid fielding of warfighter capability. We explain the reforms in greater detail in appendix IV. These acquisition reforms are summarized as follows: • The NDAA for fiscal year 2016 contained specific provisions that authorize changes in certain areas of DOD’s acquisition process. 9 For example, some of these provisions included language pertaining to the alternate pathways for rapidly prototyping and fielding new capabilities, waivers of certain acquisition-related laws, and revisions to milestone decision responsibilities for MDAPs. The NDAA for fiscal 8 Pub. L. No. 111-23. 9 Pub. L. No. 114-92. Page 9 GAO-19-336SP Weapon Systems Annual Assessment year 2016 also contained a provision that permanently codified DOD’s “other transaction” authority for prototype projects. Under this authority, programs can enter into what are commonly referred to as “other transaction agreements,” which generally are not subject to federal acquisition or assistance statutes and regulations. Consequently, this authority provides agencies more flexibility to meet project requirements and mission needs. • The NDAA for fiscal year 2017 included provisions that require new acquisition programs to be designed and developed, to the maximum extent practicable, with a modular, open systems approach to enable incremental development and enhance competition, innovation, and interoperability. 10 It also included provisions that require an independent technical risk assessment in certain instances and goals for procurement unit cost, sustainment costs, initial operational capability, technology maturation, and prototyping. • The NDAA for fiscal year 2018 contained a provision titled “expanded other transaction authority for prototype projects” that increased the dollar values of the DOD-internal approval thresholds. The 2016 and 2017 NDAAs also contained provisions that altered the roles and responsibilities for MDAP oversight to give significantly more authority to the military departments for managing acquisition programs. Among other things, these provisions required the following: • The service acquisition executive of the military department concerned be designated as the milestone decision authority for most MDAPs initiated after October 1, 2016; • A process to address requests from service acquisition executives to revert existing programs’ milestone decision authority to them; • The establishment of two new executive offices—the Offices of the Under Secretary of Defense for Research and Engineering and the Under Secretary of Defense for Acquisition and Sustainment, which replaced the Under Secretary of Defense for Acquisition, Technology and Logistics; and • Use of an alternate milestone decision authority, under certain circumstances. 10 Pub. L. No. 114-328. Page 10 GAO-19-336SP Weapon Systems Annual Assessment The movement of milestone decision authority has begun to take place, as shown in figure 2. Figure 2: Milestone Decision Authority in DOD’s Portfolio of Major Defense Acquisition Programs, 2012-2019 Note: The figure does not account for classified programs, which we excluded from our analyses. We have ongoing work to more comprehensively assess DOD’s implementation of these and other recent acquisition reforms and we expect to issue a report in spring 2019. As part of that review, we are evaluating DOD’s new organizational structure that refocuses the Office of the Secretary of Defense’s principal role from program oversight to one intended to ensure that major DOD investments produce integrated, technically superior capability that consistently outpaces global threats. Page 11 GAO-19-336SP Weapon Systems Annual Assessment Nine Observations on the Performance of DOD’s 2018 MDAP Portfolio DOD’s 2018 MDAP portfolio consists of 82 programs and will cost $1.69 trillion to acquire. 11 We identified MDAPs as being in the 2018 portfolio if they published a December 2017 SAR. The portfolio is smaller yet more expensive than last year’s portfolio. The estimated total acquisition cost of the portfolio has grown by 51 percent, or $569 billion, since the identified programs’ first full estimates. While programs initiated since 2010 continue to outperform those initiated earlier, our analysis of the current portfolio indicates that this trend has weakened. The result is portfoliowide cost growth and a loss of buying power gains. Moreover, this growth has occurred in an acquisition environment where DOD has awarded 86 of the 183 major contracts that we reviewed to five corporations or entities connected with them and where most of the major contracts as reported in the SARs were awarded without full and open competition or on a sole source basis. In addition, current MDAPs did not compete 123 of these 183 contracts. DOD’s 2018 Portfolio Invested More Money across Fewer Programs Observation 1. The 2018 portfolio consists of 82 programs that will cost $1.69 trillion to acquire. This portfolio has four fewer programs than last year’s portfolio, but will require more money than any portfolio from the prior six years. • The current portfolio is smaller yet more expensive that last year’s portfolio. This is the third time in 10 years that a decrease in the number of portfolio systems was not accompanied by a corresponding decrease in expected costs. Specifically, the 2018 portfolio is 14 percent smaller than the 2008 portfolio, yet its projected completion cost is only 3 percent less since that time. • The average total acquisition cost per program has continued to increase over the last decade. In 2008, the average cost of an acquisition program was $18.3 billion, while the comparative average cost for 2018 is $20.6 billion. Figure 3 shows the number of programs and the total cost of the 2018 portfolio in relation to previous years. 11 All dollar figures are in fiscal year 2019 constant dollars, unless otherwise noted. Page 12 GAO-19-336SP Weapon Systems Annual Assessment Figure 3: Historical DOD Portfolio Comparison Note: DOD did not issue Selected Acquisition Reports (SAR) in 2009, which prevents us from obtaining the cost baseline information necessary to include 2009 in this analysis. • In 2018, two programs entered the portfolio and six exited. DOD approved the two entering programs for system development starts between January and December 2018. For five of the six exited programs, 90 percent of planned expenditures have been made. The Army terminated the remaining program, Warfighter Information Network—Tactical (WIN-T) Increment 3, due to affordability challenges and changes to the Army’s network strategy. Figure 4 shows programs entering and exiting the 2018 portfolio. Page 13 GAO-19-336SP Weapon Systems Annual Assessment Figure 4: Programs that Entered and Exited the 2018 Portfolio Note: The two entering programs produced their first Selected Acquisition Reports in December 2017, while the six exiting programs produced their final Selected Acquisition Reports in December 2016. Page 14 GAO-19-336SP Weapon Systems Annual Assessment Observation 2. The average age of programs in the portfolio has increased. While the number of programs in the portfolio has declined since last year, the average age of the programs that have remained has steadily increased since 2012. The current portfolio is approximately 4 months older than last year’s and nearly 3 years older than that from 2012. Figure 5 displays the increasing average age of programs in the portfolio since 2012. Figure 5: Annual Change in Average Program Age in DOD’s Portfolio of Major Defense Acquisition Programs, 2012-2018 The growing portfolio age is a byproduct of multiple factors, including the following: • DOD has introduced new capabilities and upgrades through additions to existing programs rather than by starting new programs—an Page 15 GAO-19-336SP Weapon Systems Annual Assessment approach inconsistent with best practices. 12 Best practices recommend an incremental approach in which new development efforts are structured and managed as separate acquisition programs with their own requirements and acquisition program baselines. If the effort is not established as a separate acquisition program, transparency is limited and it may be more difficult for Congress to hold the new effort accountable for achieving its cost, schedule, and performance requirements. MDAPs to which DOD has added warfighting capabilities under the existing program structure include the DOD-wide F-35 Lightning II Joint Strike Fighter and the Navy’s P8A, Arleigh Burke Class Destroyer (DDG 51), and Virginia Class Submarine (SSN 774). • Twenty programs have remained in the DOD MDAP portfolio for more than 20 years. Programs using DOD’s middle tier rapid prototyping and rapid fielding pathways, established pursuant to Section 804 of the fiscal year 2016 NDAA, are expressly excluded from the statutory definition of MDAP and therefore will not join the portfolio. For example, during the course of our review, six programs that DOD previously classified as pre-MDAPs are now proceeding via the alternate rapid prototyping and rapid fielding acquisition pathways. As newer programs choose this alternate path, the portfolio takes in fewer new programs than it otherwise would, which skews the average age of the portfolio toward older programs. Observation 3. Future funding needs remain within historic norms. • The current amount of future funding the portfolio needs decreased from $701.8 billion to $688.8 billion over the past year. 13 This decrease is largely due to a nearly $7 billion decrease in development and procurement costs since 2017 in the Army’s WIN-T Increment 2 program, which the Army has descoped. For the 2018 portfolio, DOD has programmed $32.6 billion for development and $656.2 billion for procurement activities. 12 GAO, F-35 Joint Strike Fighter: Continued Oversight Needed as Program Plans to Begin Development of New Capabilities, GAO-16-390 (Washington, D.C.: Apr. 14, 2016); GAO, Arleigh Burke Destroyers: Delaying Procurement of DDG 51 Flight III Ships Would Allow Time to Increase Design Knowledge, GAO-16-613 (Washington, D.C.: Aug 4, 2016). 13 Future funding is the amount of investment the portfolio requires to complete all of its development activities and to procure all of its planned quantities. Page 16 GAO-19-336SP Weapon Systems Annual Assessment • Total development and procurement costs already invested in the portfolio have increased from $962 billion to $981 billion over the past year. Both invested and future funding needs remain within the historical norms. • The Navy’s 40 MDAPs comprise 46 percent of the portfolio, but are responsible for 60 percent of the portfolio’s total invested development and procurement costs. In addition, the Navy’s share of future portfolio costs will likely grow because of its plan to grow its fleet size by 25 percent, starting in fiscal year 2019 and continuing through 2048. 14 • Importantly, our analysis of DOD’s MDAP portfolio excluded classified programs, such as the Air Force’s new B-21 Raider program. Figure 6 displays the current portfolio’s development and procurement funding (invested versus remaining) by military department. 14 GAO, Navy and Marine Corps: Rebuilding Ship, Submarine, and Aviation Readiness Will Require Time and Sustained Management Attention, GAO-19-225T (Washington, D.C.: Dec 12, 2018). Page 17 GAO-19-336SP Weapon Systems Annual Assessment Figure 6: Major Defense Acquisition Program Investments among Military Departments and DOD-wide within the 2018 Portfolio Note: The figure does not account for classified programs, which we excluded from our analyses. The figure also excludes $7.17 billion in spent development and procurement funding and $5.04 billion in development funding needed to complete the Chemical Demilitarization— Assembled Chemical Weapons Alternatives program managed by DOD. Page 18 GAO-19-336SP Weapon Systems Annual Assessment A Small Number of Aging Programs Are Driving Portfolio Cost Growth Observation 4. Over the past year, the total estimated acquisition cost of the portfolio increased by $26.6 billion, despite its net loss of four programs. • One year of cost growth for the current, smaller portfolio (plus the addition of two new systems, the Army’s CH-47F Block II and the Navy’s T-AO 205) was large enough to offset the cost decrease that resulted from the aforementioned exit of six systems, one being the Navy’s MH-60R Helicopter, with its estimated cost of $15 billion. • Of the $26.6 billion increase, nearly $4 billion is in research and development funds, while $21 billion is attributed to procurement costs. The remainder ($1.6 billion) is for costs associated with acquisition-related operational maintenance and system-specific military construction activities. Table 1 details the annual change to funding and average cycle times between 2017 and 2018 for initial capability delivery portfolios. Table 1: Cost and Schedule Changes to DOD’s 2018 Portfolio of 82 Major Defense Acquisition Programs over the Past Year (fiscal year 2019 dollars in billions) Estimated portfolio cost in 2017 Estimated portfolio cost in 2018 Estimated portfolio change since 2017 Percentage change since 2017 300.50 304.47 3.97 1.3 Total estimated procurement cost 1,344.25 1,365.29 21.04 1.5 Total estimated acquisition cost 1,659.03 1,685.65 26.62 1.6 120.1 121.9 1.8 1.5 Total estimated research and development cost Average cycle time to deliver initial capabilities (in months) Source: GAO analysis of Department of Defense data. GAO-19-336SP Note: In order to make the two portfolios comparable, we added the first full estimates of the two entering programs to last year’s portfolio and removed funding and schedule information of the six programs that exited the portfolio since last year. See appendix V for program- and military department-specific cost performance information, including figures related to the performance of individual programs over the past year and since first full estimates. • Over the past year, 60 programs experienced total acquisition cost increases totaling over $41 billion. While most programs realized an Page 19 GAO-19-336SP Weapon Systems Annual Assessment increase of less than 5 percent, total cost increases for three programs—the Navy’s Ford Class Aircraft Carrier (CVN 78) and DDG 51 and the Army’s Guided Multiple Launch Rocket Alternative Warhead System —comprise 60 percent, or nearly $25 billion, of the total growth. In all three cases, the preponderance of cost growth was caused by quantity increases. • At the same time, 22 programs experienced total acquisition cost decreases, totaling $14.5 billion. The Army’s WIN-T Increment 2 program quantity reductions accounted for $6.3 billion, or 43 percent, of that savings. The Army scaled down the program for the same reason it canceled WIN-T Increment 3—the platforms’ inability to meet the Army’s evolving networking needs. Figure 7 displays the portfolio’s total acquisition cost change from 2017 to 2018 by percentage change intervals, irrespective of changes in quantity. Page 20 GAO-19-336SP Weapon Systems Annual Assessment Figure 7: Total Acquisition Cost Increases and Decreases in Major Defense Acquisition Programs, 2017-2018 In December 2008, we reached agreement with DOD and the Office of Management and Budget on a set of metrics, which we have used to assess DOD’s performance since 2011. These metrics measure cost growth of DOD MDAPs since first full estimates and over the last 5-year and 1-year periods. Specifically, cost growth of less than 2 percent that occurs over a period of 1 year is “acceptable”, while growth occurring over 5 years and since program initiation must fall below 10 and 15 percent, respectively, to meet metric goals. DOD officials stated the metrics are dated and do not reflect current thinking within the department for tracking cost growth. Consequently, they no longer endorse their use. We have yet to agree on an alternative way of measuring performance over the specified periods. Such metrics Page 21 GAO-19-336SP Weapon Systems Annual Assessment provide DOD with a means to monitor and independently validate the effectiveness and sustainability of corrective measures—key criteria needed to remove DOD Weapon Systems Acquisition from our high-risk list. 15 In July 2018, DOD initiated discussions with us, which will continue in 2019, to develop a new set of agreed-upon metrics to track the portfolio’s performance over time. We believe the metrics ultimately chosen should allow for the department to identify specific programs whose costs are approaching unacceptable levels and that may require added attention. Until then, we will continue to rely on the previous agreed-upon standards, which show that programs continue to fall short of the metric for acceptable cost growth over all three recognized periods. Figure 8 presents the percentage of programs that satisfied the cost growth metrics. As the figure shows, programs have more successfully minimized cost growth over 1- and 5-year periods, as compared to cost growth since first full estimate. Appendix VI provides additional details on the portfolio’s historic cost performance against these metrics. 15 GAO-19-157SP. Page 22 GAO-19-336SP Weapon Systems Annual Assessment Figure 8: Major Defense Acquisition Programs’ Performance against Selected Cost Growth Metrics MDAP Schedules and Costs Have Grown by 35 Percent and 51 Percent, Respectively, since First Full Estimates Observation 5. Programs in the 2018 portfolio experienced a $569 billion increase in total acquisition cost since first full estimates and average schedule delays of more than 27 months, largely driven by aging programs. The majority of cost growth occurred after programs began production. • Programs’ average delay in delivering initial capabilities has increased by over 27 months since their first full estimates. Further, MDAPs show, on average, 51 percent total acquisition cost increase since their first full estimates, more than half of which is attributable to three programs—the Navy’s DDG 51 and SSN 774 and the DOD-wide F-35 Lightning II Joint Strike Fighter. • Programs that have been in the portfolio for over 20 years are the primary drivers of these outcomes. Within these programs, average delays in initial capability exceed 36 months and cost changes from first full estimates have increased by over 83 percent. The portfolio’s 20 oldest programs have amassed total acquisition cost growth of Page 23 GAO-19-336SP Weapon Systems Annual Assessment $474 billion since first full estimates. On the other hand, programs that have been in the portfolio for fewer than 10 years recorded decreasing costs of $3.4 billion since first full estimates and initial capability delays of less than 12 months. Table 2 breaks down the portfolio’s total acquisition cost and schedule changes since first full estimates, which are not adjusted for quantity changes. Table 2: Portfolio Cost and Schedule Changes since Major Defense Acquisition Programs’ First Full Estimates (fiscal year 2019 dollars in billions) Dollar change Percentage change Total research and development cost 103.49 51.5 Total procurement cost 461.42 51.0 4.50 39.6 569.41 51.0 27.4 34.9 Total other (military construction and operations and maintenance) acquisition cost Total acquisition cost Average delay in delivering initial capabilities (in months) Source: GAO analysis of Department of Defense data. GAO-19-336SP While the current portfolio’s costs have grown across all phases of the acquisition cycle, the majority of growth occurred after production start. As we have previously reported, cost growth during this acquisition phase may be symptomatic of programs entering production without attaining key knowledge about technology maturity, design stability, and production readiness. 16 We assess these parameters later in this report. Sixty-one of 82 portfolio programs are in production. Figure 9 displays the current portfolio’s cost growth across three acquisition phases. 16 GAO, Weapon Systems Annual Assessment: Knowledge Gaps Pose Risks to Sustaining Recent Positive Trends, GAO-18-360SP (Washington, D.C.: Apr. 25, 2018). Page 24 GAO-19-336SP Weapon Systems Annual Assessment Figure 9: Portfolio Cost Growth by Acquisition Phase (fiscal year 2019 dollars in billions) Note: In order to accommodate shipbuilding programs in this analysis, we correlated detail design contract awards, fabrication starts, and lead ship deliveries with development start, critical design review, and production start, respectively. Quantity Increases and Cost Reporting In the cases of SBIRS High and AEHF, the Air Force increased quantities by two additional units late into each programs’ acquisition cycle, which the Air Force called a “block buy.” Each program reported separate cost and funding information for the block buys and eventually removed the first four satellite units in each program’s Selected Acquisition Report after they were 90 percent complete. This reduced our visibility into each program’s cost performance and, as a result, our numbers do not reflect DOD’s total investment in them. Other contributing factors to high cost growth after programs enter production include increases in quantities or added capabilities after first full estimates are finalized. For example, satellite systems such as the Air Force’s Space-Based Infrared System (SBIRS) High and the Advanced Extremely High Frequency (AEHF) added additional satellite units in the form of “block buy” subprograms well after the programs’ first full estimates. Source: GAO analysis of Department of Defense data. GAO 19-336SP Programs’ Cost Performance Varied Considerably When Controlling for Quantity Changes Observation 6. For some programs, cost performance reversed when factoring in quantity changes. We analyzed the total amount of acquisition cost changes, not due to quantity changes, since first full estimates. We then ranked the programs in terms of highest and lowest performing, by both total dollars and total percentage, in tables 3 and 4 below. Page 25 GAO-19-336SP Weapon Systems Annual Assessment • Navy programs showed the widest variability in cost changes, with the best performing program and the second-worst performing program when adjusted for quantities. • Some programs showed improved cost performance when we accounted for quantity changes, as was the case of the Navy’s DDG 51 program. This program was one of three that drove overall portfolio cost growth since first full estimate. Yet, when we accounted for quantity changes, our analysis showed program cost reductions of more than $25 billion since first full estimate. Likewise, the Air Force Joint Direct Attack Munition (JDAM) program’s unadjusted total cost growth since first full estimate is almost $8.8 billion. When adjusted for quantities, it had a cost reduction of more than $3.3 billion since its first full estimate. • At least one program, the Navy’s DDG 1000, showed decreased costs, but not at a rate commensurate with the decreased quantities. In 1997, the Navy set an initial baseline for the program that required a quantity of 32 ships. In 2008, the Navy truncated the program at three ships. This substantial reduction in quantity has driven an overall reduction in program costs totaling $13.5 billion. Nonetheless, the Navy is now only procuring three ships for what it originally planned to spend on 20 ships. Tables 3 and 4 rank the 82 current MDAPs by the 10 programs with the lowest and highest total acquisition cost changes—unrelated to quantity changes—since programs’ first full estimates. Table 3 ranks the programs by changes in total dollars, while Table 4 ranks the programs by total percentage changes. Page 26 GAO-19-336SP Weapon Systems Annual Assessment Table 3: Ten Major Defense Acquisition Programs with the Lowest and Highest Total Acquisition Cost Change since First Full Estimates (fiscal year 2019 dollars in millions) 10 Programs with the Lowest Total Acquisition Cost Changes Since First Full Estimate Rank Program name Cost change Lead component 1. DDG 51 -$25,066.81 2. KC-46A -$7,851.91 Air Force 3. Multifunctional Information Distribution System -$5,039.60 Navy 12/15/1993 4. Joint Direct Attack Munition -$3,346.45 Air Force 10/15/1993 5. Joint Light Tactical Vehicle -$2,761.52 Army 12/22/2007 6. Airborne & Maritime/Fixed Station -$2,407.79 Army 12/15/2007 7. CVN 78 -$2,213.62 Navy 4/23/2004 8. HC/MC-130 Recap -$1,577.44 Air Force 2/15/2010 9. AH-64E New Build -$930.40 Army 7/15/2010 10. AIM-9X Blk II -$871.69 Navy 6/15/2011 Navy Program start date 5/15/1983 2/15/2011 10 Programs with the Highest Total Acquisition Cost Changes Since First Full Estimate Rank Program name Cost change 1. F-35 $147,689.74 2. V-22 3. Evolved Expendable Launch Vehicle 4. Lead component Program start date Joint Department of Defense(DOD) wide 11/15/1996 $40,945.63 Navy 12/15/1982 $39,607.54 Air Force 12/15/1996 SSN 774 $34,542.19 Navy 8/15/1994 5. DDG 1000 $19,451.31 Navy 12/15/1997 6. Trident II $16,068.45 Navy 10/15/1977 7. Advanced Medium Range Air-to-Air Missile $13,427.50 Air Force 11/15/1978 8. UH-60M Black Hawk $11,786.87 Army 4/15/2001 9. Guided Multiple Launch Rocket System and Alternate Warhead $11,234.73 Army 3/15/1998 10. Chemical Demilitarization—Assembled Chemical Weapons Alternatives $10,783.10 Joint DOD-wide 3/31/1998 Source: GAO analysis of Department of Defense data. GAO-19-336SP Page 27 GAO-19-336SP Weapon Systems Annual Assessment Table 4: Ten Major Defense Acquisition Programs with the Lowest and Highest Total Acquisition Cost Percentage Change since First Full Estimates 10 Programs with the Lowest Total Acquisition Cost Percentage Changes since First Full Estimate Rank Program name Percentage change Lead component Program start date 1 Multifunctional Information Distribution System -345 Navy 2 DDG 51 -147 Navy 3 Joint Direct Attack Munition -87 Air Force 4 AH-64E New Build -35 Army 7/15/2010 5 Airborne & Maritime/Fixed Station -26 Army 12/15/2007 6 AIM-9X Blk II -19 Navy 6/15/2011 7 HC/MC-130 Recap -17 Air Force 2/15/2010 8 KC-46A -16 Air Force 2/15/2011 9 B61 Mod 12 LEP TKA -15 Air Force 11/15/2012 10 Small Diameter Bomb II -14 Air Force 7/29/2010 12/15/1993 5/15/1983 10/15/1993 10 Programs with the Highest Total Acquisition Cost Percentage Changes since First Full Rank Program name Percentage change Lead component Program start date 1 Guided Multiple Launch Rocket System and Alternate Warhead 567 Army 3/15/1998 2 Chemical Demilitarization—Assembled Chemical Weapons Alternatives 364 Joint Department of Defense(DOD) wide 3/31/1998 3 CH-47F 280 Army 11/15/1997 4 MQ-1C Gray Eagle 263 Army 4/15/2005 5 H-1 Upgrades 216 Navy 9/15/1996 T6 Littoral Combat Ship 204 Navy 5/15/2004 T6 Evolved Expendable Launch Vehicle 204 Air Force 8 Ground/Air Task Oriented Radar 111 Navy 9 Advanced Medium Range Air-to-Air Missile 110 Air Force 11/15/1978 10 MQ-9 Reaper 108 Air Force 2/15/2004 12/15/1996 3/15/2005 Source: GAO analysis of Department of Defense data. GAO-19-336SP Appendix VII contains an expanded set of rankings that utilize the same criteria as the tables above, but identifies the best/worst performing five programs within each individual military department. Page 28 GAO-19-336SP Weapon Systems Annual Assessment Programs Initiated Since 2010 Continue to Outperform Others, but Positive Performance Continues to Slow Observation 7. Programs initiated since 2010 have realized a total acquisition cost increase of $1.4 billion since last year, whereas last year we found no growth in these programs. Those initiated before 2010 recorded an increase of $25.2 billion over the same period. Table 5 outlines changes in both the total estimated acquisition cost and the average cycle time to deliver initial capabilities for the programs initiated before and since 2010. Page 29 GAO-19-336SP Weapon Systems Annual Assessment Table 5: Programs’ Total Acquisition Cost and Cycle Time Changes, 2017 to 2018 (fiscal year 2019 dollars in billions) Estimated portfolio cost in 2017 Estimated portfolio cost in 2018 Estimated portfolio change since 2017 Percentage change since 2017 Programs initiated since 2010 Total estimated research and development cost 55.71 57.07 1.36 2.4 Total estimated procurement cost 219.18 219.06 -0.11 -0.1 Total estimated acquisition cost 278.41 279.84 1.43 0.5 92.2 93.7 1.5 1.6 Average cycle time to deliver initial capabilities (in months) Programs initiated before 2010 Total estimated research and development cost 244.79 247.40 2.61 1.1 Total estimated procurement cost 1,125.07 1,146.22 21.15 1.9 Total estimated acquisition cost 1,380.61 1,405.80 25.19 1.8 134.8 136.5 1.7 1.3 Average cycle time to deliver initial capabilities (in months) Source: GAO analysis of Department of Defense data. GAO-19-336SP While the two groups of programs initiated before and since 2010 both experienced total acquisition cost increases since last year, the percentage increase for older programs is more than triple that of the newer programs. This performance disparity supports the observation from prior years that acquisition reforms, namely the Weapon Systems Acquisition Reform Act of 2009 and three Better Buying Power initiatives, have driven reduced cost growth. 17 On the other hand, the cost growth now realized by the newer programs reflects a reversal since our prior assessment, when this group showed overall cost decreases. Over the past year, we observed a total of $1.4 billion in cost growth among these newer programs. Thus, instead of helping offset cost increases within the portfolio, as they have done in the past, newer programs directly contributed to the portfolio’s cost growth between 2017 and 2018. 17 Weapon Systems Acquisition Reform Act of 2009, Pub. L. No. 111-23; Between 2010 and 2015, the Under Secretary of Defense for Acquisition, Technology, and Logistics implemented “Better Buying Power” initiatives based on the principle that continuous improvement is the best approach to improving the performance of the defense acquisition enterprise. Page 30 GAO-19-336SP Weapon Systems Annual Assessment Similarly, while newer programs continue to minimize short-term cost growth more efficiently than older programs, that effectiveness has diminished. In particular, newer programs as a whole show additional cost growth between development start and critical design review (CDR) than in our prior assessment. This is likely because half of the newer programs have already passed, or are approaching, CDR. Figure 10 shows the cost performance by phase for both newer and older programs, as measured since first full estimate. Figure 10: Programs’ Total Acquisition Cost Changes since First Full Estimate by Acquisition Phases Note: To accommodate shipbuilding programs in this analysis, we correlated detail design contract awards, fabrication starts, and lead ship deliveries with development start, critical design review, and production start, respectively. Our analysis further showed that: • Overall, the current cost estimates for the 28 programs initiated since 2010 have decreased from their first full estimates, which is illustrated above as cost decreases incurred after both CDR and production start. While we acknowledge the decreases in projected future spending, only 15 of the 28 since-2010 programs have conducted their CDR, and only 11 are in production, which is the phase where cost growth has historically been the greatest. At this point, it is unclear whether the newer programs can maintain lower spending projections as they proceed through the acquisition process. • Despite the positive changes to cost projections beyond CDR and production for since-2010 programs, those same programs reported cost growth of $485 million between development start and CDR, almost doubling the growth we identified last year over the same acquisition timeframe. Page 31 GAO-19-336SP Weapon Systems Annual Assessment • In comparison, cost growth reported by the 54 programs initiated prior to 2010 occurred during all phases of the acquisition process, with the vast majority occurring after CDR or start of production. Figure 11 illustrates how the current portfolio’s total acquisition cost is apportioned between older programs that were initiated prior to 2010, newer ones initiated since 2010, and the F-35 Lightning II Joint Strike Fighter—DOD’s largest acquisition program, initiated in 2001. Figure 11: Distribution of Total Acquisition Costs among Various Groups of Programs • The deteriorating performance of newer programs can also be seen in their diminished buying power gains. Buying power is the amount of goods or services that can be purchased given a specified level of funding. To determine changes in buying power, the effects of quantity changes must be isolated from other factors that affect cost. • A program’s cost can increase because of additional quantities. While that does represent a cost increase, it does not necessarily indicate acquisition problems or a loss of buying power. Alternatively, a program’s cost can decrease due to a reduction in quantity and it may still experience a buying power gain or loss. Page 32 GAO-19-336SP Weapon Systems Annual Assessment • While newer, post-2010 programs still recorded a buying power gain of $263 million since last year, it is merely a fraction of past gains. For instance, post-2010 programs’ buying power gains amounted to nearly $5 billion in 2017. • On the other hand, the portfolio’s overall buying power gain of $3.9 billion was largely driven by the performance of older programs, 16 of which experienced a combined buying power increase of nearly $11 billion. Table 6 showcases buying power changes across the two groups of programs. Negative numbers indicate decreased costs and a gain in buying power, while positive numbers indicate the opposite. Page 33 GAO-19-336SP Weapon Systems Annual Assessment Table 6: Buying Power Changes Since 2017 (fiscal year 2019 dollars in millions) Buying Power since 2017 for Programs Initiated before 2010 Number of programs Increased buying power Actual procurement cost change GAO calculated cost change attributable to quantity changes GAO calculated cost change not attributable to quantity changes 16 9,525.75 20,508.76 -10,983.01 Procurement cost decreased with no quantity change 7 -4,178.24 0.00 -4,178.24 Quantity increased with less cost increase than anticipated 8 20,299.79 26,812.46 -6,512.67 Quantity decreased with more cost decrease than anticipated 1 -6,516.96 -6,303.70 -213.26 Decreased buying power 36 11,621.06 4,352.69 7,268.37 Procurement cost increased with no quantity change 23 3,213.37 0.00 3,213.37 Quantity increased with more cost increase than anticipated 6 11,534.59 9,511.20 2,023.39 Quantity decreased with less cost decrease than anticipated 7 -3,205.74 -5,158.51 1,952.77 No change in buying power 2 0.00 0.00 0.00 54 21,146.81 24,861.45 -3,714.64 Increased buying power 9 -1,178.12 76.92 -1,255.04 Procurement cost decreased with no quantity change 6 -1,064.50 0.00 -$1,064.50 Quantity increased with less cost increase than anticipated 2 55.46 173.81 $118.35 Quantity decreased with more cost decrease than anticipated 1 -169.08 -96.89 $72.19 Decreased buying power 17 1,066.87 75.02 991.85 Procurement cost increased with no quantity change Sub-portfolio Totals Buying Power since 2017 for Programs Initiated since 2010 16 952.58 0.00 952.58 Quantity increased with more cost increase than anticipated 1 114.29 75.02 39.27 Quantity decreased with less cost decrease than anticipated 0 0.00 0.00 0.00 No change in buying power 2 0.00 0.00 0.00 Sub-portfolio Totals 28 -111.25 151.94 -263.19 Portfolio Totals 82 21,035.56 25,013.39 -3,977.83 Source: GAO analysis of Department of Defense data. │GAO-19-336SP Page 34 GAO-19-336SP Weapon Systems Annual Assessment DOD MDAPs Operate in an Environment of Limited Competition and a Constrained Defense Contractor Base Observation 8. Current MDAPs did not compete 67 percent of their currently reported major development and procurement contracts. We have reported that competition is the cornerstone of a sound acquisition process and a critical tool for achieving the best return on investment for taxpayers. 18 Generally, a low competition rate can contribute to increased costs of goods and services and decreased buying power. According to the Office of Federal Procurement Policy, “ineffective competition”—when one offer is received in response to a competitive solicitation—deprives the programs and their managing agencies of the ability to consider alternative solutions in a reasoned and structured manner, thereby creating opportunities for increased costs, substandard products and prolonged acquisition cycles. In July 2009, the Office of Management and Budget (OMB) issued a memorandum to department and agency heads in which it categorized noncompetitive contracts as high-risk because, absent competition, agencies must negotiate contracts without the benefit of a direct market mechanism to help set the contract price. Our prior work has identified various factors that affect competition rates, including the government’s preference for a specific vendor, inadequate acquisition planning, and overly restrictive government requirements. Moreover, we identified DOD-specific factors, such as reliance on original manufacturers and entry barriers resulting from the complexity of DOD’s contracting process and lack of data rights—legal rights a contractor and/or the government have to use technical, intellectual property and software data—as contributing to DOD’s particularly low competition rates. 19 Creation and sustainment of competitive environments was a recurring theme in DOD’s Better Buying Power policy memorandums issued between 2010 and 2015. In order to analyze DOD’s performance in generating competition in contracting within the 2018 MDAP portfolio, we compiled a list of all 183 active major contracts and orders identified in individual programs’ 2017 SARs and compared them to information in the 18 GAO, Defense Contracting: Early Attention in the Acquisition Process Needed to Enhance Competition, GAO-14-395 (Washington, D.C.: May 5, 2014). 19 GAO, Federal Acquisitions: Congress and the Executive Branch Have Taken Steps to Address Key Issues, but Challenges Endure, GAO-18-627 (Washington, D.C.: Sept. 12, 2018). Page 35 GAO-19-336SP Weapon Systems Annual Assessment FPDS-NG database. 20 All but three of these contracts and orders were for development or procurement. 21 Of the 183 contracts and orders included in our review: • the department awarded 180 for development and procurement efforts, • the Navy awarded 85 contracts that totaled $147.1 billion, • the Air Force awarded 51 contracts that totaled $132.6 billion, • the Army awarded 37 contracts that totaled $50.5 billion, and • Joint, DOD-wide offices awarded 10 contracts that totaled $33.8 billion. Figure 12 shows the percentage of the 183 contracts in our analysis that DOD either competed or did not compete, as well as the reported number of offers DOD received for each category. Appendix II provides more details on the selection of contract actions included in this analysis, as well as how we define “competitive” and “non-competitive” actions. 20 By statute, SARs include an MDAP’s “major contracts,” which are defined as “each of the six largest prime, associate, or Government-furnished equipment contracts under the program that is in excess of $40,000,000 and that is not a firm, fixed price contract.” An MDAP’s SAR requirements cease after 90 percent of the total quantity of items to be purchased under the program are delivered or 90 percent of planned expenditures are made. DOD guidance states that once a contract is 90 percent complete, it is no longer reported in the SAR. 21 Of the 183 contracts and orders in our analysis, DOD reporting indicated that 115 were for procurement; 65 were for research, development, test, and evaluation; two were for acquisition-related operations and maintenance; and one was for system-specific military construction. Page 36 GAO-19-336SP Weapon Systems Annual Assessment Figure 12: Competition in Currently Reported Major Contracts and Number of Offers Received Note: In order to analyze DOD’s performance in generating competition in contracting within the 2018 major defense acquisition program portfolio, we compiled all 183 contracts listed on individual programs’ 2017 SARs and compared them to information in the Federal Procurement Data SystemNext Generation database. Federal statutes and acquisition regulations generally require that contracts be awarded on the basis of full and open competition. However, they also permit agencies to award noncompetitive contracts in certain circumstances, such as: • only one responsible source and no other supplies or services will satisfy agency requirements; • unusual and compelling urgency; and • when authorized or required by statute (for example, sole source awards to small businesses.) 22 Additionally, Federal Acquisition Regulation (FAR) part 34, which establishes policies for the acquisition of major systems, directs agencies and program managers to promote and sustain effective competition among alternative system concepts and sources for “as long as it is 22 FAR §§ 6.302-1, 6.302-2, 6.302-5. Page 37 GAO-19-336SP Weapon Systems Annual Assessment economically beneficial and practicable to do so.” 23 Thus, the FAR reflects that there may be a point in a major system acquisition at which promoting and sustaining competition among alternative sources is no longer economically beneficial and practicable. Our analysis of the 183 contract awards also found the following: • Competition rates for research and development contract awards were considerably higher, at 81 percent competed, than those for procurement contract awards, where the competition rate was barely over 15 percent. This dichotomy may be linked to the government’s tendency to rely on original weapon systems equipment manufacturers for follow-on procurements, to include sustainment contracts. 24 • In March 2017, we reported on overall trends in defense and civilian agencies’ contract obligations, such as trends in competition. 25 In that report, we noted that DOD produces annual competition reports, which recognize that competition achievement varies by organization based on the mission and types of products and services being procured. For example, competition rates are lower in organizations that buy major weapon systems—such as those covered in this annual assessment—particularly as systems move into production. 23 FAR §§ 34.002(b), 34.005-1(a). In FAR part 34, “effective competition” is defined as “a market condition that exists when two or more contractors, acting independently, actively contend for the Government's business in a manner that ensures that the Government will be offered the lowest cost or price alternative or best technical design meeting its minimum needs.” FAR § 34.001. 24 The additional systems or sustainment are often procured through contract modifications or the exercise of contract options. This situation is partly attributable to the unique relationship that DOD has with the defense industry that differs from the commercial marketplace. The combination of a single buyer (DOD), few very large prime contractors in each segment of the industry, and a limited number of weapon programs constitutes a structure for doing business that is altogether different from a classic free market. For instance, there is less competition and once a contract is awarded, the contractor often remains the sole vendor capable of providing additional systems and sustainment. These long-term contractual relationships with weapon system contractors limit opportunities for competition. See GAO-14-395. 25 GAO, Contracting Data Analysis: Assessment of Government-wide Trends, GAO-17-244SP (Washington, D.C.: Mar. 9, 2017). Page 38 GAO-19-336SP Weapon Systems Annual Assessment Figure 13 displays the percentage of the 183 contracts that DOD awarded competitively and noncompetitively—by military department and Joint, DOD-wide—we analyzed. Figure 13: Competition Rates for 183 Currently Reported Major Contract Awards within DOD’s 2018 Portfolio of Major Defense Acquisition Programs Figure 14 shows total dollar amounts of 180 major development and procurement contracts that DOD awarded competitively and noncompetitively—by military department and Joint, DOD-wide. Page 39 GAO-19-336SP Weapon Systems Annual Assessment Figure 14: Dollars Associated with Competition Results for 180 Currently Reported Major Development and Procurement Contract Awards within DOD’s 2018 Portfolio of Major Defense Acquisition Programs A Small Group of Contractors Received the Majority of MDAP Contract Award Dollars Observation 9. DOD awarded 47 percent of its currently reported 183 major development and procurement contract awards to five corporations and entities connected with them. This 47 percent constituted 72 percent of the dollars associated with the 183 major contracts; the remaining 28 percent was distributed among 30 other companies. • Of the 183 major contract awards we analyzed, five corporations and entities connected with them received a combined 86 awards totaling over $262 billion. The prime contractors are Boeing, Lockheed Martin, General Dynamics, United Technologies, and Northrop Grumman. DOD awarded 97 contracts totaling almost $100 billion to 30 other Page 40 GAO-19-336SP Weapon Systems Annual Assessment companies. Figure 15 shows how DOD divided the 183 contract awards we reviewed among the top five prime contractors and remaining companies. Figure 16 shows the total dollars associated with these contract awards by prime contractor. Figure 15: Distribution of Contract Awards among Prime Contractors for 183 Currently Reported Major Contracts within DOD’s 2018 Portfolio of Major Defense Acquisition Programs Note: Percentages for each of the five prime contractors identified in the figure also include awards to entities connected with them. Values may not sum to 100 percent due to rounding. Page 41 GAO-19-336SP Weapon Systems Annual Assessment Figure 16: Distribution of Dollars among Prime Contractors for 183 Currently Reported Major Contracts within DOD’s 2018 Portfolio of Major Defense Acquisition Programs Notes: Dollar values for each of the five prime contractors identified in the figure also include those from entities connected with them. These contract values include the base contract and all options, regardless of whether the option has been, or will be, exercised. Our Observation on DOD’s Application of Key Acquisition Reforms Our analysis of questionnaire responses from 51 MDAPs shows that these programs have applied, to varying degrees, six selected acquisition reforms included in NDAAs between fiscal years 2016 and 2018. We completed this analysis to identify the extent to which reforms have taken hold across DOD, not to evaluate programs’ compliance in implementing them. Prior to passage of these newer acquisition reforms, numerous programs had already taken steps related to the areas addressed by some of the reforms. For instance, many programs that were well into either development or production had crafted an acquisition strategy that called for the use of a modular, open-systems design prior to the related reform described below. Others had previously established goals for numerous acquisition elements, such as procurement unit costs, sustainment costs, and an initial operational capability date. However, these reforms are still relatively new, and hence will apply primarily to newer systems that have yet to begin development. In other cases, the reforms offer options—as opposed to requirements—for the department and programs to consider utilizing during program execution. Page 42 GAO-19-336SP Weapon Systems Annual Assessment Observation 1. Six acquisition reforms we reviewed have gained varying degrees of traction within the 51 programs we surveyed; even though many of these programs predate these reforms, programs have taken action in areas related to the reforms. Forty-five of the 51 programs are well into either development or production and beyond several of the key acquisition phases where application of many of the reforms might be expected to occur. At the same time, most, but not all, of the six reforms we reviewed built upon concepts that already existed in policy or law. For example, DOD’s Better Buying Power initiatives have emphasized the use of modular, open systems architecture. Table 7 presents an overview of the six acquisition reforms we included in our analysis. Appendix IV contains more detailed descriptions of these acquisition reforms. Table 7: Overview of Selected, Recent Acquisition Reforms Legislative provision Name of reform Description Fiscal Year (FY) 2016 Middle tier of acquisition for rapid National Defense prototyping and rapid fielding Authorization Act (NDAA) § 804 Calls for the Department of Defense (DOD) to issue guidance for a “middle tier” of acquisition programs to be completed in two to five years through two types of “acquisition pathways:” rapid prototyping and rapid fielding. FY 2016 NDAA § 806 Secretary of Defense waiver of acquisition laws to acquire vital national security capabilities Authorizes the Secretary of Defense to waive any provision of certain types of acquisition laws and regulations to, among other things, meet vital national security needs. FY 2016 NDAA § a 823 Requires a milestone decision authority to make various written determinations before granting Milestone A approval, including that an MDAP: (1) fulfills an approved Initial Capabilities Document, (2) is being developed in light of appropriate market research, (3) has a risk reduction plan for any identified areas of risk (4) addresses sustainment planning, and (5) has a cost estimate and the level of resources is sufficient for successful program execution. Revision of Milestone A decision authority responsibilities for major defense acquisition programs (MDAP) FY 2017 NDAA § 805 Modular, open-systems approach in development of MDAPs Page 43 Requires that MDAPs that receive Milestone A or Milestone B approval after January 1, 2019 must be designed and developed, to the maximum extent practicable, with a modular, opensystems approach to enable incremental development and enhance competition, innovation, and interoperability. GAO-19-336SP Weapon Systems Annual Assessment Legislative provision Name of reform Description FY 2017 NDAA § 807 Cost, schedule and performance goals of b MDAPs Requires that (1) DOD set goals for MDAPs’: a) procurement unit cost, b) sustainment costs, c) initial operational capability date, d) technology maturity, and e) prototyping, and (2) ensure that MDAPs complete an independent technical c risk assessment FY 2018 NDAA § 864 Other transaction (OT) authority for certain prototype projects Increased the service acquisition executive and Under Secretary of Defense approval thresholds for use of OTs for prototype projects to $100 million and $500 million, respectively. Source: GAO analysis of the NDAAs for fiscal years 2016-2018 and of MDAP responses to GAO’s questionnaire. │GAO-19-336SP a Prior legislation required a milestone decision authority to certify to some of the same or similar elements as those in FY2015 NDAA § 823. See 10 U.S.C. 2366a (2014), amended by FY17 NDAA, Pub. L. No. 114-92, § 823 (2015). b We did not include Air Force systems within the tallies for this acquisition reform. c We collected data associated with this acquisition reform, but later identified information that led us to conclude this data is unreliable. As a result, we excluded this information from our analysis and we will obtain this information for future reporting. Table 8 illustrates instances where programs reported taking actions in an area addressed by the acquisition reforms. In certain cases, these actions preceded the reforms. Page 44 GAO-19-336SP Weapon Systems Annual Assessment Table 8: Actions in Areas Related to Recent Acquisition Reforms as Reported by 51 Current and Future Major Defense Acquisition Programs (MDAP) Areas addressed by acquisition reform Number of MDAPs that reported actions in the area Secretary of Defense waivers of acquisition laws and regulations 3 Revision of milestone A decision authority responsibilities for MDAPs, to include milestone decision authority written determinations that the program 1. fulfilled an approved Initial Capabilities Document, 2. is being developed in light of appropriate market research, 3. has a risk reduction plan in place for any identified areas of risk 4. addresses sustainment planning, and a 5. has a cost estimate and the level of resources is sufficient for successful program execution 6 5 8 8 7 Modular, open-systems approach in development of MDAPs 33 Cost, schedule, and performance goals of MDAPs’: a. procurement unit cost, b. sustainment costs, c. initial operational capability date, d. technology maturity, and e. prototyping, and ensure that MDAPs complete an independent technical risk assessment 22 21 29 12 7 — Other transactions authority for certain prototype projects Middle tier of acquisition for rapid prototyping and rapid fielding 2 b 0 Source: GAO analysis of Department of Defense data. │GAO-19-336SP a Prior legislation required a milestone decision authority to certify to some of the same or similar elements as those in FY2015 NDAA § 823. See 10 U.S.C. 2366a (2014), amended by FY17 NDAA, Pub. L. No. 114-92, § 823 (2015 ). While many programs provided questionnaire responses regarding areas addressed by the acquisition reforms, only 22 current MDAPs reported having held a milestone A review. Further, only one program (Improved Turbine Engine Program) held its milestone A review after enactment of the FY16 NDAA. One pre-MDAP (Precision Strike Missile) was initiated at milestone A, and this also occurred after enactment of the FY16 NDAA. b The statute that defines what constitutes an MDAP states that MDAPs do not include programs or projects carried out using the middle tier rapid prototyping or rapid fielding acquisition pathway. 10 U.S.C. § 2430(a)(2)(A). However, section 804 of the FY16 NDAA contemplates that the rapid prototyping pathway will include a process for transitioning successful prototypes to new or existing programs for production and fielding, under either the rapid fielding pathway or the traditional acquisition system. FY16 NDAA, Pub. L. No. 114-92, § 804 (2015) (codified, as amended, at 10 U.S.C. § 2302 note). Some examples of programs’ reports regarding areas addressed by these acquisition reforms include the following: • The Army’s Improved Turbine Engine Program was the only program among 45 current MDAPs we surveyed that was scheduled to enter Page 45 GAO-19-336SP Weapon Systems Annual Assessment system development after January 1, 2019. The program reported having an acquisition strategy with a modular, open-systems approach, as well as applying many features in the acquisition reform provisions cited in our questionnaire. • None of the 45 current MDAPs in our review reported using any other transaction authority instruments in connection with Section 864 of the NDAA for Fiscal Year 2018. Only two of the 6 future MDAPs reported using other transaction agreements. The Army’s Precision Strike Missile reported awarding two separate OTs totaling $240.5 million, although the awards occurred prior to enactment of this provision. • Although our other ongoing work has identified that DOD has begun over 30 middle-tier acquisition programs under the authority granted in DOD guidance implementing Section 804 of the NDAA for Fiscal Year 2016, MDAPs did not report any plans to use prototypes associated with this authority. For all 51 current and future MDAPs we surveyed, program officials could not identify any middle tier prototypes that, as of February, 2019, had transitioned into their offices for production or fielding. Further, for the 45 current MDAPs we surveyed, officials from 44 of the programs reported that they either did not anticipate receiving any such prototypes in time for critical design review or production decision, or that the question was not applicable. Only one program reported that it did not know whether it would receive such prototypes in the future. We have ongoing work to assess DOD’s efforts to more broadly implement many of these acquisition reforms. We will continue to assess DOD’s progress going forward. Eight Observations from Our Assessment of Knowledge Attained at Key Decision Points and Related Implications for Testing and Software Development Based on our analysis of 51 selected weapon acquisition programs, we found that while a few programs implemented elements of knowledgebased acquisition practices, implementation across the entire portfolio remains inconsistent. Numerous acquisition programs, both current and future, have proceeded, or plan to proceed, through key acquisition points without the requisite knowledge called for in best practices. For instance, several programs proceeded through key milestones without attaining the knowledge recommended for the event—such as maturing technologies and conducting a preliminary design review prior to starting system development, or testing a system-level integrated prototype to inform a critical design review. Several programs do not plan to deliver a baseline software capability until after production start, and many others plan to declare an initial operational capability in advance of testing. Actions like these increase the risk of cost growth and schedule delays. Page 46 GAO-19-336SP Weapon Systems Annual Assessment For the second year in a row, we analyzed programs that have progressed through all three knowledge points—development start, critical design review, and a production decision. This year’s analysis includes 17 programs for which we have previously collected information for prior assessments. This year our analysis revealed that MDAPs that implemented key knowledge practices saw substantially better cost performance and less schedule growth than programs that have not implemented those same practices. This analysis emphasizes the importance of knowledge-based acquisition practices and the positive effects they can have on acquisition outcomes. Current Programs Are Inconsistently Following Knowledge-Based Approaches 1. MDAPs have inconsistently applied key knowledge practices, even newer programs that have more recently progressed through early acquisition events. Few programs held preliminary design reviews in advance of development start decisions or demonstrated critical technologies in a realistic environment. Table 9 presents key knowledge practices and the percent of current acquisition programs that attained the requisite knowledge for each practice. Table 9: Major Defense Programs Have Not Consistently Implemented Knowledge-Based Acquisition Practices Practices associated with the three key knowledge points (KP) Programs that completed the KP before this assessment period KP 1 practices Programs that completed the KP during this assessment period 38 programs Four programs Demonstrate all critical technologies are very close to final form, fit, and function within a relevant environment ◐ ◐ Demonstrate all critical technologies are in form, fit, and, function within a realistic environment ○ ○ Completed preliminary design review before system development start ◐ ○ 33 programs Two programs Release at least 90 percent of design drawings to manufacturing ○ ● Test a system-level integrated prototype ○ ○ 15 programs Three programs Demonstrate critical manufacturing processes are in statistical control ○ ○ Demonstrate critical processes on a pilot production line ◐ ● KP 2 practices KP 3 practices Page 47 GAO-19-336SP Weapon Systems Annual Assessment Practices associated with the three key knowledge points (KP) Programs that completed the KP before this assessment period Test a production-representative prototype in its intended environment ○ Programs that completed the KP during this assessment period ● Legend: ● Practice implemented in 75 – 100 percent of programs ◐ Practice implemented in 50 – 74 percent of programs ○ Practice implemented in 0 – 49 percent of programs Source: GAO analysis of Department of Defense data. GAO-19-336SP Notes: To accommodate shipbuilding programs in this analysis, we correlated detail design contract award with development start. We assessed knowledge-based acquisition practices as not applicable in situations when programs submitted responses to our questionnaire that identified events or circumstances that made completion of a practice impractical. For example, we assessed the technology demonstration practice as not applicable in instances where a program did not have any critical technologies. Observation 2: Programs’ definitions of critical technologies varied. Most programs did not fully demonstrate mature technologies, and only about half completed system engineering reviews, before starting development. Establishing a consistent definition of the term “critical technology” is important for acquisition programs so they can correctly identify all critical technology elements. As we have previously reported, not doing so could result in an underrepresentation of the technical risk facing the program. 26 GAO’s Technology Readiness Assessment Guide establishes criteria for identifying critical technology elements. 27 It states that a critical technology element is one that is new or novel, and necessary for a system to meet its anticipated operational performance requirements; or poses a major cost, schedule, or performance risk during design or demonstration. We observed that while the major acquisition programs in our review define the term “critical technology” differently from one another, these reported definitions frequently included several common elements. For instance, the three most common elements program officials included in their definitions were that 26 GAO, Columbia Class Submarine: Immature Technologies Present Risks to Achieving Cost, Schedule, and Performance Goals, GAO-15-158 (Washington, D.C.: Dec. 21, 2017). 27 GAO, GAO Technology Readiness Assessment Guide: Best Practices for Evaluating the Readiness of Technology for Use in Acquisition Programs and Projects--Exposure Draft, GAO-16-410G (Washington, D.C.: Aug. 11, 2016). Page 48 GAO-19-336SP Weapon Systems Annual Assessment • the system being acquired depends on the technology element to meet requirements, • the technology is new or novel, and • the technology poses a major technological risk. Eighteen of 51 programs included these three elements in their definition of “critical technology.” According to knowledge-based best practices, programs should fully demonstrate critical technologies in a realistic environment and conduct a preliminary design review before starting system development. DOD and commercial technology development cases show the more mature technology is at the start of the program, the more likely the program will succeed in meeting its objectives. 28 Technologies that were included in a product development before they were mature later contributed to cost increases and schedule delays in those products. 29 Table 10 presents the number of programs that had fully matured technologies when they began system development as well as the number of programs that had technologies approaching maturity at that time. The table presents the information for 42 current MDAPs that began development before or during our assessment period. 30 See appendix VIII for additional details on technology readiness levels. 28 GAO-16-410G. 29 Ibid. 30 Three of the current MDAPs we assessed entered the acquisition cycle postdevelopment start. We therefore excluded these three programs from this analysis. Page 49 GAO-19-336SP Weapon Systems Annual Assessment Table 10: Technology Readiness Levels (TRL) Reported by 42 Current Major Defense Acquisition Programs Thirty-eight programs that began development before our assessment period System prototype demonstration in an operational environment (TRL 7) System/subsystem model or prototype demonstration in a relevant environment (TRL 6) Number of systems Percentage of systems Number of systems Percentage of systems 3 8 percent 24 63 percent 30 79 percent 9 24 percent 5 13 percent 5 13 percent Met Not met Not applicable or information not available Four programs that began development during our assessment period System prototype demonstration in an operational environment (TRL 7) System/subsystem model or prototype demonstration in a relevant environment (TRL 6) Number of systems Percentage of systems Number of systems Percentage of systems Met 0 0 percent 2 50 percent Not met 2 50 percent 0 0 percent Not applicable or information not available 2 50 percent 2 50 percent Source: GAO analysis of Department of Defense data. GAO-19-336SP Note: A technology achieves TRL 7 when it has achieved form, fit, and function, and has been demonstrated in an operational environment. A technology that achieves TRL 6 is close to form, fit, and function, and has been demonstrated in a high-fidelity laboratory environment. Progressing from TRL 6 to TRL 7 is a significant step for a critical technology. While GAO’s best practices work has shown that a TRL 7 is the level of technology maturity that constitutes a low risk for starting development, DOD’s policy, permits development to start at TRL 6. DOD’s policy is based on a statute that generally prohibits a major defense acquisition program from receiving approval for development start until the milestone decision authority certifies—based on an independent review and technical risk assessment—that the technology in the program has been demonstrated in a relevant environment. 10 U.S.C. § 2366b(a)(2). Page 50 GAO-19-336SP Weapon Systems Annual Assessment Impact of Immature Technologies In two extreme cases, the Family of Advanced Beyond Line-of-Sight Terminals (FAB-T) and the F-35 Joint Strike Fighter, the programs still had immature critical technologies that, at the start of development, were basically advanced breadboard demonstrators in a laboratory environment, which does not reach the maturity level associated with best practices for starting development. The current cost growth for the FAB-T and F-35 since their first full estimates is 31 percent and 51 percent, respectively. Each program also had significant schedule delays for declaring initial operational capabilities. The FAB-T IOC date slipped 96 months and the F-35 IOC date slipped 62 months. Source: GAO analysis of Department of Defense data. GAO 19-336SP According to knowledge-based practices, DOD policy, and statute, programs should conduct a preliminary design review (PDR) before beginning system development. 31 DOD policy also generally requires programs to complete formal system reviews in advance of beginning development. Further, best practices require programs to constrain their development periods to 6 years or less. 32 These reviews are intended to demonstrate that requirements are well understood and that the systems’ functional baselines can satisfy those requirements. Table 11 identifies the number of programs that held preliminary design reviews, key system reviews, and constrained development phases to 6 years or less. The table presents information for 42 current MDAPs that began development before and during our assessment period. 33 31 The purpose of a PDR is to assess the maturity of the preliminary design, supported by the results of requirements trades, prototyping, and critical technology demonstrations. The PDR establishes the allocated baseline (a definition of the configuration items making up a system, and then how system function and performance requirements are allocated across lower level configuration items) and confirms that the system under review is ready to proceed into detailed design. In detailed design, the program office will develop build-to drawings, software code-to documentation, and other fabrication documentation. By statute, a major defense acquisition program generally may not receive approval for development start until the milestone decision authority has received a preliminary design review, conducted a formal assessment of the preliminary design review, and certifies, based on that assessment, that the program has a high likelihood of accomplishing its intended mission. 10 U.S.C. § 2366b(a)(1). Under certain circumstances, this requirement may be waived. Id. § 2366b(d) 32 GAO-18-360SP. 33 Three of the current MDAPs we assessed entered the acquisition cycle postdevelopment start. We therefore excluded these three programs from this analysis. Page 51 GAO-19-336SP Weapon Systems Annual Assessment Table 11: Number of 42 Selected, Current Programs that Met Key Knowledge Practices Prior to Beginning Development Thirty-eight programs that began development before our assessment period Held a preliminary design review before starting development Held system reviews before starting development Constrain development phase to 6 years or less Yes 19 16 19 No 17 12 8 2 10 11 Held a preliminary design review before starting development Held system reviews before starting development Constrain development phase to 6 years or less Yes 0 2 2 No 1 0 0 Not applicable or information not available 3 2 2 Not applicable or information not available Four programs that began development during our assessment period Source: GAO analysis of Department of Defense data. GAO-19-336SP • As in prior years, many programs in the 2018 portfolio began development without conducting a preliminary design review. DOD policy implementing related statute requires that PDR occur in advance of development start, unless the program’s milestone decision authority grants a waiver for this requirement. Of the 38 programs that began development before our assessment period, only half conducted a preliminary design review prior to beginning system development. 34 • Of the four programs that began development during our assessment period, two do not plan to conduct a preliminary design review, and one has yet to identify a date for its preliminary design review. Only one program—the Presidential Aircraft Recapitalization (PAR) — has conducted a PDR, but it occurred three months after beginning development, which is inconsistent with best practices. • Also, 12 programs did not hold system reviews in advance of starting development, as generally required by DOD policy. Three of those 34 We did not evaluate why these programs did not hold a preliminary design review in advance of beginning development. In some cases, programs may have received a waiver from their milestone decision authority. Page 52 GAO-19-336SP Weapon Systems Annual Assessment programs had cost growth of greater than 55 percent, while two other programs had modest cost growth of 10 percent or less. Two other programs, the Army’s JLTV program and the Air Force’s KC-46 program, had cost reductions of 5 percent and 16 percent, respectively. Observation 3. Most programs did not fully demonstrate system design stability prior to conducting critical design reviews. We found that 35 programs held a knowledge point 2 event (critical design review) either before or during our assessment period. Table 12 shows the number of those that satisfied the two associated key knowledge-based acquisition practices. Only two programs held a critical design review (or, for ships, began construction) during our assessment period. Table 12: Number of Programs Demonstrating Sufficient Design Knowledge Prior to Knowledge Point 2 Thirty-three programs that held a critical design review before our assessment period Number of programs that tested an early system-level integrated prototype Number of programs that released at least 90 percent of design drawings Yes 4 11 No 22 17 7 5 Not applicable or information not available Two programs that held a critical design review during our assessment period Number of programs that tested an early system-level integrated prototype Number of programs that released at least 90 percent of design drawings Yes 0 2 No 1 0 Not applicable or information not available 1 0 Source: GAO analysis of Department of Defense data. GAO-19-336SP • Seventeen of the 35 programs had not released 90 percent of their design drawings by the time of the critical design review, and five of those programs still had not released 90 percent of the design drawings by the time low-rate production began. Combined, those five programs experienced almost 27 percent average cost growth above their first full estimates. • The Navy’s CH-53K program did not test a system-level integrated prototype until 45 months after its critical design review. At the same Page 53 GAO-19-336SP Weapon Systems Annual Assessment time, the program’s acquisition cycle time slipped 51 months and total acquisition costs grew by more than $10 billion—an increase of more than 55 percent. The Joint Precision Approach and Landing System system-level integrated prototype test was 85 months late and, consequently, the program delayed its initial operational capability declaration by 72 months and total acquisition costs grew by more than 69 percent. • The one non-ship program that conducted a critical design review during our assessment period, the B-2 Defensive Management System Modernization (DMS-M) program, did not test an early system-level prototype prior to its critical design review, but did so 7 months afterward. The Navy’s Fleet Replenishment Oiler (T-AO 205) began lead-ship construction during our assessment period. Both the B-2 DMS-M and T-AO 205 programs delivered 90 percent of their design drawings to manufacturing in advance of their knowledge point 2 events, and the programs experienced cost increases over their first full estimates of only 2.7 percent and 0.5 percent, respectively. Both programs also held system requirements reviews and system functional reviews prior to beginning development. • Of the six future acquisition programs we assessed, only one, the B52 Radar Modernization Program, identified a date for testing an early system-level integrated prototype. However, this event is scheduled more than two years after the program’s critical design review. Observation 4. Only one program we reviewed reported that its manufacturing processes were in statistical control prior to starting production. Programs that held production decisions during our assessment period demonstrated higher levels of production-related knowledge than programs that started production in advance of our assessment period. In fact, all three programs that began production during our assessment period satisfied the knowledge requirements for two of the key practices associated with production—testing a production-representative prototype in its intended environment and demonstrating critical processes on a pilot production line. However, none of those three programs demonstrated the knowledge associated with a third key productionrelated practice—demonstrating manufacturing processes are in statistical control. Table 13 identifies the number of programs that satisfied those three key production-related practices. The table presents the information for Page 54 GAO-19-336SP Weapon Systems Annual Assessment programs that both began production before our assessment period and those that began production during our assessment period. Table 13: Program Completion of Key Production-Related Knowledge Practices Fifteen programs that began production before our assessment period Test a productionrepresentative prototype in its intended environment Demonstrate critical processes on a pilot production line Demonstrate critical manufacturing processes are in statistical control Yes 7 8 1 No 8 5 13 Not applicable 0 2 1 Three programs that began production during our assessment period Test a productionrepresentative prototype in its intended environment Demonstrate critical processes on a pilot production line Demonstrate critical manufacturing processes are in statistical control Yes 3 3 0 No 0 0 3 Not applicable 0 0 0 Source: GAO analysis of Department of Defense data. GAO-19-336SP Production Knowledge Of the 15 programs that began production in advance of our assessment period, only one, the Infrared Search and Track, reported compliance with all three production-related knowledge practices contained in table 13. This program also satisfied most, but not all, other knowledge practices throughout its acquisition cycle and experienced virtually no cost changes since its first full estimate. The Common Infrared Countermeasure, Joint Air to Ground Missile, and the Amphibious Combat Vehicle all began production during our assessment period. Each program satisfied two of the three production-related knowledge practices noted above. These programs also completed most, but not all, of the knowledge practices throughout their lifecycles. The Amphibious Combat Vehicle reported a cost reduction from its first full estimate, while the Joint Air to Ground Missile and the Common Infrared Countermeasure reported modest cost growth since first full estimate of 2.3 percent and 0.1 percent, respectively. Observation 5. Future programs provided mixed signals on their planned implementation of knowledge-based acquisition practices. The six future programs we surveyed reported that they plan to incorporate some, but not all, of selected knowledge-based acquisition practices associated with development start. A few of these future programs, which remain in their early planning stages, have yet to establish dates for events that serve as key indicators for compliance. Table 14 presents the number of future MDAPs that plan to meet three key knowledge practices associated with beginning system development. Source: GAO analysis of Department of Defense data. GAO 19-336SP Page 55 GAO-19-336SP Weapon Systems Annual Assessment Table 14: Planned Implementation of Selected Knowledge-Based Acquisition Practices at Development Start among Six Future Major Defense Acquisition Programs Plan to demonstrate all critical technologies in a realistic environment Plan to complete all system Plan for a development phase engineering reviews of less than six years Yes 0 2 2 No 0 0 0 To be determined 3 1 1 Information not available 3 0 0 Not applicable 0 2 2 Not applicable—ship 0 1 1 Source: GAO analysis of Department of Defense data. GAO-19-336SP • Of the six future acquisition programs we surveyed, three have identified specific critical technologies. Our prior work has shown that programs beginning development with immature technologies run a greater risk of cost growth and schedule delays than programs that satisfy the knowledge practices. • Of the three programs that identified critical technologies, the Air Force’s Weather System Follow-on program has the nearest planned system development start date (March 2019). The program has identified eight critical technologies, three of which the program reported as being equivalent to a high-fidelity breadboard demonstration in a laboratory environment, which does not reach the maturity level associated with best practices for starting development. • Four of the six plan to complete PDR prior to starting development. However, other schedule dates were either not yet available or not applicable, as is the case with Weather System Follow-on, which will not have a production decision. Observation 6. Certain knowledge-based acquisition practices correspond with better program outcomes. For the second consecutive year, we conducted an exploratory statistical correlation analysis to determine whether a statistically significant link exists between non-ship programs’ cost and schedule performance and their implementation of knowledge-based acquisition practices. We found that, generally, programs that complete certain knowledge practices have better cost and schedule outcomes than programs that do not implement those practices. Page 56 GAO-19-336SP Weapon Systems Annual Assessment This year, we analyzed 17 programs—an increase of two programs as compared to our 2018 analysis—that have completed system development, held a critical design review, and started production (i.e., completed knowledge points 1 through 3). 35 We observed that MDAPs that completed preliminary design review before system development start had an estimated cost savings of 0.5 percent compared to an estimated 65.6 percent cost increase for MDAPs that did not. Further, we observed that this best practice is also associated with shorter schedules to deliver capabilities to the warfighter. Namely, our analysis found a 33.5 percent increase in schedule when programs implement this practice as compared to an 80.5 percent increase when programs do not implement it. Programs with technologies that were at least approaching maturity also realized, on average, improved cost and schedule performance. Programs implementing this practice realized an average cost reduction of 5.4 percent, while those not implementing the practice had cost growth of 24.0 percent. Programs implementing this practice also realized less growth in acquisition cycle times, 21.5 percent, versus 42.8 percent for those programs that did not implement this practice. Further, in our 2018 exploratory analysis, we identified a third knowledge practice—delivery of 90 percent of design drawings to manufacturing prior to critical design review—as statistically significant in producing better cost and schedule outcomes in programs. 36 This year’s analysis did not revalidate our 2018 analysis, but we plan to continue assessing this measure in future statistical correlation analyses we conduct. These results support our work and offer continued validation that MDAPs that follow these particular knowledge-based practices achieve lower cost increases and less schedule growth than those programs that do not. Our prior work demonstrates that completion of all of the knowledge-based practices by the time programs reach their knowledge points underpins a 35 We conducted a means test comparing averages across systems that did and did not complete knowledge-based acquisition practices using a 90 percent confidence interval. See appendix II for additional details. 36 GAO-18-360SP. Page 57 GAO-19-336SP Weapon Systems Annual Assessment sound business case that positions programs to better meet their cost and schedule goals. 37 As with last year’s annual assessment, the small sample size and the unique characteristics of the 17 MDAPs in the data set limit our analysis. As more programs within our scope enter production, we will be able to offer a more statistically significant analysis of differences between MDAPs that completed or did not complete specific knowledge-based practices. Observation 7. Programs continue to identify software as a risk area and many do not plan to deliver a baseline software capability prior to production start. Current weapon systems are dependent upon software for providing warfighter capabilities, as we have reported in prior work. 38 Yet despite this reality, many programs are not tracking software development costs, are underestimating the difficulty of software development efforts, or are not utilizing key software metrics. According to a 2018 Defense Science Board report, “software is a crucial and growing part of weapons systems and the national security mission.” 39 Software is increasingly driving the amount and types of capability that weapon systems provide to the warfighter, and software is an item that often represents considerable risk for weapons acquisition programs. In order to develop that software, programs can choose from a number of different approaches, and there is no single approach that is best for every situation. Table 15 presents a few of the approaches that are more frequently utilized among DOD acquisition programs and their corresponding definitions. 37 In general, a business case is a justification for a proposed project or undertaking. We have reported that a sound business case for successful defense acquisition programs contains key elements, including firm requirements, mature technologies, a knowledgebased acquisition strategy, a realistic cost estimate, and sufficient funding. 38 GAO, Weapon Systems Cybersecurity: DOD Just Beginning to Grapple with Scale of Vulnerabilities, GAO-19-128 (Washington, D.C.: Oct. 9, 2018). 39 DOD, Defense Science Board: Design and Acquisition of Software for Defense Systems (Washington, D.C.: Feb. 14, 2018). Page 58 GAO-19-336SP Weapon Systems Annual Assessment Table 15: Software Development Models Employed by 51 Selected Current and Future Major Defense Acquisition Programs Software development life cycle model Description Waterfall This model relies on strict phases, and each phase needs to be completed before going to the next phase. The phases include requirements definition, design, execution, testing, and release. Each phase relies on information from the previous phase. This model is a linear sequential flow in which progress is seen as flowing steadily downwards (like a waterfall) through the phases of software implementation. Incremental This model sets high level requirements early in the effort, and functionality is delivered in stages. Multiple increments delivery part of the overall required program capability. Several builds and deployments are typically necessary to satisfy approved requirements. Spiral This model takes ideas from the incremental model and its repetition but also combines the structured and systematic development of the waterfall model with a heavy emphasis on risk analysis. The project passes through four phases (identification, design, build, evaluation and risk analysis) over and over in a “spiral” until completed, allowing for multiple rounds of refinement. Agile This model breaks a product into components where on each cycle or iteration, a working model of a component is delivered. The model produces ongoing releases, each time adding small changes to the previous release. During each iteration, as the product is being built, it is also tested to ensure that at the end of the iteration the product is shippable. The Agile model emphasizes collaboration, as the customers, developers, and testers work together throughout the project. Hybrid This approach is a combination of two or more different methodologies or systems to create a new model. Source: GAO analysis of Department of Defense and software industry documentation. │GAO-19-336SP Programs we surveyed reported using a variety of these approaches, and many programs use multiple software development approaches when a single approach is not the best option for the capabilities being developed. Figure 17 shows the various software development models that selected current and future MDAPs have employed. Page 59 GAO-19-336SP Weapon Systems Annual Assessment Figure 17: Distribution of Software Development Models among 51 Selected Current and Future Major Defense Acquisition Programs Note: “Other” category represents any software development approach that programs identified to us that does not fall into one of the five DOD-identified models. In addition, some programs reported using multiple approaches. As a result, the columns in the figure do not sum to 51. • Among the 51 programs we surveyed, the most frequently utilized software development model was the incremental approach, with 17 programs reporting its use. The Air Force’s Combat Rescue Helicopter is an example of a program using multiple software models for various subsystems, and one of those models is the incremental approach. The spiral model is the least utilized. Only one program, the Navy’s DDG 1000, reported using this approach. • Some program officials reported using “other” software development methods, for a variety of reasons. The Navy’s Virginia Class Submarine (SSN 774) initially used a waterfall-like approach to deliver required software. However, program officials reported that the SSN 774 program no longer employs that approach because software Page 60 GAO-19-336SP Weapon Systems Annual Assessment deliveries are complete and the Navy is modifying existing software to accommodate revised ship characteristics. • Eleven Navy programs reported utilizing the agile software development method, the most among any military department. Only two Air Force programs and one Army program reported using the agile software method. Table 16 identifies the number of current major programs we surveyed that identified software as a risk at any point during the program’s history, as well as the number of programs that satisfied key software-related practices we identified in prior work. 40 Table 16: Software Risk Identification and Development Practices among 45 Selected Current Major Defense Acquisition Programs Software identified as a risk area Baseline capability Baseline capability completed before completed before initial production operational capability Software development costs tracked Yes 33 16 27 15 No 11 15 4 30 Do Not Know 1 0 0 0 Information Not Available 0 14 14 0 Source: GAO analysis of Department of Defense data. GAO-19-336SP • • Programs cited three main reasons why they identified software as a risk area: • completing the software effort in order to conduct developmental testing successfully, • changes for cybersecurity needs leading to additional software development efforts, and • completing the originally planned software development effort has proven more difficult than expected. Software development has similar phases to that of hardware and—in the case of new systems—occurs in parallel with hardware development until software and hardware components are integrated. 40 GAO, Defense Acquisitions: Stronger Management Practices Are Needed to Improve DOD’s Software-Intensive Weapon Acquisitions, GAO-04-393 (Washington, D.C.: Mar. 1, 2004). Page 61 GAO-19-336SP Weapon Systems Annual Assessment According to DOD policy, major contracts and subcontracts for contractors developing or producing software elements for MDAPs are subject to software resource data reporting requirements if the software effort is projected to be greater than $20 million. 41 However, when programs do not track the cost of software development separately from the overall development effort, DOD’s ability to impose the reporting requirement is constrained. We have previously found that programs that defer completion of software development activities to post-production phases—such as after initial operational capability or initial operational test and evaluation—risk delivering systems to the warfighter that do not meet their minimum performance requirements. 42 • Figure 18 shows the points in the acquisition process when the 45 current MDAPs we surveyed completed, or plan to complete, software development. 41 Specifically, all major contracts and subcontracts, regardless of contract type, for contractors developing or producing software elements within current and future MDAPs, as well as other certain acquisitions, such as those of certain automated information systems, for any software development element with a projected software effort greater than $20 million (then-year dollars) are required to complete a Software Resources Data Report. DOD Instruction 5000.02, Operation of the Defense Acquisition System (Jan. 2015) [incorporating change 4 (Aug. 2018)] (“DOD Instruction 5000.02”). 42 GAO-18-360SP. Page 62 GAO-19-336SP Weapon Systems Annual Assessment Figure 18: Plans for Completing Software Development among 45 Selected Current Major Defense Acquisition Programs DOD acquisition policy identifies six acquisition models that serve as examples of defense program structures tailored to the type of product being acquired or to the need for accelerated acquisition. 43 Among other things, these models emphasize software development to varying degrees. The policy directs acquisition programs to use these models as a starting point in structuring a program to acquire a specific product. Table 17 identifies these six models in more detail. 43 DOD Instruction 5000.02. Page 63 GAO-19-336SP Weapon Systems Annual Assessment Table 17: Department of Defense’s (DOD) Six Acquisition Models and Their Basic Characteristics Acquisition Model Basic Characteristics Hardware intensive program This is the classic model for a major weapon program and has existed in some form in all previous editions of DOD’s acquisition policy. Defense unique software intensive program Uses a series of planned “software builds”—testable, integrated subsets of the overall capability— which together with clearly defined decision criteria, ensure adequate progress is being made before the program commits to subsequent builds. Incrementally deployed software intensive program Deploys full software capability in multiple increments as new capability is developed and delivered, nominally in 1- to 2-year cycles. Accelerated acquisition program Prioritizes schedule considerations over cost and technical risk considerations by compressing or eliminating phases of the acquisition process and accepting the potential for inefficiencies in order to achieve a deployed capability on a compressed schedule. Hybrid program A (hardware dominant) Combines a basic structure hardware development program with a simultaneous, software intensive development effort. Design, fabrication, and testing of physical hardware prototypes may determine the program’s overall schedule, decision points, and milestones, but software development will often dictate the pace of program execution and must be tightly integrated and coordinated with hardware development decision points. Hybrid program B (software dominant) Centers on a software intensive product development that can include a mix of incrementally deployed software products or releases that include intermediate software builds. Source: GAO analysis of DOD Instruction 5000.02. GAO-19-336SP We asked each program we surveyed to identify the specific model that was most applicable to their program. Figure 19 identifies the distribution of development models among 45 major defense programs. Page 64 GAO-19-336SP Weapon Systems Annual Assessment Figure 19: Plans for Utilizing System Development Models among 45 Current Major Defense Acquisition Programs We compared the acquisition models and software development plans that programs reported and observed the following: • Of the 15 programs that identified Hybrid Program A (hardware dependent) as their model, 13 reported to us that they had identified software as a risk area for the program. Further, nine of these 15 programs tracked the cost associated with their software development efforts, while six of those 15 programs reported that a baseline software capability would be available in time for the start of production. • Of the 14 programs that identified Hardware Intensive Program as their model, half the programs either reported that they did not know or do not track the estimated total acquisition costs specifically for software. At the same time, six of these programs reported software Page 65 GAO-19-336SP Weapon Systems Annual Assessment as a risk area, and only four reported that a baseline software capability would be ready for a production decision. Observation 8. Many programs we surveyed declared, or plan to declare, initial operational capability without conducting initial operational test and evaluation. Initial operational capability occurs when a unit or organization has been equipped and trained and is determined to be capable of conducting operations with a newly fielded system. However, each individual program declares its initial operational capability date, and programs do not apply consistent standards when determining that date. Initial operational test and evaluation is a separate event that is intended to evaluate a system’s effectiveness and suitability under realistic operational conditions before a program makes a full-rate production decision. We have observed programs declaring initial operational capability on the basis of full, partial, or no initial operational test and evaluation. Programs that declare initial capability before completing initial operational test and evaluation risk fielding systems to warfighters that are not operationally effective or suitable for the missions they will be tasked to perform. Figure 20 details our analysis of the relationship between initial operational capability and initial operational test and evaluation in 45 current MDAPs. Figure 20: Plans for Declaring Initial Operational Capability among 45 Selected Current Major Defense Acquisition Programs Page 66 GAO-19-336SP Weapon Systems Annual Assessment Assessments of Individual Programs This section contains (1) 51 assessments of individual weapon programs and (2) four summary analyses—each segmented by military department. Each assessment presents data on the extent to which programs are following a knowledge-based acquisition approach to product development. 44 Each military department’s summary analysis page presents aggregated information about selected programs’ acquisition phases, current estimated funding needs, knowledge attained, cost and schedule performance, oversight authorities, software characteristics, and competition in contracting. To develop the individual program assessments, we collected program information using a web-based questionnaire, obtained and reviewed agency documents, and interviewed program officials. For 39 programs, all of which were in development or early production, we produced two-page assessments discussing the technology, design, and manufacturing knowledge obtained, as well as other program issues. Each two-page assessment also contains a comparison of total acquisition cost from the first full estimate for the program to the current estimate. The first full estimate is generally the cost estimate established at development start; however, for a couple of programs that did not have such an estimate, we used the estimate at production start. For shipbuilding programs, we used their planning estimates if those estimates were available. For programs that began as non-MDAPs, we used the first full estimate available. See figure 21 for an illustration of the layout of each two-page assessment. 44 The assessments also contain basic information about the program, including the prime contractor(s) and contract type(s). We abbreviated the following contract types in the individual assessments: cost-plus-award-fee (CPAF), cost-plus-fixed-fee (CPFF), costplus-incentive-fee (CPIF), firm-fixed-price (FFP), fixed-price incentive (FPI), and indefinitedelivery / indefinite-quantity (IDIQ). Page 67 GAO-19-336SP Weapon Systems Annual Assessment Figure 21: Illustration of Program Two-Page Assessment In addition, we produced one-page assessments on the current status of 12 programs, which include (1) six future MDAPs; (2) one MDAP that was in development, but released its cost baseline very late in our review, and Page 68 GAO-19-336SP Weapon Systems Annual Assessment (3) five MDAPs that were well into production, but planned to introduce new increments of capability. See figure 22 for an illustration of the layout of each one-page assessment. Figure 22: Illustration of Program One-Page Assessment Page 69 GAO-19-336SP Weapon Systems Annual Assessment For 45 of our one- and two-page assessments, we used scorecards to depict the extent of knowledge that a program has gained. 45 These scorecards display key knowledge-based acquisition practices that should be implemented by certain points in the acquisition process. In our prior and current work, we found that the more knowledge a program has attained by each of these key points, the more likely the weapon system will be delivered within its estimated cost and schedule. In a current MDAP, knowledge deficits signal that the program is proceeding without sufficient knowledge about its technologies, design, or manufacturing processes, and faces unresolved risks that could lead to cost growth and schedule delays. For each scorecard, we used a closed circle to denote a knowledgebased practice the program implemented. We used an open circle to denote a knowledge-based practice the program did not, or has not yet, implemented. For future MDAPs only, we used a partially closed circle to denote a knowledge-based practice that the program reported it plans to implement. If the program did not provide us with enough information to make a determination, we showed this with a dashed line. We also marked as “NA” any scorecard field that corresponded with a knowledgebased practice that was not applicable to the program. A knowledgebased practice may not be applicable to a particular program if it has not yet reached the point in the acquisition cycle when the practice should be implemented, or if the particular practice is not relevant to the program. For shipbuilding programs, we assessed different key points in the acquisition cycle and applicable knowledge-based practices, which were informed by our prior work. 46 Appendix II provides additional detail on our scorecard methodology. Figures 23 and 24 provide examples of the knowledge scorecards we used in our assessments. 45 We did not use scorecards in five of our one-page assessments of MDAPs that were well into production, but planned to introduce new increments of capability, because our metrics on knowledge attainment were incongruent with the acquisition strategies these programs employed. We did not use a scorecard in a sixth one-page assessment—for VC-25B Presidential Aircraft Recapitalization—which released a baseline late in our review. 46 GAO-09-322. Page 70 GAO-19-336SP Weapon Systems Annual Assessment Figure 23: Examples of Knowledge Scorecards on Two-Page Assessments Figure 24: Example of Knowledge Scorecard on One-Page Assessments Page 71 GAO-19-336SP Weapon Systems Annual Assessment AR MY ROG RAM GAO Assessed 7 of the The 7 Current Programs GAO Army?s 23 Current Major Assessed Represent 41 Percent of ASSESSMENTS Defense Acquisition the Army?s $189 Billion Estimated Programs Total Acquisition Costs (Fiscal Year 2019 dollars in billions) Current program 1 (Cost baseline not established) Current $78 $111 programs Other programs (Not assessed) I We also assessed two future major defense acquisition programs. We did not assess programs that had yet to enter technology development or had already progressed into full-rate production. Most Current Programs GAO Assessed Have Either Cost Growth, Schedule Delays, or Both Program complete Programs in production Programs in system development CIRCM Program age . Cost growth of more than 15 percent andfor schedule delays of more than 6 months Cost growth of 15 percent or less andlor schedule delays of 6 months or less . Cost growth of 0 percent or less and schedule delays of 0 months or less Note: Bubble size is based on each program's currently estimated future acquisition funding needed, Total Acquisition Cost Performance Most Current Programs Did Not Attain in Current Programs Varied Among Knowledge at Key Points Knowledge Points (KP) (Fiscal Year 2019 dollars in billions) KP1 0 of 7 current programs attained knowledge 3 $3current programs $61 attained KP1 to KP2 to Post - knowledge KP2 KP3 KP3 KP3 of 5 current I A cost decrease due to a signi?cant quantity i I i decrease in one current program?HMS?accounted knowledge for most of the cost decrease between KP1 and KP2. Average Acquisition Cycle Time of Current Programs Increased by 16 Months from First Full Estimate (FFE) to Current Estimate Programs' Programs' average average FFE 88 cycle time change from cycle time FFE to current estimate Number of months I One current program?lTEP?entered system development after December 2017 and therefore ls excluded from our cost, schedule. knowledge attainment, and contract analysis. I Cost and schedule analysis is primarily based on estimates from December 2017 selected acquisition reports. This information may differ from the amounts reported in the individual assessments. which are based on more recent program estimates in some cases. Please see appendix It for details. Source: GAO analysis of Department of Defense data. GAO-19-3365P Page 72 GAO-19-3368P Weapon Systems Annual Assessment ARMY PROGRAM ASSESSMENTS Most Current and Future Army The Army was the Milestone Decision Authority in 2017 for Programs Report that Less than 20 9 of its Current and Future Programs in Review Percent of Planned Acquisition Costs are Specifically for Software Army programs Percentage of planned acquisition costs programs I The milestone decision authority for AMPV, CIRCM, HMS, JAGM, 0200/ and changed from the 0 Under Secretary of Defense for 1 Acquisition and Sustainment to the Army in 2017. 21-40% The Army Received One Offer on Solicitations for Just Over Half of its NAlnot Currently Reported Major Contracts for Development and Procurement tracked 5 10 15 20 Number of programs Number of contracts Most Current Army Programs Report that Less than 20 Percent of Work Tasks are Software-Related Percentage of work tasks (13-20% 21-40% C3l Combat Ground Helicopter Missile Other Vehicle Combat 0 5 10 15 20 33?5?? type Number of programs cal is short for command, 2 or more Offers - 1 Offer control, communications and intelligence Source: GAO analysis of Department of Defense data. GAO-19-3365P Page 73 GAO-19-3368P Weapon Systems Annual Assessment Army Program Assessmentsa 2-page assessments Armored Multi-Purpose Vehicle (AMPV) CH-47F Modernized Cargo Helicopter (CH-47F Block II) Common Infrared Countermeasure (CIRCM) Handheld, Manpack, and Small Form Fit Radios (HMS) Integrated Air and Missile Defense (IAMD) Improved Turbine Engine Program (ITEP) Joint Air-to-Ground Missile (JAGM) Joint Light Tactical Vehicle (JLTV) 1-page assessments Indirect Fire Protection Capability Increment 2 – Intercept Block 1 (IFPC Inc 2-I Block 1) Precision Strike Missile (PrSM) Page number 75 77 79 81 83 85 87 89 91 92 a We abbreviate the following contract types in the individual assessments: cost-plus-award-fee (CPAF), cost-plus-fixed-fee (CPFF), cost-plus-incentivefee (CPIF), firm-fixed-price (FFP), fixed-price incentive (FPI), and indefinite-delivery / indefinite-quantity (IDIQ). Page 74 GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Army Common Name: AMPV Armored Multi-Purpose Vehicle (AMPV) The Army's AMPV is the replacement to the M113 family of vehicles at the brigade level and below. The AMPV will replace the M113 in five mission roles: general purpose, medical evacuation, medical treatment, mortar carrier, and mission command. The Army determined that development of the AMPV is necessary due to mobility, survivability, and force protection deficiencies identified with the M113, as well as space, weight, power, and cooling limitations that prevent the incorporation of future technologies. Program Essentials Milestone decision authority: Army Program office: Warren, MI Prime contractor: BAE Systems Contract type: CPIF (development) Software development approach: Incremental Next major milestone: Start operational testing (February 2021) Program Performance (fiscal year 2019 dollars in millions) First full estimate (05/2015) Latest (06/2018) Percentage change Development $1,050.90 $1,051.30 0.0% Procurement 10,354.8 $10,466.20 +1.1% $3.89 $3.95 +1.8% 87 87 0.0% 2,936 2,936 0.0% Unit cost Acquisition cycle time (months) Total quantities Total quantities comprise 39 development quantities and 2,897 procurement quantities. Funding and Quantities (fiscal year 2019 dollars in millions) Attainment of Product Knowledge As of January 2019 Status at Resources and requirements match • Demonstrate all critical technologies are very close to final form, fit and function within a relevant environment • Demonstrate all critical technologies in form, fit and function within a realistic environment • Complete a system-level preliminary design review Product design is stable ● ● ○ ● ● ● Design Review • Release at least 90 percent of design drawings • Test a system-level integrated prototype Manufacturing processes are mature ● ○ ● ● Production Start • Demonstrate Manufacturing Readiness Level of at least 9, or critical processes are in statistical control • Demonstrate critical processes on a pilot production line • Test a production-representative prototype in its intended environment Current Status Development Start ○ ○ ● ○ ○ ● ● Knowledge attained, ○ Knowledge not attained, … Information not available, NA Not applicable Page 75 GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Army AMPV Program Technology Maturity and Design Stability The AMPV program has matured its critical technologies and stabilized its design. The program entered system development in December 2014 with its critical technologies deemed fully mature by an independent review team. Program officials noted that although the AMPV uses a new hull design, a majority of subsystems are derived from existing vehicles. For example, the AMPV uses a common powertrain with the Bradley infantry fighting vehicle. While the program has released over 90 percent of expected design drawings to manufacturing, the overall number has fluctuated due to vehicle configuration changes. Following its critical design review (CDR) in June 2016, the program’s number of expected drawings increased by nearly 19 percent. In our 2018 assessment, we reported that this increase was due to the contractor underestimating the number and complexity of design drawings needed. For our current assessment, however, we found that the program has now lowered its total number of design drawings by 19 percent. According to program officials, this decrease reflected a changed design due to testing results. This lowered number re-establishes that the program released over 90 percent of its design drawings to manufacturing by CDR. We have updated our assessment of product knowledge table to reflect this latest calculation. To maintain schedule, the program office delayed some of its developmental testing, which it now expects to complete in July 2019—over 5 months after it entered low-rate initial production and contrary to best practices. Remaining technical risk areas include the automatic fire extinguishing system and hard transmission shifts, both of which may require redesigns if testing reveals further deficiencies. Production Readiness The AMPV program entered low-rate initial production in January 2019. The program’s manufacturing readiness level did not indicate that its production processes were in statistical control, as recommended by best practices. Further, the program office did not demonstrate its critical manufacturing processes on a pilot production line prior to production start. The program attributed this to the contractor’s need to address manufacturing process deficiencies discovered during production of prototype AMPV units. The contractor, BAE Systems, delivered all 29 prototype vehicles by March 2018—5 months later than earlier program plans, which called for delivery of all vehicles by October 2017. As a result, the contractor had less time than initially planned between the prototype deliveries and the start of low rate initial production to resolve production issues. Problems with parts Page 76 Common Name: AMPV shortages and changes to engineering drawings, among other things, caused the late deliveries. In light of these challenges, BAE Systems is modifying its overall manufacturing plan to include changes to assembly and facility processes. For example, it is implementing robotic welding to improve quality and manufacturing throughput. In a program assessment report, the Defense Contract Management Agency (DCMA) noted that although these manufacturing modifications should improve the overall process, some changes will, in fact, introduce new risks. Our review of program documentation shows that the program’s ability to meet increased production quantities remains a key risk area. In September 2017, the Army increased the program’s low-rate initial production quantity from 289 to 551 vehicles spread over 4 years to support the European Deterrence Initiative (EDI)—an effort intended to reassure allies and partners of U.S. commitment to their security. According to program officials, this quantity has been lowered to 469 vehicles spread over 3 years with the final EDI vehicles being procured as part of full rate production. Persisting AMPV technical challenges, coupled with contractor plans to produce AMPV in the same facility as two other major platforms—the M109A7 Family of Vehicles, and M88A2 HERCULES recovery vehicles— will together constrain AMPV deliveries. Other Program Issues Based on recent results from ongoing reliability tests, the program reported that it is slightly below its reliability goal for the start of low-rate initial production. Program officials stated the contractor has addressed AMPVunique issues, but legacy powertrain problems remain. Program Office Comments We provided a draft of this assessment to the program office for review and comment. The program office provided technical comments, which we incorporated where appropriate. Program officials stated that AMPV began production ahead of schedule in January 2019 and is executing activities to show manufacturing processes are nearing maturity. According to the program, production processes will reach full maturity to support full-rate production in fiscal year 2022. To date, the program reports that it has implemented production process changes that improve vehicle quality and provide capacity for increased production quantities. In addition, program officials stated that BAE Systems produced vehicle hulls ahead of the first low-rate production vehicle to validate these process changes. At the same time, program officials stated that they continue to improve the AMPV design based on recent test results. They also stated that vehicle developmental tests will finish in July 2019, and they assess the risk of additional re-design of the vehicle to be low. Further, program officials said that AMPV is meeting its overall availability requirement, but reliability is a watch item. GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Army Common Name: CH-47F Block II CH-47F Modernized Cargo Helicopter (CH-47F Block II) The Army’s CH-47F Block II program upgrades existing CH-47F aircraft intended to provide additional capability, greater reach, and increased payload capacity. Improvements include a strengthened airframe and drive train, improved flight controls, upgraded fuel and electrical systems, and advanced rotor blades that the Army expects will increase lift in hot weather conditions. The Army also expects the improved fuel and rotor components to reduce operating and support costs. CH-47F helicopters provide the Army’s only heavy lift capability and are scheduled to remain in service through 2060. Program Essentials Milestone decision authority: Army Program office: Redstone Arsenal, AL Prime contractor: Boeing Contract type: CPIF (development) Software development approach: Mixed Next major milestone: Production start (August 2021) Program Performance (fiscal year 2019 dollars in millions) First full estimate (02/2018) Latest (06/2018) Percentage change Development $791.70 $761.60 -3.8% Procurement $15,714.80 $15,768.20 +0.3% $30.92 $30.97 +0.2% 88 88 0.00% 542 542 0.00% Unit cost Acquisition cycle time (months) Total quantities Total quantities comprise 3 development quantities and 539 procurement quantities. Funding and Quantities (fiscal year 2019 dollars in millions) Attainment of Product Knowledge As of January 2019 Status at Resources and requirements match • Demonstrate all critical technologies are very close to final form, fit and function within a relevant environment • Demonstrate all critical technologies in form, fit and function within a realistic environment • Complete a system-level preliminary design review Product design is stable ● ○ ● ● ○ ● Design Review • Release at least 90 percent of design drawings • Test a system-level integrated prototype Manufacturing processes are mature ● ○ ● ○ Production Start • Demonstrate Manufacturing Readiness Level of at least 9, or critical processes are in statistical control • Demonstrate critical processes on a pilot production line • Test a production-representative prototype in its intended environment Current Status Development Start NA NA NA NA NA NA ● Knowledge attained, ○ Knowledge not attained, … Information not available, NA Not applicable Page 77 GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Army CH-47F Block II Program Technology Maturity CH-47F Block II critical technologies continue to approach maturity at the same time that the program reports that it has released 99 percent of planned drawings, which constitutes a stable design. As the program continues to mature each technology into a final form, fit, and function, the program may need to revise its system drawings to accommodate necessary changes, which could compromise the program’s design stability. The program office has identified two critical technologies—Ferrium C61 steel and the advanced Chinook rotor blade (ACRB)—that it assesses as approaching maturity. Ferrium C61 is a new, high strength steel designed to create an improved drive train (shafts) that can operate under increased horsepower, without needing to redesign the CH-47F transmission. According to the program office, the CH47F Block II will be the first to use this material in this type of application. Other applications include power transmission shafts and gears in the aerospace and energy industries, as well as off-road vehicles. Program officials expect the production cost of Ferrium C61 shafts to be higher than the steel used in existing CH47F aircraft, but they stated that they will not be able to estimate the cost until production processes are established. The ACRB is a carbon fiber blade intended to improve the aircraft’s lift performance while also minimizing effects on forward flight. The Block II program is tracking technical risks with ACRB. Program officials stated they have demonstrated the ACRB in an operational environment near or at planned operational configuration. Nonetheless, the program characterizes the ACRB design as more complex than legacy blades due to a new tip design, which has introduced manufacturing risk. According to officials, prototypes and process changes will help limit that risk. Design Stability Common Name: CH-47F Block II prime contractor, Boeing, is producing the three system development aircraft in the same final assembly facility as the current Block I aircraft using many of the same processes, tools, and management procedures. The program plans to demonstrate statistical control of critical manufacturing processes as part of a production readiness assessment before starting production, which aligns with GAO best practices. Program officials noted that Boeing—as the original designer and prime manufacturer of the CH-47 series aircraft since 1962— has demonstrated statistical control of manufacturing processes in previous models. The program office further stated that Boeing is the only capable source for Block II production. According to the program office, Boeing’s significant cargo helicopter experience, possession of CH-47F-unique tooling and test equipment, and existing Block I production line preclude use of full and open competition for Block II production. Program Office Comments We provided a draft of this assessment to the program office for review and comment. The program office provided technical comments, which we incorporated where appropriate. The program office stated that the CH-47F Block II program is ahead of schedule and within expected cost. Specifically, the program office stated that following the development contract award, it has consistently reported favorable cost and schedule performance and met program milestones ahead of schedule, which the program office considers indicative of overall positive program health. The program office also stated that testing of the first system-level integrated prototype could begin in July 2020. It also said that it conducted two test flights that confirmed ACRB design for the development phase of the CH-47F program. Further, the program office stated that the similarities between the legacy CH-47F and the new Block II design allows for demonstration of critical technologies before prototype testing. According to the program office, this constituted a low-risk approach to reduce program cost and schedule. CH-47F Block II released 90 percent of its design drawings by critical design review (CDR) in December 2017. Prior to the CDR, however, the program did not elect to developmentally test a fully configured, production representative prototype in its intended environment. Instead, the program plans to initiate this prototype testing in August 2019 (20 months after CDR)—an approach inconsistent with best practices. Until the program completes this testing, it cannot know whether its design is actually stable. Production Readiness The program plans to enter production in August 2021 following delivery of three aircraft that officials report were procured as part of system development. The Page 78 GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Army Common Name: CIRCM Common Infrared Countermeasure (CIRCM) The Army's CIRCM is the next generation lightweight, laser-based infrared countermeasure system for rotary-wing, tilt-rotor, and small fixed-wing aircraft across DOD. CIRCM consists of three major items— a system processor unit, a pointer tracker, and an infrared laser. CIRCM receives input from the Army’s Common Missile Warning System and employs the pointer tracker to track incoming missiles. It jams the missile by using laser energy, thus causing the missile to miss the aircraft. CIRCM is to replace the Advanced Threat Infrared Countermeasures system and enable integration of laser-based infrared countermeasures onto smaller aircraft. Program Essentials Milestone decision authority: Army Program office: Huntsville, AL Prime contractor: Northrop Grumman Contract type: CPFF/FPI/FFP (development and low-rate initial production) Software development approach: Incremental Next major milestone: Start operational testing (June 2019) Program Performance (fiscal year 2019 dollars in millions) First full estimate (07/2016) Latest (11/2018) Percentage change Development $802.50 $880.60 +9.7% Procurement $1,895.70 $3,094.50 +63.2% $2.40 $2.18 -9.2% 115 117 +1.7% 1,124 1,829 +62.7% Unit cost Acquisition cycle time (months) Total quantities Quantities comprise 48 development quantities and 1,781 procurement quantities. Funding and Quantities (fiscal year 2019 dollars in millions) Attainment of Product Knowledge As of January 2019 Status at Resources and requirements match • Demonstrate all critical technologies are very close to final form, fit and function within a relevant environment • Demonstrate all critical technologies in form, fit and function within a realistic environment • Complete a system-level preliminary design review Product design is stable ● ○ ● ● ● ● Design Review • Release at least 90 percent of design drawings • Test a system-level integrated prototype Manufacturing processes are mature ● ● ● ● Production Start • Demonstrate Manufacturing Readiness Level of at least 9, or critical processes are in statistical control • Demonstrate critical processes on a pilot production line • Test a production-representative prototype in its intended environment Current Status Development Start ○ ● ● ○ ● ● ● Knowledge attained, ○ Knowledge not attained, … Information not available, NA Not applicable Page 79 GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Army CIRCM Program Technology Maturity, Design Stability, and Production Readiness The CIRCM program has fully matured its six critical technologies and stabilized its design. However, 4 months into production, the program had yet to demonstrate that its critical manufacturing processes were in statistical control—an approach inconsistent with best practices. Following development start, the program demonstrated its six critical technologies by integrating a prototype CIRCM system on an Army aircraft and testing it in a realistic environment. The program has also released 100 percent of design drawings, which indicates a stable design. Nonetheless, persistent reliability shortfalls with CIRCM units—and the design changes potentially required to remedy these shortfalls—pose risk to CIRCM’s design stability. In April 2017, the DOD Inspector General found deficiencies with the CIRCM program office’s plan for demonstrating minimum reliability requirements. The Inspector General recommended that the Army revisit CIRCM performance requirements and test plans prior to production start. In response, the Army took actions that included updating system reliability requirements in the CIRCM’s master test plan. This update set a new goal for the end of the system’s initial operational test and evaluation phase in November 2019—achieving150 hours “mean time between operational mission failure” with an 80 percent likelihood of operating for 150 hours without a failure. Program officials stated that meeting this goal will position the CIRCM to demonstrate 214 hours mean time between operational mission failure as required for the program’s June 2020 full-rate production decision. At the time of its low-rate initial production start in September 2018, the CIRCM demonstrated 148.5 hours reliability at a 50 percent likelihood of performing reliably for 148.5 hours without failure. More recently, the program reported that it has increased CIRCM’s reliability performance to 151 hours within the same 50 percent likelihood. In a September 2018 briefing it prepared for the CIRCM production decision, the Army’s Aircraft Survivability Equipment project management office that oversees the CIRCM program office stated that, should CIRCM ultimately fall short of the 214-hour requirement, the consequences will likely be delayed entry into full-rate production and postponement in equipping the first Army unit with CIRCM. Common Name: CIRCM those processes until the full-rate production decision. Our prior work has found that this DOD standard falls short of the industry best practice. Further, a June 2018 production readiness review identified concerns with (1) Northrop Grumman’s ability to oversee subcontractors and (2) the pointer tracker subcontactor’s ability to deliver reliable components on schedule. The Office of the Secretary of Defense expressed similar concerns and found that the pointer tracker subcontractor may be challenged to increase production from 6 units per month to the 26 per month needed to support low-rate production and testing. The Army also is tracking risks that Northrop Grumman cannot provide timely delivery of CIRCM assets for testing and that production facilities for the pointer tracker component do not have the capacity to meet demand for full-rate production. The program has mitigation efforts for both risks under way, which include, among other things, increasing repair capability at Northrop Grumman and adding to the workforce at the pointer tracker production facility. It expects to retire the testing assets risk in May 2019 and the pointer tracker production risk in October 2019. Program Office Comments We provided a draft of this assessment to the program office for review and comment. The program office stated that CIRCM completed all engineering and manufacturing development activities and entered production in September 2018. According to the program office, CIRCM exceeded all key performance requirements as demonstrated through rigorous laboratory, aircraft flight, and live fire missile testing. The program office also stated that the Army Test and Evaluation Command Director assessed CIRCM to be an effective system and on track to be suitable and survivable. Further, the office stated that the program has awarded a low-rate initial production contract and identified plans to begin initial operational test and evaluation in June 2019. According to the program office, it continues to aggressively manage contractor performance to facilitate a successful transition to fullrate production. Additionally, the Army stated that the CIRCM program is on track to achieve the 214-hour requirement necessary for full-rate production, and that it will reassess program strategy going forward if CIRCM does not achieve this goal. The program office reported that, at the time of CIRCM’s entry into low-rate initial production, the contractor, Northrop Grumman, had achieved manufacturing readiness at the level recommended by DOD guidance. That guidance calls for programs to demonstrate critical manufacturing processes on a pilot production line, but does not require statistical control of Page 80 GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Army Common Name: HMS 0 Handheld, Manpack, and Small Form Fit Radios (HMS) Through its HMS program, the Army is procuring software-defined radios that will connect with existing radios and increase the service’s communications and networking capabilities. The program continues efforts begun under the former Joint Tactical Radio System program to procure multiple radios, including the Handheld (Leader and Rifleman) and the Manpack. A subset of Manpack radios will operate with the Mobile User Objective System (MUOS)—a worldwide, multiservice Navy satellite communication system. In 2017, the Army deferred its acquisition of one-channel Rifleman radios in favor of two-channel Leader radios. Program Essentials Milestone decision authority: Army Program office: Aberdeen Proving Ground, MD Prime contractors: General Dynamics C4 Systems, Inc.; Harris Corporation; Collins Aerospace; Thales Defense and Security Contract types: CPAF (development); FFP/IDIQ (low-rate initial and full-rate production) Software development approach: Other (modifications to existing systems) Next major milestone: Full-rate production (February 2021) Funding and Quantities (fiscal year 2019 dollars in millions) Program Performance (fiscal year 2019 dollars in millions) First full estimate (05/2004) Latest (06/2018) Percentage change Development $609.90 $1,421.00 +133.0% Procurement $10,629.60 $7,977.20 -25.0% $0.03 $0.04 +2.4% 85 124 +45.88% 328,674 271,202 -17.5% Unit cost Acquisition cycle time (months) Total quantities Acquisition cycle time reflects development start to initial operational capability for the Manpack radio (August 2014). Total quantities comprise 833 development quantities and 270,369 procurement quantities. Attainment of Product Knowledge As of January 2019 Status at Resources and requirements match • Demonstrate all critical technologies are very close to final form, fit and function within a relevant environment • Demonstrate all critical technologies in form, fit and function within a realistic environment • Complete a system-level preliminary design review Product design is stable ○ ○ ○ NA NA NA Design Review • Release at least 90 percent of design drawings • Test a system-level integrated prototype Manufacturing processes are mature ○ ● NA NA Production Start • Demonstrate Manufacturing Readiness Level of at least 9, or critical processes are in statistical control • Demonstrate critical processes on a pilot production line • Test a production-representative prototype in its intended environment Current Status Development Start ○ ○ ● NA NA NA ● Knowledge attained, ○ Knowledge not attained, … Information not available, NA Not applicable Page 81 GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Army HMS Program Technology Maturity, Design Stability, and Production Readiness The HMS program office is now acquiring the Leader and Manpack radios as non-developmental items, which has precluded the program’s use or tracking of any critical technologies. When the HMS program started development in 2004, it did not assess the maturity of the technologies it then deemed critical—an approach inconsistent with best practices. The program had also not fully matured its critical technologies by its 2011 production start. Instead, the program completed development of individual critical technologies in May 2015. This divergence from best practices likely contributed to the over 130 percent increase in development costs since the first full estimate. At its 2008 critical design review, the program office had completed fewer than half of its planned design drawings, which did not meet best practices criteria for design stability. Most importantly, the program’s persistent technology immaturity between 2004 and 2011, including at the critical design review, contributed to radio designs that did not fully accommodate the final form, fit, and function of critical technologies as they matured. These design shortfalls became evident as HMS radios entered testing, with testers identifying significant reliability and suitability problems with radios at that time. With the move to a non-developmental acquisition— including their decision to shift to a two-channel radio and introduce an additional waveform—the program has developed a new test plan. The Army has started working with the contractors to perform customer and qualification testing to ensure their radios meet Army specifications. These tests will support preparations for operational testing, which is scheduled to begin in June 2020. The HMS program has not yet finalized its assessments of contractor manufacturing readiness levels under the restructured program. To date, the HMS program office completed preliminary assessments of the Harris Corporation’s and Collins Aerospace’s readiness to begin production for the Manpack radio and found them to be on track to support production decisions. Further, program officials stated that they already held a followup assessment of Harris Corporation and plan to hold one with Collins Aerospace in June 2019. Officials reported that the program added Leader radios to the existing program in summer 2018 and that they plan to assess production readiness of Leader in late fiscal year 2019. Common Name: HMS deficiencies found in initial operational testing in November 2015. Over the past several years, problems with MUOS testing have repeatedly delayed availability of the waveform. Although the program office has not identified MUOS as a critical technology, without this waveform, affected Manpack radios will be able to communicate only through legacy communications, which limit the capacity of the network. The Navy currently plans additional MUOS testing for mid-2019, but further delays to the waveform could lead to delays in fielding affected Manpack radios by March 2020, which is already delayed from initial plans by nearly 2 years. Program officials told us Manpack radio low-rate initial production will move forward regardless of the MUOS waveform performance during the Navy's testing scheduled in 2019. Program Office Comments We provided a draft of this assessment to the program office for review and comment. The program office provided technical comments, which we incorporated where appropriate. According to the program office, in September 2018, it added the two-channel Leader Radio capabilities to the existing Handheld indefinitedelivery/indefinite-quantity contracts with two vendors, Harris Corporation and Thales Defense & Security. The program office also said that, based on the Army Network Review, it amended the Leader Radio capability production document to address threshold radio waveforms, weight, and size. Additionally, it stated that it is also amending the Manpack capability production document to address waveform clarifications, and that this documentation is currently pending approval from Army leadership. According to the program office, these amended documents will validate and stabilize requirements consistent with current non-developmental item capabilities and the HMS acquisition strategy. The program also said that it awarded low-rate initial production delivery orders to procure 3,800 Leader radios and an additional low-rate delivery order to procure 2,258 Generation 2 Manpack radios. Other Program Issues At present, warfighters are not able to use the MUOS waveform—which some Manpack radios will rely on— because the MUOS program is still addressing Page 82 GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Army Common Name: IAMD Integrated Air and Missile Defense (IAMD) The Army's IAMD program plans to network sensors, weapons, and a common battle command system across an integrated fire control network. Its purpose is to support the engagement of air and missile threats. The IAMD battle command system will provide a capability for the Army to control and manage IAMD sensors and weapons, such as the Sentinel radar and Patriot launcher and radar, through an interface module that supplies battle management data and enables networked operations. Program Essentials Milestone decision authority: Under Secretary of Defense for Acquisition and Sustainment Program office: Redstone Arsenal, AL Prime contractor: Northrop Grumman Space & Mission Systems Corporation and Raytheon Contract type: CPIF/CPFF/FPI (development) Software development approach: Mixed Next major milestone: Low-rate initial production (September 2020) Funding and Quantities (fiscal year 2019 dollars in millions) Program Performance (fiscal year 2019 dollars in millions) First full estimate (12/2009) Latest (06/2018) Percentage change Development $1,786.00 $3,314.20 +85.6% Procurement $3,844.20 $3,872.70 +0.7% $19.02 $15.14 -20.4% 80 148 +85.0% 296 479 61.8% Unit cost Acquisition cycle time (months) Total quantities Total quantities comprise 25 development quantities and 454 procurement quantities. Attainment of Product Knowledge As of January 2019 Status at Resources and requirements match • Demonstrate all critical technologies are very close to final form, fit and function within a relevant environment • Demonstrate all critical technologies in form, fit and function within a realistic environment • Complete a system-level preliminary design review Product design is stable ● ○ ● ● ● ● Design Review • Release at least 90 percent of design drawings • Test a system-level integrated prototype Manufacturing processes are mature ○ ○ ● ● Production Start • Demonstrate Manufacturing Readiness Level of at least 9, or critical processes are in statistical control • Demonstrate critical processes on a pilot production line • Test a production-representative prototype in its intended environment Current Status Development Start NA NA NA NA NA NA ● Knowledge attained, ○ Knowledge not attained, … Information not available, NA Not applicable Page 83 GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Army IAMD Program Technology Maturity and Design Stability The IAMD program has demonstrated that its four critical technologies are mature and that its system design is stable. Since our 2018 assessment, the program changed two IAMD command and control components that program officials said were reaching end-of-life and becoming unreliable. This change contributed to a 29 percent reduction in the program’s total number of expected design drawings. To date, the program has completed 96 percent of its revised total number of expected design drawings, which constitutes a stable design. IAMD battle command system (IBCS) software performance indicated the need for several improvements including more soldier training. Program officials stated that the first exercises to address this and involve warfighters in August 2017 and October 2017 were successful and that the results of 2018 tests also demonstrated progress over previous tests. At the same time, program officials acknowledged that upcoming tests will require additional refinements. Despite these mixed results, the program office is confident that software reliability is on track in part because the number of software failures the program has experienced has decreased and software deliveries are on schedule. However, our review of analysis of the test results indicated that the software continues to need several improvements, including to display and tracking capabilities.These and additional developmental tests are in preparation for a second limited user test (referred to as limited testing) in early 2020. The program is conducting a second test because it demonstrated an unsatisfactory performance of the IBCS software in 2016 during the initial limited testing. The program office plans to begin IBCS qualification testing in mid-2019, followed by the start of the next major software build. To help further reduce the risk of IBCS deficiencies during the 2020 test, program officials stated the contractor is testing IBCS software with tactical network and weapon/sensor interfaces prior to government acceptance so that issues can be addressed before acceptance. Common Name: IAMD during the limited test in a shorter timeframe. This strategy does not fully account, however, for any hardware deficiencies, or for more severe software deficiencies, that may be discovered during the test. Should either of those occur, the program will likely face a choice: postpone its low-rate initial production decision to afford time to resolve deficiencies or enter low-rate initial production with system performance that, at present, it would deem unsatisfactory. Other Program Issues After experiencing cost growth in excess of the limits authorized in its acquisition program baseline, in December 2017 the Army received approval from the Under Secretary of Acquisition, Technology, and Logistics to restructure the program. The Army is now executing the program within revised cost and schedule parameters and, in accordance with the Under Secretary’s direction, plans to update its acquisition program baseline at the time of the low-rate initial production decision in September 2020. At that point, program officials said, a new independent cost estimate will also be available. The program also noted that hardware delays for the government software integration lab may threaten the schedule leading up to its low-rate initial production decision. According to program officials, the program has mitigated hardware shortages by implementing incremental deliveries of hardware. Program Office Comments We provided a draft of this assessment to the program office for review and comment. The program office provided technical comments, which we incorporated where appropriate. Program officials stated that the program has received hardware and software on schedule to support training and testing, and is continuing to develop key performance capabilities needed to support the low-rate production decision. Production Readiness The program plans to conduct a manufacturing readiness assessment in fiscal year 2020, in preparation for the September 2020 low-rate initial production decision. However, the program has allotted only a few months between the planned completion of the limited test in mid-2020 and the September 2020 production decision. Officials stated that the program recently began piloting an Agile software development approach, which provides opportunities for the program office to correct software deficiencies that may arise Page 84 GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Army Common Name: ITEP Improved Turbine Engine Program (ITEP) The Army’s ITEP is developing a replacement engine for the Black Hawk and Apache helicopter fleets. The new engine is being designed to provide increased power, performance, and fuel efficiency; enhanced reliability; increased service life; and a lower maintenance burden. The Army plans to field the new engine in fiscal year 2027. Program Essentials Milestone decision authority: Army Program office: Redstone Arsenal, AL Prime contractor: Advanced Turbine Engine Company; General Electric Company Contract type: FPI (design) Software development approach: Agile development Next major milestone: Design review (March 2020) Program Performance (fiscal year 2019 dollars in millions) First full estimate (N/A) Latest (10/2018) Percentage change Development N/A $1,913.30 N/A Procurement N/A $8.322.50 N/A Unit cost N/A $1.73 N/A Acquisition cycle time (months) N/A 103 N/A Total quantities N/A 5,994 N/A ITEP received approval to enter development in January 2019, but has yet to approve a program baseline, which would identify a first full estimate. Total quantities comprise 69 development and 5,925 procurement quantities. Funding and Quantities (fiscal year 2019 dollars in millions) Attainment of Product Knowledge As of January 2019 Status at Current Status Development Start Resources and requirements match • Demonstrate all critical technologies are very close to final form, fit and function within a relevant environment • Demonstrate all critical technologies in form, fit and function within a realistic environment • Complete a system-level preliminary design review Product design is stable NA NA NA ● ○ ● Design Review • Release at least 90 percent of design drawings NA NA • Test a system-level integrated prototype NA NA Manufacturing processes are mature Production Start • Demonstrate Manufacturing Readiness Level of at least 9, or critical processes are in statistical control • Demonstrate critical processes on a pilot production line • Test a production-representative prototype in its intended environment NA NA NA NA NA NA ● Knowledge attained, ○ Knowledge not attained, … Information not available, NA Not applicable Page 85 GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Army ITEP Program Technology Maturity The ITEP program office received approval to enter system development in January 2019 with five critical technologies approaching full maturity. Because ITEP had not established a baseline as of the end of January 2019, the program’s attainment of product knowledge could not be assessed at development start. ITEP entered technology development in August 2016 with the award of two contracts to General Electric and the Advanced Turbine Engine Company. During this phase, the program office identified three critical technologies: the advanced inlet particle separator, compressor advanced aerodynamics, and hybrid bearings. Both contractors incorporated these technologies into their prototypes for full-engine testing. A technology readiness assessment by the Office of the Assistant Secretary of the Army (Acquistion, Logistics, and Technology) (ASA (ALT)) in June 2018 concluded that these three technologies were nearing full maturity. After completing an independent review of preliminary designs, the Office of the Chief Systems Engineer within ASA (ALT) identified two additional engine components as critical technologies, based on the contractors’ use of additive manufacturing. Additive manufacturing relies on 3D printing, and is often used in rapid prototyping and direct digital manufacturing. Because both contractors are using additive manufacturing in other programs, the review team concluded that these two technologies are also approaching full maturity. The Army plans to further mature all five critical technologies through engine testing beginning in fiscal year 2021, 2 years after the start of system development. Design Stability In spring 2018, the ITEP program office conducted individual preliminary design reviews with both contractors to assess each design. In August 2018, an independent review team from the Office of the Chief Systems Engineer, ASA (ALT), also assessed each design. The independent review concluded that preliminary design reviews lacked objective and quantifiable entrance criteria. In addition, it found that the design reviews of lower level elements could not fully inform the system-level review because the schedule was compressed. Despite these findings, the Page 86 Common Name: ITEP independent review team concluded that the program office’s efforts to close deficiencies identified in the preliminary design reviews were sufficient and concluded the program was ready to proceed into system development. Following the start of system development, the program office plans to select one vendor and hold a critical design review in March 2020. Production Readiness An independent technical review conducted by the Under Secretary of Defense for Research and Engineering in October 2018 assessed manufacturing risk as low. This review concluded that the program demonstrated all manufacturing processes—to include additive manufacturing—in a production-relevant environment, on the basis that each contractor has used these processes in conventional manufacturing. The program currently plans to begin production in July 2024. Other Program Issues ITEP is highly dependent on concurrent development efforts by the engine vendors and the aircraft manufacturers, and this dependency has limited early design analysis. During technology development, both engine vendors conducted trade studies with the aircraft manufacturers to develop a proposed interface control document for their respective designs. However, the vendors could not fully assess how their designs conformed with the structural and environmental constraints of the aircraft.The program plans to have the proposed interface control document finalized during system development when the selected engine vendor and the aircraft manufacturer conducts a detailed structural and environmental analysis to support integration. Program Office Comments We provided a draft of this assessment to the program office for review and comment. The program office stated that it awarded the engineering and manufacturing development contract to General Electric on February 1, 2019. According the program office, it is providing incentives to the contractor to accelerate ITEP’s schedule and thus allow integration into the Black Hawk and Apache helicopters and other aircraft. GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Army Common Name: JAGM Joint Air-to-Ground Missile (JAGM) The Joint Air-to-Ground Missile is an Army-led program with joint requirements from the Navy and Marine Corps. The missile is designed to be air launched from helicopters and unmanned aircraft systems to target tanks, light armored vehicles, missile launchers, bunkers, and buildings. It is intended to provide precision attack capabilities no matter the time of day or weather conditions. JAGM will replace all Hellfire missile variants. Program Essentials Milestone decision authority: Army Program office: Redstone Arsenal, AL Prime contractor: Lockheed Martin Contract type: FPI (development and low-rate initial production) Software development approach: Incremental Next major milestone: Start operational test (March 2019) Program Performance (fiscal year 2019 dollars in millions) First full estimate (09/2015) Latest (12/2018) Percentage change Development $1,040.60 $1,113.20 +7.0% Procurement $4,989.30 $5,958.00 +19.4% $0.23 $0.27 +18.0% 38 44 +15.8% 26,437 26,437 0.00% Unit cost Acquisition cycle time (months) Total quantities Total quantities comprise 118 development quantities and 26,319 procurement quantities. Funding and Quantities (fiscal year 2019 dollars in millions) Attainment of Product Knowledge As of January 2019 Status at Resources and requirements match • Demonstrate all critical technologies are very close to final form, fit and function within a relevant environment • Demonstrate all critical technologies in form, fit and function within a realistic environment • Complete a system-level preliminary design review Product design is stable ● ○ ○ ● ● ○ Design Review • Release at least 90 percent of design drawings • Test a system-level integrated prototype Manufacturing processes are mature ○ ● ● ● Production Start • Demonstrate Manufacturing Readiness Level of at least 9, or critical processes are in statistical control • Demonstrate critical processes on a pilot production line • Test a production-representative prototype in its intended environment Current Status Development Start ○ ● ○ ○ ● ● ● Knowledge attained, ○ Knowledge not attained, … Information not available, NA Not applicable Page 87 GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Army JAGM Program Technology Maturity and Design Stability The JAGM program has matured its three critical technologies and stabilized its system design. The program office assessed all technologies—guidance seeker assembly/sensor platform, sensor software, and mission software—as mature in 2016 following a series of successful ground tests and one flight test from a MQ-1C Gray Eagle. JAGM components—including the motor, warhead, and electronics—have already been fully matured for the existing Hellfire missile, thus decreasing development and manufacturing risks. The program reported that JAGM had a stable design at its January 2016 critical design review. Since our 2018 assessment, however, the program changed the total number of JAGM drawings, including adding 10 drawings related to updated parts, operations, and calibration procedures. This is the third year in a row in which the program has reported an increase in total design drawings. As a result of this drawing growth, we found that the program office had released 89 percent of JAGM drawings at the critical design review—an amount that falls just short of the 90 percent level recommended by best practices. We have updated our Attainment of Product Knowledge table to reflect this change in design stability from our previous assessment. The program reported that it has now released all planned drawings for JAGM. Production Readiness The JAGM program entered low-rate initial production in June 2018, at which time it had achieved manufacturing readiness at the level recommended by DOD guidance. That guidance calls for programs to demonstrate critical manufacturing processes on a pilot production line, but does not require statistical control of those processes until the full rate production decision. Our prior work has found that this DOD standard falls short of the industry best practice. The program exercised contract options at the end of fiscal year 2018 and beginning of fiscal year 2019 for low-rate production totaling 1,423 missiles. A program official told us that the contractor began delivering the first lot of 373 low-rate production missiles in December 2018. The program expects 1,050 missiles from the second and third lots to be delivered by November 2021. The program plans to exercise another contract option for 796 low-rate production missiles in the fourth quarter of fiscal year 2019. Program officials told us that due to limited funding, they will purchase fewer missiles in the fourth lot than originally planned. Common Name: JAGM Other Program Issues In April 2016, the Army delayed initial operational testing and evaluation for JAGM by 2 years as the AH64 Apache helicopter's software—used to launch Hellfire missiles—required more pilot input than expected. The Apache program office has since developed and tested new platform software that will enable pilots to more easily select the full range of options. The program began integrated testing of the missile in January 2019. The program completed 49 developmental test firings in March 2018—including 10 firings during a limited test under operational conditions—to support its June 2018 low-rate production decision. The program also identified several JAGM deficiencies during the test. According to program officials, all of the deficiencies identified have been addressed and successfully resolved, including a missile failure. They also stated that the program was able to successfully demonstrate firing two missiles consecutively in January 2019, and that they had determined that a previous missile failure was due to the incorrect sequencing of actions in the cockpit. In December 2018, the program office reported a significant cost increase from its prior estimate, which it stated was because of changes in program requirements, changes in cost estimating methodologies, and an extended procurement schedule. Program Office Comments We provided a draft of this assessment to the program office for review and comment. The program provided technical comments, which we incorporated where appropriate. The program office stated that the 7.5month delay in the full-rate production decision has not negatively affected the program in any way. The program office also said that it will use the additional cost and software information it is now gathering to better inform that full-rate production decision. At the same time, the program office reported that it remains on track to award a full-rate production contract in September 2020, as previously planned. Since our 2018 assessment, the program office delayed its full-rate production decision by 7.5 months to allow for the collection and review of additional cost and software data. This additional information was needed to inform the full-rate production decision. Page 88 GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Army Common Name: JLTV Joint Light Tactical Vehicle (JLTV) The Army and Marine Corps’ JLTV is a family of vehicles developed to replace the High Mobility Multipurpose Wheeled Vehicle (HMMWV) for some missions. The JLTV is expected to provide protection for passengers against current and future battlefield threats, increased payload capacity, and improved automotive performance over the uparmored HMMWV variant, which includes an armor package. It is designed to be transported by air or ship. Two- and four-seat variants are planned with multiple mission configurations. Program Essentials Milestone decision authority: Army Program office: Harrison Township, MI Prime contractor: Oshkosh Defense, LLC Contract type: FFP (production) Software development approach: Waterfall Next major milestone: Full-rate production (May 2019) Program Performance (fiscal year 2019 dollars in millions) First full estimate (10/2012) Latest (06/2018) Percentage change Development $1,065.90 $988.50 -7.3% Procurement $24,127.20 $22,839.20 -5.3% $0.46 $0.41 -10.5% 125 144 +15.2% 54,730 58,306 +6.5% Unit cost Acquisition cycle time (months) Total quantities Total quantities comprise 116 development quantities and 58,190 procurement quantities. Funding and Quantities (fiscal year 2019 dollars in millions) Attainment of Product Knowledge As of January 2019 Status at Resources and requirements match • Demonstrate all critical technologies are very close to final form, fit and function within a relevant environment • Demonstrate all critical technologies in form, fit and function within a realistic environment • Complete a system-level preliminary design review Product design is stable ● ○ ● ● ● ● Design Review • Release at least 90 percent of design drawings • Test a system-level integrated prototype Manufacturing processes are mature ○ ○ ● ● Production Start • Demonstrate Manufacturing Readiness Level of at least 9, or critical processes are in statistical control • Demonstrate critical processes on a pilot production line • Test a production-representative prototype in its intended environment Current Status Development Start ○ ● ● ● ● ● ● Knowledge attained, ○ Knowledge not attained, … Information not available, NA Not applicable Page 89 GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Army JLTV Program Technology Maturity and Design Stability The JLTV program has matured its two critical technologies and stabilized the system design. However, the program’s discovery of significant deficiencies during operational testing—and the corrections that those now require—pose risk to maintaining that design stability. Following our 2018 assessment, the program’s total number of drawings increased to accommodate needed retrofits, capability changes, and delayed release of vehicle parts that support JLTV fielding. These drawing increases caused the program to fall just short of the best practice of 90 percent design drawings released at critical design review. We have updated our Attainment of Product Knowledge table to reflect this change. Production Readiness Program officials conducted a manufacturing readiness assessment in September 2018 that found that the prime contractor, Oshkosh Defense, had matured production processes and demonstrated high manufacturing readiness levels. Program officials stated that, during low-rate initial production, Oshkosh significantly reduced the number of defects per manufactured vehicle, from 14.6 in September 2016 to 1.3 in September 2018. Program officials also stated that Oshkosh has provided on-time deliveries for 6 consecutive months and is now producing vehicles 2 months ahead of schedule at a rate of about 11 per day. The program is also utilizing statistical process controls to demonstrate ongoing JLTV production readiness. Common Name: JLTV delayed the full-rate decision from December 2018 to May 2019 to review new potential requirements. According to officials, the review will not affect fielding and existing vehicles will be retrofitted if the requirements are incorporated. Officials also said they addressed some of the deficiencies discovered during testing by extending training, adjusting vehicle maintenance tasks, authenticating manuals, and correcting door and latch safety issues. Program officials previously reported to us that the JLTV engineering drawings were inadequate to facilitate open competition for production in fiscal year 2021 and beyond. Following our 2018 assessment, program officials stated they were concerned about the quality of the engineering drawings and therefore changed management and engineering processes in order to give Oshkosh improved feedback for fixing the drawings. As a result, program officials stated that they have seen significant improvements to the drawings. Program officials stated they have validated 90 percent of the drawing modifications and anticipate completing validation by March 2019. Program officials stated the drawings are now primarily government-validated and can be used in future competitions for contracts to build or support vehicles to government specifications. Program Office Comments We provided a draft of this assessment to the program office for review and comment. The program office provided technical comments, which we incorporated where appropriate. In October 2018 program officials had yet to accept 39 vehicles because of deficiencies related to their Warfighter Information Network-Tactical (WIN-T) kit configuration, which provides satellite communication capabilties to the user. Program officials stated they will not accept these vehicles until Oshkosh and the WIN-T program resolve the deficiencies. Other Program Issues The Army and Marine Corps recently concluded operational testing for JLTV and found the vehicles to be survivable for the crew and effective for small combat and transport missions, but not operationally suitable because of their high maintenance needs, low reliability, training and manual deficiencies, and safety shortcomings. The Director, Operational Test and Evaluation, made the same findings as the Army and Marine Corps. Army operational testers recommended a conditional release to full-rate production for most of the vehicle variants, pending resolution of all suitabilityrelated deficiencies. Marine Corps operational testers, however, found the JLTV could support their mission and advocated for the program’s unconditional transition into full-rate production. However, the Army Page 90 GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Army Common Name: IFPC Inc 2-1 Block 1 Indirect Fire Protection Capability Increment 2-Intercept Block 1 (IFPC Inc 2-I Block 1) The Army’s IFPC Inc 2-I is a follow-on effort to enhance and extend the range of the first IFPC increment, which provided a short-range capability to counter threats from rockets, artillery, and mortars. IFPC Inc 2-I consists of four separate subsystems—an existing sensor, interceptor (missile), and fire control system, and a new multi-mission launcher the Army plans to develop—in three blocks. We assessed Block 1, which the Army expects will counter cruise missiles and unmanned aircraft. Program Essentials Current Status Milestone decision authority: Army Program office: Huntsville, AL Prime contractor: Rafael Advanced Defense Systems Contract type (planned): FFP (procurement of interim system) Software development approach: NA Next major milestone: Procurement contract award for interim system (May 2019) The Army is pursuing a two-phased approach to acquiring IFPC Inc 2-I Block 1 capabilities. According to program officials, the Army identified two critical problems in February 2018 as the program prepared to enter system development. Specifically, Raytheon—the developer of the original planned interceptor—determined that the interceptor would shut down during operations to avoid overheating, resulting in a failure to launch. In addition, Army warfighters expressed concerns regarding the difficult procedures needed to reload the interceptor into the launcher. As a result of these discoveries, the Army’s Air and Missile Defense cross-functional team reviewed the program and recommended alternative approaches. Estimated Program Cost (FY 2019 dollars in millions) In addition, in response to legislation the Army certified that there is a need for an interim missile defense capability and plans to deploy two batteries capable of fixed-site cruise missile defense—the required capability of IFPC Inc 2-I—by September 30, 2020. In the short term, to meet this schedule and provide interim capabilities, the Army plans to acquire two Israeli Iron Dome batteries from Rafael. Program officials stated that they are planning interoperability experimentation for these batteries in fiscal year 2019 with the Sentinel radar and Integrated Battle Command System. Depending on the outcome of these experiments, the Army may elect to develop a new, interoperable launcher for the long term, which it would plan to field by 2023. Attainment of Technology Maturation Knowledge Planned Quantities As of January 2019 ○ ○ Conduct competitive prototyping Validate requirements ● Knowledge attained, ◐ Knowledge planned, Complete technology readiness assessment Complete preliminary design review ○ ○ ○ Knowledge not attained, … Information not available, NA Not applicable Program Office Comments Note: Estimated program costs reflect fiscal year 2019 and 2020 amounts only. Planned quantities reflect Iron Dome quantities only. Page 91 We provided a draft of this assessment to the program office for review and comment. The program office provided technical comments, which we incorporated where appropriate. GAO-19-336SP Weapon Systems Annual Assessment Precision Strike Missile (PrSM) The Army’s Precision Strike Missile (PrSM) (formerly known as Long Range Precision Fires) will be a ballistic missile designed to attack area and point targets at planned ranges of 400 kilometers or more. The Army anticipates that each PrSM missile container will hold two missiles for launch. The Army plans to design PrSM, as one of a family of munitions, to be compatible with existing M142 and M270A2 rocket launcher systems and to comply with statutory requirements for insensitive munitions and DOD policy on cluster munitions. Program Essentials Current Status Milestone decision authority: Army Program office: Redstone Arsenal, AL Prime contractors: Lockheed Martin; Raytheon Contract type: Other transaction agreement (technology maturation and risk reduction) Software development approach: Agile development Next major milestone: Development start (June 2021) The Army worked to maintain PrSM competitive prototyping amidst funding constraints. The Army plans to down-select from two prime contractor teams and prototypes to one in 2021 prior to development start. According to the program office, it requested and the Army provided funding to facilitate these competitive prototyping plans. The Army identified PrSM as a priority and has accelerated the missile’s acquisition schedule to provide an early capability in fiscal year 2023. However, the missile will be usable only from the M142 launcher by then. A program official stated that legacy systems on the M270 launcher need to be updated before PrSM software can be integrated. Estimated Program Cost The Army continues to refine PrSM performance requirements. In December 2018, the Army increased the range requirement from 300 to 400 kilometers and launcher quantities from one to two. (FY 2019 dollars in millions Attainment of Technology Maturation Knowledge As of January 2019 ● ◐ Conduct competitive prototyping Validate requirements ● Knowledge attained, ◐ Knowledge planned, Complete technology readiness assessment Complete preliminary design review ◐ ● ○ Knowledge not attained, … Information not available, NA Not applicable Program Office Comments Planned Quantities Page 92 We provided a draft of this assessment to the program office for review and comment. The program office provided technical comments, which we incorporated where appropriate. Program officials stated that the Army continues to maintain competition and accelerate the program. According to the program office, the current timeline reflects acceleration of the approved schedule, in accordance with Army leadership’s desire to field a capability in fiscal year 2023, complete testing in fiscal year 2024, and begin full rate production. The program office said that it continues efforts to accelerate PrSM’s integration with the M270 launcher. GAO-19-336SP Weapon Systems Annual Assessment Page 93 GAO-19-33GSP Weapon Systems Annual Assessment Navy and Marine Corps Program Assessments Page 94 GAO-19-336SP Weapon Systems Annual Assessment MARHHECORPS PROGRAM ASSESSMENTS Most Current and Future Navy Programs Report that Less than 20 Percent of Planned Acquisition Costs are Specifically for Software Percentage of planned acquisition costs 0-20% 21-40% Unknown NA/not tracked 0 5 1O 15 20 Number of programs Most Current Navy Programs Report that Less than 20 Percent of Work Tasks are Software-Related Percentage of work tasks 0-20% 21-40% 41-60% Unknown NAlnot tracked 0 5 10 15 20 Number of programs The Navy was the Milestone Decision Authority in 2017 for 22 of its Current and Future Programs in Review Navy programs I The milestone decision authority programs for AMDR, CH-53K, CVN 78. DDG 51, MQ-4C Triton, NGJ-LB, NGJ Mid-Band, and VH-92A 2 changed from the Under Secretary of Defense for Acquisition and Sustainment to the Navy in 2017. The Navy Received One Offer on Solicitations for Just Over Two Thirds of its Currently Reported Major Contracts for Development and Procurement Number of contracts System type 2 or more offers - 1 offer Source: GAO analysis of Department of Defense data. GAO-19-3365P Page 95 GAO-19-3368P Weapon Systems Annual Assessment Navy and Marine Corps Program Assessmentsa 2-page assessments Amphibious Combat Vehicle Increment 1.1 (ACV 1.1) Air and Missile Defense Radar (AMDR) CH-53K Heavy Lift Replacement Helicopter (CH-53K) CVN 78 Gerald R. Ford Class Nuclear Aircraft Carrier (CVN 78) DDG 1000 Zumwalt Class Destroyer (DDG 1000) Ground/Air Task Oriented Radar (G/ATOR) F/A-18E/F Infrared Search and Track (IRST) Joint Precision Approach and Landing System (JPALS) Littoral Combat Ship-Mission Modules (LCS Packages) MQ-4C Triton Unmanned Aircraft System (MQ-4C Triton) MQ-25 Unmanned Aircraft System (MQ-25 Stingray) Next Generation Jammer Mid-Band (NGJ Mid-Band) Offensive Anti-Surface Warfare Increment 1 (OASuW Inc 1) SSBN 826 Columbia Class Ballistic Missile Submarine (SSBN 826) Ship to Shore Connector Amphibious Craft (SSC) John Lewis Class Fleet Replenishment Oiler (T-AO 205) VH-92A Presidential Helicopter Replacement Program (VH-92A) 1-page assessmentsb DDG 51 Arleigh Burke Class Destroyer, Flight III (DDG 51 Flight III) Guided Missile Frigate (FFG(X)) LHA 8 Amphibious Assault Ship (LHA 8) LPD 17 San Antonio Class Amphibious Transport Dock, Flight II (LPD 17 Flight II) [formerly LX(R)] Next Generation Jammer – Low Band (NGJ-LB) P-8A Poseidon, Increment 3 (P-8A Increment 3) SSN 774 Virginia Class Submarine Block V (SSN 774 Block V) Page number 97 99 101 103 105 107 109 111 113 115 117 119 121 123 125 127 129 131 132 133 134 135 136 137 a We abbreviate the following contract types in the individual assessments: cost-plus-award-fee (CPAF), cost-plus-fixed-fee (CPFF), cost-plus-incentivefee (CPIF), firm-fixed-price (FFP), and fixed-price incentive (FPI). b DDG 51 Flight III, LHA 8, LPD 17 Flight II, P-8A Increment 3, and SSN 774 Block V are current major defense acquisition programs that are well into production, but we have assessed them in a one-page format because they are planning to introduce new increments of capability. Page 96 GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Navy Common Name: ACV 1.1 Amphibious Combat Vehicle Increment 1.1 (ACV 1.1) The Marine Corps’ ACV is the successor program to the canceled Expeditionary Fighting Vehicle. The ACV is intended to transport Marines from ship to shore and provide them with improved mobility and high levels of protection. The ACV acquisition approach calls for three increments of development (1.1, 1.2, and 2.0) and leverages work accomplished under the EFV program. We assessed increment 1.1. Late in our assessment period, in January 2019, the Navy merged ACV 1.1 and ACV 1.2 into a single program, the ACV Family of Vehicles. Program Essentials Milestone decision authority: Navy Program office: Stafford, VA Prime contractors: BAE Systems and Land Armaments LP; Science Applications International Corporation (SAIC) Contract types: FPI/FFP/CPFF (development—SAIC) (development and low-rate initial production—BAE Systems and Land Armaments LP) Software development approach: NA Next major milestone: Start of operational testing (February 2020) Funding and Quantities (fiscal year 2019 dollars in millions) Program Performance (fiscal year 2019 dollars in millions) First full estimate (05/2016) Latest (07/2018) Percentage change Development $821.70 $696.40 -15.3% Procurement $1,091.80 $1,094.70 +0.3% $8.18 $7.64 -6.6% 57 57 0.00% 240 240 0.00% Unit cost Acquisition cycle time (months) Total quantities Total quantities comprise 36 development quantities and 204 procurement quantities. Attainment of Product Knowledge As of January 2019 Status at Resources and requirements match • Demonstrate all critical technologies are very close to final form, fit and function within a relevant environment • Demonstrate all critical technologies in form, fit and function within a realistic environment • Complete a system-level preliminary design review Product design is stable ● ○ ○ ● ● ● Design Review • Release at least 90 percent of design drawings • Test a system-level integrated prototype Manufacturing processes are mature ● ○ ● ○ Production Start • Demonstrate Manufacturing Readiness Level of at least 9, or critical processes are in statistical control • Demonstrate critical processes on a pilot production line • Test a production-representative prototype in its intended environment Current Status Development Start ○ ● ● ○ ● ● ● Knowledge attained, ○ Knowledge not attained, … Information not available, NA Not applicable Page 97 GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Navy ACV 1.1 Program Technology Maturity, Design Stability, and Production Readiness The ACV 1.1 program has matured its critical technologies and stabilized its system design. However, the ACV 1.1 contractor is building vehicles prior to bringing critical manufacturing processes into statistical control—an approach inconsistent with best practices. The ACV 1.1’s two critical technologies—the driver vision enhancement system and the remote weapon station—are fully mature. According to program officials, the ACV 1.1 has exceeded its reliability growth targets in its most recent tests, excluding failures related to the government-furnished remote weapon station. Program officials added that they are conducting dedicated testing to localize and determine a root cause for the remote weapon station failures. The ACV 1.1 program released 100 percent of its design drawings before the production decision in June 2018. The Navy approved the program’s entry into lowrate initial production with the selected contractor’s (BAE Systems) manufacturing readiness below the level recommended by the Department of Defense guidance and best practices. An assessment made by the Defense Contract Management Agency for the ACV 1.1 production readiness review identified high-risk areas such as manufacturing personnel and process capability and control. Three months before the production decision, the program conducted a risk assessment and reported two areas as risks to the overall ACV 1.1 production schedule: facilities and personnel. Program officials said that, in June 2018, BAE Systems submitted a risk mitigation plan that addresses these manufacturing risks, and the program is monitoring its implementation. The program reported that following the production decision, the Marine Corps exercised a contract option with BAE Systems for 30 low-rate initial production vehicles. Officials also reported that in December 2018, the Marine Corps exercised another contract option with BAE Systems for a second lot of 30 vehicles. Common Name: ACV 1.1 any systemic problems that would necessitate a major redesign, and live fire testing indicated that prototypes met key force protection requirements. However, the Director, Operational Test and Evaluation, made recommendations as a result of the tests, including to investigate ways to prevent damage to steering and suspension from debris, and to modify the troop commander’s station to make it easier to move between the hatch and seat. Since the operational assessment, the program has held corrective action periods to implement minor design modifications in response to testing and plans to continue developmental testing into December 2019. Officials stated that the program has awarded a production contract to BAE Systems that includes options for the production of the second increment of the ACV (ACV 1.2). The primary intended upgrade for ACV 1.2 over ACV 1.1 was improved amphibious capability, including the ability to self-deploy from an amphibious ship and travel to shore. However, the program developed and tested these capabilities during the development of ACV 1.1. As result, the ACV program officials said that they expect the transition to the next increment will be relatively seamless. Program Office Comments We provided a draft of this assessment to the program office for review and comment. The program office provided technical comments, which we incorporated where appropriate. The program office noted that, in January 2019, it received approval from the Navy to merge the ACV programs into a single program, the ACV Family of Vehicles. The program office said that this merger includes acquiring ACV 1.1 and ACV 1.2 under a single set of performance requirements, developing additional ACV models with different mission profiles, and updating acquisition documentation for the full-rate production decision review. The program office also stated that it is continuing low-rate initial production and plans to test production representative vehicles beginning in the third quarter of fiscal year 2019. Other Program Issues The Marine Corps selected BAE Systems from two competing contractors for ACV 1.1 low-rate initial production. During the development phase, two contractors designed and produced 16 prototypes each. Starting in March 2017, the program conducted developmental and live fire testing of the prototypes to determine the extent to which they met capability requirements such as technical performance, reliability, force protection, and system survivability. The Marine Corps subsequently conducted an operational assessment from January to March 2018 to inform the contractor selection and the decision to enter production. The operational assessment did not identify Page 98 GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Navy Common Name: AMDR Air and Missile Defense Radar (AMDR) The Navy's AMDR is a next-generation radar program supporting surface warfare and integrated air and missile defense. The Navy expects AMDR’s radar—known as AN/SPY-6(V)1—to provide increased sensitivity for long-range detection to improve ballistic missile defense against advanced threats. The program office is also developing a radar suite controller that will interface with an upgraded Aegis combat system to provide integrated air and missile defense for DDG 51 Flight III destroyers. Program Essentials Milestone decision authority: Navy Program office: Washington Navy Yard, DC Prime contractor: Raytheon Contract types: CPIF (development) FPI (low-rate initial production) Software development approach: Agile development Next major milestone: First production radar delivery (April 2020) Funding and Quantities (fiscal year 2019 dollars in millions) Program Performance (fiscal year 2019 dollars in millions) First full estimate (10/2013) Latest (06/2018) Percentage change Development $2,029.70 $1,911.10 -5.8% Procurement $4,197.80 $3,634.90 -13.4% $284.50 $253.51 -10.9% 156 161 +3.2% 22 22 0.00% Unit cost Acquisition cycle time (months) Total quantities Attainment of Product Knowledge As of January 2019 Status at Current Status Development Start Resources and requirements match • Demonstrate all critical technologies are very close to final form, fit and function within a relevant environment • Demonstrate all critical technologies in form, fit and function within a realistic environment • Complete a system-level preliminary design review Product design is stable ● ○ ○ ● ○ ● Design Review • Release at least 90 percent of design drawings • Test a system-level integrated prototype Manufacturing processes are mature ○ ● ● ● Production Start • Demonstrate Manufacturing Readiness Level of at least 9, ○ ○ • Demonstrate critical processes on a pilot production line NA NA • Test a production-representative prototype in its intended ○ ○ or critical processes are in statistical control environment ● Knowledge attained, ○ Knowledge not attained, … Information not available, NA Not applicable Page 99 GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Navy AMDR Program Technology Maturity and Design Stability The program office reported that AMDR’s four critical technologies are mature—although we disagree—and that the system design is stable. Since our 2018 assessment, the program office has further demonstrated the radar system’s performance and capabilities through live testing and simulation. However, based on industry best practices, the program cannot fully demonstrate all critical technologies until the Navy tests them in their realistic, at-sea environment with the Aegis combat system. According to the AMDR program schedule, such testing will occur in 2023 during operational testing with a DDG 51 Flight III ship. Until the Navy completes this testing, the program’s design stability remains at risk for disruption. Specifically, any performance deficiencies the Navy discovers during at-sea testing could require it to revise existing design drawings to remedy issues. As part of developmental testing, the program office tested a full-scale, single-face radar array at the Navy’s Pacific Missile Range Facility (PMRF) beginning in September 2016. The program office successfully completed several live ballistic missile defense, anti-air, and anti-surface warfare tests. However, in March 2018, the array failed a ballistic missile test because of a defective software update that caused the array to stop tracking a live target. Officials said a software update corrected the issue and they verified the array’s performance through a successful retest in January 2019. Officials said the single-face array, originally scheduled to support Aegis combat system equipment testing, will undergo additional testing at PMRF through 2019. As a result, the Navy has revised the acquisition schedule and will instead divert the delivery of a new array to support land-based Aegis combat system equipment testing sometime in 2019. The program has completed software development to support core radar capabilities and will continue to develop radar updates to support system improvements, cybersecurity, and combat system integration through 2021. In parallel to the radar’s software development, significant software development remains to integrate AMDR with the Aegis combat system. Program officials said this software development must complete before both systems can be fully integrated and tested. While the Navy plans to test the radar and initial Aegis combat system software at a land-based site, the Navy will not test the radar and final Aegis combat system until both are installed on the lead ship. Any issues identified after the systems are installed on the lead ship could require retrofits to the radar or ship. Page 100 Common Name: AMDR Production Readiness Nearly 18 months after entering production, the program has not demonstrated that all of its critical manufacturing processes are in statistical control. The program reported that it exercised a contract option for the fourth low-rate initial production unit in April 2018 and was authorized to procure five additional low-rate production units in February 2019. However, in August 2018, the contractor reported early cost growth and schedule variance for the first three low-rate production units because of increased material costs and other production delays. Officials said the delays are partly due to a problem with a digital receiver component, which the contractor is testing. As a result, contractor delivery of the first production radar is at risk of delay from December 2019 to April 2020. The AMDR program office plans to procure more than two-thirds of its 22 total radars prior to completing operational testing. The Navy deliberately planned for AMDR to begin production prior to the start of Aegis upgrade software development to allow time for key radar technologies to mature and for the design to stabilize. However, this concurrency means any deficiencies identified during combat system integration or operational testing may lead to retrofitting after production is underway or complete for many of the radars. Any required retrofitting is likely to increase program costs or delay radar deliveries. Other Program Issues DOD’s Director, Operational Test and Evaluation (DOT&E) has yet to approve the AMDR Test and Evaluation Master Plan. DOT&E stated that the proposed test approach for the AMDR and DDG 51 Flight III programs does not provide realistic operational conditions without the use of an AMDR- and Aegisequipped unmanned self-defense test ship. Because the Navy has elected not to request funds for a test ship, DOT&E and the Navy are revising the DDG 51 Flight III operational test strategy to include AMDR operational requirements and an updated simulation strategy. DOT&E cautioned, however, that DDG 51 Flight III’s self-defense and survivability capabilities will not be fully known until the program completes operational testing. Program Office Comments and GAO Response We provided a draft of this assessment to the program office for review and comment. In its comments, the program office disagreed with our assessment of the program’s technology maturity, stating that combat system testing is not required to demonstrate mature radar technologies since the technologies have been tested and proven at the land-based PMRF site. We disagree. The PMRF site does not provide a realistic, at-sea environment to test the fit and function of the radar and combat system on a ship. GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Navy Common Name: CH-53K CH-53K Heavy Lift Replacement Helicopter (CH-53K) The Marine Corps' CH-53K heavy-lift helicopter is intended to transport armored vehicles, equipment, and personnel to support operations deep inland from a sea-based center of operations. The CH-53K is expected to replace the legacy CH-53E helicopter and provide increased range and payload, survivability and force protection, reliability and maintainability, and coordination with other assets, while reducing total ownership costs. Program Essentials Program Performance (fiscal year 2019 dollars in millions) Milestone decision authority: Navy Program office: Patuxent River, MD Prime contractor: Sikorsky Aircraft First full estimate (12/2005) Latest (07/2018) Percentage change Contract type: CPIF Development $4,902.9 $7,465.8 +52.3% (development) Procurement $13,635.6 $21,366.1 +56.7% $118.84 $144.23 +21.4% Acquisition cycle time (months) 117 187 +59.8% Total quantities 156 200 +28.2% FFP/CPFF (engine development) FPI/CPFF/FFP (LRIP) Software development approach: Mixed Next major milestone: Start of operational testing (May 2021) Funding and Quantities (fiscal year 2019 dollars in millions) Unit cost Total quantities comprise 6 development and 194 procurement quantities. Latest initial capability date reflects a 16month delay from the July 2018 estimate. The program has not yet determined any associated cost changes. Attainment of Product Knowledge As of January 2019 Status at Resources and requirements match • Demonstrate all critical technologies are very close to final form, fit and function within a relevant environment • Demonstrate all critical technologies in form, fit and function within a realistic environment • Complete a system-level preliminary design review Product design is stable ○ ○ ○ ● ● ● Design Review • Release at least 90 percent of design drawings • Test a system-level integrated prototype Manufacturing processes are mature ○ ○ ● ● Production Start • Demonstrate Manufacturing Readiness Level of at least 9, or critical processes are in statistical control • Demonstrate critical processes on a pilot production line • Test a production-representative prototype in its intended environment Current Status Development Start ○ ● ● ○ ● ● ● Knowledge attained, ○ Knowledge not attained, … Information not available, NA Not applicable Page 101 GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Navy CH-53K Program Technology Maturity, Design Stability, and Production Readiness The CH-53K program entered production in March 2017 with mature critical technologies and a stable design, but without demonstrated production processes. Due to a number of technical deficiencies identified through developmental testing, since last year’s assessment, the program office has delayed the start of operational testing by almost 2 years, to May 2021, and postponed initial operational capability by 20 months, to September 2021. The program office identified two critical technologies— the main rotor blade and the main gearbox—for CH53K. Although,the program office reported that both critical technologies are mature, there are technical issues with the main gear box causing low service life projections. The program office also noted that while there are parts shortages with the main gearbox,the supplier has recently improved its manufacturing processes in an effort to reduce the backlog of needed parts. It is too soon to tell if this will reduce the parts shortages. The program office also reported that it has released 99 percent of CH-53K design drawings, which constitutes a stable design. Nonetheless, recent deficiencies discovered in developmental testing may require the program to undertake design changes that could disrupt that design stability.Developmental tests have revealed high engine bay temperatures, which could cause exhaust gas to enter the aircraft fuselage during operations. The program extended its developmental testing into January 2021. Program officials stated that all technical problems likely will not be addressed before operational testing begins in May 2021. The program office is currently amending its test plan and is considering different scenarios to address this situation. Delivery of production aircraft is scheduled for the fourth quarter of fiscal year 2020. The program has yet to demonstrate that its critical manufacturing processes are in statistical control—an approach inconsistent with best practices. Since our 2018 assessment, the program delayed award of a planned second low-rate initial production contract for four aircraft. The program planned to award this contract in March 2018, but changes to the program’s test plan have caused the program to delay this award. Further, the CH-53K contractor has delivered to the Navy only three of six planned system demonstration test articles. The contractor was scheduled to deliver the fourth test article to the Navy in the first quarter of fiscal year 2019. However, the contractor used the fourth test article to supply parts for the first three due to parts shortages. The contractor now plans to deliver the Page 102 Common Name: CH-53K fourth aircraft by December 2020. According to the program office, the first two test articles are currently being used for developmental testing while the third was delivered to the Marines for logistics demonstration projects. The program plans to use the first four test articles in operational testing. However, the first three will need to be modified first, which is not expected to be completed by the start of testing. If the fourth aircraft delivery is delayed again, the Navy will have to delay operational testing until the aircraft are ready. Other Program Issues The program office has increasingly discovered software deficiencies over the past year. Specifically, developmental testing showed a failure in the software to detect the transition from ground to flight causing increased safety concerns. The program is currently working to solve this problem through additional software development. The program office stated they generally revised the software plan so they could be more flexible in addressing software deficiencies. The program office also reported increased risk in completing additional software development efforts to meet cyber security needs. According to the office of the Director, Operational Test and Evaluation, the program updated its test and evaulation plan to emphasize cyber security test strategies. In addition, the contractor moved the CH-53K assembly line from West Palm Beach, Florida, to Stratford, Connecticut, in June 2018. The final two test articles will be produced in Connecticut, but the program office does not expect the relocation to affect production. Program Office Comments We provided a draft of this assessment to the program office for review and comment. The program office provided technical comments, which we incorporated where appropriate. The program office stated that, because of technical issues during developmental testing, the program has completed test events slower than planned, which has affected schedule and cost. The program will miss four key baseline dates, and the program is establishing a new schedule baseline. The program office noted the planned dates for initial capability, end operational test, and full-rate decision are tentative pending a revised acquisition program baseline. The program’s top priorities for the start of operational testing are to resolve the remaining technical issues and complete airworthiness certification testing. Specifically, the program office stated that it has prioritized the flight test plan to address the top technical issues, correct other technical deficiencies, and deploy the fleet as planned in 2024. GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Navy Common Name: CVN 78 CVN 78 Gerald R. Ford Class Nuclear Aircraft Carrier (CVN 78) The Navy developed the CVN 78 (or Ford Class) nuclear-powered aircraft carrier to introduce new propulsion, aircraft launch and recovery, and survivability capabilities to the carrier fleet. The Ford Class is the successor to the Nimitz Class aircraft carrier. Its new technologies are intended to create operational efficiencies while enabling a 33 percent increase in sustained operational aircraft flights over legacy carriers. The Navy also expects the new technologies to enable Ford Class carriers to operate with reduced manpower. Program Essentials Milestone decision authority: Navy Program office: Washington, DC Prime contractor: Huntington Ingalls Industries Contract types: CPFF/CPIF (CVN 79 construction preparation); CPFF/FPI (CVN 79 detail design and construction); CPFF/undefinitized (CVN 80 advance procurement); FPI (CVN 80/81 detail design and construction) Next major milestone: Initial operational capability (July 2019) Software development approach: NA Funding and Quantities (fiscal year 2019 dollars in millions) Program Performance (fiscal year 2019 dollars in millions) First full estimate (04/2004) Latest (06/2018) Percentage change Development $5,378.60 $6,096.50 +13.3% Procurement $34,456.50 $42,568.60 +23.5% Unit cost $13,278.37 $12,213.29 -8.0% 137 183 +33.6% 3 4 +33.3% Acquisition cycle time (months) Total quantities Attainment of Product Knowledge As of January 2019 Status at Resources and requirements match • Demonstrate all critical technologies are very close to final form, fit and function within a relevant environment • Demonstrate all critical technologies in form, fit and function within a realistic environment • Complete a system-level preliminary design review Product design is stable ○ ○ ○ ● ○ ● Fabrication Start • Complete basic and functional design to include 100 percent of 3D product modeling Current Status Detail Design Contract Award ○ ● ● Knowledge attained, ○ Knowledge not attained, … Information not available, NA Not applicable Page 103 GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Navy CVN 78 Program Technology Maturity, Design Stability, and Production Readiness The Navy accepted delivery of the lead ship, CVN 78, in May 2017 despite challenges related to immature technologies and struggles to demonstrate the reliability of mature systems. The Navy reports that 10 of the Ford Class’s 12 critical technologies are fully mature—the advanced arresting gear (AAG) and one of the ship’s missile systems are not yet mature. The advanced weapons elevators are among the systems deemed mature by the Navy; however, according to Navy officials, only 2 of the 11 elevators installed on the ship can bring munitions to the flight deck—a key element of operational flights. The shipbuilder is working to correct the system during its first post-delivery maintenance period, now scheduled to end in October 2019, and the Navy plans to create a land-based site to test the elevators, which will come at an additional cost. Shipboard testing is ongoing for several critical systems and could delay future operational testing. Those systems include the electromagnetic aircraft launch system (EMALS), AAG, and dual band radar (DBR). Although the Navy is testing EMALS and AAG on the ship with aircraft, the reliability of those systems remains a concern. If these systems cannot function safely, CVN 78 will not demonstrate it can rapidly deploy aircraft—a key requirement for these carriers. Recent shipboard testing revealed that the Navy is struggling to get DBR to operate as planned. Moreover, DBR poses a greater radiation hazard to personnel and systems on an aircraft carrier than the Navy anticipated, which could restrict certain types of flight operations. The remaining challenges the Navy faces in maturing CVN 78’s critical technologies could lead to their redesign or replacement on later ships. This would include CVN 79, which is currently 55 percent complete, as well as the third and fourth ships, CVNs 80 and 81. CVN 79 repeats the CVN 78 design with some modifications and replaces DBR with the Enterprise Air Surveillance Radar (EASR), which is in development. The Navy does not identify this new system as a critical technology in the Ford Class program because it derives from the pre-existing Air and Missile Defense Radar (AMDR) program. However, EASR is a different size and performs a different mission than the AMDR systems, which are designed for destroyers. Therefore, EASR may still require design and development efforts to function on the carrier.The Navy plans to procure two EASR units for CVNs 79 and 80 and install the CVN 79 unit during that ship’s second phase of delivery. CVNs 80 and 81 will repeat the design of CVN 79. Other Program Issues CVN 78’s procurement costs increased by 23 percent over its initial cost cap and as a result of continuing technical deficiencies, the Navy may still require more Page 104 Common Name: CVN 78 funding to complete this ship. The Navy increased the current $12.9 billion cost cap for CVN 78 by $120 million in May 2018 to account for additional postdelivery work, but added work and cost changes may result in an additional cost increase. Costs for CVN 79 are also likely to increase as a result of optimistic cost and labor targets, putting the ship at risk of exceeding its $11.4 billion cost cap. The CVN 79 cost estimate assumes unprecedented construction efficiency—labor hours will be 18 percent lower than CVN 78. However, our analysis shows the shipbuilder is not meeting this goal and is unlikely to improve performance enough to meet cost and labor targets. Congress raised the cost cap for CVN 80 and later ships to $12.6 billion and approved the Navy’s plans to buy two carriers—CVNs 80 and 81—at the same time, based on the shipbuilder’s estimate that this strategy will save the Navy over $2 billion. However, it is unclear whether the Navy can meet this cost cap, even with the estimated savings from a two-ship buy, because it assumes further reductions in subsystem costs, construction change orders, and labor hours. The Navy projects a further reduction in labor hours compared to CVN 79—about 25 percent fewer labor hours than CVN 78—will contribute to cost savings for these ships. The program office indicated that it does not separately track or report information on software development to integrate the various subsystems of the ship. These subsystems include CVN 78’s combat control systems, which rely on integrating systems through software intensive development. Program Office Comments We provided a draft of this assessment to the program office for review and comment. The program office provided technical comments, which we incorporated where appropriate. The program office stated that, in July 2018, CVN 78 entered a year-long maintenance period. It also said that, as of February 2019, two advanced weapons elevators are operating, and it continues to improve developmental system reliability. The program also stated that, with CVN 79 construction 55 percent complete, shipbuilder cost performance remains stable, but slightly below the level needed to achieve production labor hour reduction targets. The program stated that the shipbuilder continues to work through the effects of material shortfalls that disrupted performance. The program said that the Navy plans to deliver a complete, deployable ship as scheduled and within the cost cap to maintain an 11-carrier fleet. The program office also stated that the Navy awarded the CVN 80/81 procurement contract in January 2019 and expects to save $4 billion, compared to if it had purchased each ship individually. According to the program, the contract limits the Navy’s liability and incentivizes the shipyard’s best performance. GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Navy Common Name: DDG 1000 DDG 1000 Zumwalt Class Destroyer (DDG 1000) The DDG 1000 destroyer is a multi-mission surface ship initially designed to provide advanced capability to support forces on land. DDG 1000 class ships feature a stealth design, integrated power system, and total ship computing environment. The Navy adopted a phased acquisition strategy, which separates delivery and acceptance of hull, mechanical, and electrical (HM&E) systems from combat system activation and testing. The Navy has recently changed DDG 1000’s primary mission from land attack to offensive surface strike. Program Essentials Milestone decision authority: Navy Program office: Washington, DC Prime contractors: General Dynamics Bath Iron Works; BAE Systems; Huntington Ingalls Industries; Raytheon Contract types: FPI/FFP/CPFF (ship construction); FPI/CPFF(advanced gun systems equipment); CPFF/CPAF (mission systems equipment) Software development approach: Mixed Next major milestone: Lead-ship final delivery (May 2019) Funding and Quantities (fiscal year 2019 dollars in millions) Program Performance (fiscal year 2019 dollars in millions) First full estimate (01/1998) Latest (06/2018) Percentage change Development $2,550.53 $11,876.75 +365.7% Procurement $36,414.13 $13,538.91 -62.8% 1,217.65 8,471.89 +595.8% 128 273 +113.3% 32 3 -90.63% Unit cost Acquisition cycle time (months) Total quantities Attainment of Product Knowledge As of January 2019 Status at Resources and requirements match • Demonstrate all critical technologies are very close to final form, fit and function within a relevant environment • Demonstrate all critical technologies in form, fit and function within a realistic environment • Complete a system-level preliminary design review Product design is stable ○ ○ ● ● ○ ● Fabrication Start • Complete basic and functional design to include 100 percent of 3D product modeling Current Status Detail Design Contract Award ○ ● ● Knowledge attained, ○ Knowledge not attained, … Information not available, NA Not applicable Page 105 GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Navy Common Name: DDG 1000 DDG 1000 Program the ship’s main turbine generators and found that the generator was damaged by a woodscrew. The damage was extensive enough that the Navy chose to replace the engine and send it for repair. Officials report that the shipbuilder delivered the ship in April 2018 and the Navy replaced the engine in September 2018 at its expense. Technology Maturity and Design Stability The DDG 1000 program has fully matured most, but not all, of its nine current critical technologies and reports a stable design. According to the Navy, the fire suppression system, hull form, deckhouse, power system, and undersea warfare suite technologies are all mature. At the same time, the vertical launch system, infrared signature, multi-function radar, and total ship computing environment technologies each continue to approach maturity. The Navy expects to fully mature these systems as it completes ship construction, certification, and testing over the next 2 years. The program originally had 12 critical technologies, but in the last several years, the Navy removed three, including two technologies associated with the advanced gun system—the projectile and the gun— because of the projectile’s high cost per round. The Navy planned to rely on these munitions for precision fires and offensive operations. Following an evaluation of five other munition options, the Navy determined that no viable replacement, guided or unguided, was feasible. As a result, the guns will remain inoperable on the ships for the foreseeable future. Lastly, the Navy will use a modified multi-function radar in place of a volume search radar, which the Navy removed from the class. As we have previously reported, the Navy and its shipbuilders had not stabilized DDG 1000’s design by lead ship fabrication start in 2009—an approach inconsistent with best practices. This approach contributed to numerous design changes after the fabrication start and significant cost increases and schedule delays. Nearly 10 years later, development and shipboard testing of technologies continues, each of which could lead to discovery that could disrupt the design stability the Navy currently claims. The Navy plans to complete software development for the class in September 2020—a delay of 24 months since our 2018 assessment. As a result, the Navy has had to delay some testing. Also that month, the program plans to complete its cyber security vulnerability evaluation along with the remainder of a 2-year regimen of certifications and several different tests. The Navy expects this regimen to demonstrate the full functionality of the ship’s systems. Production Readiness The DDG 1000 shipbuilder is approaching completion of the hull, mechanical, and electrical (HM&E) systems for all three ships of the class. Shipbuilder delivery of the lead ship's HM&E occurred 18 months behind schedule, in part because of problems completing electrical work associated with the ship’s power system. The shipbuilder also experienced problems completing the power system for DDG 1001, the second ship in the class. Following sea trials, the Navy inspected one of Page 106 The Navy has scheduled DDG 1000’s final delivery, including HM&E and combat systems, for May 2019. The Navy has scheduled DDG 1001’s final delivery to follow in September 2020. However, the Navy is still working to correct serious deficiencies that its Board of Inspection and Survey has identified on both ships. Specifically, the board found over 320 serious deficiencies when the shipbuilder delivered DDG 1000’s HM&E in May 2016, and 246 serious deficiences after the Navy conducted acceptance trials for DDG 1001 in January and February 2018. This increases the likelihood that the ship will not be fully capable and sustainable when provided to the fleet. To limit further delays to DDG 1000 and DDG 1001 construction, the Navy has authorized its shipbuilder to take parts from DDG 1002—the third and final ship of the class, which is under construction. The Navy does not yet know the full extent to which these actions will delay DDG 1002’s construction schedule, but stated that these parts typically can be borrowed and replaced without causing a delay. The Navy has scheduled the ship’s HM&E delivery in March 2020 followed by final delivery in September 2022. Other Program Issues In a January 2018 decision memorandum, the Navy changed DDG 1000’s primary mission from land attack to offensive surface strike. Navy officials are in the process of determining the operational concept for the ship within its new mission. The Navy has yet to establish testing plans to evaluate these future mission sets. According to Navy officials, the Navy’s planned modifications to support the new mission will cost about $1 billion, from non-acquisition accounts. Program Office Comments We provided a draft of this assessment to the program office for review and comment. The program office provided technical comments, which we incorporated where appropriate. The program office also stated that it is making good progress delivering the Zumwalt class. The Navy said that, since our assessment, DDG 1000 completed combat systems availability, combat tests are underway, and final delivery is now planned for September 2019. The program office also said that DDG 1001 started combat systems availability in April 2019, and DDG 1002 is 84 percent constructed. The program office further noted that future addition of new systems onto Zumwalt-class ships will provide offensive fire capabilities. GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Navy Common Name: G/ATOR Ground/Air Task Oriented Radar (G/ATOR) The Marine Corps' G/ATOR is a three-dimensional, short-to-medium range, multi-role radar designed to detect, identify, and track threats such as incoming cruise missiles, rockets, and artillery. It will replace five legacy radars. G/ATOR is being acquired in blocks, with later blocks focused on software upgrades. We assessed Block 1, which has an air defense and surveillance role, and have made observations on Block 2, which will determine enemy firing positions and point of impact for incoming fire. Program Essentials Milestone decision authority: Navy Program office: Quantico, VA Prime contractor: Northrop Grumman Contract type: FPI/CPFF/FFP (low-rate initial production) Software development approach: Incremental Next major milestone: Initial operational capability for Block 2 (February 2019) Funding and Quantities (fiscal year 2019 dollars in millions) Program Performance (fiscal year 2019 dollars in millions) First full estimate (08/2005) Latest (07/2018) Percentage change Development $396.50 $1,167.00 +194.4% Procurement $1,244.90 $1,928.60 +54.9% $25.65 $68.79 +168.2% Acquisition cycle time (months) 61 150 +145.9% Total quantities 64 45 -29.7% Unit cost Attainment of Product Knowledge As of January 2019 Status at Resources and requirements match • Demonstrate all critical technologies are very close to final form, fit and function within a relevant environment • Demonstrate all critical technologies in form, fit and function within a realistic environment • Complete a system-level preliminary design review Product design is stable ⋯ ● ● ● ○ ○ ● ● ⋯ ○ Design Review • Release at least 90 percent of design drawings • Test a system-level integrated prototype Manufacturing processes are mature Production Start • Demonstrate Manufacturing Readiness Level of at least 9, or critical processes are in statistical control • Demonstrate critical processes on a pilot production line • Test a production-representative prototype in its intended environment Current Status Development Start ○ ● ● ● ● ● ● Knowledge attained, ○ Knowledge not attained, … Information not available, NA Not applicable Page 107 GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Navy G/ATOR Program Technology Maturity, Design Stability, and Production Readiness According to the G/ATOR program office, the radar’s six critical technologies are mature, design is stable, and production processes are in control as the program approaches its full-rate production decision in April 2019. We were unable to assess G/ATOR’s technology maturity at development start because the program did not generate the type of information needed for such an assessment. The program office stated that the system is on track to demonstrate all of its key performance requirements; however, system reliability—while improved—remains a concern due mostly to software issues. The program’s operational assessment, conducted from March to May 2018, raised a series of reliability and usability deficiences that the program is working to address. Program officials stated that the Marine Corps decided to wait to field the Block 2 radar until 2019 due to the issues raised during the operational assessment. During that assessment, Marine Corps testers noted that while Block 2 radar offered greater detection ranges than current radars, those advantages were offset by reliability concerns, among other issues. According to program officials, several of the issues raised in the assessment resulted from the Marine testers being inexperienced and unfamiliar with operating the radar system. In other cases, they were due to known software problems, which the program said that it has addressed. The operational assessment resulted in eight recommendations, and program officials stated they have taken actions to address them. The program completed operational testing, which assessed the effectiveness, suitability, and survivability of the Block 1 and Block 2 radars, in October and December 2018, respectively. The testing report for Block 1 found it to be operationally effective, suitable, and survivable. However, the testing report for Block 2 was not complete at the time of our review. Common Name: G/ATOR Beginning with radars produced in 2016, the program upgraded G/ATOR’s “transmit/receive modules”—key components that process signals from and to the radar—to a new, but mature gallium nitride (GaN) semiconductor technology. The GaN semiconductors fit inside the G/ATOR system the same way as the older gallium arsenide (GaAs) semiconductors they replaced. Program officials expect the GaN technology to achieve better performance with higher reliability at lower cost by reducing the number of modules required. The program office has budgeted $19.8 million from fiscal years 2022 through 2024 to refurbish the first six GaAs radars with GaN technology, and to bring all 15 low rate initial production systems up to current full-rate production configuration. The Marine Corps also continues to fund initiatives to add new capabilities to the radar. Program Office Comments We provided a draft of this assessment to the program office for review and comment. The program office stated that the G/ATOR program continues to meet its cost, schedule, and performance commitments following establishment of a new acquisition program baseline in 2010. The program office stated that it has accepted delivery of eight low-rate initial production systems and has seven additional systems under contract. According to the program office, G/ATOR has demonstrated all Block 1 and Block 2 key performance requirements. The program office stated that G/ATOR will meet initial operational capability with two Block 1 systems fielded in 2018 and four Block 2 systems fielded in 2019. According to the program office, the program is on track for a full-rate production decision in 2019 as well as delivery of the required capability to the warfighter. The program office also stated that G/ATOR's operational availability has exceeded the system’s requirement and early software quality challenges addressed. The G/ATOR program is well into low-rate initial production and its production processes have matured. At the program’s March 2014 production decision, the contractor had demonstrated G/ATOR production processes to the DOD recommended level, but had not brought them into statistical control, which was inconsistent with best practices. In 2018, the program completed a production readiness review to support fullrate production and reported that its production processes are now in statistical control. The Marine Corps accepted delivery of eight production radars in 2018 as scheduled, and the contractor delivered the first two radars incorporating new semiconductor technology about one month ahead of schedule. Page 108 GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Navy Common Name: IRST F/A-18E/F Infrared Search and Track (IRST) The Navy is developing an infrared search and track (IRST) sensor to integrate onto the F/A-18E/F fuel tank. IRST will allow F/A-18s to detect and track objects from a distance in environments where radar is not effective. The Navy is acquiring IRST with an evolutionary acquisition approach. It used existing IRST systems to develop Block I and is implementing Block II to deliver an improved sensor, upgraded processor, and additional software. We assessed IRST Block II. Program Essentials Milestone decision authority: Navy Program office: Patuxent River, MD Prime contractor: Boeing Contract type: CPIF (Block II development), FPIF (Block II low-rate production) Software development approach: Mixed Next major milestone: Start of operational testing (August 2020) Program Performance (fiscal year 2019 dollars in millions) First full estimate (02/2017) Latest (12/2018) Percentage change Development $899.00 $941.00 +4.7% Procurement $1,354.00 $1,403.20 +3.6% $12.59 $13.55 +7.7% Acquisition cycle time (months) 123 123 0.00% Total quantities 179 173 -3.4% Unit cost The Navy approved a new acquisition program baseline in December 2018. Total quantities comprise 3 development quantities and 170 procurement quantities. Funding and Quantities (fiscal year 2019 dollars in millions Attainment of Product Knowledge As of January 2019 (Block II activities only) Status at Resources and requirements match • Demonstrate all critical technologies are very close to final form, fit and function within a relevant environment • Demonstrate all critical technologies in form, fit and function within a realistic environment • Complete a system-level preliminary design review Product design is stable ● ○ ○ ● ● ● Design Review • Release at least 90 percent of design drawings • Test a system-level integrated prototype Manufacturing processes are mature ○ ○ ○ ○ Production Start • Demonstrate Manufacturing Readiness Level of at least 9, or critical processes are in statistical control • Demonstrate critical processes on a pilot production line • Test a production-representative prototype in its intended environment Current Status Development Start ○ ○ ○ ○ ○ ○ ● Knowledge attained, ○ Knowledge not attained, … Information not available, NA Not applicable Page 109 GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Navy IRST Program Technology Maturity, Design Stability, and Production Readiness IRST Block II entered low-rate production in December 2018 with mature critical technologies, but an unstable design and undemonstrated critical manufacturing processes. The IRST program’s only critical technology is its “passive ranging algorithm” tracking software. The program expects this software to track targets better than legacy system software. In a July 2017 technology maturation assessment, an independent review panel assessed the software as fully functional in an operational environment. Although in production, IRST does not yet have a stable design. At the time of the program’s November 2018 critical design review, the contractor had released about 77 percent of Block II design drawings—a level that falls short of the 90 percent recommended by best practices as a mark of design stability. As of January 2019, the program had increased its drawing releases to about 82 percent of planned total. The program also has yet to test a system-level integrated prototype. According to program officials, they have tested hardware and software in configurations similar to those planned for the Block II design—but short of a prototype—to validate design efforts for a future Block II system-level prototype. Our prior work has shown that carrying a stable design into production reduces the risk of costly design changes to already-produced systems. Common Name: IRST exceeds 10 percent of the total production quantity, the program must provide a rationale for these quantities in a report to Congress. For IRST, the Navy’s rationale hinges on establishing an initial production base and allowing an orderly increase in production leading to full-rate production. Other Program Issues A May 2014 report from DOD’s Inspector General found the the Navy had inappropriately planned to use IRST procurement funds to fund Block II development activities and recommended the funds be reprogrammed. Following the reprogramming, the Navy reclassified IRST as a major defense acquisition program based on its adjusted total research, development, technology, and evaluation costs. Program Office Comments We provided a draft of this assessment to the program office for review and comment. The program had no comments on the assessment. The program also entered production without demonstrating critical manufacturing processes both on a pilot production line and within statistical process controls—inconsistent with best practices. Program officials said they will review or modify Block I manufacturing processes, which they consider mature, that are used for building Block II prototype hardware, with deliveries starting in late fiscal year 2019. The program also plans for IRST development and production to overlap in order to achieve the accelerated initial operational capability. As we have found in our evaluations of numerous other DOD programs, this type of concurrent approach increases risk of program cost increases and schedule delays. Officials reported that in December 2018, the program awarded an undefinitized contract action, under fixedprice incentive firm target terms, for low-rate initial production of six Block II units. According to the program office, this contract action supports F/A-18 operations by accelerating IRST’s initial operational capability by over 2 years, from January 2024 to September 2021. The program plans to award two additional contracts—one in fiscal year 2020 for 12 units and another in fiscal year 2021 for 25 units—for low-rate initial production. Consequently, the program intends to acquire over 28 percent (43 of 152) total production quantities through low-rate initial production. If a program’s low-rate initial production quantity Page 110 GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Navy Common Name: JPALS Joint Precision Approach and Landing System (JPALS) JPALS is a program to develop a Global Positioning System (GPS)based aircraft landing system that will allow aircraft such as the F-35 Lightning II and the MQ-25 Unmanned Aircraft System to operate from aircraft carriers and amphibious assault ships. With JPALS, the Navy intends to provide a reliable, sea-based precision approach and landing capability that is effective in adverse weather conditions. JPALS functionality is primarily software-based, although it will also feature off-the-shelf hardware such as antennas and racks. Program Essentials Milestone decision authority: Navy Program office: Lexington Park, MD Prime contractor: Raytheon Contract type: CPIF (development) Software development approach: Mixed Next major milestone: Low-rate initial production (March 2019) Program Performance (fiscal year 2019 dollars in millions) First full estimate (07/2008) Latest (06/2018) Percentage change Development $886.90 $1,494.70 +68.5% Procurement $238.80 $415.10 +73.9% $30.63 $58.11 +89.7% Acquisition cycle time (months) 77 146 +89.6% Total quantities 37 33 -10.8% Unit cost Total quantities comprise 10 development quantities and 23 procurement quantities. Funding and Quantities (fiscal year 2019 dollars in millions) Attainment of Product Knowledge As of January 2019 Status at Resources and requirements match • Demonstrate all critical technologies are very close to final form, fit and function within a relevant environment • Demonstrate all critical technologies in form, fit and function within a realistic environment • Complete a system-level preliminary design review Product design is stable ● ○ ○ ● ○ ● Design Review • Release at least 90 percent of design drawings • Test a system-level integrated prototype Manufacturing processes are mature ○ ○ ● ● Production Start • Demonstrate Manufacturing Readiness Level of at least 9, or critical processes are in statistical control • Demonstrate critical processes on a pilot production line • Test a production-representative prototype in its intended environment Current Status Development Start NA NA NA NA NA NA ● Knowledge attained, ○ Knowledge not attained, … Information not available, NA Not applicable Page 111 GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Navy JPALS Program Technology Maturity and Design Stability Both of JPALS’s two critical technologies are approaching maturity, and the program has released 100 percent of its design drawings, which corresponds with a stable design. However, as the program continues to mature its critical technologies through testing, the program may need to revise its design drawings to accommodate these changes, which could compromise design stability. JPALS originally entered system development in July 2008 and held a critical design review (CDR) in December 2010, but the design later proved unstable. The program proceeded with development and accepted delivery of eight prototypes. As JPALS approached its original production decision in 2013, other military departments and civilian agencies decided to continue using their current landing systems instead of investing their resources in JPALS. As a result, the Navy restructured the JPALS program from seven increments to one. Because of the restructure, the Navy revised its schedule and milestones and conducted a new systemlevel preliminary design review in March 2016, a new development start in June 2016, and a new CDR in May 2017. Because the program repeated these three events, our attainment of product knowledge table assesses the program’s knowledge at its original development start and original CDR events, which formed the basis for the program’s original business case. This methodology is consistent with how we have previously assessed JPALS and other programs that have repeated key program events. In June 2016, Navy leadership authorized the restructured JPALS program to enter the engineering and manufacturing development phase. The program office reported that it awarded a contract in September 2016 to upgrade the eight original prototypes, as well as to procure two additional prototypes for developmental testing. The contractor delivered these prototypes during the second quarter, of fiscal year 2018, according to program officials. Both the new and upgraded prototypes are intended to be production representative. According to program officials, these prototypes will allow the program to demonstrate the JPALS critical technologies in a realistic environment, which the program plans to do prior to entering production. Common Name: JPALS shortening the procurement schedule. As a result, the program updated its baseline in March 2018 to reflect that it would not execute a full-rate production decision. Program officials reported that they completed an operational test readiness review in April 2018 and attained early operational capability with their prototypes in June 2018 to support F-35 Lightning II operational testing. For fiscal year 2018, program officials reported a combined total of 78 aircraft approaches for integrated and operational testing. They also stated the program successfully completed its production readiness review in December 2018 ahead of the planned March 2019 low-rate initial production decision. Other Program Issues Because JPALS is GPS-based, it will need to be compliant with with any updates to GPS systems, such as the integration of M-code, a new military GPS signal designed to further improve anti-jamming and secure access to GPS signals for military users. JPALS program officials stated they contracted for a trade study to determine future M-code integration and implementation options. Program officials expect the study to be delivered in early 2019. Program Office Comments We provided a draft of this assessment to the program office for review and comment. The program office stated that JPALS is part of a family of systems that provide capability to naval aviation and its partners. According to the program office, in fiscal year 2018 and early fiscal year 2019, JPALS successfully deployed on the amphibious ships LHD 1 and LHD 2, supporting F-35 operational deployments. The program stated that, in fiscal year 2018, it received approval to compress the JPALS production schedule from five to four lots, which it anticipated would save costs over the program lifetime and accelerate deployment. The program also stated that JPALS entered the production and deployment phase on March 25, 2019, which it said provides authority to award a low-rate initial production contract for 23 JPALS quantities. The program said that it expects to complete some integrated testing and an operational assessment in April 2019 in support of JPALS’s integrated operational test and evaluation phase. Additionally, the program stated that restructured and accelerated requirements drove changes to design drawings during JPALS development. Production Readiness JPALS does not have any critical manufacturing processes, according to the program, because the hardware is primarily off-the-shelf. In December 2017, the Navy approved the JPALS program to procure the entirety of its 23 production units through low-rate initial production because it anticipates cost savings through Page 112 GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Navy Common Name: LCS Packages Littoral Combat Ship-Mission Modules (LCS Packages) The Navy’s LCS packages—weapons, helicopters, boats, and sensors launched and recovered from LCS seaframes—will provide mine countermeasure (MCM), surface warfare (SUW), and antisubmarine warfare (ASW) capabilities to the LCS seaframe. The Navy originally planned to swap mission packages among LCS, but has since decided that each LCS will be designated with a specific mission package. The Navy is currently delivering specific systems as they become available, rather than complete packages. We assessed the current capability of the delivered systems against the threshold requirements that define the baseline capabilities for the complete mission package. Program Essentials Milestone decision authority: Navy Program office: Washington Navy Yard, DC Prime contractor: Northrop Grumman Contract types: FFP/FPI (production) Software development approach: Incremental Next major milestone: Surface warfare package initial capability with surface-tosurface missile (second quarter, fiscal year 2019) Funding and Quantities (fiscal year 2019 dollars in millions) Program Performance (fiscal year 2019 dollars in millions) First full estimate (08/2007) Latest (09/2018) Percentage change Development NA $2,724.90 NA Procurement $3,649.49 $3,771.82 +3.4% Unit cost NA $133.43 NA Acquisition cycle time (months) NA NA NA Total quantities 64 49 -23.4% The Navy approved an acquisition program baseline for LCS Packages in September 2018. Total quantities comprise 5 development quantities and 44 procurement quantities. Attainment of Product Knowledge As of January 2019 Status at Resources and requirements match • Demonstrate all critical technologies are very close to final form, fit and function within a relevant environment • Demonstrate all critical technologies in form, fit and function within a realistic environment • Complete a system-level preliminary design review Product design is stable ○ ○ ○ ○ ○ ● Design Review • Release at least 90 percent of design drawings ⋯ ● ● NA NA NA NA NA NA ○ • Test a system-level integrated prototype Manufacturing processes are mature Production Start • Demonstrate Manufacturing Readiness Level of at least 9, or critical processes are in statistical control • Demonstrate critical processes on a pilot production line • Test a production-representative prototype in its intended environment Current Status Development Start ● Knowledge attained, ○ Knowledge not attained, … Information not available, NA Not applicable Page 113 GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Navy LCS Packages Program Mine Countermeasure The Navy accepted systems that comprise about seven MCM packages prior to demonstrating that they meet threshold MCM requirements. It continues to fund numerous retrofits and configuration changes due to reliability and performance failures. Rather than complete packages, the Navy is now delivering MCM systems individually. Navy officials stated that they no longer plan to conduct testing to demonstrate that the packages meet MCM requirements because it believes that as long as each system meets its specific requirements, collectively it will meet overarching MCM requirements. The Navy has yet to demonstrate that meeting specific system requirements satisfies overarching MCM requirements. The Navy projects that it will not meet its requirement for Deep Volume Focused Mine Hunting, which involves the clearing of all moored mines. Under the current requirement, the package must clear a certain number of squared nautical miles each day. Officials said that the original requirement was optimistic and that the technology has not performed as needed to meet it, despite previous claims by the Navy that the systems were fully mature. The program office is currently working with stakeholders to change the requirement and align it with operator needs. Officials said they are seeking approval for an updated requirement in March 2019. Following our 2018 assessment, the program delayed initial operational capability (IOC) for the MCM package from 2020 to 2022. The Navy plans to start fielding eight LCS with the MCM package between 2020 and 2022. These ships will deploy with only selected MCM systems that do not provide the package’s full capability. Officials stated that because of the variability within MCM missions, they can conduct some operations with the MCM systems that have already achieved IOC. Navy officials are also considering fielding these systems on other ships apart from LCS. To realize these plans, the Navy plans to procure 24 MCM packages, but will use only 15 of them on LCS. Surface Warfare Through operational testing completed in fiscal year 2015, the Navy met its incremental performance requirements for the SUW package and is now fielding it. To meet full performance requirements, the Navy is adding the Army’s Longbow Hellfire missile to the package, which the Navy has adapted and is currently testing for use within a maritime environment. According to officials, test ship availability issues delayed IOC for the missile from the second quarter of fiscal year 2018 to the second quarter of fiscal year 2019. The Navy completed testing on one LCS variant (Freedom). The Page 114 Common Name: LCS Packages Navy plans to test the missile on the other variant (Independence) in 2020. Antisubmarine Warfare Following our 2018 assessment, the program delayed IOC for the package by 1 year to the end of fiscal year 2020 because of budget limitations, according to program officials. To meet this new date, the Navy accepted delivery of the Escort Mission Module test article, one of the package’s main systems, and plans to begin testing in the fourth quarter of fiscal year 2019 on board LCS 3. To achieve this, the Navy needs to make alterations on LCS 3 to outfit it with the ASW package. However, the program cannot begin the alterations until LCS 3 completes ongoing maintenance and subsequent sea trials, according to ship officials. The ASW alterations are scheduled to begin in April 2019, a 2month delay from the previous start date. At this point, the Navy has little to no schedule margin remaining between when it expects to complete the LCS 3 alterations and begin ASW package testing. Any further delays will jeopardize the Navy’s ability to meet its planned ASW package testing and IOC dates. Other Program Issues The Navy recently reduced its procurement quantities of LCS mission packages from 64 to 44, trimming the planned 24 SUW and 16 ASW packages to 10 each. This is in line with reductions to the total number of LCS from 52 to 32. Additionally, the Navy recently decided that it will semi-permanently install mission modules on LCS seaframes—an approach that it expects will add simplicity and stability to the program but will reduce the mission flexibility that the Navy previously intended the program to have. Program Office Comments We provided a draft of this assessment to the program office for review and comment. The program office provided technical comments, which we incorporated where appropriate. Officials said that the Navy is delivering LCS capability and deploying mission package systems that have attained IOC. They stated that the Navy successfully completed surface-to-surface missile module testing 2 months early, that the missile will achieve IOC as scheduled in the second quarter of fiscal year 2019, and that it will deploy on LCS 7 in the fourth quarter of fiscal year 2019. Officials also said that the Navy took delivery of the ASW mission package in November 2019, plans for it to embark on LCS 3, and to begin formal testing of it in fiscal year 2019 to achieve the scheduled fiscal year 2020 IOC. According to the program, the Navy has established IOC for three MCM mission package airborne systems. The program stated that it expects the full package’s IOC to occur in fiscal year 2022 due to funding limitations for the MCM unmanned surface vehicle. GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Navy Common Name: MQ-4C Triton MQ-4C Triton Unmanned Aircraft System (MQ-4C Triton) The Navy's MQ-4C Triton is an unmanned aircraft system, based on the design of the Air Force's RQ-4B Global Hawk air vehicle. It will be operated from five land-based sites worldwide and provide the Navy with persistent maritime intelligence, surveillance, and reconnaissance (ISR) data collection and dissemination capability. The Triton is integral to a family of maritime patrol and reconnaissance systems and part of the Navy's plan to recapitalize its airborne ISR assets by the end of the decade. Program Essentials Milestone decision authority: Navy Program office: Patuxent River, MD Prime contractor: Northrop Grumman Contract types: Cost-sharing (development) FPI (low-rate initial production) FFP (low-rate initial production spares) Software development approach: Mixed Next major milestone: Baseline early capability ( July to September 2019) Funding and Quantities (fiscal year 2019 dollars in millions) Program Performance (fiscal year 2019 dollars in millions) First full estimate (02/2009) Latest (10/2018) Percentage change Development $3,517.70 $5,668.00 +61.1% Procurement $10,439.20 $9.760.90 -6.5% $205.50 $225.24 +9.6% Acquisition cycle time (months) 92 156 +69.6% Total quantities 70 70 0.0% Unit cost Total quantities comprise 5 development quantities and 65 procurement quantities. Since last year’s report, one unit shifted from procurement to development as part of an agreement with the contractor to share cost growth. Attainment of Product Knowledge As of January 2019 Status at Resources and requirements match • Demonstrate all critical technologies are very close to final form, fit and function within a relevant environment • Demonstrate all critical technologies in form, fit and function within a realistic environment • Complete a system-level preliminary design review Product design is stable ● ○ ○ NA NA ● Design Review • Release at least 90 percent of design drawings • Test a system-level integrated prototype Manufacturing processes are mature ○ ○ ● ● Production Start • Demonstrate Manufacturing Readiness Level of at least 9, or critical processes are in statistical control • Demonstrate critical processes on a pilot production line • Test a production-representative prototype in its intended environment Current Status Development Start ○ ● ○ ● ● ● ● Knowledge attained, ○ Knowledge not attained, … Information not available, NA Not applicable Page 115 GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Navy MQ-4C Triton Program Technology Maturity, Design Stability, and Production Readiness The MQ-4C Triton program currently relies on no critical technologies and has stabilized its system design as it continues low-rate initial production. Program officials acknowledged that although the wing contractor continues to improve its manufacturing processes, production remains out of statistical control, which poses continued risk to the Triton’s production schedule, quality, and cost. For example, program officials said the contractor reduced defects by 68 percent between the wings manufactured for the last system development test article—the sixth aircraft to be produced—and those manufactured for the tenth aircraft. In addition, the 10th and 11th sets of wings, which are the most recently produced, were each delivered 2 to 4 weeks ahead of schedule. Program officials emphasized that continued improvement will depend on the contractor’s ability to successfully implement corrective actions, such as updating engineering processes and establishing a dedicated team of mechanics to address specific quality issues. Common Name: MQ-4C Triton Management System software from the contractor, as originally planned. This system, according to program officials, provides health and maintenance data on aircraft subsystems, including real-time notification of part failures as they occur during flight. While program officials stated the Navy is still designing and testing the software, the program plans for the software to be operational prior to initial operational capability in April 2021. Program Office Comments We provided a draft assessment to the program office for review and comment. The program office provided technical comments, which we incorporated where appropriate. Program officials reported the start of full-rate production has been extended an additional 2 months, until July 2021. In addition, these problems have increased the program’s total number of design drawings by nearly 14 percent since receiving Milestone C approval in September 2016. The program plans to increase the quantity of aircraft acquired during low-rate initial production from 15 to 18 to maintain operational availiability requirements. However, this strategy also increases the quantity of aircraft that could require design changes due to testing discoveries as the Triton completes developmental testing and begins operational testing. Other Program Issues Program officials reported they potentially face further delays to the start of full-rate production after an aviation mishap—an unplanned event or series of events that results in damage to property or injury to personnel—occurred on September 12, 2018, during early operational testing. While program officials report that the Navy has completed an internal investigation that did not detect any system design deficiencies, the program paused flight test of all aircraft for 3 months out of an abundance of caution while the internal investigation was underway. The program also made updates to the training materials and operation procedures to address findings from their investigation. Program officials reported that the Triton returned to flight during the week of December 17, 2018, and early operational testing resumed on January 23, 2019. Program officials also told us that they have taken over configuration and development of the Triton’s Logistics Page 116 GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Navy Common Name: MQ-25 Stingray MQ-25 Unmanned Aircraft System (MQ-25 Stingray) The Navy’s MQ-25 is a catapult-launched unmanned aircraft system that will operate from aircraft carriers. The Navy expects MQ-25 to provide a refueling capability for the carrier air wing and the intelligence, surveillance, and reconnaissance capabilities needed to identify and report on surface targets, such as ships. The system is made up of an aircraft segment, a control station segment, and a carrier modification segment. The aircraft development represents about 90 percent of the Navy’s planned investment in the MQ-25 system as a whole over the next 5 years. Program Essentials Milestone decision authority: Navy Program office: Patuxent River, MD Prime contractor: Boeing Contract type: FPI (development) Software development approach: Mixed Next major milestone: Design review (December 2019) Program Performance (fiscal year 2019 dollars in millions) First full estimate (08/2018) Latest (08/2018) Percentage change Development $3,548.60 $3,548.60 0.00% Procurement $8,915.00 $8,915.00 0.00% $168.85 $168.85 0.00% Acquisition cycle time (months) 72 72 0.00% Total quantities 76 76 0.00% Unit cost The Navy approved an acquisition program baseline for the MQ-25 Stingray program in August 2018. Total quantities comprise 4 development quantities and 72 procurement quantities. Funding and Quantities (fiscal year 2019 dollars in millions) Attainment of Product Knowledge As of January 2019 Status at Resources and requirements match • Demonstrate all critical technologies are very close to final form, fit and function within a relevant environment • Demonstrate all critical technologies in form, fit and function within a realistic environment • Complete a system-level preliminary design review Product design is stable ● ○ ○ ● ○ ○ Design Review • Release at least 90 percent of design drawings • Test a system-level integrated prototype Manufacturing processes are mature NA NA NA NA Production Start • Demonstrate Manufacturing Readiness Level of at least 9, or critical processes are in statistical control • Demonstrate critical processes on a pilot production line • Test a production-representative prototype in its intended environment Current Status Development Start NA NA NA NA NA NA ● Knowledge attained, ○ Knowledge not attained, … Information not available, NA Not applicable Page 117 GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Navy MQ-25 Stingray Program Technology Maturity and Design Stability The Navy identified no critical technologies for MQ-25. The Navy approved the MQ-25 technology readiness assessment in June 2018 as part of the program’s development start decision. The MQ-25 program has other critical technologies not resident in the MQ-25 aircraft. The Navy is developing those two critical technologies, which are currently approaching maturity, under its Joint Precision Approach and Landing System program. Our Attainment of Product Knowledge table accounts for these technologies. The program has initiated system design activities ahead of a planned review in December 2019. The Navy did not hold a preliminary design review for MQ-25 prior to the start of development—an approach inconsistent with best practices. Instead, the program has relied on preliminary design knowledge attained under the previous Unmanned Carrier-Launched Airborne Surveillance and Strike (UCLASS) program to inform MQ-25 development. Nonetheless, the Navy has set different performance requirements for MQ-25 as compared to UCLASS, which necessitates design differences between the two aircraft. In an effort to accelerate acquisition and fielding, the Navy designated MQ-25 a Maritime Accelerated Acquisition program. This designation is intended to enable the MQ-25 program to deliver capabilities more rapidly than it could under the traditional DOD acquisition process. The Navy emphasized schedule during the source selection process, and, according to program officials, Boeing presented a schedule that would allow the program to achieve initial operational capability 2 years earlier than originally planned. Program officials stated that, as part of this acquisition approach, they plan to hold periodic design reviews as MQ-25 subsystems are completed. The reviews will conclude with a final system design review in December 2019, at which point the program expects to have completed its final design. Common Name: MQ-25 Stingray building the aircraft. Until the program develops the knowledge that its critical manufacturing processes are in statistical control, there is increased risk that the design may not be producible at the program’s cost, schedule, and quality targets. Best practices call for programs to attain this knowledge prior to entering production. Other Program Issues Officials reported that in August 2018, the Navy awarded a fixed price incentive development contract with a ceiling price of $805 million to Boeing, which includes four MQ-25 aircraft. Prior to the award, the Navy estimated the total development cost at $3.8 billion, and the Office of the Secretary of Defense (OSD) estimated its cost at $5.1 billion. Based on the disparity between the development contract’s ceiling price and the DOD cost estimates, OSD cost estimators emphasized the importance of avoiding requirements changes that might lead to the renegotiation of the development contract’s terms, including the ceiling price. Program officials stated the development contract includes options for up to three additional test aircraft, but does not include options for any of the 72 planned production aircraft. Officials stated that while they do not plan to compete the production contract, Boeing will provide cost, technical, and programmatic data during the development phase, which is intended to help inform the Navy’s negotiation of a fair and reasonable price under the production contract. Program Office Comments We provided a draft of this assessment to the program office for review and comment. The program office provided technical comments, which we incorporated where appropriate. Production Readiness The Navy plans to authorize MQ-25’s entry into low-rate initial production in October 2023. Boeing plans to use a commercially demonstrated approach to manufacture the aircraft in modules, which program officials say will be faster and more efficient than a typical assembly line. In this approach, suppliers will use detailed models to produce the various aircraft modules, including predrilling holes. The suppliers will then send the completed modules to Boeing to assemble. Under MQ-25’s approach, the program could find that critical manufacturing processes reside with the suppliers responsible for module fabrication. According to program officials, Boeing intends to conduct manufacturing readiness level assessments when Page 118 GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Navy, Common Name: NGJ Mid-Band Next Generation Jammer Mid-Band (NGJ Mid-Band) The Navy’s Next Generation Jammer (NGJ) is an external jamming pod system that will be integrated on EA-18G Growler aircraft. It will augment, then replace, the ALQ-99 jamming system and provide enhanced airborne electronic attack capabilities to disrupt adversaries’ use of the electromagnetic spectrum for radar detection, among other purposes. The Navy plans to field the system that jams mid-band radio frequencies in 2022. The Navy has a separate program for low-band frequencies and will roll out a high-band frequencies program at a later date. We assessed the Mid-Band program. Program Essentials Milestone decision authority: Navy Program office: Patuxent River, MD Prime contractors: Raytheon; Boeing Contract type: CPIF (development— Raytheon) (development and integration—Boeing) Software development approach: Mixed Next major milestone: Low-rate initial production (August 2020) Program Performance (fiscal year 2019 dollars in millions) First full estimate (04/2016) Latest (06/2018) Percentage change Development $3,629.80 $4,064.20 +12.0% Procurement $4,206.20 $4,062.3 -3.4% $58.10 $60.25 +3.7% 98 110 +12.2% 135 135 0.00% Unit cost Acquisition cycle time (months) Total quantities Total quantities comprise 7 development quantities and 128 procurement quantities. Funding and Quantities (fiscal year 2019 dollars in millions) Attainment of Product Knowledge As of January 2019 Status at Resources and requirements match • Demonstrate all critical technologies are very close to final form, fit and function within a relevant environment • Demonstrate all critical technologies in form, fit and function within a realistic environment • Complete a system-level preliminary design review Product design is stable ● ○ ● ● ○ ● Design Review • Release at least 90 percent of design drawings • Test a system-level integrated prototype Manufacturing processes are mature ● ○ ● ○ Production Start • Demonstrate Manufacturing Readiness Level of at least 9, or critical processes are in statistical control • Demonstrate critical processes on a pilot production line • Test a production-representative prototype in its intended environment Current Status Development Start NA NA NA NA NA NA ● Knowledge attained, ○ Knowledge not attained, … Information not available, NA Not applicable Page 119 GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Navy, NGJ Mid-Band Program Technology Maturity and Design Stability Nearly 3 years after development start, the NGJ MidBand program’s seven critical technologies continue to approach maturity—an approach inconsistent with best practices. The program also reports that it has released all of its planned design drawings following a redesign of the jamming pod to address structural issues. However, until the program fully matures its critical technologies—by demonstrating each in a final form, fit, and function within a realistic environment—existing design drawings face risk of change. Further, the program has yet to test a system-level integrated prototype, which best practices hold as a key criterion for achieving design stability. The program entered system development in April 2016 with its critical technologies approaching maturity. These technologies include two separate arrays—each with different transmit/receive modules, circulators, and apertures—as well as a power generation system. The program plans to have its critical technologies fully mature, integrated, and in testing in January 2020. The NGJ Mid-Band program discovered design deficiencies with the jamming pod structure at its April 2017 critical design review, which caused a 1-year schedule delay and contributed to an over $400 million increase in the program’s development cost. The program identified several other deficiencies in modeling, assumptions, and methodologies regarding the structural analysis and design of the pod structure, which required redesign. According to the program office, the pod redesign did not affect the program’s critical technologies, subsystems, or software. As of September 2018, the contractor had released 100 percent of the design drawings. Program officials said that they considered the design of the pod structure to be stable, although the contractor continues to make minor changes to it. In addition, the program office has yet to test a system-level integrated prototype of the jamming pod, which runs counter to the GAO-identified best practice for demonstrating design stability. The program office plans to begin testing the redesigned pod on an EA-18G in January 2020. Production Readiness The program office plans to demonstrate its critical manufacturing processes prior to the start of production in August 2020, which would be an approach consistent with best practices. However, the program office does not plan to test a production-representative prototype or complete system-level developmental testing (which includes demonstrating the full functionality of the system) until 7 and 15 months, respectively, after production starts. DOD policy allows some concurrency between developmental testing and initial production, but we have previously found that starting production Page 120 Common Name: NGJ Mid-Band before demonstrating that a system will work as intended increases the risk of deficiencies that require substantial and costly design changes. Program officials told us that they plan to mitigate the risk associated with the concurrency between developmental testing and initial production to what they consider to be an acceptable level by gathering extensive data about pod performance in specialized ground test chambers and through flight testing engineering development models. Further, according to program officials, the delay caused by the pod redesign will allow the program to conduct additional ground testing before committing to production. In addition, the delay associated with the pod redesign introduced a potential production gap in between the last engineering developmental model and the first production system. The program added three additional system demonstration test articles to mitigate this gap and verify that the system can meet all its performance requirements prior to the start of operational testing in January 2022. Other Program Issues The Under Secretary of Defense for Acquisition, Technology, and Logistics approved NGJ Mid-Band as the first program in the “Skunk Works” pilot in September 2015, which aims to streamline processes in order to deliver capabilities on time and within budget. NGJ Mid-Band officials said that the pilot helped reduce the length of time spent on the decision making process by providing officials direct access to management and allowing them to focus on execution. Nonetheless, the program has still exceeded original estimates for development cost, unit cost, and schedule. Program Office Comments We provided a draft of this assessment to the program office for review and comment. The program office provided technical comments, which we incorporated where appropriate. According to program officials, as of January 2019, all subsystems that contain critical technologies have been manufactured and assembled and are undergoing subsystem-level tests. Program officials reported that this progress has reduced program risk to an acceptable level for system development. Additionally, program officials told us that they plan to test a system-level integrated prototype of the pod on an EA-18G in a sound- and electromagneticabsorbent chamber beginning in the fourth quarter of fiscal year 2019. GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Navy, Common Name: OASuW Inc 1 Offensive Anti-Surface Warfare Increment 1 (OASuW Inc 1) The Navy’s OASuW Inc 1 program plans to develop an air-launched, long-range, anti-surface warfare missile to address an urgent operational need. The program is using an accelerated acquisition approach and has leveraged previous technology demonstration efforts. It fielded an early operational capability on Air Force B-1 bombers in 2018 and plans to do so on Navy F/A-18 aircraft in 2019. DOD also plans to develop an additional capability with Increment 2 to address future threats. We assessed Increment 1. Program Essentials Milestone decision authority: Navy Program office: Patuxent River, MD Prime contractor: Lockheed Martin Contract type: CPIF (development) FPIF (low-rate initial production) Software development approach: Incremental approach Next major milestone: Early operational capability with F/A-18 (September 2019) Program Performance (fiscal year 2019 dollars in millions) First full estimate (03/2016) Latest (02/2019) Percentage change Development $1,263.30 $1,466.80 +16.1% Procurement $314.30 $1,319.30 +319.8% $12.72 $7.14 -43.8% 42 42 0.00% 124 390 +214.5% Unit cost Acquisition cycle time (months) Total quantities We calculated acquisition cycle time using the program’s early operational capability with F/A-18 as its initial operational capability. Total quantities comprise 16 development quantities and 374 procurement quantities. Funding and Quantities (fiscal year 2019 dollars in millions) Attainment of Product Knowledge As of January 2019 Status at Current Status Development Start Resources and requirements match • Demonstrate all critical technologies are very close to final form, fit and function within a relevant environment • Demonstrate all critical technologies in form, fit and function within a realistic environment • Complete a system-level preliminary design review Product design is stable ○ ○ ● ● ● ● Design Review • Release at least 90 percent of design drawings • Test a system-level integrated prototype Manufacturing processes are mature ● ○ ● ● Production Start • Demonstrate Manufacturing Readiness Level of at least 9, ○ ○ • Demonstrate critical processes on a pilot production line NA NA • Test a production-representative prototype in its intended ○ ● or critical processes are in statistical control environment ● Knowledge attained, ○ Knowledge not attained, … Information not available, NA Not applicable Page 121 GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Navy, OASuW Inc 1 Program Technology Maturity, Design Stability, and Production Readiness The OASuW Inc 1 program has mature technologies, a stable design, and production practices approaching maturity. The Navy chose to move forward earlier in the program without the technology, design, or production knowledge recommended by best practices. In 2016, the program office acknowledged the risk its strategy posed, but pointed to an urgent maritime warfighting need as well as steps taken to manage the risk, such as leveraging the Air Force’s Joint Air-to-Surface Standoff Missile-Extended Range (JASSM-ER) airframe and production facilities. According to the program office, the Navy assessed OASuW’s manufacturing processes as mature prior to production. However, the program’s last formal assessment in March 2017 indicated its processes were only approaching maturity. The program’s top priority has been to produce missiles to field an early operational capability on the B-1 and F/A-18 aircraft. The program delivered the last missiles needed to achieve that capability on the B-1 later than planned in 2018. Consequently, the B-1 achieved early operational capability in December 2018—3 months later than the program’s goal, but 9 months earlier than required. The program is now planning future changes to the missile to improve performance, counter evolving threats, and address parts obsolesence. Some changes will involve hardware updates that are currently in development; others will leverage upgrades from the JASSM-ER program. For example, according to program officials, they plan to incorporate an enhanced weapon data link. They expect to begin incorporating this data link into missiles starting with the fourth OASuW production lot, which the contractor is scheduled to begin in the third quarter of fiscal year 2020. The program office does not plan to upgrade already fielded missiles with this link. Other Program Issues The OASuW program relies on a combination of missile tests and modeling and simulation to demonstrate and characterize the system’s performance. One key modeling and simulation facility, however, was not yet operational as of November 2018. According to program officials, the program uses modeling and simulation in part because replicating the operational environment is challenging. The program completed six flight tests in 2017 and 2018, among other activities to demonstrate missile performance. Navy officials stated that, because it did not have all the anticipated modelling and simulation capabilities available before fielding the OASuW early operational capability on the B-1, the program did not collect as much information as planned about the weapon’s performance in a wide range of conditions. Officials expect the new modeling Page 122 Common Name: OASuW Inc 1 and simulation facility will be available to support testing for the F/A-18 early operational capability. Since the program’s 2016 development start, DOD has significantly increased OASuW Inc 1 quantities due to Increment 2 delays and evolving threats. Most recently, in February 2019, DOD increased planned procurement quantities from 110 to 374. According to program officials, the increases will provide sufficient quantities for mission needs until a future capability becomes available. Officials stated the Navy is examining options for addressing future threats and Increment 2 by building on its Next Generation Land Attack Weapon Analysis of Alternatives. The Navy plans to complete its OASuW-related analysis in 2019. The program office expects OASuW Increment 2 to achieve initial operational capability as early as fiscal year 2028. Program Office Comments We provided a draft of this assessment to the program office for review and comment. The program office provided technical comments, which we incorporated where appropriate. The program office stated that OASuW Inc 1 is an accelerated acquisition program intended to fill an urgent, unmet maritime warfighting requirement where schedule is a priority. The program office said that a small, integrated team of government and industry experts manages the program, including the risks it faces related to concurrent weapon system development, test, and production. It also stated that monthly executive steering boards chaired by the Assistant Secretary of the Navy for Research, Development, and Acquisition provide oversight. The program office reiterated its position that the OASuW’s production processes are mature. It also stated that OaSuW successfully completed both a production readiness review and physical configuration audit, and the contractor has delivered 10 missiles to date. Additionally, the program office said that it met the early operational capability date for fielding OASuW on the B1 bomber aircraft, and the program is ahead of schedule for fielding OASuW Inc 1 on F/A-18 fighter aircraft no later than the fourth quarter of fiscal year 2019. According to the program office, the Navy intends to procure OaSuW quantities based on the available funding, which may cause purchased quantities to differ from current projections. GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Navy, Common Name: SSBN 826 SSBN 826 Columbia Class Ballistic Missile Submarine (SSBN 826) The Navy's Columbia class (SSBN 826) will replace its current fleet of Ohio class ballistic missile submarines, which the Navy plans to retire starting in 2027. The submarine will serve as a sea-based, strategic nuclear deterrent that will remain in service through 2080. According to the Navy’s current acquisition plan, the lead ship will make its first patrol in October 2030. Program Essentials Milestone decision authority: Under Secretary of Defense for Acquisition and Sustainment Program office: Washington Navy Yard, DC Prime contractor: General Dynamics Electric Boat Contract type: Combination of CPIF and CPFF (design and development) Software development approach: Mixed Next major milestone: Design review (April 2020) Funding and Quantities (fiscal year 2019 dollars in millions) Program Performance (fiscal year 2019 dollars in millions) First full estimate (01/2017) Latest (06/2018) Percentage change Development $13,068.90 $13,103.00 +0.3% Procurement $90,335.30 $89,932.20 -0.4% $8,629.70 $8,599.85 -0.3% 231 237 +2.6% 12 12 0.00% Unit cost Acquisition cycle time (months) Total quantities Attainment of Product Knowledge As of January 2019 Status at Resources and requirements match • Demonstrate all critical technologies are very close to final form, fit and function within a relevant environment • Demonstrate all critical technologies in form, fit and function within a realistic environment • Complete a system-level preliminary design review Product design is stable ○ ○ ● ○ ○ ● Fabrication Start • Complete basic and functional design to include 100 percent of 3D product modeling Current Status Detail Design Contract Award NA NA ● Knowledge attained, ○ Knowledge not attained, … Information not available, NA Not applicable Page 123 GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Navy, Common Name: SSBN 826 SSBN 826 Program efficiencies with new design software. The shipbuilder increased its design staff by 18 percent in an effort to reach the design goal on schedule. However, the program’s plan for achieving design stability is premised on assumptions about the final form, fit, and function of critical technologies—and how those technologies will perform in a realistic environment—that the program has yet to demonstrate. Technology Maturity The Columbia class program identified two critical technologies—a carbon dioxide removal system and the stern area system, the details of which are classified. The program expects the carbon dioxide removal system to reach full maturity in late 2019, while the stern area system is still immature. In December 2017, we reported that several Columbia class technologies that met GAO’s definition of a critical technology element were not identified by the Navy as critical technologies. Specifically, the Navy did not follow best practices for assessing critical technologies. When we applied these best practices, we identified four additional critical technologies that the Navy excluded. These include the integrated power system, the propulsor/coordinated stern, the common missile compartment (CMC), and the nuclear reactor. Of these, only the nuclear reactor is fully mature as of late 2018. The Navy expects the CMC to reach full maturity in 2019. However, officials reported that in July 2018 the shipbuilder identified significant weld defects in CMC missile tubes from one of three suppliers after the supplier had already delivered seven tubes to the shipyard and installation work had begun, resulting in rework. Officials further report that the shipbuilder found defects affected five additional tubes. Program officials attributed these defects to inexperienced welders and weld inspectors. The Navy estimates that, as of January 2019, the CMC consumed 52 percent of its schedule margin. Should the Navy discover additional CMC deficiencies, the planned construction sequence for the lead submarine will be jeopardized. Further, manufacturing defects have delayed delivery of the integrated power system’s (IPS) first productionrepresentative motor. The Navy plans to recover the motor’s schedule margin by testing it while the supplier updates the motor’s production design. Consequently, any new deficiencies discovered in testing may require the supplier to modify its design, which could delay the lead ship’s IPS motor production schedule. Design Stability The program office plans to complete the basic and functional design prior to the lead submarine’s scheduled construction start, in October 2020. However, Navy officials report the shipbuilder has already begun building sections of the submarine, with 95 percent of the basic and functional design complete—a level slightly below best practices. Further, the Navy has determined that the shipbuilder needs to complete 83 percent of the detail design—the most complex design phases down to the lowest level of the submarine—by October 2020 to meet its cost and schedule goals. Currently, the shipbuilder is behind schedule because it has yet not achieved planned Page 124 Production Readiness By beginning to build sections of the submarine starting in December 2018, the Navy believes that the builder can achieve an aggressive 84-month construction schedule. However, this is 2 years prior to the planned request for fiscal year 2021 authorization to start construction of the lead ship. Other Program Issues In a April 2019 report, we made several recommendations to improve the program’s cost estimate. Specifically, we found that the program’s $115 billion procurement cost estimate is not reliable because its estimate is based on overly optimistic assumptions about the labor hours needed to construct Columbia class submarines and did not include any cost margin in case these assumptions are not met. While the Navy analyzed program cost risks, it did not include enough margin in its estimate for likely cost growth. The Navy plans to update the cost estimate for the lead ship, but it may not complete this update in time for its fiscal year 2021 budget request, which will seek authorization and funding for lead submarine construction. Program Office Comments We provided a draft of this assessment to the program office for review and comment. The program office provided technical comments, which we incorporated where appropriate. The program office stated that it intends to provide needed capabilities on schedule and at an affordable price by committing to stable requirements, achieving high design maturity at the start of construction for the lead submarine, improving manufacturing and construction readiness, and aggressively working to reduce costs. It also said it plans to complete 83 percent of the design by construction start—more than other recent submarine programs. The program also stated that it plans to update its cost estimate in 2019 to inform lead submarine funding. The program noted that the Navy recognizes its supplier base remains a high risk to construction readiness and continues to devote increased oversight on manufacturing issues and readiness assessments. The program said it continues to comply with all Navy, Department of Defense, and statutory requirements for managing critical technologies. GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Navy, Common Name: SSC Ship to Shore Connector Amphibious Craft (SSC) The Navy's SSC is an air-cushioned landing craft intended to transport personnel, weapon systems, equipment, and cargo from amphibious vessels to shore. SSC is the replacement for the Landing Craft, Air Cushion, which is approaching the end of its service life. The SSC is designed to deploy in and from Navy amphibious ships that have well decks, such as the LPD 17 class, and will support assault and nonassault operations. Program Essentials Milestone decision authority: Navy Program office: Washington, DC Prime contractor: Textron Inc. Contract type: FPI (detail design and construction) CPFF (long lead materials and early production) Software development approach: Modified waterfall Next major milestone: Start operational testing (summer 2020) Funding and Quantities (fiscal year 2019 dollars in millions) Program Performance (fiscal year 2019 dollars in millions) First full estimate (07/2012) Latest (06/2018) Percentage change Development $623.20 $586.20 -5.9% Procurement $3,782.20 $4,122.30 +9.0% $60.63 $64.72 +6.7% 135 135 0.0% 73 73 0.00% Unit cost Acquisition cycle time (months) Total quantities Total quantities comprise 1 development quantity and 72 procurement quantities. Attainment of Product Knowledge As of January 2019 Status at Resources and requirements match • Demonstrate all critical technologies are very close to final form, fit and function within a relevant environment • Demonstrate all critical technologies in form, fit and function within a realistic environment • Complete a system-level preliminary design review Product design is stable ● ● ● ● ● ● Design Review • Release at least 90 percent of design drawings • Test a system-level integrated prototype Manufacturing processes are mature ○ ○ ● ● Production Start • Demonstrate Manufacturing Readiness Level of at least 9, or critical processes are in statistical control • Demonstrate critical processes on a pilot production line • Test a production-representative prototype in its intended environment Current Status Development Start ○ ● ○ ○ ● ○ ● Knowledge attained, ○ Knowledge not attained, … Information not available, NA Not applicable Page 125 GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Navy, SSC Program Technology Maturity and Design Stability SSC’s one critical technology, the fire supression system, is mature. The SSC’s design was not stable at its design review in 2014, but the program now reports that it has achieved stability. However, software development—a modified waterfall approach, which includes iterative testing—is not yet finished. Additionally, according to the program office, problems with two components have required design changes and delayed the start of operational testing. The program office stated the gearbox, which had previous problems with premature wear, showed gear slippage during testing. The program is now working on separate engineering solutions for the propulsion and lift elements. This is the third iteration of design changes related to problems with the gearbox. The program expects that once the revised design is finalized and manufactured, the contractor will have to retrofit the craft already in production. In October 2018, during pre-delivery testing, Craft 100—the test and training prototype—lost propulsion and drifted into a bridge, causing damage to one side of the craft. Program officials said the Navy found the root cause was overheating in the propulsion power control module. The prime contractor since developed a heat exchanger and, in November 2018, demonstrated it could significantly lower the module’s internal temperature. The program office said it plans to apply this solution to all craft currently in production. The program office will have the opportunity to demonstrate that the SSC design meets performance requirements through testing when the contractor delivers Craft 100, expected in late fiscal year 2019. The Navy and contractor will test Craft 100 while the contractor is producing the eight other SSC craft. This concurrency increases the risk that the program will discover deficiencies that could require costly design changes and modifications to units in production. Production Readiness The program entered low-rate initial production in May 2015 after demonstrating that all materials, manpower, tooling, and facilities were proven and available to meet the production schedule, as recommended by DOD guidance. However, best practices indicate programs should also demonstrate that manufacturing processes are in statistical control prior to production start, which the program has not done. Critical processes should be repeatable, sustainable, and consistent in producing parts within the quality standards, which provides confidence that the product can be produced within cost, schedule, and quality targets. The aforementioned technical problems have caused delays to the SSC production schedule. Program Page 126 Common Name: SSC officials reported nine SSC craft are currently under contract, and they are uncertain about the production schedule because solutions to the technical problems are pending. However, they expect delivery of Craft 100 in July 2019, a 15-month delay compared to last year’s assessment. The program expects delivery of the second craft several months after Craft 100. Other Program Issues The program office will not verify in realistic operational conditions that it has fully addressed all known deficiencies until operational testing of Craft 100. However, SSC’s operational testing—which cannot take place until the Navy takes delivery of Craft 100—will not begin until summer 2020. The program plans to conclude testing about 2 months before it declares initial capability in August 2020, when the first SSC craft are scheduled to deploy. Should the Navy discover deficiencies during operational testing, it may face a choice to delay initial capability or deliver SSC craft that are operationally ineffective or unsuitable, as it has done in other programs. The program office said it has yet to award its follow-on contract for the 15 additional craft funded in fiscal years 2017, 2018, and 2019. The program delayed negotiations because of prior funding uncertainty, as well as the effort to address the gearbox problems. Program Office Comments We provided a draft of this assessment to the program office for review and comment. The program office provided technical comments, which we incorporated where appropriate. The program office stated that since our assessment, the program has delayed operational testing by 12 months, initial capability by 12 months, and full-rate production by 13 months from the dates shown in our program timeline. It said that the SSC program has made notable progress, but testing of Craft 100 continues to pose challenges, and electrical system stability and command, control, communications, computers and navigation integration challenges must be resolved before it can complete pre-delivery testing. The program said that Craft 100 is scheduled to complete on-water, pre-delivery testing and deliver in summer 2019, with subsequent craft delivering in late 2019. The program also stated that the Navy’s 2020 budget request has reduced fiscal year 2020 craft quantities from eight craft to zero. Additionally, it said that, although one test and training craft and 72 operational craft remain the requirement, the abovementioned change in requested funding shifts SSC's full capability to fiscal year 2032, 3 years later than previously planned. The program stated that Textron has indicated in the past that eight craft per year are necessary to achieve economies of scale, but with zero craft allotted in fiscal year 2020, the program foresees increased risk for growth in acquisition cost, lifecycle cost, and industrial base instability. GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Navy, DR Common Name: T-AO 205 John Lewis Class Fleet Replenishment Oiler (T-AO 205) The John Lewis Class Fleet Replenishment Oiler (T-AO 205) will replace the Navy’s 15 existing Henry J. Kaiser Class Fleet Oilers (T-AO 187), which are nearing the end of their service lives. The primary mission of the oiler is to replenish bulk petroleum products, dry stores and packaged cargo, fleet freight, mail, and personnel to other vessels at sea. The Navy plans to procure a total of 20 ships. Construction of the lead ship began in September 2018. The Navy plans to proceed at a rate of one to two ships per year until 2033. Program Essentials Milestone decision authority: Navy Program office: Washington Navy Yard, DC Prime contractor: General Dynamics National Steel and Shipbuilding Company (NASSCO) Contract type: FPI (detail design and construction) Software development approach: NA Next major milestone: Lead ship delivery (November 2020) Funding and Quantities (fiscal year 2019 dollars in millions) Program Performance (fiscal year 2019 dollars in millions) First full estimate (09/2017) Latest (12/2018) Percentage change Development $71.00 $70.80 -0.3% Procurement $8,907.00 $10.735.90 +20.5% $528.12 $540.34 +2.3% Acquisition cycle time (months) 46 52 +13.0% Total quantities 17 20 +17.6% Unit cost Attainment of Product Knowledge As of January 2019 Status at Resources and requirements match • Demonstrate all critical technologies are very close to final form, fit and function within a relevant environment • Demonstrate all critical technologies in form, fit and function within a realistic environment • Complete a system-level preliminary design review Product design is stable ● ● ○ ● ● ● Fabrication Start • Complete basic and functional design to include 100 percent of 3D product modeling Current Status Detail Design Contract Award ● ● ● Knowledge attained, ○ Knowledge not attained, … Information not available, NA Not applicable Page 127 GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Navy, DR T-AO 205 Program Technology Maturity and Design Stability The Navy has matured all Lewis class critical technologies and stabilized the ships’ design. In 2014, the Navy identified three critical technologies for the Lewis class, all of which involved a new system for transferring cargo at sea. Prior to initiating detail design activities in June 2016, the Navy completed prototype tests of the critical technologies and found that they were fully mature—an approach consistent with shipbuilding best practices. In 2017, the Navy removed one critical technology—the Heavy e-STREAM cargo delivery system—from the Lewis class design. The Navy had intended to use this system to deliver F-35 Lightning II power modules. The Navy subsequently decided to deliver these by air, which precluded any need for the Heavy system. Common Name: T-AO 205 program officials stated that they plan to compete a new contract for the remaining 12 ships using the construction knowledge gained from efforts under the existing contract. Program Office Comments We provided a draft of this assessment to the program office for review and comment. The program office provided technical comments, which we incorporated where appropriate. The program office stated that it continues to follow GAO shipbuilding best practices and has leveraged commercial vessel design practices to minimize risk. The program office also stated that it is currently revising its acquisition baseline to reflect the update in total quantities to 20 ships. In addition, the program office noted that, in fiscal year 2019, it fully funded the third and fourth ships and funded advance procurement for the fifth ship. Lead ship construction began in September 2018 with 95 percent of the ship’s total design effort complete. Program officials stated that this figure meant that 100 percent of the ship’s basic and functional design were by then complete—an approach consistent with best practices. Throughout detail design and now into construction, the Navy has not changed the Lewis class program’s performance requirements. The Navy also leveraged commercial vessel designs to minimize design and construction risks. The Lewis class features a modern double-hull construction, an environmentalbased design standard for commercial tankers, to ensure the ships can dock at ports-of-call. This design was included in the final three Kaiser class oilers. Production Readiness The program office has largely kept to its construction schedule to date for the first ship, but a flooding incident at a NASSCO graving dock in July 2018 has affected the delivery of future ships. The program office stated that this incident has not affected current ship fabrication activities. However, the dock’s unavailability while repairs are planned and implemented has disrupted the contractor’s schedule for future ships. According to the program office, the incident has resulted in some delays to certain delivery dates for ships two through six. Other Program Issues As part of the Navy’s plan to expand the fleet, the Navy concluded that it would need an additional three Lewis class ships. The Navy’s budget request for fiscal year 2019 increased its planned one-ship-per-year buy to two for fiscal years 2019, 2021, and 2023. The Congress provided appropriations for the additional fiscal year 2019 ship in support of the Navy’s request. To account for the additional ships in fiscal years 2019 and 2021, the Navy plans to add two more ships to the low-rate initial production phase. Subsequently, Page 128 GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Navy, Common Name: VH-92A VH-92A Presidential Helicopter Replacement Program (VH-92A) The Navy's VH-92A program provides new helicopters for safe, reliable, and timely transportation in support of the presidential airlift mission. It will replace the current Marine Corps fleet of VH-3D and VH-60N aircraft and supersedes the VH-71 program, which DOD canceled due to cost growth, schedule delays, and performance shortfalls. The VH-92A is expected to provide improved performance, survivability, and state-of-the-art communications capabilities, while offering increased passenger capacity. Program Essentials Milestone decision authority Navy Program office: Patuxent River, MD Prime contractor: Sikorsky Aircraft Corporation, a Lockheed Martin Company Contract type: FPIF (development) FFP (production; planned) Software development approach: Mixed Next major milestone: Low-rate initial production (June 2019) Funding and Quantities (fiscal year 2019 dollars in millions) Program Performance (fiscal year 2019 dollars in millions) First full estimate (04/2014) Latest (06/2018) Percentage change Development $2,802.00 $2,695.20 -3.8% Procurement $2,197.20 $2,140.00 -2.6% $217.35 $210.23 -3.3% Acquisition cycle time (months) 75 78 +4.0% Total quantities 23 23 0.00% Unit cost Total quantities comprise 6 development quantities and 17 procurement quantities. Attainment of Product Knowledge As of January 2019 Status at Resources and requirements match • Demonstrate all critical technologies are very close to final form, fit and function within a relevant environment • Demonstrate all critical technologies in form, fit and function within a realistic environment • Complete a system-level preliminary design review Product design is stable NA NA NA NA ○ ● Design Review • Release at least 90 percent of design drawings • Test a system-level integrated prototype Manufacturing processes are mature ● ○ ● ○ Production Start • Demonstrate Manufacturing Readiness Level of at least 9, or critical processes are in statistical control • Demonstrate critical processes on a pilot production line • Test a production-representative prototype in its intended environment Current Status Development Start NA NA NA NA NA NA ● Knowledge attained, ○ Knowledge not attained, … Information not available, NA Not applicable Page 129 GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Navy, VH-92A Program Technology Maturity and Design Stability The VH-92A program has released all of its planned design drawings for this configuration that relies on no critical technologies. The program has a governmentdesigned mission communications system (MCS), which consists of both government and commercial hardware and software. The program expects this system to provide the VH-92A passengers, pilot, and crew with the ability to conduct simultaneous short- and long-range secure and non-secure voice and data communications. The MCS is not in use in any other aircraft. The Navy plans to test MCS version 2.0 with the first two VH-92A developmental aircraft during an operational assessment scheduled to start in March 2019. MCS version 2.0 has limitations connecting to secure networks, following a change to the network security required protocols. For the operational assessment, the Navy intends to connect to existing networks that do not use the new security protocols. This will allow the operational assessment to proceed, but will limit the scope of testing. The program is working on the problem, which the Navy anticipates will be resolved by December 2019, about 3 months prior to the start of operational testing scheduled for March 2020. Once resolved, the Navy plans to upgrade the production representative aircraft with MCS version 3.0 retrofit kits and use this version for operational testing. Common Name: VH-92A Production Readiness The Navy is using the Federal Aviation Administration’s air-worthiness certification process, including audits and inspections, to demonstrate production readiness. Further, the program plans to demonstrate manufacturing processes on a pilot production line prior to production start. Ahead of production start, the contractor continues to outfit the four production representative aircraft with VH92A’s unique modifications. According to the contractor, despite some parts shortages, it plans to deliver each of these aircraft on or ahead of its contractual delivery date. Program Office Comments We provided a draft of this assessment to the program office for review and comment. The program office provided technical comments, which we incorporated where appropriate. The program office stated that, with regard to prototype testing, it expects to complete testing of a system-level integrated prototype in April 2019. Based on new information provided by the program office, we have updated our attainment of product knowledge table to show that the program met the best practices metric of at least 90 percent of design drawings released to manufacturing at the time of its 2016 critical design review. However, the program did not test a system-level integrated prototype at the time of its critical design review, which is required by best practices to confirm design stability. The program currently identifies two VH-92A performance attributes as high risks to the aircraft’s development. Specifically, the aircraft has yet to demonstrate performance requirements related to (1) the propulsion system and (2) landing zone suitability, which includes a requirement to land on the White House south lawn without causing damage to the lawn. Additionally, the program plans to implement design changes outside the baseline program that will allow for improved visibility from one of the aircraft’s doors. The aforementioned development activities are expected to be completed after the June 2019 production start. This approach introduces the potential for discovery of design deficiencies in testing at the same time that the contractor is manufacturing aircraft—an approach that could require costly rework and retrofits. Page 130 GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Navy Common Name: DDG 51 Flight III DDG 51 Arleigh Burke Class Destroyer, Flight III (DDG 51 Flight III) The Navy’s DDG 51 Flight III destroyer will be a multi-mission ship designed to operate against air, surface, and underwater threats. Compared to existing Flight IIA ships of the same class, the new Flight III ships will provide the fleet with increased ballistic missile and air defense capabilities. Flight III’s changes include replacing the current SPY-1D(V) radar with the Air and Missile Defense Radar (AMDR) program’s SPY-6 radar and upgrading the destroyer’s Aegis combat system. The Navy currently plans to procure 20 Flight III ships. Program Essentials Current Status Milestone decision authority: Navy Program office: Washington, DC Prime contractors: Huntington Ingalls Industries; General Dynamics-Bath Iron Works Contract type: FPI (construction) Software development approach: Mixed Next major milestone: Fabrication start for second Flight III ship (DDG 126) (April 2019) The Navy and the shipbuilders completed Flight III detail design activities in December 2017. As compared to Flight IIA, the Flight III design included considerable changes to the ship’s hull, mechanical, and electrical systems to incorporate the AMDR program’s SPY-6 radar, and changes to restore ship weight and stability safety margins. To reduce technical risk, the Navy plans to field all but one—the SPY-6 radar—of the program’s four mature critical technologies on other ship classes before integration with Flight III. In 2018, however, the Navy identified software-related deficiencies affecting SPY-6 that delayed delivery of a radar array for power and integration testing with the Aegis combat system by at least 1 year. Despite these delays, the Navy plans to complete testing, install the radar on the ship, and activate the combat system for shipboard testing by January 2022. Estimated Program Cost (FY 2019 dollars in millions) Planned Quantities The Navy expects to complete a draft test and evaluation master plan for Flight III by early 2022. The Navy and the Director, Operational Test and Evaluation continue to disagree on whether the use of a self-defense test ship equipped with Aegis and the SPY-6 radar is necessary to validate performance during operational test and evaluation. Huntington Ingalls began construction of the first Flight III ship in May 2018 and plans delivery in fiscal year 2023. Bath Iron Works will begin construction of the second Flight III ship in April 2019. Congress provided the Navy with multiyear procurement authority, which allows the Navy to procure up to 15 Flight III ships on one or more multiyear contracts. In September 2018, the Navy awarded multiyear procurement contracts for 10 Flight III ships: six to Huntington Ingalls and four to Bath Iron Works, with options for up to five additional ships split between the yards. The program reported that the Navy exercised one of these options in fiscal year 2019 for an 11th ship. In fiscal year 2019, the Navy added six ships to its planned Flight III quantities to work toward its goal of a 355 ship Navy. Program Office Comments Note: Cost and quantity estimates reflect amounts reported in the fiscal year 2019 President’s Budget, which are procurement amounts only. Page 131 We provided a draft of this assessment to the program office for review and comment. The program office provided technical comments, which we incorporated where appropriate. The program office said that it has delivered 67 DDG 51 class ships since its inception in 1985 and the class remains in serial production at both new construction shipyards. GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Navy, Common Name: FFG(X) Guided Missile Frigate (FFG(X)) The Navy’s guided missile frigate program is intended to develop and deliver a small surface combatant with enhanced lethality and survivability as compared to the Littoral Combat Ship (LCS). The Navy expects FFG(X) to be an agile, multi-mission ship that provides local air defense, maximizes anti-surface and -submarine warfare capabilities, and delivers capability to protect and enable communications in hostile environments. Program Essentials Current Status Milestone decision authority: Navy Program office: Washington Navy Yard, DC Prime contractors (conceptual design): Austal USA; Bath Iron Works; Ingalls Shipbuilding; Marinette Marine Corporation; Lockheed Martin Contract types: FFP (conceptual design); FPI (detail design and construction; planned) Software development approach: TBD Next major milestone: Preliminary design review (June 2019) The FFG(X) program continues conceptual design work ahead of planned award of a lead ship detail design and construction contract in September 2020. In May 2017, the Navy revised its plans for a new frigate derived from minor modifications of an LCS design. The current plan is to select a design and shipbuilder through full and open competition to provide a more lethal and survivable small surface combatant. Estimated Program Cost (FY 2019 dollars in millions) As stated in the FFG(X) acquisition strategy, the Navy awarded conceptual design contracts in February 2018 for development of five designs based on ships already demonstrated at sea. The tailoring plan indicates the program will minimize technology development by relying on governmentfurnished equipment from other programs or known-contractor-furnished equipment. In November 2018, the program received approval to tailor its acquisition documentation to support development start in February 2020. This included waivers for several requirements, such as an analysis of alternatives and an affordability analysis for the total program life cycle. FFG(X) also received approval to tailor reviews to validate system specifications and the release of the request for proposals for the detail design and construction contract. Following the development start decision, the program office will complete its evaluation of FFG(X) proposals and plans to award a contract in September 2020 to a single shipbuilder for detail design and construction of the lead frigate, with options for nine additional ships. Planned Quantities Attainment of Technology Maturation Knowledge As of January 2019 ○ ◐ Conduct competitive prototyping Validate requirements ● Knowledge attained, ◐ Knowledge planned, Complete technology readiness assessment Complete preliminary design review ◐ ◐ ○ Knowledge not attained, … Information not available, NA Not applicable Program Office Comments Note: Development costs include fiscal years 20192023 and procurement costs include fiscal years 2020-2023, covering 6 of the planned quantities. The Navy has yet to identify funding needs beyond 2023. Page 132 We provided a draft of this assessment to the program office for review and comment. The program office did not have any comments. GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Navy, Common Name: LHA 8 LHA 8 Amphibious Assault Ship (LHA 8) The Navy’s LHA 8, third ship of the LHA 6 class of ships, will help replace retired LHA 1 Tarawa-class amphibious assault ships. The LHA 8 incorporates significant design changes from earlier ships in the LHA 6 class to provide enhanced aviation capabilities and a well deck that can accommodate two landing craft. The ship is designed to transport about 1,350 Marines and their equipment onto hostile shores. The LHA 8 is under contract and began construction in October 2018. Program Essentials Current Status Milestone decision authority: Navy Program office: Washington, DC Prime contractor: Huntington Ingalls Industries Contract type: FPI (detail design and construction) Software development approach: Mixed Next major milestone: LHA 8 ship delivery (January 2024) In June 2017, the Navy exercised a contract option for detail design and construction of the LHA 8. The LHA 8 incorporates significant design changes from earlier ships in the LHA 6 class, but Navy officials were unable to quantify the changes. The Navy started construction in October 2018 and LHA 8 is scheduled to be delivered in January 2024. Estimated Program Cost (FY 2019 dollars in millions) Planned Quantities The LHA 8 program office has not identified any critical technologies. However, the ship is relying on technology that is currently being developed by another Navy program, the Enterprise Air Surveillance Radar (EASR), with delivery expected in August 2021. EASR, intended to provide selfdefense and situational awareness capabilities, is derived from the preexisting Air and Missile Defense Radar program, but will be a different size and will rotate. LHA 8 program officials have identified the radar as the program’s highest development risk. If the radar is not delivered on schedule, Navy officials report that this could lead to out-of-sequence design and delayed installation and testing. Officials responsible for developing the radar, however, stated that the radar is approaching maturity and is on schedule to be delivered to the shipbuilder when needed. The Navy began construction with about 61 percent of the LHA 8 product model completed—an approach inconsistent with shipbuilding best practices. These best practices call for 100 percent completion of 3D product modeling prior to construction start to minimize the likelihood of costly re-work and out of sequence work that can drive schedule delays. The Navy, however, estimates that the LHA 8 shipbuilder will not complete 100 percent of the ship’s 3D product model until June 2019, almost 8 months after the start of construction. Program Office Comments We provided a draft of this assessment to the program office for review and comment. The program office provided technical comments, which we incorporated where appropriate. The program office stated that the Navy understands all design changes incorporated on the LHA 8, such as reintroducing the well deck and incorporating EASR. According to the program office, the Navy does not begin construction on any section of the LHA 8 ship before completing that respective section’s design. / Page 133 GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Navy Common Name: LPD 17 Flight II bn LPD 17 San Antonio Class Amphibious Transport Dock, Flight II (LPD 17 Flight II) [formerly LX(R)] The Navy’s LPD 17 Flight II program will replace existing amphibious ships that the Navy is retiring. The Navy intends to use LPD 17 Flight II ships to transport Marines and equipment to support expeditionary operations ashore, as well as non-combat operations for storage and transfer of people and supplies. The Flight II ships will include a larger hull than the retiring ships, but the Navy expects them to provide similar capabilities. The Navy plans to develop Flight II incrementally over three ships and acquire 13 ships beginning in fiscal year 2019. Program Essentials Current Status Milestone decision authority: Navy Program office: Washington Navy Yard, DC Prime contractor: Huntington Ingalls Incorporated Contract type: CPFF (long lead material purchasing) FPI (detail design and construction) CPFF (life cycle and engineering support; planned) Software development approach: NA Next major milestone: Detail design and construction contract award (March 2019) The Navy planned to accelerate purchase of LPD 30—the first fully configured Flight II ship—after Congress appropriated $1.8 billion above the fiscal year 2018 budget request, according to program officials. The Navy reported that it awarded contracts in August 2018 for LPD 30 long lead time materials and in March 2019 for lead ship construction. Estimated Program Cost (FY 2019 dollars in millions) Planned Quantities The Navy based the Flight II design on Flight I, with modifications to reduce costs and meet new requirements. According to program officials, roughly 200 design changes will distinguish the two flights including replacing the composite mast with a steel stick. Officials stated that the design would not rely on any new technologies. However, the Navy plans to install a new radar, the Enterprise Air Surveillance Radar, which is still in development. The Navy expects live radar system testing through November 2019, with a complete radar prototype in February 2020. Although program officials consider these activities to be low risk, the Navy will make its decision to begin ship construction by December 2019 without incorporating lessons learned from radar testing into the design. Starting construction before stabilizing the design could require the Navy to absorb costly design changes and rework during ship construction. The Navy initially pursued a limited competition for LX(R), but now has a non-competitive acquisition strategy for LPD 17 Flight II. The Navy plans to award sole-source contracts to Huntington Ingalls—the only shipbuilder of Flight I ships—for Flight II construction. Further, the program did not request a separate independent cost estimate for Flight II prior to awarding the LPD 30 detail design and construction contract. At the same time, the Navy identified no plans to establish a cost baseline specific to Flight II. Without this baseline, the Navy would report full LPD 17 program costs— rather than Flight II specific costs—constraining visibility into Flight II. Program Office Comments We provided a draft of this assessment to the program office for review and comment. The program office provided technical comments, which we incorporated as appropriate The program office stated that LPD Flight II is included under the existing LPD 17 acquisition program baseline, and that no other viable contractor responded to a public notice regarding the Navy’s plan to award Huntington Ingalls the LPD 30 construction contract. Page 134 GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Navy Common Name: NGJ Low-Band Next Generation Jammer – Low Band (NGJ-LB) The Navy’s NGJ-LB is an external jamming pod system that will be fitted on EA-18G Growler aircraft. It is expected to replace the ALQ-99 jamming system and provide enhanced airborne electronic attack capabilities to disrupt adversaries’ use of the electromagnetic spectrum for radar detection, among other purposes. The Navy plans to field a system that jams low-band radio frequencies. The Navy plans separate programs for mid- and high-band frequencies. We assessed the Low Band program. Program Essentials Current Status Milestone decision authority: Navy Program office: Tactical Aircraft Programs, Patuxent River, MD Prime contractor: TBD Contract type: TBD Software development approach: TBD Next major milestone: Completion of two demonstration of existing technologies contracts (June 2020) The Navy is determining the acquisition strategy for the NGJ-LB program and analyzing potential solutions to meet its capability needs. In October 2018, the Assistant Secretary of the Navy (Research, Development, and Acquisition) directed the NGJ-LB program manager to proceed with planning to execute the program as a middle tier acquisition. A middle tier acquisition—also referred to as a Section 804 program—is a program that uses a streamlined acquisition process to rapidly prototype or field capabilities within a 5-year period. Estimated Program Cost (FY 2019 dollars in millions) In October 2018, the program awarded two demonstration of existing technologies (DET) contracts to assess maturity of technologies, identify potential materiel solutions, and inform acquisition strategy development. Both contractors—L3 Technologies and Northrop Grumman—are required to provide technology demonstration prototypes and demonstrate technology maturity in a relevant test environment. Program officials stated that the results of these demonstrations will determine whether the program will move forward as a middle tier acquisition. According to program officials, the Navy plans to conduct a preliminary design review and a technology readiness assessment in the future, but will not schedule most program events until after the DET contractors complete their work. The Navy intends to commence a follow-on development or rapid prototype program by the end of fiscal year 2020. Attainment of Technology Maturation Knowledge Planned Quantities As of January 2019 ● ● Conduct competitive prototyping Validate requirements ● Knowledge attained, ◐ Knowledge planned, Complete technology readiness assessment Complete preliminary design review ◐ ◐ ○ Knowledge not attained, … Information not available, NA Not applicable Program Office Comments We provided a draft of this assessment to the program office for review and comment. The program office provided technical comments, which we incorporated where appropriate. Page 135 GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Navy Common Name: P-8A Increment 3 P-8A Poseidon, Increment 3 (P-8A Increment 3) The Navy’s P-8A Increment 3 is intended to provide enhanced capabilities to the P-8A aircraft in four sets of improvements. The first two sets include communication, radar, and weapons upgrades, which will be incorporated into the existing P-8A architecture. The following sets will establish a new open systems architecture, improve the combat system’s ability to process and display classified information, and enhance the P-8A’s search, detection, and targeting capabilities. DOD made Increment 3 part of the P-8A baseline program in 2016. Program Essentials Current Status Decision authority level: Navy Program office: Patuxent River, MD Prime contractor: Various Contract type: CPFF (design and integration) Software development approach: Mixed Next major milestone: Increment 3 design review (January 2020) The P-8A program has delivered the first set of Increment 3 improvements and is scheduled to complete the second set in fiscal year 2019. Since our 2018 assessment, the P-8A program has delayed the development and fielding of the last two sets of capabilities in Increment 3. Specifically, the program office reports it deferred events such as platform integration contract award, P-8A Increment 3 critical design review, and fielding of remaining capabilities by 3 to 11 months. Program officials attributed these new delays to receiving 30 percent less in development funds than requested in fiscal year 2018. In total, program officials report that limited funding in the last two fiscal years has delayed the fielding of the last two sets of capabilities by 12 to 18 months. Estimated Program Cost (FY 2019 dollars in millions) Planned Quantities To upgrade the combat system for P-8A Increment 3, the Navy will integrate new hardware and software on the aircraft. The P-8A acquisition program baseline provides for 117 aircraft to be retrofitted with Increment 3 capabilities. According to program officials, P-8A Increment 3 capabilities are based on mature technologies. The government will act as the lead integrator for the combat system. The combat system upgrade includes an applicationbased open system architecture that will allow the program to compete the development and integration of future capabilities. The program office conducted a preliminary design review for the combat system upgrade in July 2018. The program office stated that, among the issues that the review identified, it has resolved those associated with certain combat system components that exceeded their size, weight, power, and cooling allocations. According to program officials, the P-8A Increment 3 critical design review will largely focus on integration. Program Office Comments We provided a draft of this assessment to the program office for review and comment. The program office provided technical comments, which we incorporated as appropriate. The program office stated that P-8A Increment 3 has continued to make technical progress to support the combat system's critical design review. According to the program office, this progress includes closing all technical actions from the preliminary design review conducted in July 2018. Page 136 GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Navy Common Name: SSN 774 Block V SSN 774 Virginia Class Submarine Block V (SSN 774 Block V) SSN 774 is a nuclear-powered attack submarine capable of performing multiple missions, with enhanced capabilities for special operations and intelligence collection and surveillance. The Navy has implemented major upgrades to the class in blocks. The most recent, Block V, includes enhanced undersea acoustic improvements called acoustic superiority, and increases strike capacity for Tomahawk cruise missiles by inserting a new mid-body section called the Virginia Payload Module (VPM). Program Essentials Current Status Milestone decision authority: Navy Program office: Washington, DC Prime contractor (planned): General Dynamics Electric Boat Contract type (planned): FPI (construction) Software development approach: Other Next major milestone: Block V contract award (calendar year 2019) In 2019, the Navy plans to award a multibillion dollar, multiyear contract for construction of 10 Block V submarines. Under the Navy’s plan, all Block V ships will include acoustic superiority improvements, while the VPM will be added starting with the second Block V submarine. Estimated Block V Cost (FY 2019 dollars in millions) Planned Quantities According to program officials, the design of Block V submarines will differ from Block IV by approximately 20 percent. Of this 20 percent, the program office considers 70 percent to constitute major changes. The program office plans to complete basic and functional designs for VPM by construction start—an approach consistent with best practices. However, the shipbuilder is currently behind schedule in completing detail design work, where (1) the design advances to the highest level of fidelity, (2) specific fabrication and installation instructions for the shipyard are developed, and (3) required production materials are identified. The program now plans to complete 76 percent of this work by construction start, compared to the 86 percent it initially planned, in part due to the shipbuilder’s challenges in using a new design tool. Going forward, the Navy and shipbuilder will need to balance staffing levels for the remaining Block V design work with design efforts for the new Columbia class ballistic missile submarine. Construction of Block V and the Columbia class will coincide beginning in fiscal year 2021.This will require the Navy and its shipbuilder to manage staffing demands and other resources across both programs. In addition, program officials said vendor quality issues with welding on VPM have caused a 3.5-month delay in the schedule for the payload tubes for the first two submarines with VPM. The Navy plans to recover some time by accelerating tube manufacturing with a second vendor, but this approach may increase program costs. The Block V effort is subsumed under the SSN 774 major defense acquisition program, and is not managed as a separate program. In 2015, the Office of the Secretary of Defense shifted the program’s oversight to the Navy. SSN 774 had already completed its required defense acquisition system milestone reviews before Block V started, but program officials said the Navy continues to conduct regular oversight of the Block V. Note: In our 2018 report, we reflected Block V estimated costs for fiscal years 2019-2022. At that time, the Navy had yet to determine funding needs beyond fiscal year 2022. The cost estimates above, however, reflect information provided by the Navy in November 2018, which include funding needs for prior years as well as fiscal year 2023. Page 137 Program Office Comments In commenting on a draft of this assessment, the program office provided technical comments, which we incorporated where appropriate. GAO-19-336SP Weapon Systems Annual Assessment AHRFORCE PROGRAM ASSESSMENTS GAO Assessed 10 of the Air Force?s 37 Current Major Defense Acquisition Programs Current program ?4 (Cost baseline not established) 23 Other programs (Not assessed) Current programs The 10 Current Programs GAO Assessed Represent 33 Percent of the Air Force?s $238 Billion Estimated Total Acquisition Costs (Fiscal Year 2019 dollars in billions) I We also assessed two future major defense acquisition programs. We did not assess programs that had yet to enter technology development or had already progressed into full-rate production. Most Current Programs GAO Assessed Have Either Cost Growth, Schedule Delays, or Both Program comptete Programs in production . FAB-T CPT SDB . see in Programs in system development B-2 DMS F-15 EPAWSS .MGUE moi OSpace Fence OCX Program age . Cost growth of more than 15 percent andicr schedule delays of more than 6 months Cost growth of 15 percent or less andlor schedule delays of 6 months or less . Cost growth of 0 percent or less and schedule delays of 0 months or less Note: Bubble size is based on each program?s currently estimated future acquisition funding needed. Total Acquisition Cost Performance in Current Programs Varied Among Knowledge Points (KP) (Fiscal Year 2019 dollars in billions) Hi1 KP1 to KP2 to Post - KP2 KP3 KP3 I A cost decrease in one current program?KC-dBA? accounted for most of the cost decrease between KP2 and KP3. Current Programs Did Not Attain Knowledge at Key Points KP1 Oct!) 5 1.1 ll: i 1-: current programs attained i 2" knowledge KP2 Oof8 in: a; current programs attained knowledge KP3 Oof4 current It I programs attained knowledce Average Acquisition Cycle Time of Current Programs Increased by 25 Months from First Full Estimate (FFE) to Current Estimate Programs' average FFE cycle time Number of months Programs' average 25 cycle time change from FFE to current estimate I We could not calculate cycle time for GPS because initiai capability depends on the availability of complementary systems. I Four current programs?APT, GPS Replacement. and VC-ZSB?entered system development after December 2017 and therefore are excluded from our cost. schedule. knowledge attainment, and contract analysis. I Cost and schedule analysis is primarily based on estimates from December 2017 selected acquisition reports. This information may differ from the amounts reported in the individual assessments. which are based on more recent program estimates in some cases. Please see appendix II for details. Source; GAO analysis of Department of Defense data. GAO-19-33BSP Page 138 GAO-19-3368P Weapon Systems Annual Assessment AIR FORCE PROGRAM ASSESSMENTS Half of the Current and Future Air Force Programs Report that Less than 20 Percent of Planned Acquisition Costs are Specifically for Software Percentage of planned acquisition costs 0-20% 21-40% 81-100% Unknown NA/not tracked 0 2 4 6 Number of programs 03 Most Current Air Force Programs Report that Less than 40 Percent of Work Tasks are Software-Related Percentage of work tasks 0-20% 21-40% 81-100% Unknown NA/not tracked 2 4 6 8 Number of programs The Air Force was the Milestone Decision Authority in 2017 for 13 of its Current and Future Programs in Review Air Force programs programs 3 13 The milestone decision authority for APT, FAB-T CPT, GPS KC- 46A, MGUE Inc 1, SDB II, and WSF changed from the Under Secretary of Defense for Acquisition and Sustainment to the Air Force in 2017. The Air Force Received One Offer on Solicitations for Just Over Half of its Currently Reported Major Contracts for Development and Procurement Number of contracts 2 4 1 1 1 A Aircraft- Aircraft- C3l Helicopter Munitions Radar Satellite Unspecified Bomber Other System type Cl 2 or more offers - 1 offer C3I is short for command, control, communications and intelligence Source: GAO analysis of Department of Defense data. GAO-19-33GSP Page 139 GAO-19-3368P Weapon Systems Annual Assessment Air Force Program Assessmentsa 2-page assessments Advanced Pilot Training (APT) B-2 Defensive Management System Modernization (B-2 DMS-M) Combat Rescue Helicopter (CRH) F-15 Eagle Passive Active Warning Survivability System (F-15 EPAWSS) Family of Advanced Beyond Line-of-Sight Terminals Command Post Terminals (FAB-T CPT) Global Positioning System III (GPS III) Global Positioning System III Follow-On (GPS IIIF) KC-46 Tanker Modernization Program (KC-46A) Military GPS User Equipment (MGUE) Increment 1 Next Generation Operational Control System (OCX) Small Diameter Bomb Increment II (SDB II) Space Fence Ground-Based Radar System Increment 1 (Space Fence Inc 1) Utility Helicopter (UH-1N) Replacement 1-page assessmentsb B-52 Radar Modernization Program (B-52 RMP) VC-25B Presidential Aircraft Recapitalization (VC-25B) Weather System Follow-On (WSF) Page number 141 143 145 147 149 151 153 155 157 159 161 163 165 167 168 169 a We abbreviate the following contract types in the individual assessments: cost-plus-award-fee (CPAF), cost-plus-fixed-fee (CPFF), cost-plus-incentivefee (CPIF), firm-fixed-price (FFP), and fixed-price incentive (FPI). b The VC-25B program has entered system development but released a baseline late in our review, so we assessed it in a one-page format. Page 140 GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Air Force, Common Name: APT Advanced Pilot Training (APT) With its APT program, the Air Force is replacing its legacy T-38C trainer fleet and related ground equipment by developing and fielding newer, more technologically advanced trainer aircraft and an associated ground-based training system. The APT program responds to the Air Force’s advanced fighter pilot training needs and seeks to close training gaps that the T-38C cannot fully address. Program Essentials Program office: Wright-Patterson Air Force Base, OH Prime contractor: Boeing Contract type: Indefinite-delivery indefinite-quantity with delivery orders: FPI (development) Software development approach: TBD Next major milestone: Design review (March 2020) Program Performance (fiscal year 2019 dollars in millions) First full estimate (09/2018) Latest (09/2018) Percentage change Development $1,258.40 $1,258.40 0.0% Procurement $6,782.30 $6,782.30 0.0% $23.40 $23.40 0.0% 85 85 0.0% 351 351 0.0% Unit cost Acquisition cycle time (months) Total quantities The Air Force approved an acquisition program baseline for APT in September 2018. Total quantities comprise 5 development quantities and 346 procurement quantities. Funding and Quantities (fiscal year 2019 dollars in millions) Attainment of Product Knowledge As of January 2019 Status at Current Status Development Start Resources and requirements match • Demonstrate all critical technologies are very close to final form, fit and function within a relevant environment • Demonstrate all critical technologies in form, fit and function within a realistic environment • Complete a system-level preliminary design review Product design is stable NA NA NA NA ○ ○ Design Review • Release at least 90 percent of design drawings NA NA • Test a system-level integrated prototype NA NA Manufacturing processes are mature Production Start • Demonstrate Manufacturing Readiness Level of at least 9, or critical processes are in statistical control • Demonstrate critical processes on a pilot production line • Test a production-representative prototype in its intended environment NA NA NA NA NA NA ● Knowledge attained, ○ Knowledge not attained, … Information not available, NA Not applicable Page 141 GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Air Force, APT Program Technology Maturity The APT program office reported that APT does not rely on any critical technologies. According to the program’s 2016 acquisition strategy, the program expected all competing vendors to either (1) not employ any critical technologies in their proposals or (2) only employ critical technologies that were mature and ready to demonstrate in a field environment. The strategy also said that the program office conducted a preliminary technology readiness assessment in 2015 that showed the technologies required for the major components of the APT—including airframe, propulsion, and ejection seat—were mature and demonstrated in operating environments similar to that for the APT. The strategy also acknowledged, however, that some new APT capabilities, such as embedded training systems, cockpit displays, and software, might need to be developed or integrated during the program’s development phase. Common Name: APT Program Office Comments We provided a draft of this assessment to the program office for review and comment. The program office provided technical comments, which we incorporated where appropriate. The program did not conduct a preliminary design review (PDR) before its September 2018 development start. Instead, it plans to hold the PDR in approximately June 2019, some 9 months after development start. The program received a statutory waiver from the Assistant Secretary of the Air Force (Acquisition, Logistics, and Technology) that allowed it to delay the PDR. The waiver stated that conducting PDR-related activities prior to development start would prevent the program from meeting national security objectives because it would delay transition of pilots to 4th and 5th generation fighter aircraft. Further, the waiver noted that holding PDR post-development start was low-risk because APT would leverage mature systems and technologies and a PDR would increase the program’s cost as well as delay the program by 12-18 months with “no appreciable decrease” in technical risk. However, as we have observed in prior work, a PDR conducted prior to development start helps ensure a system’s design is feasible—which in turn contributes to a match between customer needs and available time, funding, and other resources. Additionally, we have found that major defense acquisition programs that held a PDR prior to starting development had fewer schedule delays than other programs. Design Stability The APT program has initiated system design activities, but is not currently tracking design drawings. The program’s acquisition strategy called for vendors to provide flight test data for their systems in order to be considered for contract award. The strategy further stated that the program office would structure the acquistion to require the use of mature systems. Looking ahead, the program expects to hold a critical design review no later than March 2020. Page 142 GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Air Force, Common Name: B-2 DMS-M B-2 Defensive Management System Modernization (B-2 DMS-M) The Air Force’s B-2 DMS-M program plans to upgrade the aircraft’s 1980s-era defensive management system to a more capable system. This system detects and locates enemy radar systems to provide threat warnings and avoidance information. This upgrade is expected to improve the system’s frequency coverage and sensitivity, update pilot displays, and enhance in-flight rerouting capabilities. It will improve the reliability and maintainability of the DMS system and the B-2’s readiness. Program Essentials Milestone decision authority: Under Secretary of Defense for Acquisition and Sustainment Program office: Wright-Patterson Air Force Base, OH Prime contractor: Northrop Grumman Contract type: FFP (development) Software development approach: Agile development Next major milestone: Low-rate initial production decision (June 2020) Funding and Quantities (fiscal year 2019 dollars in millions) Program Performance (fiscal year 2019 dollars in millions) First full estimate (05/2016) Latest (06/2018) Percentage change Development $1,914.60 $2,250.40 +17.5% Procurement $771.6 $776.50 +0.6% $134.31 $151.34 +12.7% 124 130 +4.8% 20 20 0.00% Unit cost Acquisition cycle time (months) Total quantities The Air Force released a new service cost position for the B-2 DMS-M program in June 2018. Total quantities comprise 4 development quantities and 16 procurement quantities. Attainment of Product Knowledge As of January 2019 Status at Resources and requirements match • Demonstrate all critical technologies are very close to final form, fit and function within a relevant environment • Demonstrate all critical technologies in form, fit and function within a realistic environment • Complete a system-level preliminary design review Product design is stable ● ○ ● ● ● ● Design Review • Release at least 90 percent of design drawings • Test a system-level integrated prototype Manufacturing processes are mature ● ○ ● ○ Production Start • Demonstrate Manufacturing Readiness Level of at least 9, or critical processes are in statistical control • Demonstrate critical processes on a pilot production line • Test a production-representative prototype in its intended environment Current Status Development Start NA NA NA NA NA NA ● Knowledge attained, ○ Knowledge not attained, … Information not available, NA Not applicable Page 143 GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Air Force, B-2 DMS-M Program Technology Maturity and Design Stability The B-2 DMS-M program has fully matured its critical technologies and stabilized its design. The program entered system development in March 2016 with four critical technologies approaching maturity. Since then, Northrop Grumman proposed, and the Air Force accepted, an alternative system that provides additional capabilities. The alternative system is leveraged from other programs and has the same four critical technologies. Because the Air Force has tested the major subsystems on an existing program, it now considers its critical technologies mature. The Air Force’s decision to pursue an alternative system led to some initial program delays. For example, although the Air Force held a preliminary design review before entering system development, it completed a second preliminary design review for the alternative system in March 2018. The program also held its critical design review in November 2018, nearly 2 years later than originally planned. Consistent with acquisition best practices, the program released more than 90 percent of its design drawings by the critical design review. However, the program did not test a system-level integrated prototype—an approach inconsistent with acquisition best practices. This decision puts the program at risk for costly design changes following system-level integration testing if the program identifies deficiencies. The program continues to face delays and increased integration risk in its software development, a critical factor for achieving required B-2 DMS-M capabilities. The contractor is utilizing an Agile software development process for the first time on this program and has taken longer than expected to achieve the development efficiency to complete the functionality of a software block, known as PD 7.1. The program requires certification of software block PD 7.1 functionality to support developmental flight testing. Program officials stated that over the last year the contractor has made significant progress in addressing software development risk by increasing the number of software developers and expanding overtime for existing staff. However, software version PD 7.1 may still be at risk of not being certified with full functionality by June 2019, when the program has scheduled developmental flight tests. Lack of software certification could delay these tests. Program officials continue to monitor the effectiveness of the contractor’s mitigation plans and possible schedule delays. Other Program Issues Program officals report that, in May 2017, due to the selection of the alternative DMS system, the program office and contractor agreed to an undefinitized contract action that changed the development contract from a Page 144 Common Name: B-2 DMS-M cost-type contract to a firm-fixed-price contract. The program office stated that the undefinitized contract action included a not-to-exceed amount of $741 million until the contract scope and price could be definitized. Because there were issues with the contractor delivering a complete proposal, the program has delayed definitization. The program office reports approximately 50 percent of the $741 million not-toexceed amount has been obligated, and officials do not expect to definitize the contract action until the spring of 2019, nearly 2 years after award. As program officials began negotiations for using the alternative DMS system, they recognized they could not execute the related development work within the existing Air Force budget. As a result, the Air Force conducted a cost-versus-capability trade analysis of the program and ultimately decided on a course of action that continued with the alternative system but required hardware changes on the aircraft. In June 2018, the Air Force released an updated service cost position to reflect the latest acquisition approach, which identified a 13 percent unit cost increase over the program’s acquisition baseline. This increase was largely due to hardware changes and integration updates as a result of the increased capabilities of the alternative DMS system. Program Office Comments We provided a draft of this assessment to the program office for review and comment. The program office provided technical comments, which we incorporated where appropriate. In reviewing the draft, program officials noted that the revised program strategy leverages development from other Air Force programs, implements additional capability to improve weapon system survivability, and provides risk reduction for other programs. In addition, program officials stated the cost-versuscapability analysis ensured that while hardware requirements were decreased in order to reduce procurement costs, the program will still meet performance requirements.They also stated that the service cost position now estimates a 12.5 percent increase in program acquisition unit cost as well as the potential for delays.This estimate includes the increased capabilities and associated risk reduction, and is not constrained to the not-to-exceed value. Program officials also noted that design efforts are complete as presented at the critical design review. They said that DMS-M installation into the B-2 flight test aircraft is underway. According to the officials, the critical path to beginning flight test is completion of software development and PD 7.1 certification efforts. The program office stated that it continues to work closely with the prime contractor to achieve schedule milestones. GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Air Force, Common Name: CRH Combat Rescue Helicopter (CRH) The Air Force's CRH program will replace the Air Force’s aging HH60G Pave Hawk rescue helicopter fleet. It will provide 113 new air vehicles, related training systems, and support for increased personnel recovery capability. CRH uses a derivative of the operational UH-60M helicopter. Planned modifications to the existing design include a new mission computer and software, a higher capacity electrical system, larger capacity main fuel tanks, armor for crew protection, a gun mount system, and situational awareness enhancements. Program Essentials Milestone decision authority: Air Force Program office: Wright-Patterson Air Force Base, OH Prime contractor: Sikorsky Aircraft Corporation Contract type: FPI/FFP (development) Software development approach: Mixed Next major milestone: Low-rate initial production (September 2019) Program Performance (fiscal year 2019 dollars in millions) First full estimate (06/2014) Latest (12/2018) Percentage change Development $2,106.00 $2,013.80 -4.4% Procurement $6,567.50 $6,240.80 -5.0% $77.67 $73.66 -5.2% 75 75 +0.0% 112 113 0.9% Unit cost Acquisition cycle time (months) Total quantities Acquisition cycle time as of November 2018. Cycle time is based on required asset availability date. Initial capability is not a formal program milestone. Quantities comprise 10 development quantities and 103 procurement quantities. Funding and Quantities (fiscal year 2019 dollars in millions) Attainment of Product Knowledge As of January 2019 Status at Resources and requirements match • Demonstrate all critical technologies are very close to final form, fit and function within a relevant environment • Demonstrate all critical technologies in form, fit and function within a realistic environment • Complete a system-level preliminary design review Product design is stable ○ ○ ○ ● ○ ● Design Review • Release at least 90 percent of design drawings • Test a system-level integrated prototype Manufacturing processes are mature ● ○ ● ○ Production Start • Demonstrate Manufacturing Readiness Level of at least 9, or critical processes are in statistical control • Demonstrate critical processes on a pilot production line • Test a production-representative prototype in its intended environment Current Status Development Start NA NA NA NA NA NA ● Knowledge attained, ○ Knowledge not attained, … Information not available, NA Not applicable Page 145 GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Air Force, CRH Program Technology Maturity and Design Stability The CRH program has matured its one critical technology—a radar warning receiver—and stabilized its design. At development start in June 2014, however, the program had not demonstrated the receiver to final form, fit, or function in either a relevant or realistic environment—an approach inconsistent with best practices. Program officials stated that CRH’s one critical technology, the radar warning receiver, was proven mature through Integrated Demonstrations and Applications Laboratory testing in May 2018 that tested based on over 100 simulated real-world threats. As indicated by our attainment of production knowledge section, we believe that the program cannot demonstrate the full maturity of critical technologies until they are tested in their realistic environment, such as on a helicopter in flight. Approximately 72 percent of the CRH design is based on the fully operational and fielded UH-60M helicopter, while the remainder is based on new design drawings that are unique to the CRH. Although the program reports that it has released all its planned design drawings, the program continues to resolve and test fixes for system deficiencies identified in recent testing. Program officials noted that, for the identified deficiencies—including those that involve the mission computer, fuel gauge, and ballistic armor—they are either implementing corrections or awaiting testing to confirm resolution. These deficiencies may require the program to revise currently completed design drawings to accommodate necessary changes, which could compromise the CRH’s design stability. The CRH program also has yet to test a system-level integrated prototype to demonstrate its design, although best practices criteria state that such a prototype should be tested by a program’s critical design review. Instead, CRH program officials reported that, in February 2017, they began lab-based prototype tests using a partial CRH system. However, these tests will not fully demonstrate certain CRH subsystems and software, which continue to pose technical risk in the program. Production Readiness In accordance with best practices, program officials plan to conduct system-level developmental prototype testing and demonstrate critical manufacturing processes on a pilot production line prior to the low-rate initial production decision, which is planned for September 2019. Since our 2018 assessment, the program office has delayed planned dates for low-rate initial production, the start of developmental testing and the helicopter’s first flight. According to program officials, problems with suppliers and software integration and testing required the program to extend the developmental schedule and delay initiating flight Page 146 Common Name: CRH tests. The contractor had previously pursued an accelerated program schedule. Program officials stated that the contractor is now following the program’s approved 75-month baseline schedule, due primarily to delayed design and supplier performance issues. Program officials reported that first flight has changed from October 2018 to May 2019 with no changes in required or planned testing. Program officials stated they do not anticipate engineering-related changes to pose a serious risk after production begins, but that software development and deficiency corrections may be necessary. Program officials stated that correcting deficiencies identified during testing would not require any increase in the government’s program costs. According to these officials, the government’s liability is capped at a negotiated ceiling price under the fixed-price incentive contract for CRH development, and program officials project reaching this ceiling. Program officials expect to complete integration of CRH’s software and hardware prior to the start of operational testing in January 2021. Other Program Issues Program officials expect to meet CRH affordability requirements for average unit cost, but the program’s cost estimate for military construction currently exceeds its baseline cost constraints by $10 million due to increased trainer facilities requirements. In addition, program officials reported that this estimate may further change due to DOD pricing guide rate changes in the future. Program Office Comments We provided a draft of this assessment to the program office for review and comment. The program office provided technical comments, which we incorporated where appropriate. According to the program office, CRH has advanced significantly and the contractor has mitigated its design and supplier performance issues. The program office also said that the contractor completed testing of CRH hardware and initial software required for production flight testing in an aircraftrepresentative laboratory. According to the program, CRH’s critical technology tested successfully in a simulated environment using production hardware. The program office stated that all data required to support the production decision will be in place. The program office also expressed confidence that it will achieve its planned dates for production decision, subsequent contract award for low-rate initial production, and required asset availability. GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Air Force, Common Name: F-15 EPAWSS F-15 Eagle Passive Active Warning Survivability System (F-15 EPAWSS) The Air Force’s F-15 EPAWSS program plans to modernize the F-15’s electronic warfare (EW) system used to detect and identify threats, employ counter-measures, and jam enemy radars. The program plans to reconfigure hardware and software from other military aircraft to meet the challenges of today’s EW threat environment. The Air Force developed EPAWSS Increment 1 to replace the F-15’s legacy EW system. The Air Force has yet to budget for a proposed Increment 2, which adds a new towed decoy. We assessed Increment 1. Program Essentials Milestone decision authority: Air Force Program office: Wright-Patterson Air Force Base, OH Prime contractor: Boeing Contract types: CPIF/CPFF (technology maturation and risk reduction) CPIF/CPFF/FFP (development) Software development approach: Mixed Next major milestone: Low-rate initial production (July 2019) Funding and Quantities (fiscal year 2019 dollars in millions) Program Performance (fiscal year 2019 dollars in millions) First full estimate (11/2016) Latest (12/2018) Percentage change Development $921.10 $986.20 7.1% Procurement $3,546.70 $3,556.70 0.3% $10.82 $11.00 1.7% 83 82 -1.2% 413 413 0.0% Unit cost Acquisition cycle time (months) Total quantities Attainment of Product Knowledge As of January 2019 Status at Resources and requirements match • Demonstrate all critical technologies are very close to final form, fit and function within a relevant environment • Demonstrate all critical technologies in form, fit and function within a realistic environment • Complete a system-level preliminary design review Product design is stable ● ○ ● ● ○ ● Design Review • Release at least 90 percent of design drawings • Test a system-level integrated prototype Manufacturing processes are mature ○ ○ ● ● Production Start • Demonstrate Manufacturing Readiness Level of at least 9, or critical processes are in statistical control • Demonstrate critical processes on a pilot production line • Test a production-representative prototype in its intended environment Current Status Development Start NA NA NA NA NA NA ● Knowledge attained, ○ Knowledge not attained, … Information not available, NA Not applicable Page 147 GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Air Force, F-15 EPAWSS Program Technology Maturity Over 2 years after entering development, the EPAWSS program reported each of its four critical technologies as approaching maturity—a condition inconsistent with best practices. The program delayed ground-based testing of the critical technologies by several months in 2018 because of performance shortfalls with some components where the technologies reside. The remaining technology maturation work also delayed software development and other activities needed to support flight testing. Specifically, the first flight of an EPAWSS-equipped test aircraft is now delayed by 3 months from December 2018 to March 2019. As a result, the Air Force has yet to demonstrate the full maturity of EPAWSS technologies. Design Stability The program reported the release of all EPAWSS design drawings, but technology immaturity and the redesign of key components pose risk to maintaining design stability. For example, the program released only 87 percent of the EPAWSS drawings at its February 2017 critical design review (CDR), with the remaining drawings released in the months that followed. As a result of subsequent redesign efforts to address performance shortfalls, the program modified some released drawings in the past year. The design changes also required a subcontractor to remanufacture components of the program’s initial test articles which led the program to delay key integration and testing activities by several months. According to program officials, the components in need of redesign performed adequately during prototype testing at development start, but did not perform as needed during more rigorous testing after CDR. Officials indicated that they are still experiencing a problem with one of the desired capabilities and may need to make additional design changes after the start of flight testing. In addition, a subcontractor experienced low yield rates for the initial fabrication of a critical test article component, leaving a limited number of test articles available to support development activities before flight testing. As a result, the program plans to install EPAWSS hardware without this component on several F-15 test aircraft, limiting aircraft functionality during the early months of flight testing until the missing component is equipped. Production Readiness The program plans to test a production-representative prototype and demonstrate manufacturing processes on a pilot production-line before EPAWSS production begins.The program schedule, however, indicates concurrency between testing and production is increasing. For example, the Air Force moved up the EPAWSS low-rate initial production decision by one month, from August 2019 to July 2019. Page 148 Common Name: F-15 EPAWSS Further, the program is now following a tailored approach where its production decision involves two separate decision points. The first, in July 2019, authorizes the start of EPAWSS low-rate production and will be reported as the program’s Milestone C decision for statutory purposes. The second decision point in 2020 authorizes installation of the alreadyprocured hardware on fielded F-15 aircraft. If the second decision is delayed, the program may accrue a stockpile of unused hardware until installation is approved. Also as a result of the tailored approach, the program will not attain certain pieces of knowledge before it starts production. For example, the Air Force decided that the EPAWSS program no longer needs to complete all hardware qualification testing, conduct an operational assessment, or demonstrate full EPAWSS performance in-flight prior to the July 2019 production decision. The program will instead complete these activities in support of the later deployment decision in 2020. This approach increases the risk that the program would need to retrofit already-installed units if system testing finds EPAWSS design problems after July 2019. Other Program Issues Several ongoing modifications to other F-15 aircraft subsystems need to be accomplished before EPAWSS can be successfully installed on the aircraft. According to EPAWSS program officials, these predecessor modifications are on schedule to support EPAWSS. In addition, the Air Force now plans to procure EPAWSS for both the F-15C and F-15E after reporting last year that only the F-15E would be upgraded. As a result, the program increased its current cost estimate over the past year, but it is now in keeping with the first full estimate, which included both F-15 models. Program Office Comments We provided a draft of this assessment to the program office for review and comment. The program office provided technical comments, which we incorporated where appropriate. The program office stated that to expedite replacement of the functionally obsolete EW self-protection system on F-15s today, the EPAWSS acquisition heavily leverages mature hardware from existing U.S. and foreign military EW programs. The program office indicated that availability of critical hardware components unique to F-15 EPAWSS drives the program schedule. The program office maintained that the system’s design is now stable and noted that it expects to begin flight tests by early summer 2019. The program office contended that the technical maturation of EPAWSS components will help support a decision to purchase a small quantity of production hardware before development is complete to help accelerate initial operational capability. The program office stated that while this strategy is not without risk, it provides the Air Force an opportunity to transfer operational risk from F-15 aircrews to the acquisition community. GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Air Force, Common Name: FAB-T CPT Family of Advanced Beyond Line-of-Sight Terminals Command Post Terminals (FAB-T CPT) The Air Force's FAB-T program plans to provide a family of satellite communication terminals for airborne and ground-based users to replace legacy communication terminals. In July 2015, DOD separated FAB-T into two subprograms: command post terminals (CPT), which we assess here, and force element terminals (FET). CPT is expected to provide communications for nuclear and conventional forces through ground command posts and E-6 and E-4 aircraft. FET is expected to provide capabilities for B-52 and RC-135, at a minimum. Program Essentials Milestone decision authority: Air Force Program office: Bedford, MA Prime contractor: Raytheon Contract type: FPI (development) FFP (production) Software development approach: Incremental Next major milestone: Start of operational testing (August 2019) Program Performance (fiscal year 2019 dollars in millions) First full estimate (12/2007) Latest (10/2018) Percentage change Development $739.20 $1,239.30 +67.6% Procurement $993.50 $619.10 -37.7% $18.24 $17.05 -6.5% 129 225 +74.4% 95 109 +14.7% Unit cost Acquisition cycle time (months) Total quantities Latest estimate reflects amounts from the Defense Acquisition Executive Summary following the new acquisition program baseline in July 2018. Total quantities comprise 25 development quantities and 84 procurement quantities. Funding and Quantities (fiscal year 2019 dollars in millions) Attainment of Product Knowledge As of January 2019 Status at Development Start Resources and requirements match • Demonstrate all critical technologies are very close to final form, fit and function within a relevant environment • Demonstrate all critical technologies in form, fit and function within a realistic environment • Complete a system-level preliminary design review Product design is stable ⋯ ● ○ ● ○ ○ ● ● ⋯ ⋯ Design Review • Release at least 90 percent of design drawings • Test a system-level integrated prototype Manufacturing processes are mature Production Start • Demonstrate Manufacturing Readiness Level of at least 9, or critical processes are in statistical control • Demonstrate critical processes on a pilot production line • Test a production-representative prototype in its intended environment Current Status ○ ● ○ ○ ● ● ● Knowledge attained, ○ Knowledge not attained, … Information not available, NA Not applicable Page 149 GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Air Force, FAB-T CPT Program Technology Maturity, Design Stability, and Production Readiness The FAB-T CPT program has several critical technologies, half of which continue to approach maturity, and reports a stable design. However, until the program fully matures all of its critical technologies— and demonstrates each of them in a final form, fit, and function within a realistic environment—current design stability remains at risk of disruption. The program has also entered low-rate initial production, but has yet to demonstrate statistical control of critical manufacturing processes—an approach inconsistent with best practices. FAB-T CPT is comprised of five terminal configurations, some of which have different critical technologies. Two modification kit terminals adapt existing antennas for ground fixed and airborne platforms and are fully functional but will eventually be retrofitted with new antennas. The other three configurations use new antennas for ground fixed, ground transportable, and airborne platforms. According to the program office, the four configurations currently in production—the two modification kit terminals and the new antenna ground fixed and transportable terminals—are now mature. The program previously planned for all the new antenna terminals to be fully mature by July 2016, but delays continue in developing the final configuration, the new airborne antenna. The program office now expects to achieve initial operational capability by June 2021—an 18-month delay from the December 2019 date it previously reported—and full operational capability by March 2023—a 15 month delay from the December 2021 date it anticipated. The program’s revised acquisition program baseline was approved in July 2018. The program reported that, as of October 2018, it intends to procure approximately 63 percent of FAB-T total units during low-rate production. The program’s purchase of such a quantity corresponds with 2015 direction from the Under Secretary of Defense for Acquisition, Technology, and Logistics, which authorized the FAB-T program to purchase more than 60 percent of its total units during low-rate production. Programs generally must provide a rationale to Congress if low-rate production quantities will exceed 10 percent of total planned procurement quantities. FAB-T officials said that the higher number of units are required to demonstrate initial operational capability due to the various configurations and platforms needed. Officials said they plan to request permission to procure additional terminals before a full-rate procurement decision to take advantage of cost control opportunities from volume discounts. With the majority of terminals already approved, officials noted that initial operational testing would not significantly inform a full-rate decision. We have previously found that beginning production Page 150 Common Name: FAB-T CPT before demonstrating that a system will work as intended increases the risk of needing substantial design changes and costly modifications to alreadyproduced systems. The FAB-T program reported that its manufacturing processes are nearing maturity. Officials also reported the program has contracted for a total of 51 terminals to date: 27 modification kit terminals, 15 new antenna ground fixed terminals, and 9 ground transportable terminals. As of December 2018, the contractor has delivered 22 terminals, and the program has installed 5 to begin testing. Other Program Issues The program office is developing the acquisition strategy for FET, its other subprogram, as a middle tier acquisition—also referred to as a Section 804 program—which uses a streamlined acquisition process to rapidly prototype or field capabilities within a 5-year period. The program expects the Air Force to approve this strategy in February 2019 and to then award a development contract by June 2019. Until the FET subprogram is executed, FAB-T cannot achieve its planned capabilities that are based on the interaction of bomber aircraft with intelligence, surveillance, and reconnaissance aircraft and CPTs. FAB-T is designed to communicate through the Advanced Extremely High Frequency (AEHF) network of satellites, four of which have already been launched. As FAB-T delays mount, the first launched AEHF satellite might near the end of its projected 14-year operational lifespan by the time FAB-T is available. All six AEHF satellites are expected to be on-orbit before FAB-T is operational. The lack of synchronization between the two programs has resulted in the underutilization of costly satellite capabilities. Program Office Comments We provided a draft of this assessment to the program office for review and comment. The program office provided technical comments, which we incorporated as appropriate. The program office stated that it faced challenges resolving issues identified during testing and finalizing operational software, which delayed the completion of operational testing by 5 months. According to program officials, the program is still on track to complete operational testing by November 2019. Program officials said they installed the first operational terminal in January 2019 and plan to complete development of the new airborne antenna configuration by September 2019. However, the program office also noted that it is pursuing options to address significant funding shortfalls that could affect test and terminal procurement activities. According to program officials, the Air Force designated the FET subprogram as a middle tier acquisition, but the acquisition strategy has not yet been approved. GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Air Force, Common Name: GPS III Global Positioning System III (GPS III) The Air Force's GPS III program is building and fielding a new generation of satellites to supplement and eventually replace GPS satellites currently in use. GPS III will provide a stronger military navigation signal, referred to as M-code, to improve jamming resistance, and will provide a new civilian signal that will be interoperable with foreign satellite navigation systems. Other programs are developing the related ground system and user equipment. Program Essentials Milestone decision authority: Air Force Program office: El Segundo, CA Prime contractor: Lockheed Martin Contract types: CPAF (development) CPAF/FPI (production) Software development approach: Mixed Next major milestone: Start of operational testing (January 2020) Program Performance (fiscal year 2019 dollars in millions) First full estimate (05/2008) Latest (06/2018) Percentage change Development $2,826.50 $3,436.20 +21.6% Procurement $1,587.00 $2,381.40 +50.1% $551.68 $581.76 +5.5% N/A N/A N/A 8 10 +25.0% Unit cost Acquisition cycle time (months) Total quantities We could not calculate GPS III cycle times because the initial capability depends on the availability of complementary systems. Total quantities comprise 2 development quantities and 8 procurement quantities. Funding and Quantities (fiscal year 2019 dollars in millions) Attainment of Product Knowledge As of January 2019 Status at Development Start Resources and requirements match • Demonstrate all critical technologies are very close to final form, fit and function within a relevant environment • Demonstrate all critical technologies in form, fit and function within a realistic environment • Complete a system-level preliminary design review Product design is stable ● ● NA NA ○ ● Design Review • Release at least 90 percent of design drawings • Test a system-level integrated prototype Manufacturing processes are mature ● ○ ● ● Production Start • Demonstrate Manufacturing Readiness Level of at least 9, or critical processes are in statistical control • Demonstrate critical processes on a pilot production line • Test a production-representative prototype in its intended environment Current Status ○ ○ ○ ○ ● ○ ● Knowledge attained, ○ Knowledge not attained, … Information not available, NA Not applicable Page 151 GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Air Force, GPS III Program Technology Maturity, Design Stability, and Production Readiness The GPS III program office reported that all eight of its critical technologies are mature and that the design is stable. Lockheed Martin has delivered two of the 10 GPS III satellites to the Air Force, while the remaining eight are in various stages of production. In September 2017, the Air Force declared that the first GPS III satellite was available for launch. This declaration followed an Air Force investigation into performance anomalies observed in a propulsion subsystem common to multiple Air Force programs. Although the Air Force did not determine the root cause of the anomalies, it tested multiple GPS III propulsion subsystems and was satisfied with the performance. As a result, the program office certified that the first GPS III satellite met all documented requirements. While Defense Contract Management Agency officials indicated that there were no design changes to the subsystem, program officials stated that the Air Force introduced changes to how ground control operators fire the subsystem to raise a satellite into orbit and that the subsystem manufacturer introduced measures to ensure greater uniformity in the manufacturing process. The Air Force declared in August 2018 that the second satellite was available for launch, and the third satellite will be available for launch in May 2019. After prior problems with delayed deliveries of navigation payload components, the program office has noted improvements in the navigation payload subcontractor’s management and schedule performance over the past year. The program office continues to face delayed deliveries of certain other satellite components, such as a power regulation unit and the propulsion subsystem. The program office is working to reduce potential effects of propulsion subystem delivery delays by temporarily installing a non-flight propulsion subsystem on the affected satellites. This step allows the program to progress with assembly, integration, and testing while awaiting flight unit deliveries. Common Name: GPS III additional testing, including operational testing of the modified OCS with the GPS III satellite for all currently available GPS navigation signals beginning in January 2020. The Air Force plans for a second OCS modification in early 2020 to enable some of the GPS III M-code capabilities. Because of delays to the GPS Next Generation Operational Control System (OCX)—needed to enable the full range of GPS III capabilities—the GPS III program expects to accept delivery of at least the first nine satellites before beginning developmental and operational testing with OCX Block 1. The Air Force anticipates that these tests, planned to begin in 2023, will confirm GPS III’s modernized signal capabilities. This sequencing, however, introduces the possibility that testers will discover deficiencies to already produced or launched satellites—thus constraining the Air Force’s corrective options—and carries risk to overall GPS III cost, schedule, and performance. Program Office Comments and GAO Response We provided a draft of this assessment to the program office for review and comment. The program office provided technical comments, which we incorporated where appropriate. The program office indicated that it is focused on GPS III production, launch, and mission operations. It stated that the first GPS III satellite successfully launched on December 23, 2018, and continues to progress through on-orbit checkout and testing. The program office also said the second GPS III satellite recently arrived at the Astrotech Space Operations facility in Titusville, Florida, in preparation for a planned July 2019 launch. According to the program office, the third GPS III satellite is on track to be available for launch in May 2019, and satellites 4 through 10 are progressing through various stages of production. In addition, the program office stated that GPS III’s production is currently “robust” and has overcome past technical problems. We plan to seek further detail on the program’s manufacturing metrics as part of our next annual assessment. Other Program Issues The Air Force launched the first GPS III satellite on December 23, 2018, on a SpaceX Falcon 9 rocket. In preparation for the launch, the Air Force conducted a series of mission readiness events, culminating in a final mission rehearsal in November. Next, the Air Force will conduct 6 months of on-orbit testing, including testing the satellite’s non-navigation functions and turning on its navigation payload, according to program officials. The Air Force will not conduct operational testing of the navigation payload, however, until it modifies the Operational Control Segment (OCS) to operationally control the payload. That modification is currently scheduled to be fielded in October 2019. Afterward, the Air Force will begin 6 months of Page 152 GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Air Force, Common Name: GPS IIIF Global Positioning System III Follow-On (GPS IIIF) The Air Force's GPS IIIF program will build upon the efforts of the GPS III program to develop and field next generation GPS satellites to modernize and replenish the GPS satellite constellation. In addition to the capabilities built into the original GPS III design, GPS IIIF will provide new capabilities. These include a steerable, high-power military code (M-code) signal, known as Regional Military Protection, to provide warfighters with greater jamming resistance in contested environments. Program Essentials Milestone decision authority: Air Force Program office: El Segundo, CA Prime contractor: Lockheed Martin Contract types: FPI (development) Software development approach: Mixed Next major milestone: Design review (March 2020) Program Performance (fiscal year 2019 dollars in millions) First full estimate (09/2018) Latest (09/2018) Percentage change Development $3,214.50 $3,214.50 0.00% Procurement $6,216.80 $6,216.80 0.00% $428.70 $428.70 0.00% N/A N/A N/A 22 22 0.00% Unit cost Acquisition cycle time (months) Total quantities We could not calculate cycle time because initial capability depends on the availability of complementary systems. Total quantities comprise 2 development quantities and 20 procurement quantities. Funding and Quantities (fiscal year 2019 dollars in millions) Attainment of Product Knowledge As of January 2019 Status at Current Status Development Start Resources and requirements match • Demonstrate all critical technologies are very close to final form, fit and function within a relevant environment • Demonstrate all critical technologies in form, fit and function within a realistic environment • Complete a system-level preliminary design review Product design is stable ● ● NA NA NA NA Design Review • Release at least 90 percent of design drawings NA NA • Test a system-level integrated prototype NA NA Manufacturing processes are mature Production Start • Demonstrate Manufacturing Readiness Level of at least 9, or critical processes are in statistical control • Demonstrate critical processes on a pilot production line • Test a production-representative prototype in its intended environment NA NA NA NA NA NA ● Knowledge attained, ○ Knowledge not attained, … Information not available, NA Not applicable Page 153 GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Air Force, GPS IIIF Program Technology Maturity The GPS IIIF program office reports that its two critical technologies—an L-band traveling wave tube amplifier (TWTA) and a digital waveform generator—are mature to the level generally required for the program to begin development. The Air Force certified that the program has demonstrated the system’s technologies in a relevant environment prior to development start. The TWTA is designed to provide the signal amplification required for the GPS IIIF satellite’s regional military protection capability. The Air Force’s Space and Missile Systems Center (SMC) conducted an independent assessment prior to the program’s development start and found that the TWTA provides lower technical risk and is more mature than the alternative amplifier the program office had previously considered. Program officials note that TWTAs have proven performance on orbit in other DOD space programs, such as the Advanced Extremely High Frequency satellite. Although the TWTAs have not been proven in space at the power level required for GPS IIIF, SMC reports that the contractor has successfully tested an engineering model in a representative environment. The GPS IIIF navigation payload will incorporate a new digital waveform generator to produce the military and civil waveforms, or signals, that provide the basis of GPS navigation. SMC assessed the digital waveform generation as a low-risk function, but noted that integrating the navigation signal functions into a single electronics box presents a higher risk. Of note, integrating signal functions proved difficult in developing the GPS III program’s navigation payload. The contractor demonstrated the digital waveform generator with hardware that was approaching maturity and which SMC assessed as being close to flight design. Program officials noted that the shift from the partially analog navigation payload of GPS III to the completely digital payload of GPS IIIF streamlines aspects of the manufacturing process. Common Name: GPS IIIF development start is an acquisition best practice associated with lower cost and schedule growth. The GPS IIIF program schedule plans for the contractor to deliver a production-ready satellite design soon after contract award. The program office is currently working toward a critical design review in spring 2020 to demonstrate design stability in advance of a production decision, planned for 6 months later. According to program officials, the GPS IIIF satellite will heavily leverage mature technologies from GPS III, primarily in the satellite bus design. The program plans to test a system-level integrated prototype that includes all key subsystems and components, but with less redundancy than the final configuration, prior to integrating and testing the first GPS IIIF satellite. Program Office Comments We provided a draft of this assessment to the program office for review and comment. According to the program office, since the Air Force awarded Lockheed Martin the fixed-price-type contract for GPS III FollowOn satellites in 2018, the program has been working closely with the contractor to validate contractor delivery milestones to ensure no schedule growth occurs. The program office also reported that it recently completed an integrated baseline review in March 2019 and is currently pursuing a critical design review to validate a production-ready satellite design. Design Stability The Air Force waived the requirement that the program conduct a preliminary design review prior to development start, in part to expedite a contract award given DOD’s critical national security need for the GPS IIIF’s unique capabilities. As part of required certifications at development start, the Air Force certified that GPS IIIF had already demonstrated a level of technological maturity beyond preliminary design review. According to the Air Force, the program demonstrated this level of maturity in the production readiness feasibility work it conducted between May 2016 and June 2017. However, we have found that completing a preliminary design review prior to Page 154 GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Air Force, Common Name: KC-46A KC-46 Tanker Modernization Program (KC-46A) The Air Force’s KC-46A program plans to convert an aircraft designed for commercial use into an aerial refueling tanker for operations with Air Force, Navy, Marine Corps, and allied aircraft. The program is in the first of three planned phases to replace roughly a third of the Air Force’s aging aerial refueling tanker fleet, comprised mostly of KC135s. The Air Force has designed the KC-46A to improve on the KC135’s refueling capacity, efficiency, and capabilities for cargo and aeromedical evacuation, and to integrate defensive systems. Program Essentials Milestone decision authority: Air Force Program office: Wright-Patterson Air Force Base, OH Prime contractor: Boeing Contract types: FPI (development) FFP (production) Software development approach: Mixed Next major milestone: Start of operational testing (May 2019) Program Performance (fiscal year 2019 dollars in millions) First full estimate (02/2011) Latest (01/2019) Percentage change Development $7,671.90 $6,313.80 -17.7% Procurement $37,253.70 $31,450.10 -15.6% $274.12 $225.76 -17.6% 78 112 +43.6% 179 179 0.0% Unit cost Acquisition cycle time (months) Total quantities Total quantities comprise 4 development quantities and 175 procurement quantities. Funding and Quantities (fiscal year 2019 dollars in millions) Attainment of Product Knowledge As of January 2019 Status at Development Start Resources and requirements match • Demonstrate all critical technologies are very close to final form, fit and function within a relevant environment • Demonstrate all critical technologies in form, fit and function within a realistic environment • Complete a system-level preliminary design review Product design is stable ● ○ ○ ● ● ● Design Review • Release at least 90 percent of design drawings ⋯ NA ○ ● ● ○ ● ● ○ • Test a system-level integrated prototype Manufacturing processes are mature ● Production Start • Demonstrate Manufacturing Readiness Level of at least 9, or critical processes are in statistical control • Demonstrate critical processes on a pilot production line • Test a production-representative prototype in its intended environment Current Status ● Knowledge attained, ○ Knowledge not attained, … Information not available, NA Not applicable Page 155 GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Air Force, KC-46A Program Technology Maturity and Design Stability The KC-46A’s three critical technologies—two software modules related to situational awareness and a display that allows the crew to monitor aerial refueling—are fully mature. At its 2013 critical design review (CDR), the program had released over 90 percent of design drawings. However, as reflected in our attainment of product knowledge graphic, we lack insight into design stability from CDR forward because (1) the discovery of aircraft wiring deficiencies post-CDR and consequent re-design of the wiring system called into question the accuracy of the design drawing information available at CDR, and (2) after CDR, the program stopped using drawings to assess design and instead began using other methods to measure design status. Based on developmental testing, the Air Force identified three critical refueling deficiencies that require design changes. First, refueling operators are not aware of instances when the boom strikes receiver aircraft. These strikes are due to a shortcoming with the system that provides information to their display screens and may damage the low observable aircraft coating on some receivers, possibly making stealth receiver aircraft visible to radar. Second, according to program officials, the aerial refueling operator’s screen does not provide sufficient visual sharpness and adaptation to changing background and lighting to allow for safe refueling in all environmental conditions. The officials stated Boeing will make software or hardware changes related to these two deficiencies without cost to the government. Third, program officials reported that the boom remains too stiff during refueling attempts with lighter receiver aircraft, which could cause it to strike the receiver aircraft. Boeing is working to resolve this issue, but program officials said the Air Force will have to fund implementation of any corrective fixes because the system was designed to an international standard and the Air Force needed it to be designed to a lower stiffness standard. Program officials estimate that it will take 3 to 4 years to correct these deficiencies. As of February 2019, Boeing has completed 92 percent of the development test program. All remaining tests relate to the wing aerial refueling pods. Until this testing is complete, Boeing may find additional technical issues that could require further design changes. Common Name: KC-46A aircraft, and is in the process of producing 40 additional low-rate initial production aircraft. Boeing is completing the majority of production work on its 767 aircraft production line. According to program officials, Boeing is completing some production work out of sequence due to the earlier wiring and other design problems and expects to resolve these by early 2019. The program intends to purchase approximately 38 percent of its total planned number of production aircraft during low-rate initial production. In 2017, the Under Secretary of Defense for Acquisition, Technology, and Logistics approved this strategy to avoid a break in the production line. Other Program Issues The original development contract required Boeing to deliver the first 18 aircraft with nine sets of wing aerial refueling pods by August 2017. However, because of wiring problems, test and certification delays, and other setbacks, the Air Force did not begin accepting aircraft until January 2019. As of that time, the Air Force has accepted four KC-46A aircraft that are capable of refueling other aircraft through two of three types of planned refueling options. According to program officials, the Air Force is withholding 20 percent of its payments to Boeing until Boeing demonstrates that it meets contract specifications. Boeing now plans to deliver the required aircraft and refueling pods in June 2020, at which time the aircraft will be capable of all three planned types of refueling. Under this plan, Boeing will complete work on the development contract 34 months later than initially planned. Program officials negotiated considerations from Boeing to account for lost military tanker capability associated with the delivery delays, such as obtaining additional training at no cost to the government for KC-46A pilots and maintenance personnel and support for the aircrew training system. Officials reported that the Air Force and Boeing finalized the contract modification, including these considerations, in December 2018. Program Office Comments We provided a draft of this assessment to the program office for review and comment. The program office provided technical comments, which we incorporated where appropriate. Production Readiness Program officials said that Boeing has not demonstrated manufacturing readiness to the desired level due to the extension of low-rate initial production. However, they said the Federal Aviation Administration has certified Boeing’s production process. As of February 2019, Boeing has manufactured four development aircraft, delivered six low-rate production Page 156 GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Air Force, Common Name: MGUE Increment 1 Military Global Positioning System (GPS) User Equipment (MGUE) Increment 1 The Air Force’s MGUE program is developing GPS receivers that will be compatible with the military’s next-generation GPS signal, military code (M-code). The modernized receiver cards will provide U.S. forces with enhanced position, navigation, and timing capabilities, and improved resistance to threats, such as jamming efforts by adversaries. Increment 1, assessed here, is developing receiver test cards for aviation, maritime, and ground platforms. The Air Force approved the Increment 2 acquisition strategy in 2018 for development of smaller receiver cards and a modernized handheld receiver. Program Essentials Milestone decision authority: Air Force Program office: El Segundo, CA Prime contractors: L-3 Technologies, Raytheon, Collins Aerospace Contract type: CPIF/CPFF/FFP (development) Software development approach: Other tailored approach Next major milestone: Start of operational testing (February 2020) Program Performance (fiscal year 2019 dollars in millions) First full estimate (01/2017) Latest (06/2018) Percentage change Development $1,555.80 $1,476.00 -5.1% Procurement $0.0 $0.0 0.0% Unit cost N/A N/A N/A Acquisition cycle time (months) N/A N/A N/A 0 0 0.0% Total quantities According to program officials, the latest estimate reflects funding approved to date, but they will seek funding to match the approved full estimate. Funding and Quantities (fiscal year 2019 dollars in millions) Attainment of Product Knowledge As of January 2019 Status at Current Status Development Start Resources and requirements match • Demonstrate all critical technologies are very close to final form, fit and function within a relevant environment • Demonstrate all critical technologies in form, fit and function within a realistic environment • Complete a system-level preliminary design review Product design is stable ● ○ ● ● ○ ● Design Review • Release at least 90 percent of design drawings NA NA • Test a system-level integrated prototype NA NA Manufacturing processes are mature Production Start • Demonstrate Manufacturing Readiness Level of at least 9, or critical processes are in statistical control • Demonstrate critical processes on a pilot production line • Test a production-representative prototype in its intended environment NA NA NA NA NA NA ● Knowledge attained, ○ Knowledge not attained, … Information not available, NA Not applicable Page 157 GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Air Force, MGUE Increment 1 Program Technology Maturity The MGUE Increment 1 program office has assessed four of its five critical technologies—M-code acquisition engine, M-code cryptography, selective availability antispoofing module cryptographic engine, and security protection—as fully mature. The program’s fifth critical technology, anti-spoof software, is nearing maturity. Anti-spoof technology is designed to prevent MGUE from acquiring and tracking false GPS signals. Program officials said that two contractors have delivered full anti-spoof capability, and the third contractor is providing incremental capabilities. They noted the difficulty of applying maturity standards to MGUE because the program is not delivering a final weapon, but rather a chip-based receiver card that individual programs will integrate into final weapons before it is ever fielded in an operational environment. For example, to assess maturity for anti-spoof technology, the program is using a laboratory simulator as a relevant environment because of restrictions regarding open air testing of spoofing signals. Program protection is included in the contractors’ security certification process—a key step for the services to procure and integrate receiver cards. Program officials said all testing has been completed for initial security certification and that two of the three contractors have received initial security certification. The remaining contractor is expected to complete the certification process in spring 2019. Design Stability We previously recommended that the program incorporate a critical design review into its acquisition plan to demonstrate the stability of the MGUE design, prior to testing the receiver card with the first platform. However, DOD formally eliminated this review from the acquisition program in January 2017. Program officials previously told us the design is stable, and any further design problems will be found in ongoing developmental testing. In last year’s assessment, we reported that contractors delivered final hardware test cards in 2017 for laboratory tests. Since that time, testing revealed deficiencies with the first ground card, which the program corrected in 2018. Additionally, the program office reports that it modified contracts with each contractor to incorporate a performance requirement known as “hot start”—the ability of receiver cards to transmit legacy signals to existing weapon systems. As of December 2018, government testing was ongoing for the two contractors’ cards that incorporated this capability. The third contractor expects to deliver this capability in spring 2019. Page 158 Common Name: MGUE Increment 1 Production Readiness The MGUE program office will not make a production decision because the program’s acquisition strategy does not include plans to procure cards beyond the final test articles. Instead, the program will end with operational testing of the first available test cards on four, military department-specific weapon systems. Integration and testing is ongoing on these systems and once completed, expected in April 2021, program offices will determine whether to undertake additional development and testing to integrate the receiver cards, identify their required quantities, and enter into a contract for production. While the military departments are developing some common solutions, individual program offices have the flexibility to pursue their own uncoordinated receiver card programs at different times and with different contractors. Other Program Issues The MGUE Increment 1 program is at risk of delaying schedule milestones because of delays from incorporating the hot start requirement and late discovery of hardware and software issues during government verification. The program office has taken steps to mitigate further delays, such as providing additional GPS simulators and test equipment to help resolve software problems. The program is also working with the Air Force Space Command to determine the extent of need for a more comprehensive hot start solution that would involve modifying the M-code signals, which could drive costs beyond the independent estimate. Such a change would also require changes to the GPS space and ground segments. Additionally, the manufacturer of the chips in the receiver cards plans to cease production as early as 2020, leading some program offices to consider procuring the chips in bulk. Program Office Comments We provided a draft of this assessment to the program office for review and comment. The program office provided technical comments, which we incorporated where appropriate. According to the program office, MGUE made progress this year demonstrating functionality and performance through card-level developmental testing of mature cards. The program stated that it expects to complete final verification testing of the first ground card in March 2019. It also said that two cards have met hardware reliability requirements, and all cards continue to demonstrate capabilities to meet user needs in various environments. The program office stated that, within the next year, it expects to integrate the final card of each type into the military departments’ lead platforms and make those cards available to the military departments for procurement. GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Air Force, Common Name: OCX Next Generation Operational Control System (OCX) Through its OCX program, the Air Force is primarily developing software to replace the existing Global Positioning System (GPS) ground control system. The Air Force intends for OCX software to help ensure reliable, secure delivery of position, navigation, and timing information to military and civilian users. The Air Force is developing OCX in blocks that each provide upgrades as they become available. We assessed the first three blocks: Block 0, for launch and limited testing of new satellites; Block 1, for satellite control and basic military signals; and Block 2, for modernized military and additional navigation signals. Program Essentials Milestone decision authority: Under Secretary of Defense for Acquisition and Sustainment Program office: El Segundo, CA Prime contractor: Raytheon Contract type: CPIF (development) Software development approach: Mixed Next major milestone: Contractor delivery (February 2021) Program Performance (fiscal year 2019 dollars in millions) First full estimate (11/2012) Latest (09/2018) Percentage change Development $3,707.60 $6,231.00 +68.1% Procurement $0.00 $0.00 0.0% $3,707.60 $6,231.00 +68.1% 55 113 +105.5% 1 1 0.0% Unit cost Acquisition cycle time (months) Total quantities The Air Force approved a third development acquisition program baseline for the restructured OCX program in September 2018. We calculated acquisition cycle time using the program’s initial capability date for Block 2. Funding and Quantities (fiscal year 2019 dollars in millions) Attainment of Product Knowledge As of January 2019 Status at Current Status Development Start Resources and requirements match • Demonstrate all critical technologies are very close to final form, fit and function within a relevant environment • Demonstrate all critical technologies in form, fit and function within a realistic environment • Complete a system-level preliminary design review Product design is stable ● ○ ● ● ○ ● Design Review • Release at least 90 percent of design drawings NA NA • Test a system-level integrated prototype NA NA Manufacturing processes are mature Production Start • Demonstrate Manufacturing Readiness Level of at least 9, or critical processes are in statistical control • Demonstrate critical processes on a pilot production line • Test a production-representative prototype in its intended environment NA NA NA NA NA NA ● Knowledge attained, ○ Knowledge not attained, … Information not available, NA Not applicable Page 159 GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Air Force, OCX Program Technology Maturity and Design Stability The OCX program continues software development activities necessary to demonstrate system performance, but has yet to fully mature the critical technologies that underpin the full OCX system. As recently as 2017, the program reported that it had matured 14 critical technologies. For this and our 2018 assessment, however, the program identified only five critical technologies, which are all nearing maturity, because nine of the 14 were delivered with Block 0. The OCX program office does not track the metrics we used for this assessment to measure design stability, such as the number of releasable design drawings, as OCX is primarily a software development effort. In September 2017, the OCX prime contractor, Raytheon, delivered Block 0. In December 2018, Block 0 successfully supported the Air Force launch of the first GPS III satellite. After launch, Block 0 began conducting 6 months of initial testing on the satellite. As of January 2019, Raytheon continues development work on Blocks 1 and 2. Officials report that under the terms of the development contract, Raytheon has approximately 2.5 years remaining to complete development of both blocks. Since development start, the program has incurred persistent cost and schedule growth, which the Air Force has attributed to numerous root causes. The causes included an unrealistic schedule, underestimated costs to fully implement the information assurance requirements, and poor contractor and government performance. In June 2016, the Secretary of the Air Force notified Congress of a critical statutory unit cost breach in the program. In October 2016, the Air Force restructured the program. Subsequently, the Air Force (1) combined delivery of Blocks 1 and 2, set a new contractor schedule, and repeated the program review associated with system development in June 2017; and (2) completed a comprehensive baseline review in April 2018. In September 2018, DOD approved the current baseline, which completed the process to put in place new cost and schedule estimates. Common Name: OCX estimates outside of the contractor and program office vary on when the contractor will deliver OCX, and when the Air Force will subsequently accept OCX, which is planned for 2 months later. Independent estimates forecast an additional 11 months or more are needed past June 2021 to reach acceptance. The program will surpass the approved OCX baseline if operations do not commence by April 2023. To achieve this milestone, the Air Force has to accept OCX delivery 7 months prior in September 2022. In 2018, the contractor reported it had increased the pace of software development activities compared to its past performance. Nonetheless, our analysis shows that the program faces numerous challenges that could negatively affect its cost and schedule. First, significant work and technical risk remain in the development plan, and the program has taken significantly more time to fix software defects than provided for in its schedule. Next, implementation of a new software development methodology took the program longer than planned and subsequently used up two-thirds of the allocated schedule reserve. At the same time, Raytheon has delayed by more than 1 year its planned decrease of OCX staffing levels, which suggests it is struggling to achieve the program’s new baseline plan. Finally, our analysis shows that the program’s 7-month developmental test schedule that occurs after acceptance is at risk of doubling in duration because of concurrent test events, test plan uncertainty, and risk of late discovery of problems. Program Office Comments We provided a draft of this assessment to the program office for review and comment. The OCX program office said that, to inform future testing, it is leveraging lessons learned from the completion of product testing on two areas of software development. Additionally, the OCX program office said that it is exploring options with the GPS Directorate’s lead development test organization to reduce schedule risk to the 7-month developmental testing period. These options, it said, include conducting a key risk reduction review and developmental test at the same time and allowing OCX to command some operational satellites before accepting OCX from the contractor. Other Program Issues Completion of the OCX program within the approved baseline requires both (1) timely delivery by the contractor and evaluation and acceptance by the Air Force, and (2) efficient completion of the planned 7month government-run post-acceptance developmental testing before beginning operations that signify the end of the program. The Air Force reported that the development contract with Raytheon ends in June 2021, which our analysis has shown is unlikely to provide enough time for the contractor to complete OCX development. Independent Page 160 GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Air Force, Common Name: SDB II Small Diameter Bomb Increment II (SDB II) The Air Force's Small Diameter Bomb Increment II (SDB II) is a joint interest program with the Navy and is designed to provide attack capability against mobile targets in adverse weather from extended range. It combines radar, infrared, and semi active laser sensors to acquire, track, and engage targets. It uses airborne and ground data links to update target locations, as well as a global positioning system and an inertial navigation system to ensure accuracy. SDB II will be integrated with Air Force and Navy aircraft, including the F-15E, F/A18E/F, and F-35. Program Essentials Milestone decision authority: Air Force Program office: Eglin Air Force Base, FL Prime contractor: Raytheon Contract types: FPI (development) FPI/FFP (low-rate initial production) Software development approach: Agile development Next major milestone: Complete operational testing (June 2019) Program Performance (fiscal year 2019 dollars in millions) First full estimate (10/2010) Latest (10/2018) Percentage change Development $1,841.50 $1,934.30 +5.0% Procurement $3,423.00 $2,571.90 -24.9% $0.31 $0.26 -14.4% 72 110 + 52.8% 17,163 17,163 0.00% Unit cost Acquisition cycle time (months) Total quantities Total quantities comprise 163 development quantities and 17,000 procurement quantities. Funding and Quantities (fiscal year 2019 dollars in millions) Attainment of Product Knowledge As of January 2019 Status at Resources and requirements match • Demonstrate all critical technologies are very close to final form, fit and function within a relevant environment • Demonstrate all critical technologies in form, fit and function within a realistic environment • Complete a system-level preliminary design review Product design is stable ● ○ ● ● ● ● Design Review • Release at least 90 percent of design drawings ⋯ ● ● ○ ● ● ○ ● ● ○ • Test a system-level integrated prototype Manufacturing processes are mature Production Start • Demonstrate Manufacturing Readiness Level of at least 9, or critical processes are in statistical control • Demonstrate critical processes on a pilot production line • Test a production-representative prototype in its intended environment Current Status Development Start ● Knowledge attained, ○ Knowledge not attained, … Information not available, NA Not applicable Page 161 GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Air Force, SDB II Program Technology Maturity, Design Stability, and Production Readiness The SDB II program has matured its critical technologies and has stabilized its system design. SDB II’s four critical technologies—guidance and control, multi-mode seeker, net-ready data link, and payload— are mature. However, in 2018 we reported that after production start, qualification and flight test failures revealed design deficiencies that would require hardware and software changes.However, the program reports that it has released 100 percent of its design drawings, though it does not track revisions to previously released drawings. Consequently, we do not have visibility into whether design changes have disrupted the design stability the program previously reported at its January 2011 critical design review. To address the deficiencies found in flight testing, the program released new software updates to improve stationary target performance. As a result, the program delayed the start of the operational testing and delayed completion of software development to January 2018, 5 months later than last year’s plans. The program office completed Government Confidence Tests (GCT) to demonstrate SDB II’s technology and design after 5 months of delays. Officials told us that the delays were due to software deficiencies and difficulties scheduling use of the test range. The program office reported that 22 of the 28 test shots were successful. Six of the GCTs did not achieve their objective because of software deficiencies, most of which involved problems with the aircraft relaying inaccurate or incomplete information to the weapon. To address the six failures, the program office conducted one re-fly test and successfully verified the system was ready for operational testing. Program officials stated that it began operational testing 3 months late because of the GCT delays. As of January 2019, officials said the program has conducted 49 of its planned 56 operational test mission scenarios and estimated completion of the remaining tests by June 2019. The program expects to declare initial operational capability in September 2019, 9 months later than anticipated. Despite these testing delays, program officials believe that a full-rate production decision in September 2022 is still achievable. SDB II has been in low-rate initial production since March 2017. However, the contractor has yet to bring its critical manufacturing process into statistical control. As of January 2019, officials said that the program office delivered a total of 598 units and reports that the contractor is currently producing the remaining 108 units for Lot 3, which are scheduled for delivery completion by August 2019. Officials stated that in 2018, the program exercised options for production Lots 4 and 5 for a total of 1,920 units. The Air Force is the Page 162 Common Name: SDB II sole customer for the first three production lots, but officials stated that both the Air Force and the Navy will procure units under Lot 4 and Lot 5. The program currently plans to purchase 58.5 percent of total SDB II production quantities during low-rate initial production. If a program’s low-rate initial production quantity exceeds 10 percent of the total production quantity, the program must provide a rationale for these quantities in a report to Congress. In December 2017, the Air Force reported that SDB II exceeded the 10 percent level due to a delay in the completion of operational test and evaluation caused by schedule revisions to the F-35 program, a threshold aircraft. The Air Force further reported that it needed the increased quantities to provide production-configured or representative articles for operational tests, to establish an initial production base for the system, and to permit an orderly increase in the production rate for the system sufficient to lead to full-rate production following operational testing. Other Program Issues In 2017, the contractor reported cost growth on the first three production lots. Program officials said they expect this growth to occur in the first five production lots because of the contractor’s overly aggressive cost proposals early in the system development phase and rework of initial production lots. The program stated that the government’s liability for cost growth is capped by the fixed-price incentive production contract’s terms and that the contractor is exploring opportunities to reduce production costs. Officials stated that the program has not yet negotiated the pricing for Lot 6 and Lot 7, and they expect it to be higher than Lots 1-5, which they reported were pre-priced options under a competively awarded contract. Program Office Comments We provided a draft of this assessment to the program office for review and comment. The program office provided technical comments, which we incorporated where appropriate. The program office stated that as of April 2019, it has seven operational test mission scenarios to complete. According to the program office, a January 2019 review of the weapon’s performance in operational testing identified deficiencies with the weapon datalink communication, and a software update is planned for operational test missions in May 2019. The program office also said that in February 2019 it reported an acquisition program baseline schedule deviation for the F-15E required assets available threshold date of January 2019, with an estimated revised date of September 2019. For Lots 6 and 7, the program office said it anticipates a proposal from Raytheon by April 30, 2019. GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Air Force, Common Name: Space Fence Inc 1 Space Fence Ground-Based Radar System Increment 1 (Space Fence Inc 1) The Air Force’s Space Fence Inc 1 program is developing a large, ground-based radar to detect and track objects in low and medium Earth orbit and provide data to a space surveillance network. Space Fence will use high radio frequencies to detect and track more and smaller objects than previous systems. The Air Force awarded a development and production contract for the first radar site in June 2014. The contract included an option that, if exercised, would enable the Air Force to acquire a second site under a separate program. Program Essentials Milestone decision authority: Air Force Program office: Hanscom Air Force Base, MA Prime contractor: Lockheed Martin Contract type: FPI (development and production) Software development approach: Incremental Next major milestone: Start operational testing (March 2019) Funding and Quantities (fiscal year 2019 dollars in millions) Program Performance (fiscal year 2019 dollars in millions) First full estimate (06/2014) Latest (06/2018) Percentage change Development $1,684.80 $1,588.6 -5.7% Procurement $0.0 $0.0 0.00% $1,684.77 $1,588.65 -5.7% 124 124 0.00% 1 1 0.00% Unit cost Acquisition cycle time (months) Total quantities Attainment of Product Knowledge As of January 2019 Status at Resources and requirements match • Demonstrate all critical technologies are very close to final form, fit and function within a relevant environment • Demonstrate all critical technologies in form, fit and function within a realistic environment • Complete a system-level preliminary design review Product design is stable ● ○ ● ● ● ● Design Review • Release at least 90 percent of design drawings • Test a system-level integrated prototype Manufacturing processes are mature ● ○ ● ● Production Start • Demonstrate Manufacturing Readiness Level of at least 9, or critical processes are in statistical control • Demonstrate critical processes on a pilot production line • Test a production-representative prototype in its intended environment Current Status Development Start NA NA NA NA NA NA ● Knowledge attained, ○ Knowledge not attained, … Information not available, NA Not applicable Page 163 GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Air Force, Space Fence Inc 1 Technology Maturity, Design Stability, and Production Readiness The Space Fence Inc 1 program has matured its critical technologies, stabilized the system design, and completed planned production and assembly activities. The Space Fence Inc 1 program did not have a production start milestone as it is producing only a single radar, and thus our production metrics do not apply to this program. In February 2015, the Air Force completed a technology readiness assessment that showed that all seven of the program’s critical technologies are fully mature. Space Fence’s critical technologies provide capabilities for transmitting and receiving radar signals from the radar array. By its June 2015 critical design review, the program attained design stability with release of 100 percent of system drawings—a level that exceeds the best practices standard. Since early 2016, the contractor has tested production-representative hardware and operational software on a prototype testbed of the Inc 1 radar. Program officials stated that while the prototype testbed was not a contract requirement, they expect the contractor will choose to keep the testbed operational throughout the life of the Space Fence program to allow for testing of future upgrades and changes. The prime contractor completed production of the radar components in February 2018, followed by installation and checkout of the radar hardware in March 2018. The program identified a production deficiency that affected the power supply cabinets. Tests showed that capacitor components within the cabinets were failing in testing at an unexpectedly high rate, which could have led to performance problems had it not been discovered prior to the start of operations. To resolve this deficiency, the program replaced the capacitors with an upgraded design. This problem did not cause any schedule delays to the overall installation and checkout of the radar as there was sufficient margin in the schedule to cover the time needed for the replacement process. Other Program Issues The Air Force expects Space Fence Inc 1 to provide performance sufficient to declare initial operational capability. However, the program only plans to declare full capability if and when a second radar—identical in capability and size to Inc 1—becomes operational. The Air Force had included development and production of a second radar, which would comprise an Inc 2 program, as a contract option. However, officials state that because this option was not exercised by August 2018, the program likely will need to renegotiate the second site pricing, and the renegotiated pricing may be higher than what was negotiated at the time of the original June 2014 contract award. Page 164 Common Name: Space Fence Inc 1 Program officials stated that the Air Force has yet to budget for an Inc 2 program. According to the program office, if the Air Force does not budget for Inc 2 by fiscal year 2021, the capability will become significantly more costly to acquire. Namely, after fiscal year 2021, the program office reports that it will have completed the Inc 1 program to include reassignment of staff and ramping down of the program office. Once operational, the Space Fence Inc 1 radar will provide a significantly increased number of space object observations than previous systems. In part to effectively utilize this data, the Air Force attempted to acquire new data processing capabilities under its Joint Space Operations Center Mission System (JMS) program. However, in late 2018, the Air Force ended development work on JMS because of numerous technical challenges the program encountered. Consequently, JMS will not have the capability to process Space Fence Inc 1 data. For the near term, the Air Force has planned an alternative for completing Space Fence operational testing and beginning operations. Specifically, the Space Fence program has coordinated with the Air Force’s Non-traditional Data Pre-Processor (NDPP) program and established a method to transfer Space Fence data to existing legacy Air Force space situational awareness data processing systems. According to Air Force officials, updates to these legacy systems will enable them to use some of the Space Fence data to perform the space situational awareness mission as soon as Space Fence Inc 1 is operational. However, to fully utilize what Space Fence will provide, the legacy systems will need to be replaced by a modernized system. The Air Force hopes to complete these modernizations and retire the legacy system in December 2020. Program Office Comments We provided a draft of this assessment to the program office for review and comment. Program officials stated that since our assessment, the program has completed contractor testing and is preparing to enter developmental testing. This represents a 4-month delay from previously reported testing dates, with operational testing now expected to run from July 2019 through September 2019, and initial operational capability (IOC) occurring in November 2019. The program noted that Space Fence is still on track to meet the IOC threshold of January 2020 established in the program’s baseline. The program also stated that the system has successfully exchanged test messages with NDPP, and that some key stakeholders have toured the Space Fence site and commended the contractor on the readiness of the program. The program said that the stakeholders also witnessed system demonstrations showing live Space Fence data, which according to the program, left the stakeholders satisfied that the system is on track to meet mission needs. GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Air Force Common Name: UH-1N Replacement Utility Helicopter (UH-1N) Replacement The UH-1N Replacement program aims to replace the Air Force’s 63helicopter fleet, initially manufactured in the 1960s. The UH-1N helicopter’s primary missions are securing intercontinental ballistic missile sites and convoys and transporting senior government officials in the National Capital Region. However, the program office reported that the current fleet does not comply with DOD’s nuclear weapons security guidance and cannot meet all mission requirements. The program plans to acquire 84 helicopters, an integration laboratory, a training system, support and test equipment, and associated software. Program Essentials Milestone decision authority: Air Force Program office: Wright-Patterson Air Force Base, OH Prime contractor: Boeing Contract type: FFP (integration) Software development approach: TBD Next major milestone: Design review (November 2019) Program Performance (fiscal year 2019 dollars in millions) First full estimate (09/2018) Latest (09/2018) Percentage change Development $579.10 $579.10 0.00% Procurement $2,463.60 $2,463.60 0.00% $40.06 $40.06 0.00% Acquisition cycle time (months) 60 60 0.00% Total quantities 84 84 0.00% Unit cost The Air Force approved an acquisition program baseline for UH-1N Replacement in September 2018. Total quantities comprise 4 development quantities and 80 procurement quantities. Funding and Quantities (fiscal year 2019 dollars in millions) Attainment of Product Knowledge As of January 2019 Status at Current Status Development Start Resources and requirements match • Demonstrate all critical technologies are very close to final form, fit and function within a relevant environment • Demonstrate all critical technologies in form, fit and function within a realistic environment • Complete a system-level preliminary design review Product design is stable NA NA NA NA NA NA Design Review • Release at least 90 percent of design drawings NA NA • Test a system-level integrated prototype NA NA Manufacturing processes are mature Production Start • Demonstrate Manufacturing Readiness Level of at least 9, or critical processes are in statistical control • Demonstrate critical processes on a pilot production line • Test a production-representative prototype in its intended environment NA NA NA NA NA NA ● Knowledge attained, ○ Knowledge not attained, … Information not available, NA Not applicable Page 165 GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Air Force UH-1N Replacement Program Technology Maturity and Design Stability The UH-1N Replacement program has identified no critical technologies and has begun system integration activities ahead of its November 2019 critical design review. The UH-1N Replacement program plans to integrate an existing helicopter certified by the Federal Aviation Administration with previously developed—or non-developmental—items. As a result, the program office expects to reduce technical risk and facilitate an expedited delivery schedule for aircraft. Program officials said that through market research, they concluded this approach could meet the program’s performance requirements and avoid development of new technologies. In September 2018, the Air Force approved an acquisition program baseline and the program entered engineering and manufacturing development. Although the program is considered non-developmental, the Air Force determined that it needed this phase to facilitate contractor modifications (integration and testing) to the existing helicopter design. The program office anticipates that these integration and test activities will require up to 3 years and has scheduled a low-rate initial production decision for September 2021. This schedule, however, is underpinned by the assumption that the existing helicopter’s engine and the propulsion system meet program requirements. Program officials stated that if tests show that either of these systems requires additional development or modification to achieve certifications, the program office will likely have to extend its development phase. Any additional development efforts would likely portend program delays and cost increases. Officials stated that they awarded the UH-1N Replacement contract to Boeing (in September 2018) based in part on the program’s assessment of Boeings’s ability to provide technical components that can be certified through testing. Common Name: UH-1N Replacement development will be important for the program to maintain its cost and schedule. We have previously found that software development efforts that occur after production start place programs at risk of schedule delays and cost growth. Other Program Issues The Air Force Cost Analysis Agency, which produced the service cost position for the program, predicted that Boeing may lose money on the contract starting in fiscal year 2023 and may seek to renegotiate pricing as a result. According to the Agency, strict adherence to requirements can help the government avoid cost increases, particularly when the contractor may seek to renegotiate pricing, because changes to contract requirements could lead to a contract renegotiation. The program office has also not yet determined whether the existing helicopter can meet DOD’s cybersecurity requirements. Opportunities to change the design to implement additional cybersecurity controls are limited under the program’s non-developmental item acquisition strategy. In addition, program documentation indicates that the Air Force may not implement some cybersecurity controls. If the helicopter is not able to meet cybersecurity requirements, the program office will have to determine whether and how to mitigate the deficiencies. Program officials said they are implementing a cybersecurity risk management framework and standing up a cyber security working group to understand the specific risks. Program Office Comments We provided a draft of this assessment to the program office for review and comment. The program office did not have any comments. Production Readiness Ahead of the low-rate initial production decision, the program plans to establish key metrics for assessing the UH-1N Replacement helicopter’s production readiness. The program office indicated that it will involve the contractor, Boeing, in these future efforts. Officials explained that they anticipate Boeing will have valuable insight into the appropriate metrics because it has already produced the baseline helicopter for commercial purposes. Program officials said these metrics will be established after critical design review, which is planned for November 2019. Similarly, the program has yet to determine a timeframe for reviewing and establishing software development metrics with the contractor, or for completing software development and integration. The timing of software Page 166 GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Air Force Common Name: B-52 RMP B-52 Radar Modernization Program (B-52 RMP) The Air Force’s B-52 RMP supports nuclear and conventional operations by replacing the current APQ-166 radar on all 76 B-52H aircraft in the Air Force inventory. This modernization will allow the Air Force to fully utilize the capabilities of the B-52H aircraft to employ an array of nuclear and conventional weapons and to perform missionessential navigation and weather avoidance functions. Program Essentials Current Status Milestone decision authority: Air Force Program office: Wright-Patterson Air Force Base, OH Prime contractor: Boeing Contract type: CPFF (risk reduction and requirements development) Software development approach: Mixed Next major milestone: Development start (September 2020) The B-52 RMP program has completed the initial subsystem specifications for the radar, radome, and crew stations, and is currently defining performance requirements in preparation for the development contract award in 2020. Additionally, according to program officials, vendors have submitted proposals for radar source selection and the program expects to make a decision in June 2019. The Air Force plans to modify existing training systems, or develop new ones, in support of the B-52 RMP. This work will affect all three Air Force Weapon System Trainers (WST)—the WST Training Systems Integration Laboratory and both B-52 Offensive Station Maintenance Trainers—which are used to help aircrews learn the proper techniques and handling of the system. Estimated Program Cost (FY 2019 dollars in millions) The program office plans to release a developmental request for proposal in late July 2019. The program has yet to identify any critical technologies, but program officials said they are developing a technology readiness assessment plan as part of the developmental request for proposal and expect to identify candidate technologies as part of that effort by the fourth quarter of fiscal year 2019. Consistent with acquisition best practices, the program plans to conduct a preliminary design review in July 2020, prior to development start in September 2020. Attainment of Technology Maturation Knowledge As of January 2019 Planned Quantities ○ ● Conduct competitive prototyping Validate requirements ● Knowledge attained, ◐ Knowledge planned, Complete technology readiness assessment Complete preliminary design review ◐ ◐ ○ Knowledge not attained, … Information not available, NA Not applicable Program Office Comments We provided a draft of this assessment to the program office for review and comment. The program office stated that B-52 RMP is executing its March 2018 approved acquisition strategy, as planned. Page 167 GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Air Force Common Name: VC-25B VC-25B Presidential Aircraft Recapitalization (VC-25B) Through its VC-25B program, the Air Force is replacing the current two VC-25A presidential aircraft with two modified Boeing 747-8 aircraft. The commercial aircraft will be uniquely modified to provide the President of the United States, staff, and guests with safe and reliable air transportation with the same level of security and communications available in the White House. Aircraft modifications will include electrical power upgrades, a mission communication system, military avionics, and other systems. Program Essentials Current Status Milestone decision authority: Under Secretary of Defense for Acquisition and Sustainment Program office: Wright-Patterson Air Force Base, OH Prime contractor: Boeing Contract type: CPFF/CPAF/FFP (studies, engineering services, commercial aircraft, and development) Software development approach: Other Next major milestone: Design review (August 2019) The program reported that in January 2016, the Air Force awarded a solesource contract to Boeing for VC-25B risk reduction activities and that it has since modified the contract based on the different phases of development work. The Air Force purchased two Boeing 747-8 aircraft in 2017. Estimated Program Cost (FY 2019 dollars in millions) Since then, the program has been refining system requirements and maturing sub-system designs. The program conducted a system-level preliminary design review in October 2018 and established a baseline for cost, schedule, and performance in December 2018. We are assessing VC25B in a one-page format this year because the program baseline was approved late in our review. We plan to assess it in a two-page format going forward. According to program officials, the Air Force is incorporating lessons learned from the KC-46 program, another commercial derivative aircraft, as appropriate. For example, the VC-25B team is implementing a new approach to better facilitate wire design, routing and installation processes. Program officials stated that they expect to definitize the engineering and manufacturing contract by the third quarter of 2019. The program office plans to start modification of both aircraft as early as 2020 after the design is stable. By fiscal year 2024, the Air Force plans to accept delivery of aircraft capable to support presidential missions. Program Office Comments Planned Quantities We provided a draft of this assessment to the program office for review and comment. The program office provided technical comments, which we incorporated where appropriate. . Page 168 GAO-19-336SP Weapon Systems Annual Assessment Lead Component: Air Force, Common Name: WSF Weather System Follow-On (WSF) The Air Force’s polar–orbiting WSF satellite is intended to contribute to a family of space-based environmental monitoring (SBEM) systems by providing 3 of 11 mission critical capabilities in support of military operations. WSF is being developed to conduct remote sensing of weather conditions, such as wind speed and direction at the ocean’s surface, and provide real-time data to be used in weapon system planning and weather forecasting models. The family of SBEM systems replaces the Defense Meteorological Satellite Program. Program Essentials Current Status Milestone decision authority: Air Force Program office: El Segundo, CA Prime contractor: Ball Aerospace and Technologies Corporation Contract types: FFP/CPIF (design, risk reduction, development, fabrication, integration, test, and operations) Software development approach: Mixed Next major milestone: Development start (March 2019) WSF is to have two payloads: (1) a microwave imager to collect data on ocean surface vector wind and tropical cyclone intensity and (2) an energetic charged particle sensor to collect space weather data. The Air Force awarded a contract in November 2017 for WSF system design, with options for development and delivery of up to two WSF satellites, if needed. The Air Force plans to start system development in March 2019 and launch a satellite in late 2023. Estimated Program Cost (FY 2019 dollars in millions) As a precursor to WSF, the Air Force undertook a technology demonstration program using an existing microwave sensor and planned to launch it in June 2018. However, the Air Force did not complete the planned launch because of problems with the flight software. The Air Force is now working toward a potential launch in 2021 to the International Space Station. The timely launches of the demonstration and WSF, according to the acquisition strategy, are critical to mitigate potential capability gaps. Currently, WindSat, a payload operating over 13 years beyond its design life, is the only capability that fully meets the service’s needs for ocean surface vector wind data. The program continues to assess the maturity level of critical technology elements. Program officials stated that they plan to complete a technology readiness assessment in February 2019, shortly before the program’s March 2019 development start. Attainment of Technology Maturation Knowledge As of January 2019 Planned Quantities Conduct competitive prototyping NA ◐ Validate requirements ● Knowledge attained, ◐ Knowledge planned, Complete technology readiness assessment Complete preliminary design review ◐ ● ○ Knowledge not attained, … Information not available, NA Not applicable Program Office Comments We provided a draft of this assessment to the program office for review and comment. The program office provided technical comments, which we incorporated where appropriate. Page 169 GAO-19-336SP Weapon Systems Annual Assessment Page 170 GAO-19-33GSP Weapon Systems Annual Assessment JOINT DOD PROGRAM ASSESSMENTS GAO Assessed 1 of 3 Joint DOD-wide Current Major Defense Acquisition Programs 1 Cu rrent program (F-35 Lightning II) Other programs (Not assessed) programs The F-35 Lightning II Program Costs are 96 Percent of the $374 Billion Joint DOD-wide Estimated Total Acquisition Costs (Fiscal Year 2019 dollars in billions) $14 Current I We assessed one current program?F35 Lightning ll. We did not assess programs that had yet to enter technology development or had already progressed into full-rate production. F-35 Lightning II had Cost Growth and Schedule Delays Program complete Programs in system Total Acquisition Costs for the F-35 Lightning II Program Increased after Knowledge Points (KP) (Fiscal Year 2019 dollars in billions) $76.7 $51.9 $9.1 KP1 to KP2 to Post - KP2 KP3 KP3 F-35 Lightning ll Did Not Attain Knowledge at Key Points KP1 Oof1 current programs attained knowledge KP2 Oof1 current programs attained knowledge KP3 Oof1 current programs attained knowledge F-35 Lightning Acquisition Cycle Time Increased by 62 Months from First Full Estimate (FFE) to Current Estimate development .2 taggers: cycle time to current estimate Number of months Program age Note: Bubble size is based on the program's currently estimated future acquisition funding needed. Cost growth of more than 15 percent andfor schedule delays of more than 6 months I Cost and schedule analysis is based on estimates from the December 2017 selected acquisition report. I The Under Secretary of Defense forAcquisition and Sustainment was the milestone decision authority in 2017 for F-35 Lightning II. I The F-35 Lightning program reported that it does not know the percent of planned acquisition costs that are speci?cally for software. I The F-35 Lightning ll program reported that it does not know the percent of work tasks that are software related. I DOD received single offers for its solicitations for its currently reported major procurement contracts for F-35 Lightning II. Source: GAO analysis of Department of Defense data. GAO-19-3368P Page 171 GAO-19-3368P Weapon Systems Annual Assessment Joint DOD Program Assessmenta 2-page assessment F-35 Lightning II (F-35) Page number 173 a We abbreviate the following contract types in the individual assessment: cost-plus-fixed-fee (CPFF), cost-plus-incentive-fee (CPIF), and fixed-price incentive (FPI). Page 172 GAO-19-336SP Weapon Systems Annual Assessment Lead Component: DOD Common Name: F-35 F-35 Lightning II (F-35) DOD is developing and fielding a family of fifth generation strike fighter aircraft integrating stealth technologies with advanced sensors and computer networking capabilities for the United States Air Force, Marine Corps, and Navy, eight international partners, and four foreign military sales customers. The family is comprised of three aircraft variants. The Air Force’s F-35A variant will complement its F-22A fleet and is expected to replace the air-to-ground attack capabilities of the F16 and A-10. The Marine Corps’ F-35B variant will replace its F/A-18 and AV-8B aircraft. The Navy’s F-35C variant will complement its F/A18E/F aircraft. Program Essentials Milestone decision authority level: Under Secretary of Defense for Acquisition and Sustainment Program office: Arlington, VA Prime contractor: Lockheed Martin; Pratt & Whitney Contract type: FPI/CPIF/CPFF (aircraft low-rate initial production) FPI/CPIF (engine low-rate initial production) Software development approach: Mixed Next major milestone: F-35C initial capability (February 2019) Funding and Quantities (fiscal year 2019 dollars in millions) Program Performance (fiscal year 2019 dollars in millions) First full estimate (10/2001) Latest (06/2018) Percentage change Development $43,642.80 $66,227.30 +51.7% Procurement $193,622.50 $288,950.50 +49.2% $83.49 $145.79 +74.6% 175 237 +35.4% 2,866 2,470 -13.8% Unit cost Acquisition cycle time (months) Total quantities Total quantities comprise 14 development quantities and 2,456 procurement quantities. Attainment of Product Knowledge As of January 2019 Status at Resources and requirements match • Demonstrate all critical technologies are very close to final form, fit and function within a relevant environment • Demonstrate all critical technologies in form, fit and function within a realistic environment • Complete a system-level preliminary design review Product design is stable ○ ○ ○ ● ● ● Design Review • Release at least 90 percent of design drawings • Test a system-level integrated prototype Manufacturing processes are mature ○ ○ ● ● Production Start • Demonstrate Manufacturing Readiness Level of at least 9, or critical processes are in statistical control • Demonstrate critical processes on a pilot production line • Test a production-representative prototype in its intended environment Current Status Development Start ○ ○ ○ ● ● ● ● Knowledge attained, ○ Knowledge not attained, … Information not available, NA Not applicable Page 173 GAO-19-336SP Weapon Systems Annual Assessment Lead Component: DOD F-35 Program Technology Maturity and Design Stability All of the F-35’s critical technologies are mature and the baseline engineering drawings are complete for all three aircraft variants. The program office completed the final development test flights for the baseline program in April 2018, but continues to address over 900 deficiencies identified with the aircraft’s performance prior to the end of development testing. For example, the program is developing a new helmet mounted display, which will resolve an existing green glow effect that can distort a pilot’s vision during night time carrier landings. Program officials expect installation of some of the new displays in 2019. The program office is also testing and integrating software updates to resolve other deficiencies, but it did not fully resolve over 800 other deficiencies prior to the start of operational testing. The program obtained a waiver from the Under Secretary of Defense for Acquisition and Sustainment to start operational testing prior to fully resolving these deficiencies. Program officials stated that they expect to continue resolving these deficiencies through the start of full rate production in October 2019. Program officials continue to identify and address technical risks, some of which are specific to individual variants of the F-35. For example, we reported last year that a problem with the F-35’s main fuel throttle valve caused the aircraft to move suddenly and without stopping until the engine is shut down. In 2018, the program implemented software changes to fix this problem. Also, across all variants, pilots have reported experiencing extreme pressure in the cockpit during certain flight maneuvers. Contractor representatives told us they have identified the root cause of the excessive cockpit pressure and will implement a minor hardware change in 2019 to address the issue. Recently, following the crash of an F-35B in October 2018, the program grounded the F-35 fleet to inspect all of its engines. An investigation determined a manufacturing defect caused an engine fuel tube to rupture during flight, resulting in a loss of power to the engine. The program office reported that it identified 117 aircraft with the same type of fuel tubes that it must replace. According to program officials, the grounding generally did not impact the delivery of the aircraft, as the contractor has provided replacement fuel tubes that were installed on a majority of the affected aircraft by the end of 2018. Common Name: F-35 contractor continues to fix gaps between adjacent aircraft surface panels attached to the airframe. These fixes are needed to meet low observable (stealth) performance requirements. The contractor is working with its supplier to resolve the problem through improved production processes as the program approaches its full-rate production decision in October 2019. Other Program Issues Following our 2018 assessment, the program delayed the start of operational testing by up to 3 months, to December 2018. This delay stemmed from software upgrades needed to assess the aircraft’s performance. To mitigate further delays, the program received authorization to complete certain operational tests in advance of the formal start of operational testing. For example, the program completed cold weather operational testing in January 2018. Because of evolving threats, the program office continues to move forward with Block 4 modernization efforts, which will modernize current capabilities and develop and integrate new capabilities onto the aircraft. In October 2018, the program office updated its acquisition strategy, providing a general schedule for future technology development and integration. The program plans to field new capabilities starting in October 2019, but it has yet to complete its acquisition program baseline. As a result, the program is concurrently testing, producing, and modernizing aircraft, which increases the risk of future schedule and cost overruns. Program Office Comments We provided a draft of this assessment to the program office for review and comment. The office provided technical comments, which we incorporated where appropriate. Program officials stated that since the start of developmental testing in 2006, more than 2,200 deficiencies have been discovered and corrected. Further, officials reported that, in coordination with the warfighting community, they have resolved the highestpriority deficiencies and have mitigated the remaining deficiencies. Finally, officials commented that aircraft deliveries have increased as planned and operational testing remains on track for completion in fall 2019. Production Readiness As of December 2018, the prime contractor has delivered 264 production aircraft. Since the start of production, F-35 contractors have refined their production processes to improve manufacturing efficiency and quality. However, the prime contractor has identified quality control and late radar deliveries as the top production risks in the program. For example, because of supplier identified limitations, the prime Page 174 GAO-19-336SP Weapon Systems Annual Assessment Agency Comments We provided a draft of this report to DOD for comment. In its comments, reproduced in appendix IX, DOD generally concurred with our observations. DOD also provided us with technical comments, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees and offices; the Acting Secretary of Defense; the Secretaries of the Army, Navy, and Air Force; and the Director of the Office of Management and Budget. In addition, the report will be made available at no charge on the GAO website at http://www.gao.gov. If you are your staff have any questions concerning this report, please contact me at (202) 512-4841. Contact points for our offices of Congressional Relations and Public Affairs may be found on the last page of this report. Staff members making key contributions to this report are listed in appendix X. Shelby S. Oakley Director, Contracting and National Security Acquisitions Page 175 GAO-19-336SP Weapon Systems Annual Assessment List of Committees The Honorable James M. Inhofe Chairman The Honorable Jack Reed Ranking Member Committee on Armed Services United States Senate The Honorable Richard Shelby Chairman The Honorable Richard J. Durbin Ranking Member Subcommittee on Defense Committee on Appropriations United States Senate The Honorable Adam Smith Chairman The Honorable Mac Thornberry Ranking Member Committee on Armed Services House of Representatives The Honorable Pete Visclosky Chairman The Honorable Ken Calvert Ranking Member Subcommittee on Defense Committee on Appropriations House of Representatives Page 176 GAO-19-336SP Weapon Systems Annual Assessment Appendix I: Individual Subcontracting Reports in the Electronic Subcontracting Reporting System Appendix I: Individual Subcontracting Reports in the Electronic Subcontracting Reporting System Table 18 shows the number of prime contractors for the 82 major defense acquisition programs (MDAP) included in our assessment as reported in the Electronic Subcontracting Reporting System’s (eSRS) individual subcontracting reports for 2018. We reviewed this information against the reported prime contract information from the programs’ December 2017 Selected Acquisition Report (SAR) submissions. eSRS individual subcontracting reports include information on contractor performance relative to small business subcontracting goals, thus are used by the government as a method to monitor small business participation. Not all prime contractors for MDAPs are required to submit individual subcontracting reports. Instead, some contractors report small business participation at a corporate level as opposed to the program level and this data is not captured in the individual subcontracting reports. In addition, although prime contractors may be required to submit reports, they may not yet have done so for the period we reviewed. Their reports could also be pending or in rejected status. Table 18: Major Defense Acquisition Programs’ Individual Subcontracting Reports in the Electronic Subcontracting Reporting System (eSRS) Program name Number of contracts listed in the December 2017 Selected Acquisition Report Contracts with an accepted individual subcontracting report (as of July 2018) Advanced Arresting Gear (AAG) 4 3 Amphibious Combat Vehicle Phase 1 Increment 1 (ACV 1.1) 2 2 Advanced Extremely High Frequency (AEHF) Satellite 2 2 AGM-88E Advanced Anti-Radiation Guided Missile (AGM-88E AARGM) 2 2 AH-64E Apache New Build (AH-64E New Build) 2 2 AH-64E Apache Remanufacture (AH-64E Remanufacture) 6 6 AIM-9X Block II Sidewinder (AIM-9X Blk II) 3 3 Air and Missile Defense Radar (AMDR) 1 1 Airborne and Maritime/Fixed Station (AMF) 0 0 Armored Multi-Purpose Vehicle (AMPV) 1 1 AIM-120 Advanced Medium Range Air-to-Air Missile (AMRAAM) 4 2 Airborne Warning and Control System Block 40/45 Upgrade (AWACS Blk 40/45 Upgrade) 1 1 B-2 Defensive Management System Modernization (B-2 DMS-M) 1 1 B61 Mod 12 Life Extension Program Tailkit Assembly (B61 Mod 12 LEP TKA) 2 1 C-130J Hercules Transport Aircraft (C-130J) 4 4 Page 177 GAO-19-336SP Weapon Systems Annual Assessment Appendix I: Individual Subcontracting Reports in the Electronic Subcontracting Reporting System Program name Number of contracts listed in the December 2017 Selected Acquisition Report Contracts with an accepted individual subcontracting report (as of July 2018) Cooperative Engagement Capability (CEC) 6 1 CH-47F Improved Cargo Helicopter (CH-47F) 1 1 CH-47F Modernized Cargo Helicopter (CH-47F Block II) 3 1 CH-53K Heavy Lift Replacement Helicopter (CH-53K) 4 2 Chemical Demilitarization-Assembled Chemical Weapons Alternatives (Chem Demil-ACWA) 2 2 Common Infrared Countermeasure (CIRCM) 1 1 Combat Rescue Helicopter (CRH) 1 1 Gerald R. Ford Class Nuclear Aircraft Carrier (CVN 78) 4 4 DDG 1000 Zumwalt Class Destroyer (DDG 1000) 1 1 DDG 51 Arleigh Burke Class Guided Missile Destroyer (DDG 51) 4 4 E-2D Advanced Hawkeye Aircraft (E-2D AHE) 4 4 EA-18G Growler Aircraft (EA-18G) 6 6 Evolved Expendable Launch Vehicle (EELV) 9 2 F-15 Eagle Passive Active Warning Survivability System (F-15 EPAWSS) 1 1 F-22 Increment 3.2B Modernization (F-22 Inc 3.2B Mod) 1 1 F-35 Lightning II Program (F-35) 6 6 Family of Advanced Beyond Line-of-Sight Terminals (FAB-T) 2 1 Ground/Air Task Oriented Radar (G/ATOR) 3 3 Guided Multiple Launch Rocket System/Guided Multiple Launch Rocket Sys Alt Warhead (GMLRS/GMLRS AW) 4 4 Global Positioning System III (GPS III) 2 2 H-1 Upgrades (4BW/4BN) (H-1 Upgrades) 3 0 HC/MC-130 Recapitalization Aircraft (HC/MC-130 Recap) 1 1 Integrated Air and Missile Defense (IAMD) 1 1 Intercontinental Ballistic Missile Fuze Modernization (ICBM Fuze Mod) 1 0 Integrated Defensive Electronic Countermeasures (IDECM) Block 4 3 2 Infrared Search and Track (IRST) 3 3 Joint Air-to-Ground Missile (JAGM) 1 1 Joint Air-to-Surface Standoff Missile (JASSM) 2 2 Joint Direct Attack Munition (JDAM) 2 1 Joint Light Tactical Vehicle (JLTV) 1 1 Joint Precision Approach and Landing System Increment 1A (JPALS Inc 1A) 1 0 Joint Tactical Radio System Handheld, Manpack, and Small Form Fit Radios (JTRS HMS) 7 5 Page 178 GAO-19-336SP Weapon Systems Annual Assessment Appendix I: Individual Subcontracting Reports in the Electronic Subcontracting Reporting System Program name Number of contracts listed in the December 2017 Selected Acquisition Report Contracts with an accepted individual subcontracting report (as of July 2018) KC-130J Transport Aircraft (KC-130J) 1 1 KC-46 Tanker Modernization Program (KC-46A) 4 3 Littoral Combat Ship (LCS) 2 2 Littoral Combat Ship - Mission Modules (LCS Packages) 1 1 LHA 6 America Class Amphibious Assault Ship (LHA 6) 2 2 LPD 17 San Antonio Class Amphibious Transport Dock (LPD 17) 1 1 M88A2 Heavy Equipment Recovery Combat Utility Lift Evacuation System (M88A2 HERCULES) 2 2 Military Global Positioning System (GPS) User Equipment Increment 1 (MGUE Inc 1) 3 2 Multifunctional Information Distribution System (MIDS) 4 4 MQ-1C Gray Eagle Unmanned Aircraft System (MQ-1C Gray Eagle) 2 2 MQ-4C Triton Unmanned Aircraft System (MQ-4C Triton) 3 3 MQ-8 Fire Scout 1 1 MQ-9 Reaper Unmanned Aircraft System (MQ-9 Reaper) 4 3 Mobile User Objective System (MUOS) 0 0 Next Generation Jammer Increment 1 (NGJ Inc 1) 2 1 Navy Multiband Terminal (NMT) 1 1 Offensive Anti-Surface Warfare Increment 1 (Long Range Anti-Ship Missile) (OASuW Inc 1 (LRASM)) 2 2 Next Generation Operational Control System (GPS OCX) 1 1 P-8A Poseidon Multi-Mission Maritime Aircraft (P-8A) 2 1 Patriot Advanced Capability-3 Missile Segment Enhancement (PAC-3 MSE) 2 2 M109A7 Family of Vehicles (M109A7 FOV) 2 2 Space Based Infrared System High (SBIRS High) 1 1 Small Diameter Bomb Increment II (SDB II) 3 3 Standard Missile-6 (SM-6) 2 2 Space Fence Ground-Based Radar System Increment 1 1 1 SSBN 826 Columbia Class Submarine(SSBN 826) 2 2 Ship to Shore Connector Amphibious Craft (SSC) 1 1 SSN 774 Virginia Class Submarine (SSN 774) 2 2 T-AO 205 John Lewis Class Fleet Replenishment Oiler (T-AO 205 Class) 1 1 Tactical Tomahawk RGM-109E/UGM 109E Missile (TACTOM) 2 2 Trident II (D-5) Sea-Launched Ballistic Missile UGM 133A (Trident II Missile) 9 7 UH-60M Black Hawk Helicopter (UH-60M Black Hawk) 2 2 Page 179 GAO-19-336SP Weapon Systems Annual Assessment Appendix I: Individual Subcontracting Reports in the Electronic Subcontracting Reporting System Program name Number of contracts listed in the December 2017 Selected Acquisition Report Contracts with an accepted individual subcontracting report (as of July 2018) V-22 Osprey Joint Services Advanced Vertical Lift Aircraft (V-22) 5 4 VH-92A Presidential Helicopter Replacement Program 1 1 Warfighter Information Network-Tactical Increment 2 (WIN-T Inc 2) 1 1 201 163 Totals 82 Source: GAO analysis of Department of Defense (DOD) and eSRS data. GAO-19-336SP Page 180 GAO-19-336SP Weapon Systems Annual Assessment Appendix II: Objectives, Scope, and Methodology Appendix II: Objectives, Scope, and Methodology This report includes observations on (1) the cost and schedule performance of the Department of Defense’s (DOD) 2018 portfolio of 82 major weapon programs and the competitive environment in which they exist, (2) implementation of key acquisition reform initiatives within 51 current and future programs, and (3) the knowledge that 51 current and future programs attained at key decision points in the acquisition process. Analysis of Portfolio Cost and Schedule Performance To develop the seven observations on the cost and schedule performance of DOD’s 2018 portfolio of current major defense acquisition programs (MDAP), we obtained and analyzed cost, quantity, and schedule data from Selected Acquisition Reports (SAR) and other information in the Defense Acquisition Management Information Retrieval system (DAMIR). To assess the reliability of the SAR data, we reviewed our previous assessment and DOD officials’ responses regarding any changes to DAMIR’s data quality control procedures. We determined that the SAR data and the information retrieved from DAMIR were sufficiently reliable for the purposes of this report. We analyzed this data to determine the number of programs in the current portfolio year. We included in the portfolio the programs that issued an unclassified December 2017 SAR. The Missile Defense Agency’s Ballistic Missile Defense System and its elements are excluded from all analyses due to the lack of an integrated long-term baseline. In general, we refer to the 82 MDAPs with December 2017 SARs as DOD’s 2018 or current portfolio and use a similar convention for prior year portfolios. We entered DAMIR-obtained SAR data into a database and verified that the data was entered correctly. We then converted all cost information to fiscal year 2019 dollars using conversion factors from DOD Comptroller’s National Defense Budget Estimates for Fiscal Year 2019 (table 5-9). Our assessment includes comparisons of cost and schedule changes over the past year, 5 years, and from baseline estimates, that utilize SAR data from December 2017, December 2016, December 2012, and from the programs’ initial SAR submissions. For our first observation, we compared the 2018 portfolio with the programs that issued SARs in December 2016 (2017 portfolio) to identify the programs that exited and entered the current portfolio, the total cost and number of programs in the current portfolio compared to previous years, and the total cost change in the current portfolio over the past year. Page 181 GAO-19-336SP Weapon Systems Annual Assessment Appendix II: Objectives, Scope, and Methodology For the second observation, we used program acquisition start dates to calculate individual program ages and the average program age of the current portfolio, then compared the results to preceding portfolios dating back to 2012. For the third observation, we aggregated DAMIR funding stream data for the total planned investment of each portfolio for each year since 2008 to determine any trends. We also calculated yearly totals for research and development, procurement, and total acquisition cost. To distinguish between the funding already invested from the funding needed to complete the programs in each portfolio since 2008, we used funding stream data obtained from DAMIR for each December SAR submission for the years 2007 (2008 portfolio) through 2017 (2018 portfolio). We define funding invested as all funding that has been provided to the programs in the fiscal year of the annual SAR submission (this includes fiscal year 2018 for the December 2017 submission) and earlier, while funding remaining is what will be provided in the fiscal years following the annual SAR submission (fiscal year 2019 and later for the December 2017 submission). Invested and remaining research and development and procurement funding totals for the 2018 portfolio were organized by the military departments overseeing portfolio programs. For our fourth observation, we calculated the total acquisition-related cost growth of the current portfolio by collecting relevant data from programs’ annual SARs. Data from the current (December 2017) and prior year (December 2016) SARs was compared and yearly trends analyzed. We then divided the programs into percent cost change categories based on their total acquisition cost changes over the past year. Finally, we totaled the number of programs in each category and the total cost change of the programs. Portfolio performance associated with acquisition cost growth of its programs was also evaluated against the high-risk criteria discussed by DOD, the Office of Management and Budget, and GAO. We calculated how many programs attained less than 2 percent increase in total acquisition cost over the past year, less than a 10 percent increase over the past 5 years, and less than a 15 percent increase from baseline estimates. We also calculated the percentage of programs meeting each of these high-risk criteria for 2013 through 2018 portfolios to identify any changes. For programs with multiple sub-programs presented in the SARs, we calculated the net effect of the sub-programs to reach an aggregate program result. In the fifth observation, we calculated the total acquisition-related cost growth and schedule delays of the current portfolio since the programs’ first full estimates by retrieving SAR cost data for the 82 programs in the portfolio. In addition, we analyzed programs’ cost growth between three Page 182 GAO-19-336SP Weapon Systems Annual Assessment Appendix II: Objectives, Scope, and Methodology acquisition phase intervals based on the program’s knowledge points: development start to critical design review, critical design review to production decision, and from production start until end of production. We then calculated the total cost growth for the entire portfolio within these intervals. For the sixth observation, cost changes related to quantity changes were factored out of the total acquisition cost change calculation used to identify programs with the largest acquisition cost increase and decrease. We ranked the ten programs with the highest overall cost change and the ten programs with the lowest overall cost change both in terms of their total cost and as a percentage since their initial estimates. Our seventh observation focuses on the performance of programs initiated after and prior to 2010. DOD implemented selected requirements of the Weapon Systems Acquisition Reform Act (WSARA) on December 4, 2009, and programs with succeeding acquisition start dates were classified as “post-2010” programs. We compared the cost and schedule changes of both groups, then demonstrated their total acquisition cost as a percentage, apportioned between the two groups and costs of F-35, DOD’s largest pre-2010 acquisition program. Furthermore, for both groups of programs, we analyzed cost growth between three acquisition phase intervals based on the program’s knowledge points: development start to critical design review, critical design review to production decision, and from production start until end of production. We then calculated the total cost growth for the entire portfolio within these intervals. In addition, we analyzed buying power performance of both groups over the past year. Data elements used to complete this calculation are the number of procurement units, procurement cost changes, and average procurement unit costs per program. We calculated cost change “due” to quantity changes as the change in quantity over the last year multiplied by the average procurement unit cost for the program a year ago. We calculated cost change “not due” to quantity changes as the current acquisition quantity times the change in average per unit costs. In practice, changes in quantity will often affect per unit cost—as discussed later in the appendix—so this is more precisely described as “Cost change due to change in quantity assuming no change in average procurement unit cost” and “Cost change due to changes in average procurement unit cost.” If changes in quantity affect per unit cost, those changes will appear in the cost change “not due” to quantity changes. We first calculated the amount of procurement cost growth attributable to quantity changes. To do this, we multiplied any change in quantity by the average procurement unit cost for the program a year ago. The resulting dollar Page 183 GAO-19-336SP Weapon Systems Annual Assessment Appendix II: Objectives, Scope, and Methodology amount is considered a change due solely to shifts in the number of units procured and may overestimate the amount of change expected when quantities increase. Additionally, it could also underestimate the expected change when quantities decrease as it does not account for other effects of quantity changes on procurement such as gain or loss of learning in production that could result in changes to unit cost over time or the use or absence of economic orders of material. However, these changes are accounted for as part of the change in cost not due to quantities. Analysis of Portfolio’s Competitive Environment We utilized data from the Federal Procurement Data System-Next Generation (FPDS-NG) and DAMIR to develop the two competitionrelated observations (observations eight and nine). More specifically, we compiled all data related to the contracts listed on each program’s December 2017 SAR, then queried the FPDS-NG database to obtain the data used to analyze the competitive environment of the 2018 MDAP portfolio. By statute, SARs include an MDAP’s “major contracts,” which are defined as “each of the six largest prime, associate, or Governmentfurnished equipment contracts under the program that is in excess of $40,000,000 and that is not a firm, fixed price contract.” 1 An MDAP’s SAR requirements cease after 90 percent of the total quantity of items to be purchased under the program are delivered or 90 percent of planned expenditures are made. 2 Additionally, DOD guidance states that once a contract is 90 percent complete, it is no longer reported in the SAR. We added data elements from DAMIR, namely the account type, to the FPDS-NG dataset. We assessed the reliability of FPDS-NG data by performing electronic testing of selected data elements as well as reviewing existing information about the system and its data. Specifically, we reviewed the data dictionary, data validation rules, and the fiscal year 2017 Federal Government Procurement Data Quality Summary for Agency Data in the Federal Procurement Data System. We found the FPDS-NG data sufficiently reliable for the purposes of this report. We defined noncompetitive obligations to include obligations through contracts that were awarded using the exceptions to full and open competition listed in Federal Acquisition Regulation (FAR) Subpart 6.3 1 10 U.S.C. § 2432(a)(3), (e)(8). 2 10 U.S.C. § 2432(g). Page 184 GAO-19-336SP Weapon Systems Annual Assessment Appendix II: Objectives, Scope, and Methodology (Other than Full and Open Competition). We also included noncompetitive orders issued under multiple award indefinite delivery/indefinite quantity contracts or under the General Service Administration’s schedules program. For noncompetitive contract actions, we included contracts and orders coded as “not competed,” “not available for competition,” and “not competed under simplified acquisition procedures,” as well as orders coded as an exception to “subject to fair opportunity,” including “urgency,” “only one source,” “minimum guarantee,” “follow-on action following competitive initial action,” “other statutory authority,” and “sole source.” For competitive contract actions, we included contracts and orders coded as “full and open competition,” “full and open after exclusion of sources,” and “competed under simplified acquisition procedures” as well as orders subject to multiple award fair opportunity coded as “subject to fair opportunity” and as “fair opportunity given,” and “competitive set aside.” We calculated competition rates as the number of competitive contracts and orders over all contracts and orders reported in the December 2017 SARs. In observation eight, we used the FPDS-NG dataset to identify the number of offers received for each contract award, as well as the extent to which the obligations were for contracts or orders that were competitively awarded. In addition, we examined the extent to which the contracts and orders in the dataset were awarded competitively across the managing military departments and, for each, calculated the total value of the competitive versus non-competitive awards. In order to conduct this analysis, we included contract or account-type identifying data from DAMIR to our FPDS-NG dataset. For observation nine, we used FPDS-NG data elements containing parent company information of contract awardees and the total base and all options values of the awards to identify the five parent corporations with the highest amounts of contract awards. We used parent company information contained in the FPDS-NG dataset in order to group companies associated with the five contractors receiving the greatest value of awards. Analysis of Selected Acquisition Reforms To determine the extent to which the department has transferred acquisition programs’ decision authority to the military departments, we obtained and analyzed the annual MDAP lists from DAMIR dating back to 2012. Each list identifies the acquisition authority for each program. For each year of data, we calculated both the number of programs for which DOD was the decision authority and the number of programs for which Page 185 GAO-19-336SP Weapon Systems Annual Assessment Appendix II: Objectives, Scope, and Methodology the military department was the acquisition authority. Based on that data, we created a column chart that illustrates how the decision authority has shifted each year across the portfolio. To develop observations on the extent to which DOD is applying recent, selected acquisition reforms, we reviewed specific acquisition reform provisions contained in the 2016, 2017, and 2018 NDAAs. We selected reforms that, in our view, involve oversight and acquisition principles consistent with our knowledge-based practices and provide mechanisms for more rapid fielding of warfighter capability. We analyzed questionnaire data received from the 45 current and six future MDAPs in our assessment to determine the extent to which acquisition reforms have been applied or gained traction within DOD. Based on the questionnaire responses, we analyzed the degree to which the programs applied specific acquisition reform provisions contained in the 2016, 2017, and 2018 NDAAs. Those provisions are summarized separately in appendix IV. Analysis of Selected DOD Programs Using Knowledge-Based Criteria and Related Implications for Testing and Software Development To collect data from current and future MDAPs—including cost and schedule estimates, technology maturity, and planned implementation of acquisition reforms—we distributed two electronic questionnaires—one questionnaire for the 45 current programs and a slightly different questionnaire for the six future programs. Both of the questionnaires were web-based so that respondents could respond and submit their answers online. We received responses from all of the programs we assessed from October 2018 to January 2019. To ensure the reliability of the data collected through our questionnaires, we took a number of steps to reduce measurement error and non-response error. These steps included conducting two pretests of the future MDAP questionnaire and three pretests for the current major defense acquisition program questionnaire prior to distribution to ensure that our questions were clear, unbiased, and consistently interpreted. Our pretests covered each branch of the military to better ensure that the questionnaires could be understood by officials within each branch. We determined that the data from the SARs and DAMIR were sufficiently reliable for the purposes of this report. Our analysis of how well programs are adhering to a knowledge-based acquisition approach focuses on 45 MDAPs that mare mostly in development or the early stages of production. To assess the knowledge attained by key decision points (system development start or detail design Page 186 GAO-19-336SP Weapon Systems Annual Assessment Appendix II: Objectives, Scope, and Methodology contract award for shipbuilding programs, critical design review or lead ship fabrication start for shipbuilding programs, and production start), we collected data from program offices about their knowledge at each point. We also provide information on how much knowledge was obtained at key decision points by programs that accomplished these previously. We also include observations on the knowledge that the six future programs expect to obtain before starting development. We did not validate the data provided by the program offices, but reviewed the data and performed various checks to determine that they were reliable for our purposes. Where we discovered discrepancies, we clarified the data accordingly. For programs that have passed a key decision point and have since been restructured, we assessed them against their original cost and schedule estimates at that milestone or decision point, such as development start. We did not reassess programs at milestones they had previously reached, as in cases where a program is repeating a key decision point or milestone, such as milestone B. We keep our original assessment of the program’s knowledge attained at the original milestone. However, we do change future milestone dates if those milestone had yet to be reached, and we assess those programs for their implementation of our best practices at that point in time. For the second consecutive year, we performed an exploratory statistical analysis that examined our identified knowledge-based acquisition practices and select programs’ cost and schedule changes. We focused the analysis on the 17 non-shipbuilding MDAPs that, prior to this assessment, completed each of the three knowledge points within the acquisition process (i.e., completed development, held a critical design review, and started production). Our statistical analysis compared average cost and schedule changes for those programs that had implemented eight key knowledge-based acquisition practices by the time they reached knowledge points 1 through 3, compared to those programs that did not complete the best practices at each knowledge point. To ensure a minimally reliable estimate of the average in each group, we limited our analysis to those knowledge-based acquisition practices for which at least three programs had engaged in the practice, and at least three programs had not engaged in the practice. Although we sought to assess the statistical significance of demonstrating technologies to form, fit, and function within a realistic environment, we observed that only one program in the sample demonstrated this level of technology maturity before it started development. This one program provided an insufficient basis to determine whether this best practice corresponded with lower Page 187 GAO-19-336SP Weapon Systems Annual Assessment Appendix II: Objectives, Scope, and Methodology cost and schedule growth. We assessed the statistical significance of the observed differences between the groups at the 90 percent confidence level. 3 With such a small sample of MDAPs, our estimates are fairly imprecise and do not meet normality assumptions. To assess programs’ developmental testing and production concurrency, we identified the programs—among those we included in our assessment—with production start dates. We used the questionnaire responses from those programs to identify the dates for the start and end of developmental testing, compared those dates to the timing of each program’s production decision, and determined whether productionrepresentative prototype testing would conclude before production start. To examine programs’ software development efforts, we identified the dates reported by programs for their software and hardware integration and compared those dates to each program’s production start date to assess each program’s degree of software development and production concurrency. To examine when programs were declaring initial operational capability compared to finishing their operational testing, we identified programs’ reported initial operational capability dates and start and end dates of their operational testing. We determined whether the initial operational capability date was before, during or after its testing dates. Based on our determination, we summed and analyzed what percent of programs were in each category. For some programs, either one or both of these dates were not available. The 53 current and future programs included in our assessment were in various stages of the acquisition cycle, and not all of the programs provided information on knowledge obtained at each point. Programs were not included in our assessments at key decision points if relevant data were not available. Our analysis of knowledge attained at each key point includes factors that we have previously identified as underpinning a knowledge-based acquisition approach, including holding early systems engineering reviews, testing an integrated prototype prior to the design review, using a reliability growth curve, planning for manufacturing, and testing a production-representative prototype prior to making a production 3 Statistical significance at the 90 percent confidence level indicates that the chances of observing a statistical difference as large or larger as observed by chance, if no difference existed, is less than 10 percent. Page 188 GAO-19-336SP Weapon Systems Annual Assessment Appendix II: Objectives, Scope, and Methodology decision. Additional information on how we collect these data is found in the production knowledge assessment section of this appendix. See also appendix III for a list of the practices that are associated with a knowledge-based acquisition approach. Individual Assessments of Weapon Programs This report presents individual assessments of 51 current and future MDAPs. A table listing these assessments is found in appendix X. Of our 51 total assessments, 39 are captured in a two-page format discussing technology, design, and manufacturing knowledge obtained and other program issues. These two-page assessments are of current MDAPs, most of which are in development or early production. The remaining 12 assessments we presented in a one-page format that describes their current status. Those one-page assessments include (1) six future MDAPs; (2) one MDAP that is in development, but released a baseline late in our review—VC-25B Presidential Aircraft Recapitalization, and (3) five MDAPs that are well into production, but planning to introduce new increments of capability—specifically, the Navy’s DDG 51 Arleigh Burke Class Destroyer, Flight III; LHA 8 America Class Amphibious Assault Ship; LPD 17 San Antonio Class Amphibious Transport Dock, Flight II; P8A Poseidon, Increment 3; and SSN 774 Virginia Class Submarine, Block V. We collected data on these programs through January 31, 2019, with the exception of one program—Offensive Anti-Surface Warfare Increment 1—which released a new baseline in early February 2019 that we included. For presentation purposes we grouped the individual assessments by lead military department—Army, Navy and Marine Corps, Air Force, and DOD-wide—and inserted a summary analysis page at the start of each grouping. These four summary analysis pages present aggregated information about selected programs’ acquisition phases, funding needs, knowledge attainment, cost and schedule performance, oversight authorities, software characteristics, and competition in contracting. We obtained this data primarily from December 2017 SARs and supplemented by program office responses to our questionnaire. We reported cost and schedule growth in the summary analysis pages in a manner that is consistent with how it is reported and described elsewhere in the report. Of note, the cost analysis in the summary analysis pages is based on cost data reported in the December 2017 SARs. These data may differ from the estimates reported in the individual assessments, which in some cases are more recent than December 2017. Page 189 GAO-19-336SP Weapon Systems Annual Assessment Appendix II: Objectives, Scope, and Methodology Over the past several years, DOD has revised policies governing weapon system acquisitions and changed the terminology used for major acquisition events. To make DOD’s acquisition terminology more consistent across our individual program assessments, we standardized the terminology for key program events. For most individual programs in our assessment, “development start” refers to the initiation of an acquisition program as well as the start of either engineering and manufacturing development or system development. This generally coincides with DOD’s milestone B. A few programs in our assessment have a separate “program start” date, which begins a pre-system development phase for program definition and risk-reduction activities. This “program start” date generally coincides with DOD’s former terminology for milestone I or DOD’s current milestone A, which denotes the start of technology maturation and risk reduction. The “production decision” generally refers to the decision to enter the production and deployment phase, typically with low-rate initial production. The “initial capability” refers to the initial operational capability—sometimes called first unit equipped or required asset availability. For shipbuilding programs, the schedule of key program events in relation to acquisition milestones varies for each program. Our work on shipbuilding best practices has identified the detail design contract award and the start of lead ship fabrication as the points in the acquisition process roughly equivalent to development start and design review for other programs. We obtained the information presented in the “Program Essentials” section of the individual assessments from the DAMIR system, program office responses to a questionnaire, program office documents, and communications with program officials. As a result, DOD is the source of the information regarding the milestone decision authority, identity of the contractors, the contract types, and the software development approach. We did not review individual contracts for each system. In their questionnaire responses, program offices self-identified the type of software development approach used based on definitions from the Defense Acquisition University. In cases where program offices identified more than one type, we describe this as a “mixed” approach. For each program we assessed in a two-page format, we present cost, schedule, and quantity data at the program’s first full estimate as well as an estimate from the latest SAR or the most recent Defense Acquisition Executive Summary (DAES) report as of June or July 2018, except in cases where significant changes subsequently occurred, such as a new program baseline. For cases in which we received updated schedule data after the date of the DAES reports, we followed up with programs to Page 190 GAO-19-336SP Weapon Systems Annual Assessment Appendix II: Objectives, Scope, and Methodology determine whether there were any cost changes associated with the schedule changes and updated the data as needed. The first full estimate is generally the cost estimate established at milestone B—development start; however, for a few programs that did not have such an estimate, we used the estimate at milestone C—production start—instead. For shipbuilding programs, we used their planning estimates if those estimates were available. For systems for which a first full estimate was not available, we only present the latest available estimate of cost and quantities. For the other programs assessed in a one-page format, we present the latest available estimate of cost and quantity from the program office. For each program we assessed, all cost information is presented in fiscal year 2019 dollars. We converted cost information to fiscal year 2019 dollars using conversion factors from the DOD Comptroller’s National Defense Budget Estimates for Fiscal Year 2019 (table 5-9). We have depicted only the program’s main elements of acquisition cost—research and development and procurement. However, the total program cost also includes military construction and acquisition-related operation and maintenance costs. Because of rounding and these additional costs, in some situations, total cost may not match the exact sum of the research and development and procurement costs. The program unit costs are calculated by dividing the total program cost by the total quantities planned. These costs are often referred to as program acquisition unit costs. In some instances, the data were not applicable, and we annotate this by using the term “not applicable (NA).” The quantities listed refer to total quantities, including both procurement and development quantities. The schedule assessment for each program is based on acquisition cycle time, defined as the number of months between program start and the achievement of initial operational capability or an equivalent fielding date. In some instances the data were not yet available, and we annotate this by using the term “to be determined (TBD)” or “NA.” As a result of updates to our methods for calculating acquisition cycle time for the 2019 assessment, some first full estimates may reflect a slight adjustment (e.g., 1 month) from what we reported in previous years. The information presented in “Funding and Quantities” is from fiscal year 2019 through completion and draws on information from the latest SAR (for quantities in particular) or the most recent DAES report as of June or July 2018 reflecting 2019 data, except in cases where significant changes subsequently occurred, such as a new program baseline. The quantities listed refer only to procurement quantities. Satellite programs, in Page 191 GAO-19-336SP Weapon Systems Annual Assessment Appendix II: Objectives, Scope, and Methodology particular, produce a large percentage of their total operational units as development quantities, which are not included in the quantity figure. The intent of these comparisons is to provide an aggregate, or overall, picture of a program’s history. These assessments represent the sum of the federal government’s actions on a program, not just those of the program manager and the contractor. DOD does a number of detailed analyses of changes that attempt to link specific changes with triggering events or causes. Our analysis does not attempt to make such detailed distinctions. We also reviewed whether individual subcontracting reports from a program’s prime contractor or contractors were accepted on the Electronic Subcontracting Reporting System (eSRS). We reviewed this information for 82 of the MDAPs in the 2018 portfolio using the contract information reported in their December 2017 SARs. See appendix I for a list of the programs we reviewed. Product Knowledge Data on Individual Two-Page Assessments In our past work examining weapon acquisition issues and knowledgebased acquisition practices for product development, we have found that leading commercial firms pursue an acquisition approach that is anchored in knowledge, whereby high levels of product knowledge are demonstrated by critical points in the acquisition process. On the basis of this work, we have identified three key knowledge points during the acquisition cycle—system development start, critical design review, and production start—at which programs need to demonstrate critical levels of knowledge to proceed. To assess the product development knowledge of each program at these key points, we reviewed questionnaires submitted by programs; however, not every program had responses to each element of the questionnaire. We also reviewed pertinent program documentation and discussed the information presented on the questionnaire with program officials as necessary. For our “Attainment of Product Knowledge” tables, we assessed the programs’ current status in implementing the knowledge-based acquisition practices criteria, as well as the programs’ progress in meeting the criteria at the time they reached the three key knowledge points during the acquisition cycle—system development start, critical design review, and production start. For programs that have passed a key decision point and have since been restructured, we continue to assess them against their original cost and schedule estimates at that milestone or decision point, such as development start. We have not reassessed a Page 192 GAO-19-336SP Weapon Systems Annual Assessment Appendix II: Objectives, Scope, and Methodology program at milestones that have already been reached if a program is repeating a key decision point or milestone, such as milestone B. We have kept our original assessment of the program’s knowledge attained at the original milestone. However, we have changed future milestone dates in instances when the program had not yet reached the affected milestone. In these instances, we assessed the program for its implementation of our knowledge-based acquisition practices criteria at that point in time. To assess a program’s readiness to enter system development, we collected data through the questionnaire on critical technologies and early design reviews. To assess technology maturity, we asked program officials to apply a tool, referred to as technology readiness levels (TRL), for our analysis. The National Aeronautics and Space Administration originally developed TRLs, and the Army and Air Force science and technology research organizations use them to determine when technologies are ready to be handed off from science and technology managers to product developers. TRLs are measured on a scale from 1 to 9, beginning with paper studies of a technology’s feasibility and culminating with a technology fully integrated into a completed product. See appendix VIII for TRL definitions. Our knowledgebased acquisition practices work has shown that a TRL 7—demonstration of a technology in its form, fit, and function within a realistic environment—is the level of technology maturity that constitutes a low risk for starting a product development program. 4 For shipbuilding programs, we have recommended that this level of maturity be achieved by the contract award for detail design. 5 In our assessment, the technologies that have reached TRL 7, a prototype demonstrated in a realistic environment, are referred to as mature or fully mature. Those technologies that have reached TRL 6, a prototype very close to final form, fit, and function demonstrated within a relevant environment, are referred to as approaching or nearing maturity. Satellite technologies that have achieved TRL 6 are assessed as fully mature due to the difficulty of demonstrating maturity in a realistic environment—space. In addition, we asked program officials to provide the date of the system-level preliminary design review. We compared this date to the system development start date. 4 GAO, Best Practices: Better Management of Technology Development Can Improve Weapon System Outcomes, GAO/NSIAD-99-162 (Washington, D.C.: July 30, 1999); GAO, Best Practices: Better Matching of Needs and Resources Will Lead to Better Weapon System Outcomes, GAO-01-288 (Washington, D.C.: Mar. 8, 2001). 5 GAO, Best Practices: High Levels of Knowledge at Key Points Differentiate Commercial Shipbuilding from Navy Shipbuilding, GAO-09-322 (Washington, D.C.: May 13, 2009). Page 193 GAO-19-336SP Weapon Systems Annual Assessment Appendix II: Objectives, Scope, and Methodology In most cases, we did not validate the program offices’ selection of critical technologies or the determination of the demonstrated level of maturity. We sought to clarify the TRLs in those cases where information existed that raised questions. If we were to conduct a detailed review, we may or may not adjust the critical technologies assessed, their readiness levels demonstrated, or both. It was not always possible to reconstruct the technological maturity of a weapon system at key decision points after the passage of many years. Where practicable, we compared technology assessments provided by the program office to assessments by officials from the Office of the Assistant Secretary of Defense for Research and Engineering. To assess design stability, we asked program officials to provide the percentage of design drawings completed or projected for completion by the design review, the production decision, and as of our current assessment in the questionnaire. In most cases, we did not verify or validate the percentage of engineering drawings provided by the program office. We clarified the percentage of drawings completed in cases where information raised questions. Completed drawings were defined as the number of drawings released or deemed releasable to manufacturing that can be considered the “build to” drawings. For shipbuilding programs, we asked program officials to provide the percentage of the threedimensional product model that had been completed by the start of lead ship fabrication, and as of our current assessment. To gain greater insights into design stability, we also asked program officials to provide the date they planned to first integrate and test all key subsystems and components into a system-level integrated prototype. We compared this date to the date of the design review. We did not assess whether shipbuilding programs had completed integrated prototypes. To assess production maturity, we asked program officials for their Manufacturing Readiness Level (MRL) for process capability and control or to identify the number of critical manufacturing processes and, where available, to quantify the extent of statistical control achieved for those processes as a part of our questionnaire. In most cases, we did not verify or validate the information provided by the program office. We clarified the number of critical manufacturing processes and the percentage of statistical process control where information existed that raised questions. We used a standard called the Process Capability Index, a processperformance measurement that quantifies how closely a process is running to its specification limits. The index can be translated into an expected product defect rate, and we have found it to be a best practice. We also used data provided by the program offices on their MRL for Page 194 GAO-19-336SP Weapon Systems Annual Assessment Appendix II: Objectives, Scope, and Methodology process capability and control, a sub-thread tracked as part of the manufacturing readiness assessment process recommended by DOD, to determine production maturity. We assessed programs as having mature manufacturing processes if they reported an MRL 9 for that sub-thread— meaning that manufacturing processes are stable, adequately controlled, and capable. To gain further insights into production maturity, we asked program officials whether the program planned to demonstrate critical manufacturing processes on a pilot production line before beginning lowrate initial production. We also asked programs on what date they planned to begin system-level developmental testing of a fully configured, production-representative prototype in its intended environment. We compared this date to the production start date. We did not assess production maturity for shipbuilding programs. Although the knowledge points provide indicators of potential risks, by themselves they do not cover all elements of risk that a program encounters during development, such as funding instability. Technology Maturation Knowledge Data on Individual One-Page Assessments For future MDAPs in this year’s assessment, we included a table, “Attainment of Technology Maturation Knowledge,” indicating whether the programs had attained or planned to attain key knowledge prior to starting development. We selected key activities programs should conduct prior to entering system development, based on DOD’s Instruction 5000.02: conduct competitive prototyping, validate requirements, complete a technology readiness assessment, and complete a preliminary design review. These are not the only activities contemplated at this stage, but the table is intended to provide insight into the extent to which a program has gained critical knowledge before milestone B. To determine whether programs had conducted or planned to conduct these activities, we obtained information through our questionnaire and clarified responses with program officials, as needed. We conducted this performance audit from May 2018 to May 2019, in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Page 195 GAO-19-336SP Weapon Systems Annual Assessment Appendix III: Knowledge-Based Acquisition Practices Appendix III: Knowledge-Based Acquisition Practices Our prior work on best product development practices found that successful programs take steps to gather knowledge that confirms their technologies are mature, their designs stable, and that their production processes are in control. Successful product developers ensure a high level of knowledge is achieved at key junctures in development. We characterize these junctures as knowledge points. The Related GAO Products section of this report includes references to the body of work that helped us identify these practices and apply them as criteria in weapon system reviews. Table 19 summarizes these knowledge points and associated practices. Table 19: Best Practices for Knowledge-Based Acquisitions Knowledge Point 1: Technologies, time, funding, and other resources match customer needs. Decision to invest in product development Demonstrate technologies to a high readiness level—Technology Readiness Level 7—to ensure technologies are fit, form, function, a and work within a realistic environment Ensure that requirements for product increment are informed by system-level preliminary design review using system engineering process (such as prototyping of preliminary design) Establish cost and schedule estimates for product on the basis of knowledge from system-level preliminary design using system engineering tools (such as prototyping of preliminary design) Constrain development phase (5 to 6 years or less) for incremental development Ensure development phase fully funded (programmed in anticipation of milestone) Align program manager tenure to complete development phase Contract strategy that separates system integration and system demonstration activities Conduct independent cost estimate Conduct independent program assessment Conduct major milestone decision review for development start Knowledge Point 2: Design is stable and performs as expected. Decision to start building and testing productionrepresentative prototypes. Complete system critical design review Complete 90 percent of engineering design drawing packages Complete subsystem and system design reviews Demonstrate with system-level integrated prototype that design meets requirements Complete failure modes and effects analysis Identify key system characteristics Identify critical manufacturing processes Establish reliability targets and growth plan on the basis of demonstrated reliability rates of components and subsystems Conduct independent cost estimate Conduct independent program assessment Conduct major milestone decision review to enter system demonstration Page 196 GAO-19-336SP Weapon Systems Annual Assessment Appendix III: Knowledge-Based Acquisition Practices Knowledge Point 3: Production meets cost, schedule, and quality targets. Decision to produce first units for customer. Demonstrate manufacturing processes on a pilot production line Build and test production-representative prototypes to demonstrate product in intended environment Test production-representative prototypes to achieve reliability goal Collect statistical process control data Demonstrate that critical processes are capable and in statistical control Conduct independent cost estimate Conduct independent program assessment Conduct major milestone decision review to begin production Source: GAO │ GAO-19-336SP a Department of Defense policy permits development to start at a technology maturity level commensurate with Technology Readiness Level 6—demonstration of program technology in a relevant environment. Therefore we have assessed programs against this measure as well. Page 197 GAO-19-336SP Weapon Systems Annual Assessment Appendix IV: Selected Acquisition Reforms Contained in the National Defense Authorization Acts for 2016, 2017, and 2018 Appendix IV: Selected Acquisition Reforms Contained in the National Defense Authorization Acts for 2016, 2017, and 2018 In the National Defense Authorization Acts (NDAA) for Fiscal Years 2016, 2017, and 2018, Congress introduced numerous acquisition reforms that pertain to the performance of weapon acquisition programs and the speed at which the Department of Defense (DOD) delivers capabilities to the warfighter. For this assessment, we reviewed major defense acquisition programs (MDAP) application of six of these reforms. 1 Section 864 of the 2018 NDAA increases the dollar thresholds at which approval by the service acquisition executive or Under Secretary for Acquisition, Technology and Logistics is required for use of other transaction authority for prototype projects. Specifically, the threshold for service acquisition executive approval was raised from $50 million to $100 million, and the threshold for Under Secretary of Acquisition, Technology and Logistics approval was raised from $250 million to $500 million. 2 Section 815 of the 2016 NDAA codified DOD’s other transaction authority for prototype projects at 10 U.S.C. § 2371b. This section, as amended, stipulated that to use the authority, one of the following conditions must be met: • There is at least one nontraditional defense contractor or nonprofit research institution participating to a significant extent in the prototype project. 3 • All significant participants, other than the federal government, must be small businesses (including small businesses participating in a program described under section 9 of the Small Business Act (15 U.S.C. § 638)) or nontraditional defense contractors. 1 Although we list seven reforms in this appendix, we asked programs about actions they took related to only six of them. We included Section 815 of the 2016 NDAA in this appendix as it provides additional context regarding conditions that must be met for DOD’s utilization of Other Transaction Authority for prototype projects. 2 Section 211 of the fiscal year 2019 NDAA replaced the Under Secretary of Defense for Acquisition, Technology and Logistics as the approval authority with the Under Secretary of Defense for Research and Engineering or the Under Secretary of Defense for Acquisition and Sustainment. Pub. L. No. 115-232, § 211 (2018). 3 Nontraditional defense contractor means an entity that is not currently performing and has not performed any contract or subcontract for DOD that is subject to full coverage under the cost accounting standards prescribed pursuant to 41 U.S.C. § 1502 and the regulations implementing such section, for at least the 1-year period preceding the solicitation of sources by DOD for the procurement. 10 U.S.C. § 2302(9). Section 815 is codified, as amended, at 10 U.S.C. § 2371b. Page 198 GAO-19-336SP Weapon Systems Annual Assessment Appendix IV: Selected Acquisition Reforms Contained in the National Defense Authorization Acts for 2016, 2017, and 2018 • At least one third of the total cost of the prototype project is paid out of funds provided by sources other than  the federal government. • The senior procurement executive for the agency determines in writing that exceptional circumstances justify the use of a transaction that provides for innovative business arrangements or structures that would not be feasible or appropriate under a contract, or would provide an opportunity to expand the defense supply base in a manner that would not be practical or feasible under a contract. Section 807 of the 2017 NDAA established that before funds are obligated for the technology development, systems development, or production of an MDAP, DOD must establish goals that include: • the procurement unit cost and sustainment cost • the date for initial operational capability, and • technology maturation, prototyping, and a modular open system approach to evolve system capabilities and improve interoperability. This section also requires an independent technical risk assessment before any decision is made to grant Milestone A approval for a program that identifies critical technologies and manufacturing processes that need to be matured. The assessment is also required prior to any decisions to grant Milestone B approval, enter low-rate production or fullrate production, or at any other time deemed appropriate by the Secretary of Defense, for a program that identifies any critical technologies or manufacturing processes that have not been successfully demonstrated in a relevant environment. Section 805 of the 2017 NDAA provides that a MDAP that receives Milestone A or Milestone B approval after January 1, 2019 must be designed and developed, to the maximum extent practicable, with a modular open system approach to enable incremental development and enhance competition, innovation, and interoperability. 4 DOD’s Better 4 The Department of Defense’s modular open systems approach (MOSA) is to design systems with highly cohesive, loosely coupled, and severable modules that can be competed separately and acquired from independent vendors. This approach allows the department to acquire warfighting capabilities, including systems, subsystems, software components, and services, with more flexibility and competition. MOSA implies the use of modular open systems architecture, a structure in which system interfaces share common, widely accepted standards, with which conformance can be verified. Page 199 GAO-19-336SP Weapon Systems Annual Assessment Appendix IV: Selected Acquisition Reforms Contained in the National Defense Authorization Acts for 2016, 2017, and 2018 Buying Power initiatives have emphasized the use of modular open systems architecture. Section 804 of the 2016 NDAA directs DOD to establish guidance for a “middle tier” of acquisition programs that are intended to be completed in a period of two to five years. The guidance was required to establish two acquisition pathways: • Rapid prototyping: provides for the use of innovative technologies to rapidly develop fieldable prototypes to demonstrate new capabilities to meet emerging military needs. The objective of an acquisition program under this pathway is to field a prototype that can be demonstrated in an operational environment and provide for a residual operational capability within five years of the development of an approved requirement. • Rapid fielding: provides for the use of proven technologies to field production quantities of new or upgraded systems with minimal development required. The objective of an acquisition program under this pathway shall be to begin production within six months and complete fielding within five years of the development of an approved requirement. Section 806 of the 2016 NDAA authorizes the Secretary of Defense to waive any provision of acquisition law or regulation addressing the establishment of a specification for the capability to be acquired; research, development, test, and evaluation of the capability to be acquired; production, fielding, and sustainment of the capability to be acquired; and solicitation, selection of sources, and award of contracts for the capability to be acquired. The Secretary may issue such waivers after determining that the acquisition of the capability is in the vital national security interests of the United States, the application of the law or regulation to be waived would impede the acquisition of the capability in a manner that would undermine the national security of the United States, and the underlying purpose of the law or regulation to be waived can be addressed in a different manner or at a different time. Section 823 of the 2016 NDAA amends 10 U.S.C. § 2366a—the statute that governs Milestone A approval—to require the milestone decision authority to provide a written determination—as opposed to the previously required certification—that the program meets certain conditions before granting Milestone A approval. Some of the conditions were carried over Page 200 GAO-19-336SP Weapon Systems Annual Assessment Appendix IV: Selected Acquisition Reforms Contained in the National Defense Authorization Acts for 2016, 2017, and 2018 from the prior version of the statute, while others were new. The conditions are summarized as follows: • The program fulfills an approved Initial Capabilities Document; • The program was developed after appropriate market research; • If the program duplicates a capability already provided by an existing system, the duplication is necessary and appropriate; • A plan is in place to reduce any identified areas of risk; • A plan for sustainment has been addressed and a determination of applicability of core logistics capabilities requirements has been made; • An analysis of alternatives has been performed consistent with the study guidance developed by the Director of Cost Assessment and Program Evaluation; and • A cost estimate for the program has been submitted, with the concurrence of the Director of Cost Assessment and Program Evaluation, and the level of resources required to develop, procure, and sustain the program is sufficient for successful program execution Page 201 GAO-19-336SP Weapon Systems Annual Assessment Appendix V: Total Acquisition Cost Changes for Programs in the Department of Defense’s 2018 Portfolio Appendix V: Total Acquisition Cost Changes for Programs in the Department of Defense’s 2018 Portfolio Table 20 shows total acquisition cost amounts from first full estimates and from December 2016 and 2017 select acquisition reports for each program in the Department of Defense’s (DOD) 2018 portfolio of major weapon programs. Programs are organized by their managing military department and changes in total acquisition cost since last year and since first full estimates are displayed as percentages. Table 20: December 2017 Cost Estimates and First Full Estimates for the Department of Defense’s 2018 Portfolio of Major Defense Acquisition Programs Fiscal year 2019 dollars (in millions) Air Force First full estimate of total acquisition cost Program name Advanced Extremely High Frequency (AEHF) Satellite December 2016 estimated total acquisition cost December 2017 estimated total acquisition cost Percentage change in total acquisition cost since December 2016 Percentage change in total acquisition cost since first full estimate 3,047.97 2,820.02 2,810.43 -0.3 -7.8 12,237.68 25,478.53 25,778.97 1.2 110.7 Airborne Warning and Control System Block 40/45 Upgrade (AWACS Blk 40/45 Upgrade) 3,126.27 3,004.21 3,077.76 2.4 -1.6 B-2 Defensive Management System Modernization (B-2 DMS-M) 2,686.21 2,666.56 2,757.46 3.4 2.7 B61 Mod 12 Life Extension Program Tailkit Assembly (B61 Mod 12 LEP TKA) 1,463.89 1,262.30 1,246.23 -1.3 -14.9 C-130J Hercules Transport Aircraft (C-130J) 1,062.53 17,704.67 16,353.50 -7.6 1439.1 AIM-120 Advanced Medium Range Air-to-Air Missile (AMRAAM) Combat Rescue Helicopter (CRH) 8,698.96 8,852.17 8,365.77 -5.5 -3.8 19,455.06 58,588.83 57,036.66 -2.6 193.2 F-15 Eagle Passive/Active Warning and Survivablity System (F-15 EPAWSS) 4,467.74 2,712.69 2,728.35 0.6 -38.9 F-22 Increment 3.2B Modernization (F-22 Inc 3.2B Mod) 1,677.87 1,549.08 1,556.64 0.5 -7.2 Family of Advanced Beyond Line-of-Sight Terminals (FAB-T) 3,570.19 4,716.69 4,705.63 -0.2 31.8 Global Positioning System III (GPS III) 4,413.46 5,841.52 5,841.86 0.0 32.4 HC/MC-130 Recapitalization Aircraft (HC/MC-130 Recap) 9,364.83 14,591.47 14,660.45 0.5 56.5 Intercontinental Ballistic Missile Fuze Modernization (ICBM Fuze Mod) 1,951.19 1,949.68 1,995.48 2.3 2.3 Joint Air-to-Surface Standoff Missile Extended Range (JASSM-ER) 2,524.44 4,413.57 4,515.70 2.3 78.9 Joint Direct Attack Munition (JDAM) 3,828.09 11,467.07 12,627.86 10.1 229.9 Evolved Expendable Launch Vehicle (EELV) Page 202 GAO-19-336SP Weapon Systems Annual Assessment Appendix V: Total Acquisition Cost Changes for Programs in the Department of Defense’s 2018 Portfolio First full estimate of total acquisition cost December 2016 estimated total acquisition cost December 2017 estimated total acquisition cost Percentage change in total acquisition cost since December 2016 Percentage change in total acquisition cost since first full estimate 49,067.76 41,669.40 41,215.85 -1.1 -16.0 Military GPS User Equipment (MGUE) Increment 1 1,555.80 1,194.15 1,460.63 22.3 -6.1 MQ-9 Reaper Unmanned Aircraft System (MQ-9 Reaper) 2,952.64 13,199.46 13,540.01 2.6 358.6 Next Generation Operational Control System (GPS OCX) 3,707.58 5,636.58 6,277.58 11.4 69.3 Small Diameter Bomb Increment II (SDB II) 5,264.52 4,443.05 4,518.77 1.7 -14.2 Space Based Infrared System High (SBIRS High) 3,977.45 3,505.93 3,507.12 0.0 -11.8 Space Fence Ground-Based Radar System Increment 1 1,684.77 1,589.08 1,588.65 0.0 -5.7 Program name KC-46 Tanker Modernization Program (KC-46A) Army Program name First full estimate of total acquisition cost December 2016 estimated total acquisition cost December 2017 estimated total acquisition cost Percentage change in total acquisition cost since December 2016 Percentage change in total acquisition cost since first full estimate AH-64E Apache New Build (AH-64E New Build) 2,653.25 1,965.61 1,959.75 -0.3% -26.1% AH-64E Apache Remanufacture (AH-64E Remanufacture) 8,109.14 14,262.22 14,498.81 1.7% 78.8% Airborne and Maritime/Fixed Station (AMF) 9,129.91 3,360.91 3,437.75 2.3% -62.3% Armored Multi-Purpose Vehicle (AMPV) 11,405.72 11,489.31 11,607.15 1.0% 1.8% CH-47F Improved Cargo Helicopter (CH-47F) 3,605.14 16,385.69 16,389.77 0.0% 354.6% CH-47F Modernized Cargo Helicopter (CH-47F Block II) 16,759.45 — 16,764.21 0.0% 0.0% Common Infrared Countermeasure (CIRCM) 2,698.18 2,723.28 2,699.78 -0.9% 0.1% Guided Multiple Launch Rocket System/Guided Multiple Launch Rocket Sys Alt Warhead (GMLRS/GMLRS AW) 1,980.00 7,827.56 15,502.47 98.0% 683.0% Integrated Air and Missile Defense (IAMD) 5,630.19 7,229.31 7,249.72 0.3% 28.8% Joint Air-to-Ground Missile (JAGM) 6,029.88 6,022.65 6,168.67 2.4% 2.3% Joint Light Tactical Vehicle (JLTV) 25,232.83 21,626.50 24,058.17 11.2% -4.7% Joint Tactical Radio System Handheld, Manpack, and Small Form Fit Radios (JTRS HMS) 11,239.51 9,756.12 9,510.22 -2.5% -15.4% M109A7 Family of Vehicles (M109A7 FOV) 7,487.88 7,584.57 7,726.43 1.9% 3.2% M88A2 Heavy Equipment Recovery Combat Utility Lift Evacuation System (M88A2 Hercules) 3,370.14 3,320.29 3,397.43 2.3% 0.8% MQ-1C Gray Eagle Unmanned Aircraft System (MQ-1C Gray Eagle) 1,136.80 6,137.44 6,336.29 3.2% 457.4% Page 203 GAO-19-336SP Weapon Systems Annual Assessment Appendix V: Total Acquisition Cost Changes for Programs in the Department of Defense’s 2018 Portfolio Patriot Advanced Capability-3 Missile Segment Enhancement (PAC-3 MSE) 8,159.63 6,965.27 8,924.85 28.1% 9.4% UH-60M Black Hawk Helicopter (UH-60M Black Hawk) 14,523.29 28,348.81 27,996.04 -1.2% 92.8% Warfighter Information Network-Tactical Increment 2 (WIN-T Inc 2) 4,152.07 11,337.44 4,791.84 -57.7% 15.4% Navy First full estimate of total acquisition cost December 2016 estimated total acquisition cost December 2017 estimated total acquisition cost Percentage change in total acquisition cost since December 2016 Advanced Arresting Gear (AAG) 2,301.92 2,115.91 2,319.49 9.6 0.8 AGM-88E Advanced Anti-Radiation Guided Missile (AGM-88E AARGM) 1,792.37 2,817.66 2,816.32 0.0 57.1 AIM-9X Block II Sidewinder (AIM-9X Blk II) 4,473.22 3,493.29 3,601.53 3.1 -19.5 Air and Missile Defense Radar (AMDR) 6,258.95 5,776.41 5,552.92 -3.9 -11.3 Program name Amphibious Combat Vehicle (ACV) CH-53K Heavy Lift Replacement Helicopter (CH-53K) Columbia Class Ballistic Missile Submarine (SSBN 826) Cooperative Engagement Capability (CEC) Percentage change in total acquisition cost since first full estimate 1,964.20 1,861.09 1,845.39 -0.8 -6.0 18,538.42 28,687.44 28,845.42 0.6 55.6 103,556.42 102,596.09 103,164.08 0.6 -0.4 3,295.61 6,533.58 6,795.16 4.0 106.2 DDG 1000 Zumwalt Class Destroyer (DDG 1000) 38,964.66 24,786.55 25,415.67 2.5 -34.8 DDG 51 Arleigh Burke Class Guided Missile Destroyer (DDG 51) 17,005.49 129,791.36 137,962.48 6.3 711.3 E-2D Advanced Hawkeye Aircraft (E-2D AHE) 16,518.04 22,734.87 22,320.77 -1.8 35.1 EA-18G Growler Aircraft (EA-18G) 10,050.63 16,982.42 17,017.58 0.2 69.3 8,978.04 — 9,024.06 0.5 0.5 Fleet Replenishment Oiler (T-AO 205 Class) Gerald R. Ford Class Nuclear Aircraft Carrier (CVN 78) 39,835.11 40,140.46 49,107.01 22.3 23.3 Ground/Air Task Oriented Radar (G/ATOR) 1,641.37 2,951.48 3,110.54 5.4 89.5 H-1 Upgrades (4BW/4BN) (H-1 Upgrades) 4,060.64 13,318.19 13,538.48 1.7 233.4 Infrared Search and Track (IRST) 2,253.00 2,240.41 2,244.76 0.2 -0.4 Integrated Defensive Electronic Countermeasures (IDECM) Block 4 777.48 1,218.17 1,273.36 4.5 63.8 Integrated Defensive Electronic Countermeasures (IDECM) Blocks 2/3 1,660.27 1,891.86 1,257.00 -33.6 -24.3 Joint Precision Approach and Landing System 1,133.44 1,919.50 1,917.61 -0.1 69.2 10,619.78 9,943.88 10,118.80 1.8 -4.7 LHA 6 America Class Amphibious Assault Ship (LHA 6) 3,560.70 10,486.82 10,674.30 1.8 199.8 Littoral Combat Ship - Mission Modules (LCS Packages) 7,336.98 7,378.49 6,418.17 -13.0 -12.5 Littoral Combat Ship (LCS) 2,512.72 21,063.14 21,281.43 1.0 746.9 KC-130J Transport Aircraft (KC-130J) Page 204 GAO-19-336SP Weapon Systems Annual Assessment Appendix V: Total Acquisition Cost Changes for Programs in the Department of Defense’s 2018 Portfolio First full estimate of total acquisition cost December 2016 estimated total acquisition cost December 2017 estimated total acquisition cost Percentage change in total acquisition cost since December 2016 Percentage change in total acquisition cost since first full estimate 13,113.42 23,650.43 23,744.07 0.4 81.1 7,526.23 7,067.16 7,075.03 0.1 -6.0 14,385.15 15,697.35 15,824.93 0.8 10.0 MQ-8 Fire Scout 2,928.35 2,999.90 3,103.84 3.5 6.0 Multifunctional Information Distribution System (MIDS) 1,460.21 5,267.83 5,661.54 7.5 287.7 Navy Multiband Terminal (NMT) 2,598.84 2,468.86 2,493.31 1.0 -4.1 Next Generation Jammer Mid-Band 7,843.32 7,856.45 8,133.25 3.5 3.7 Offensive Anti-Surface Warfare Increment 1 (OASuW Inc 1) 1,577.57 1,652.51 1,885.17 14.1 19.5 34,750.79 35,464.06 36,589.99 3.2 5.3 4,426.20 4,677.75 4,752.17 1.6 7.4 Program name LPD 17 San Antonio Class Amphibious Transport Dock (LPD 17) Mobile User Objective System (MUOS) MQ-4C Triton Unmanned Aircraft System (MQ-4C Triton) P-8A Poseidon Multi-Mission Maritime Aircraft (P-8A) Ship to Shore Connector Amphibious Craft (SSC) SSN 774 Virginia Class Submarine (SSN 774) 67,684.81 138,746.96 139,807.62 0.8 106.6 Standard Missile-6 (SM-6) 6,382.87 9,791.58 8,610.05 -12.1 34.9 Tactical Tomahawk RGM-109E/UGM 109E Missile (TACTOM) 2,369.41 7,607.94 7,698.00 1.2 224.9 Trident II (D-5) Sea-Launched Ballistic Missile UGM 133A (Trident II Missile) 52,863.96 60,356.14 60,718.46 0.6 14.9 V-22 Osprey Joint Services Advanced Vertical Lift Aircraft (V-22) 44,897.84 65,856.98 66,057.54 0.3 47.1 4,999.14 4,917.86 4,842.17 -1.5 -3.1 First full estimate of total acquisition cost December 2016 estimated total acquisition cost December 2017 estimated total acquisition cost Percentage change in total acquisition cost since December 2016 Percentage change in total acquisition cost since first full estimate 2,958.59 13,709.55 13,741.68 0.2% 364.5% 239,291.99 359,570.89 360,097.25 0.1% 50.5% VH-92A Presidential Helicopter Replacement Program Joint DOD-wide Program name Chemical Demilitarization-Assembled Chemical Weapons Alternatives (Chem Demil-ACWA) F-35 Lightning II Program (F-35) Source: GAO analysis of DOD data. GAO-19-336SP. Note: We obtained data for this table from DOD’s Selected Acquisition Reports Page 205 GAO-19-336SP Weapon Systems Annual Assessment Appendix VI: Performance against GAO, DOD, and the Office of Management Budget Cost Metrics Appendix VI: Performance against GAO, DOD, and the Office of Management Budget Cost Metrics Figure 25: Percentage of Programs in Department of Defense’s (DOD) Last Six Portfolios, the Cost Growth of Which Remained Under Acceptable Limits over the Last Year, 5 Years, and Since First Full Estimate . Page 206 GAO-19-336SP Weapon Systems Annual Assessment Appendix VII: Program Rankings based on Total Acquisition Cost Change, Not Due to Quantity Changes, since First Full Estimates Appendix VII: Program Rankings based on Total Acquisition Cost Change, Not Due to Quantity Changes, since First Full Estimates Tables 21 and 22 rank the programs in the 2018 portfolio by their total acquisition cost change across the managing military departments. We omitted amounts related to changes in quantity from the cost change calculations. Rankings are calculated based on the total cost, as well as the percentage change, in total acquisition expenditures since programs’ first full estimates. Table 21: Five Programs with the Lowest and Highest Total Acquisition Cost Changes by Total Dollars since First Full Estimates (fiscal year 2019 dollars in millions) Air Force Program Army Change (in dollars) Program Navy Change (in dollars) Program Change (in dollars) Lowest cost change 1. KC-46A -7,851.91 1. JLTV -2,761.52 1. DDG 51 2. JDAM -3,346.45 2. AMF -2,407.79 2. MIDS -25,066.81 -5,039.60 3. HC/MC-130 Recap -1,577.44 3. AH-64E New Build -930.40 3. CVN 78 -2,213.62 4. SDB II -745.75 4. IAMD -660.01 4. AIM-9X Blk II -871.69 5. SBIRS High -470.33 5. PAC-3 MSE -198.94 5. AMDR -706.02 Highest cost change 1. EELV 39,607.54 1. UH-60M Black Hawk 11,786.87 1. V-22 40,945.63 2. AMRAAM 13,427.50 2. GMLRS/ GMLRS AW 11,234.73 2. SSN 774 34,542.19 10,101.76 3. DDG 1000 19,451.31 16,068.45 3. MQ-9 Reaper 3,202.24 3. CH-47F 4. GPS OCX 2,570.00 4. AH-64E Remanufacture 5,967.27 4. Trident II 5. JASSM-ER 1,648.47 5. MQ-1C Gray Eagle 2,994.42 5. LPD 17 9,547.40 Source: GAO analysis of Department of Defense data. GAO-19-336SP Page 207 GAO-19-336SP Weapon Systems Annual Assessment Appendix VII: Program Rankings based on Total Acquisition Cost Change, Not Due to Quantity Changes, since First Full Estimates Table 22: Five Programs with the Lowest and Highest Total Acquisition Cost Changes by Percentage since First Full Estimates Air Force Program Army Change (percent) Program Navy Change (percent) Program Change (percent) Lowest percentage change 1. JDAM -87 1. AH-64E New Build -35 1. MIDS -345 2. HC/MC-130 Recap -17 2. AMF -26 2. DDG 51 -147 3. KC-46A -16 3. IAMD -12 3. AIM-9X Blk II -19 4. B61 Mod 12 LEP TKA -15 4. JLTV -11 4. AMDR -11 5. SDB II -14 5. PAC-3 MSE 1. EELV 204 1. GMLRS/ GMLRS AW 567 1. H-1 Upgrades 216 2. AMRAAM 110 2. CH-47F 280 2. LCS 204 3. MQ-9 Reaper 108 3. MQ-1C Gray Eagle 263 3. G/ATOR 111 -2 5. IDECM Block 4 -6 Highest percentage change 4. GPS OCX 69 4. UH-60M Black Hawk 81 4. V-22 91 5. JASSM-ER 65 5. AH-64E Remanufacture 74 5. CEC 78 Source: GAO analysis of Department of Defense data. GAO-19-336SP Page 208 GAO-19-336SP Weapon Systems Annual Assessment Appendix VIII: Technology Readiness Levels Appendix VIII: Technology Readiness Levels Table 23: Technology Readiness Levels and Descriptions Technology readiness level Description 1. Basic principles observed and reported Lowest level of technology None (paper studies and readiness. Scientific research analysis) begins to be translated into applied research and development. Examples might include paper studies of a technology’s basic properties. None 2. Technology concept and/or application formulated Invention begins. Once basic None (paper studies and principles are observed, practical analysis) applications can be invented. The application is speculative and there is no proof or detailed analysis to support the assumption. Examples are still limited to paper studies. None 3. Analytical and experimental critical function and/or characteristic proof of concept Active research and development is initiated. This includes analytical studies and laboratory studies to physically validate analytical predictions of separate elements of the technology. Examples include components that are not yet integrated or representative. Analytical studies and Lab demonstration of non-scale individual components (pieces of subsystem) 4. Component and/or breadboard validation in laboratory environment Basic technological components are integrated to establish that the pieces will work together. This is relatively “low fidelity” compared to the eventual system. Examples include integration of “ad hoc” hardware in a laboratory. Low-fidelity breadboard. Integration of nonscale components to show pieces will work together. Not fully functional or form or fit but representative of technically feasible approach suitable for flight articles. Lab 5. Component and/or breadboard validation in relevant environment Fidelity of breadboard technology increases significantly. The basic technological components are integrated with reasonably realistic supporting elements so that the technology can be tested in a simulated environment. Examples include “high fidelity” laboratory integration of components. High-fidelity breadboard. Functionally equivalent but not necessarily form and/or fit (size weight, materials, etc.). Should be approaching appropriate scale. May include integration of several components with reasonably realistic support elements/subsystems to demonstrate functionality. Lab demonstrating functionality but not form and fit. May include flight demonstrating breadboard in surrogate aircraft. Technology ready for detailed design studies. Page 209 Hardware/software Demonstration environment GAO-19-336SP Weapon Systems Annual Assessment Appendix VIII: Technology Readiness Levels Technology readiness level Description Hardware/software Demonstration environment 6. System/subsystem model or prototype demonstration in a relevant environment Representative model or prototype system, which is well beyond the breadboard tested for TRL 5, is tested in a relevant environment. Represents a major step up in a technology’s demonstrated readiness. Examples include testing a prototype in a high fidelity laboratory environment or in simulated realistic environment. Prototype. Should be very close to form, fit and function. Probably includes the integration of many new components and realistic supporting elements/subsystems if needed to demonstrate full functionality of the subsystem. High-fidelity lab demonstration or limited/restricted flight demonstration for a relevant environment. Integration of technology is well defined. 7. System prototype demonstration in an operational environment Prototype near or at planned operational system. Represents a major step up from TRL 6, requiring the demonstration of an actual system prototype in a realistic environment, such as in an aircraft, vehicle, or space. Examples include testing the prototype in a test bed aircraft. Prototype. Should be form, fit and function integrated with other key supporting elements/subsystems to demonstrate full functionality of subsystem. Flight demonstration in representative realistic environment such as flying test bed or demonstrator aircraft. Technology is well substantiated with test data. 8. Actual system completed and Technology has been proven to Flight-qualified hardware “flight qualified” through test work in its final form and under and demonstration expected conditions. In almost all cases, this TRL represents the end of true system development. Examples include developmental test and evaluation of the system in its intended weapon system to determine if it meets design specifications. Developmental Test and Evaluation in the actual system application. 9. Actual system “flight proven” through successful mission operations Operational Test and Evaluation in operational mission conditions. Actual application of the technology Actual system in final form in its final form and under mission conditions, such as those encountered in operational test and evaluation. In almost all cases, this is the end of the last “bug fixing” aspects of true system development. Examples include using the system under operational mission conditions. Source: GAO and its analysis of National Aeronautics and Space Administration data. GAO-19-336SP Page 210 GAO-19-336SP Weapon Systems Annual Assessment Appendix IX: Comments from the Department of Defense Appendix IX: Comments from the Department of Defense Page 211 GAO-19-336SP Weapon Systems Annual Assessment Appendix X: GAO Contacts and Staff Acknowledgments Appendix X: GAO Contacts and Staff Acknowledgments GAO Contact Shelby S. Oakley, (202) 512-4841 or oakleys@gao.gov Staff Acknowledgments Principal contributors to this report were Christopher R. Durbin, Assistant Director; Brenna Derritt, Program Assessments Analyst-in-Charge; Marcus C. Ferguson, Portfolio Analysis Analyst-in-Charge; Dennis A. Antonio, Emily Bond; Jasmina Clyburn; Tana M. Davis; Wendy P. Smythe; and Robin M. Wilson. Other key contributors included Cheryl K. Andrew, Stephen Babb, David B. Best, Edwin B. Booth, Raj Chitikila, Matthew T. Crosby, Kevin Dooley, Arthur Gallegos, Jeffrey L. Hartnett, Rich Horiuchi, Justin M. Jaynes, Helena Johnson, J. Kristopher Keener, Jill N. Lacey, Travis J. Masters, LaTonya D. Miller, Diana Moldafsky, Scott Purdy, Carl Ramirez, Beth Reed Fritts, Ronald E. Schwenn, Charlie Shivers III, Jay Tallon, Brian A. Tittle, Bruce H. Thomas, Nathan A. Tranquilli, Abby C. Volk, J. Andrew Walker, Alyssa B. Weir, and Khristi A. Wilkins. Table 24 lists the staff responsible for individual program assessments. Table 24: GAO Staff Responsible for Individual Program Assessments Program Name Primary Staff Army Programs Armored Multi-Purpose Vehicle (AMPV) Charlie Shivers III, Eli DeVan CH-47F Modernized Cargo Helicopter (CH-47F Block II) William Allbritton, Cale Jones Common Infrared Countermeasure (CIRCM) Tana M. Davis, William Allbritton Handheld, Manpack, and Small Form Fit Radios (HMS) Scott Purdy , Guisseli Reyes-Turnell Improved Turbine Engine Program (ITEP) Wendy Smythe , Jasmina Clyburn Indirect Fire Protection Capability Increment 2 - Intercept, Block 1 (IFPC Brian Smith , Zach Sivo Inc 2-I Block 1) Integrated Air and Missile Defense (IAMD) Juli Steinhouse , Julie Clark Joint Air-to-Ground Missile (JAGM) Jessica Berkholtz , Wendy Smythe Joint Light Tactical Vehicle (JLTV) Andrea Evans, David Wishard Precision Strike Missile (formerly Long Range Precision Fires) (PrSM (formerly LRPF)) Cale Jones , A. Maurice Robinson Navy and Marine Corps Programs Air and Missile Defense Radar (AMDR) Nathan P. Foster, Caryn Kuebler Amphibious Assault Ship Bougainville (LHA 8) Jeffrey L. Hartnett, Sara Rizik Amphibious Combat Vehicle Phase 1 Increment 1 (ACV 1.1) Matthew M. Shaffer, Holly Williams Amphibious Ship Program (LPD 17 Flight II) Holly Williams, Sarah Tempel Page 212 GAO-19-336SP Weapon Systems Annual Assessment Appendix X: GAO Contacts and Staff Acknowledgments Program Name Primary Staff CH-53K King Stallion (CH-53K) Lauren Wright, Victoria Klepacz CVN 78 Gerald R. Ford Class Nuclear Aircraft Carrier / Advanced Arresting Gear (CVN 78/AAG) Jessica Karnis, Burns Eckert DDG 1000 Zumwalt Class Destroyer (DDG 1000) Laurier Fish, Ann Halbert-Brooks DDG 51 Arleigh Burke Class Guided Missile Destroyer - Flight III (DDG 51 Flight III) Katherine M. Pfeiffer, Laura Jezewski Ground/Air Task Oriented Radar (G/ATOR) Joe E. Hunter, Claire Li Guided Missile Frigate (FFG(X)) Sean Merrill, Jillena Roberts Infrared Search and Track (IRST) Zachary J. Sivo, Jay Still Joint Precision Approach and Landing System (JPALS) Stephen V. Marchesani, Jennifer A. Dougherty Littoral Combat Ship Mission Modules (includes LCS) (LCS Packages) Tonya Woodbury, Chris Zakroff MQ-25 Stingray (MQ-25) Jillena Roberts, Christopher Lee MQ-4C Triton Unmanned Aircraft System (MQ-4C Triton) Erin Stockdale, James Kim Next Generation Jammer Low Band (NGJ Low Band) Raffaele Roffo, Carmen Yeung Next Generation Jammer Mid-Band (NGJ Mid-Band) Carmen Yeung, Ralph Roffo Offensive Anti-Surface Warfare Increment 1 (Long Range Anti-Ship Missile) (OASuW Inc 1 (LRASM)) Thomas P. Twambly, Leslie Ashton P-8A Poseidon, Increment 3 (P-8A Increment 3) Heather Barker Miller, Jocelyn Yin Ship to Shore Connector Amphibious Craft (SSC) Teague Lyons, Jon Muchin SSBN 826 COLUMBIA Class Submarine (SSBN 826) Nathaniel Vaught, Laura Jezewski SSN 774 Virginia Class Submarine Block V (SSN 774 Block V) Jenny Shinn, James Madar T-AO 205 John Lewis Class Fleet Replenishment Oiler (T-AO 205 Class) Matthew J. Ambrose, Jocelyn Yin VH-92A Presidential Helicopter (VH-92A) Bonita Oden, Anne McDonough Air Force Programs Advanced Pilot Training (APT) Jean Lee, Marvin Bonner B-2 Defensive Management System - Modernization (B-2 DMS-M) Megan Setser, Don Springman B-52 Radar Modernization Program (RMP) (B-52 RMP) Matthew C. Metz, Suzanne Sterling Combat Rescue Helicopter (CRH) Sean Seales, Matt Drerup F-15 Eagle Passive Active Warning Survivability System (F-15 EPAWSS) Matthew Drerup, LeAnna Parkey Family of Advanced Beyond Line-of-Sight Terminals (FAB-T) Andrew Berglund, Bruna Oliveira Global Positioning System III (GPS III) Jonathan Mulcare, Patrick Breiding Global Positioning System III Follow-On Production (GPS IIIF) Jonathan Mulcare, Andrew Berglund KC-46A Tanker Modernization (KC-46A) Katheryn Hubbell, Nathaniel Vaught Military Global Positioning System (GPS) User Equipment Increment 1 (MGUE Inc 1) Andrew Redd, Erin Carson Next Generation Operational Control System (OCX) Patrick Breiding, Juli Steinhouse Page 213 GAO-19-336SP Weapon Systems Annual Assessment Appendix X: GAO Contacts and Staff Acknowledgments Program Name Primary Staff Small Diameter Bomb Increment II (SDB II) Suzanne Sterling, John Crawford Space Fence Ground-Based Radar System Increment 1 (Space Fence Inc 1) Laura Hook, Scott Purdy Utility Helicopter Replacement Program (UH-1N Replacement) Lindsey Cross, Gina Flacco VC-25B Presidential Aircraft Recapitalization (VC-25B PAR) LeAnna Parkey, Jenny Shinn Weather System Follow-on (WSF) Maricela Cherveny, Brenna Derritt Joint Department of Defense Programs F-35 Lightning II Joint Strike Fighter (JSF) Program (F-35) Jen Baker, Desiree E. Cunningham Source: GAO. GAO-19-336SP Page 214 GAO-19-336SP Weapon Systems Annual Assessment Related GAO Products Related GAO Products F-35 Joint Strike Fighter: Action Needed to Improve Reliability and Prepare for Modernization Efforts, GAO-19-341. Washington, D.C.: April 29, 2019. Columbia Class Submarine: Overly Optimistic Cost Estimate Will Likely Lead to Budget Increases, GAO-19-497. Washington, D.C.: April 8, 2019. High-Risk Series: Substantial Efforts Needed to Achieve Greater Progress on High-Risk Areas, GAO-19-157SP. Washington, D.C.: March 6, 2019. Army Modernization: Steps Needed to Ensure Army Futures Command Fully Applies Leading Practices, GAO-19-132. Washington, D.C.: January 23, 2019. Weapon Systems Cybersecurity: DOD Just Beginning to Grapple with Scale of Vulnerabilities. GAO-19-128. Washington, D.C.: October 9, 2018. Navy Shipbuilding: Past Performance Provides Valuable Lessons for Future Investments. GAO-18-238SP. Washington, D.C.: June 6, 2018. Missile Defense: The Warfighter and Decision Makers Would Benefit from Better Communication about the System’s Capabilities and Limitations. GAO-18-324. Washington, D.C.: May 30, 2018. Weapon Systems Annual Assessment: Knowledge Gaps Pose Risks to Sustaining Recent Positive Trends. GAO-18-360SP. Washington, D.C.: April 25, 2018. Defense Acquisition Workforce: Opportunities Exist to Improve Practices for Developing Program Managers. GAO-18-217. Washington, D.C.: February 15, 2018. Global Positioning System: Better Planning and Coordination Needed to Improve Prospects for Fielding Modernized Capabilities. GAO-18-74. Washington, D.C.: December 12, 2017. Defense Science and Technology: Adopting Best Practices Can Improve Innovation Investments and Management. GAO-17-499. Washington, D.C.: June 29, 2017. Page 215 GAO-19-336SP Weapon Systems Annual Assessment Related GAO Products Ford-Class Aircraft Carrier: Follow-On Ships Need More Frequent and Accurate Cost Estimates to Avoid Pitfalls of Lead Ship. GAO-17-575. Washington, D.C.: June 13, 2017. Contracting Data Analysis: Assessment of Government-wide Trends. GAO-17-244SP. Washington, D.C.: March 9, 2017. Weapon System Requirements: Detailed Systems Engineering Prior to Product Development Positions Programs for Success. GAO-17-77. Washington, D.C.: November 17, 2016. Technology Readiness Assessment Guide: Best Practices for Evaluating the Readiness of Technology for Use in Acquisition Programs and Projects. GAO-16-410G. Washington, D.C. August, 2016. Defense Contracting: DOD’s Use of Competitive Procedures. GAO-15-484R. Washington, D.C.: May 1, 2015. Federal Subcontracting: Linking Small Business Subcontractors to Prime Contracts Is Not Feasible Using Current Systems. GAO-15-116. Washington, D.C.: December 11, 2014. Defense Contracting: Early Attention in the Acquisition Process Needed to Enhance Competition. GAO-14-395. Washington, D.C.: May 5, 2014. Defense Contracting: Actions Needed to Increase Competition. GAO-13-325. Washington, D.C.: March 28, 2013. Best Practices: DOD Can Achieve Better Outcomes by Standardizing the Way Manufacturing Risks Are Managed. GAO-10-439. Washington, D.C.: April 22, 2010. Best Practices: High Levels of Knowledge at Key Points Differentiate Commercial Shipbuilding from Navy Shipbuilding. GAO-09-322. Washington, D.C.: May 13, 2009. Defense Acquisitions: A Knowledge-Based Funding Approach Could Improve Major Weapon System Program Outcomes. GAO-08-619. Washington, D.C.: July 2, 2008. Defense Acquisitions: Stronger Management Practices are Needed to Improve DOD’s Software-Intensive Weapon Acquisitions. GAO-04-393. Washington, D.C.: March 1, 2004. Page 216 GAO-19-336SP Weapon Systems Annual Assessment Related GAO Products Best Practices: Capturing Design and Manufacturing Knowledge Early Improves Acquisition Outcomes. GAO-02-701. Washington, D.C.: July 15, 2002. Best Practices: Better Matching of Needs and Resources Will Lead to Better Weapon System Outcomes. GAO-01-288. Washington, D.C.: March 8, 2001. Best Practices: Better Management of Technology Development Can Improve Weapon System Outcomes. GAO/NSIAD-99-162. Washington, D.C.: July 30, 1999. (102769) Page 217 GAO-19-336SP Weapon Systems Annual Assessment GAO’s Mission The Government Accountability Office, the audit, evaluation, and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO’s commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony The fastest and easiest way to obtain copies of GAO documents at no cost is through GAO’s website (https://www.gao.gov). Each weekday afternoon, GAO posts on its website newly released reports, testimony, and correspondence. To have GAO e-mail you a list of newly posted products, go to https://www.gao.gov and select “E-mail Updates.” Order by Phone The price of each GAO publication reflects GAO’s actual cost of production and distribution and depends on the number of pages in the publication and whether the publication is printed in color or black and white. Pricing and ordering information is posted on GAO’s website, https://www.gao.gov/ordering.htm. Place orders by calling (202) 512-6000, toll free (866) 801-7077, or TDD (202) 512-2537. Orders may be paid for using American Express, Discover Card, MasterCard, Visa, check, or money order. Call for additional information. Connect with GAO Connect with GAO on Facebook, Flickr, Twitter, and YouTube. Subscribe to our RSS Feeds or E-mail Updates. Listen to our Podcasts. Visit GAO on the web at https://www.gao.gov. To Report Fraud, Waste, and Abuse in Federal Programs Contact FraudNet: Congressional Relations Orice Williams Brown, Managing Director, WilliamsO@gao.gov, (202) 512-4400, U.S. Government Accountability Office, 441 G Street NW, Room 7125, Washington, DC 20548 Public Affairs Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800 U.S. Government Accountability Office, 441 G Street NW, Room 7149 Washington, DC 20548 Strategic Planning and External Liaison James-Christian Blockwood, Managing Director, spel@gao.gov, (202) 512-4707 U.S. Government Accountability Office, 441 G Street NW, Room 7814, Washington, DC 20548 Website: https://www.gao.gov/fraudnet/fraudnet.htm Automated answering system: (800) 424-5454 or (202) 512-7700 Please Print on Recycled Paper.