FY19 DOD PROGRAMS F-35 Joint Strike Fighter (JSF) Executive Summary Programmatics Block 4 • The Joint Strike Fighter (JSF) program continues to carry 873 unresolved deficiencies, most of which were identified prior to the completion of System Development and Demonstration (SDD) and entry into IOT&E. Although the program is working to fix deficiencies, new discoveries are still being made, resulting in only a minor decrease in the overall number of deficiencies. There are many significant deficiencies that should be addressed to ensure the SDD baseline configuration is stable prior to introducing the large number of new capabilities planned in Block 4. • The current Continuous Capability Development and Delivery (C2D2) process has not been able to keep pace with adding new increments of capability as planned. Software changes, intended to introduce new capabilities or fix deficiencies, often introduced stability problems and adversely affected other functionality. Due to these inefficiencies, along with a large amount of planned new capabilities, DOT&E considers the program’s current Revision 13 master schedule to be high risk. • Although the program planned a greater dependence on modeling and simulation (M&S) in C2D2 than was used during SDD, no significant changes in the simulation venues have occurred. The program has established internal processes to aid in the development and enhancement of adequate M&S capabilities; however, planning and full funding are not complete. • Adequate evaluations of Block 4 capabilities will require the use of Open-Air Battle-Shaping (OABS) instrumentation, the Joint Simulation Environment (JSE), and Radar Signal Emulators (RSE). Static Structural and Durability Testing • The program secured funding and contracted to procure another F-35B ground test article, which will have a redesigned wing-carry-through structure that is production representative of Lot 9 and later F-35B aircraft. Testing of this production-representative ground test article will allow the program to certify the life of F-35B design improvements. The production and delivery dates are still to be determined. Operational Effectiveness Initial Operational Test and Evaluation (IOT&E) • DOT&E approved entering formal IOT&E on December 3, 2018, and the JSF Operational Test Team (JOTT) flew the first open-air mission trial on December 5, 2018. The JOTT completed numerous pre-IOT&E events, all previously approved by DOT&E for execution, earlier in CY18. • Formal start of IOT&E was delayed as the test teams waited for the program to deliver the final aircraft operational flight program software and associated mission data, to complete the integration of the Air-to-Air Range Infrastructure (AARI) in the F-35, and for fleet inspections and replacement of defective fuel pump tubes that had resulted in the crash of an F-35B. • The JOTT made good progress in managing test execution throughout CY19. RSE integration and operator training on the test ranges as well as suitability deficiencies that limited aircraft availability both affected schedule execution. On September 10, 2019, the JOTT completed the required open‑air testing on the Nevada Test and Training Range (NTTR). Open-air missions against the RSE-based threats on the Point Mugu Sea Range (PMSR), California, remain and are planned to be completed in early CY20. Joint Simulation Environment (JSE) • The IOT&E plan requires 64 mission trials against modern fielded threats in the JSE. • After falling significantly behind previous planned schedules, the government-led JSE team made good progress in the last half of 2019 in completing integration of the F-35 In-A-Box model (i.e., the model that represents F-35 air and mission systems in the JSE) into the high-fidelity threat environment, both of which are likely to meet requirements for IOT&E. • The ongoing IOT&E JSE verification, validation, and accreditation (VV&A) processes must be completed, and consistent independent schedule reviews must be continued throughout Block 4, to ensure they are aligned with the C2D2 processes. The Block 4 VV&A plan must ensure accreditation of the JSE for use in operational testing during the 30R07/08 F-35 software release time frame. Mission Data Load (MDL) Development and Testing • Although the program has initiatives in work, the U.S. Reprogramming Laboratory (USRL) still lacks adequate equipment to be able to fully test and optimize MDLs under JSF 19 FY19 DOD PROGRAMS realistic stressing conditions to ensure performance against current and future threats. • Significant additional investments, well beyond the recent incremental upgrades to the signal generator channels and reprogramming tools, are required now for the USRL to support F-35 Block 4 MDL development. At the time of this report, the program has budgeted for some of these hardware and software tools, but are already late to need for supporting fielded aircraft and Block 4 development. Operational Suitability Autonomic Logistics Information System (ALIS) • Although the program released several new versions of ALIS in 2019 that improved ALIS usability, these improvements did not eliminate the major problems in ALIS design and implementation. These deficiencies caused delays in troubleshooting and returning broken aircraft to mission capable status. It is unclear that new approaches, such as ALIS NEXT and “Mad Hatter” will sufficiently improve ALIS, or if more resources are needed. ALIS NEXT is a cloud-focused, government-owned re-architecture of ALIS, and Mad Hatter is an agile process designed to streamline new ALIS software through development, testing, and fielding on a nearly continual basis. Additionally, the program is working to develop a detailed plan for how these separate efforts will be integrated into a new version of ALIS while continuing to support fleet operations. Cybersecurity Operational Testing • Cybersecurity testing to date during IOT&E continued to demonstrate that deficiencies and vulnerabilities identified during earlier testing periods have not been remedied. More testing is needed to assess cybersecurity of the latest ALIS 3.5 release and in the air vehicle itself. Availability, Reliability, and Maintainability • Although the fleet-wide trend in aircraft availability showed modest improvement in 2019, it remains below the target value of 65 percent. • No significant portion of the fleet, including the combat-coded fleet, was able to achieve and sustain the DOD mission capable (MC) rate goal of 80 percent. However, individual units have been able to achieve the 80 percent target for short periods during deployed operations. • Reliability and maintainability (R&M) metrics defined in the JSF Operational Requirements Document (ORD) are not meeting interim goals needed to reach requirements at maturity for the F-35B and F-35C. The F-35A reached 75,000 flight hours in July 2018, the target flight hours referenced in the program’s reliability growth plan for meeting maturity, but still has not reached the ORD threshold values for R&M. Live Fire Test and Evaluation (LFT&E) • In FY18, Lockheed Martin completed the Vulnerability Assessment Report and the Consolidated LFT&E Report. These reports do not include results from Electromagnetic Pulse (EMP) or gun lethality testing, which were still not completed by the end of FY19. 20 JSF • DOT&E is evaluating the F-35 vulnerability data and reports, which will be documented in the combined IOT&E and LFT&E report to be published prior to the Full-Rate Production decision. • The JSF Program Office (JPO) evaluated the chemical and biological agent protection and decontamination systems during dedicated full-up system-level testing. However, the test plan to assess the chemical and biological decontamination of pilot protective equipment is not adequate because the JPO does not plan to test the decontamination process for either the Generation (Gen) III or Gen III Lite Helmet-Mounted Display System (HMDS). • Air-to-ground lethality flight tests of three variants of 25-mm round ammunition against armored and other vehicles, small boats, and plywood mannequins were conducted at the Naval Air Warfare Center Weapons Division facility, Naval Air Weapons Station China Lake, California, from August through December 2017. The target damage results are classified. DOT&E has received and is reviewing test reports containing data required for the gun lethality assessment, but is still awaiting additional data and analytical products from the Program Office to complete the evaluation. System • The F-35 JSF program is a tri-Service, multinational, single‑seat, single-engine family of strike fighter aircraft consisting of three variants: - F-35A Conventional Take-Off and Landing - F-35B Short Take-Off/Vertical-Landing - F-35C Aircraft Carrier Variant • Per the Joint Strike Fighter ORD, the F-35 is designed to operate and survive in the Initial Operational Capability (IOC) and IOC-plus-10-years threat environment (out to 2025, based on the first IOC declaration by the U.S. Marine Corps in 2015). It is also designed to have improved lethality in this environment compared to legacy multi-role aircraft. • Using an active electronically scanned array (AESA) radar and other sensors, the F-35 with Block 3F or later software is intended to employ precision-guided weapons (e.g., Laser‑Guided Bomb, Joint Direct Attack Munition (JDAM), Small Diameter Bomb, Navy Joint Stand-Off Weapon) and air-to-air missiles (e.g., AIM-120 Advanced Medium-Range Air-to-Air Missile (AMRAAM), AIM-9X infrared guided, air-to-air missile), and a 25-mm gun. • The SDD program was designed to provide mission capability in three increments: - Block 1 (initial training; two increments were fielded: Block 1A and Block 1B) - Block 2 (advanced training in Block 2A and limited combat capability with Block 2B) - Block 3 (limited combat capability in Block 3i and full SDD warfighting capability in Block 3F) • Post-SDD development is designed to address deficiencies and add planned Block 4 capabilities via software updates and hardware changes as new configurations are introduced in subsequent production lots. FY19 DOD PROGRAMS Mission Combatant Commanders will employ units equipped with F-35 aircraft in joint operations to attack fixed and mobile land targets, surface combatants at sea, and air threats, including advanced aircraft and cruise missiles, during day or night, in all weather conditions, and in heavily defended areas. Major Contractor Lockheed Martin, Aeronautics Company – Fort Worth, Texas Activity Programmatics System Development and Demonstration Activity • The program continued to evaluate and document air system performance against joint contract specification (JCS) requirements in order to close out the SDD contract. As of September 17, 2019, the program had closed out 493 of the 536 capability requirements. The 43 remaining represent either unmet requirements that require formal revision of the SDD contract (i.e., will never be met), or those requiring additional development and testing to evaluate performance (e.g., third life durability testing or capabilities planned for ALIS 3.5). Assessment • Full closure of the SDD contract may take years to complete. The effects of unmet contract specification requirements may be observed from both operational testing and fielded operations. Post-SDD Development and Modernization Activity: Block 4, 30 Series • The JPO and Lockheed Martin transitioned the development effort to a new process – referred to as C2D2 – starting in CY18 to begin to deliver the Block 4 capabilities, with the objective of correcting deficiencies and providing new capabilities incrementally on 6-month intervals. • The program changed software nomenclature for the initial increments of Block 4 from “3F” used during SDD to “30RXX” for development and “30PXX” for fielding software. The 30 series of software is compatible with the Block 3F aircraft hardware configuration and is being used to address deficiencies and add some Service-prioritized capabilities. • The program recently updated its software release schedule to reflect a delivery process termed “agile.” This process culminates in the delivery of a “Minimum Viable Product” (MVP) to the Services every 6 months. During this 6-month cycle, an aggressive integrated developmental test/operational test (IDT/OT) is to be conducted, resulting in an integrated test team assessment from both DT and OT 7 days after completion of flight test, well before the capability of either DT or OT to fully assess data from flight test missions. This process is then to be followed by delivery of mission planning, mission data, ALIS, joint technical data, flight series data, training simulators, and other support capabilities that were still in development and not tested during the 6-month IDT/OT window. The operational flight program software and support products are then to be bundled together into the MVP (planned to be within 6 months after completion of IDT/OT, but will likely take longer for deliveries that update training simulators and mission data), and delivered to the Services. • The program added Automatic Ground Collision Avoidance System (AGCAS), a priority capability from the Services, in the 30R03 sequence of software. This capability was tested and then fielded in 30P03.03 with the U.S. F-35A and F-35B aircraft. Testing of AGCAS was not yet complete for the F-35C, so it was not fielded in 30P03.03 for that variant. Activity: Block 4, 40 Series • Block 4 development includes the new Technical Refresh (TR)-3 hardware configuration, which will begin developmental testing in CY21 in order to deliver Lot 15 production aircraft starting in CY23. Block 4 is planned to continue to use the C2D2 process, initiated by the program following SDD, to integrate the remaining Decision Memorandum (DM) 90 capabilities. • The program is developing a Block 4 Test and Evaluation Master Plan (TEMP). The draft TEMP is expected to be staffed after the classified and unclassified versions are aligned and ready for delivery to the F-35 Program Executive Officer (PEO), likely by the end of CY19. Assessment • F-35 Block 4 is on OT&E oversight. DOT&E reviews the content of each Block 4 increment and, if the increment contains significant new capabilities or new hardware, it will require a tailored formal OT&E. DOT&E routinely oversees OT for other “agile” programs, and is working to ensure the OT of F-35 capability releases will be as efficient as possible, while maintaining test adequacy. To accomplish this, OT will leverage integrated testing as much as possible while ensuring full system evaluation of the final integrated MVP release. • Adequate mission-level evaluations of Block 4 capabilities will require the use of OABS instrumentation, the JSE, and RSEs. The current OABS instrumentation, in use since F-22 IOT&E in 2004 and now for F-35 IOT&E, is AARI. The OABS, RSEs, and other open-air test capabilities must be used to gather flight test data that will also be used for VV&A of the JSE. Without the open-air test data to validate the modeling, the JSE may not be an accurate representation of F-35 performance and could provide misleading results to acquisition decision-makers, the warfighter, and Congress. JSF 21 FY19 DOD PROGRAMS • DOT&E is coordinating funding for the DOD Test Resource Management Center (TRMC) to provide program management of OABS. The government JSE team, composed of participants of the F-35 JPO and of Naval Air Systems Command (NAVAIR), remains responsible for development and delivery of the F-35 JSE for testing. Use of JSE for adequate testing of near-term Block 4 capabilities is scheduled for the 30R07/08 and 40R02/03 increments of capability. Upgrades to, and reprogramming of, the RSEs will be carried out by the Service range program managers in coordination with DOT&E. The program and Services should fully fund RSE, JSE, and OABS upgrades to meet test adequacy requirements in time for planned test periods. • Operational testing of other DOD tactical and strike aircraft will also require OABS to ensure an adequate evaluation of capabilities in open-air test venues. These aircraft will also require integration in the JSE for operational testing. • With the completion of F-35 IOT&E trials at NTTR, 12 RSEs are being transported to PMSR to support the remaining IOT&E trials there. When the PMSR trials are complete, five RSEs will become the property of the Navy and remain based at PMSR. Two of the 11 RSEs that will remain the property of the Air Force will be transferred to Eglin AFB, Florida, to support ongoing testing on the Eglin ranges, leaving 9 based at NTTR. Neither the nine at NTTR nor the five at PMSR will be sufficient to support some of the future test scenarios necessary for adequate operational testing of the Block 4 F-35. It will be necessary at times to move RSEs between ranges to achieve sufficient numbers for a test. The RSEs are readily capable of moving from range to range, but Block 4 test planning must account for the timing and costs of implementing these moves and the Navy and Air Force ranges must be prepared to coordinate the logistical actions to support these events. • The program is still carrying a large number of deficiencies, most of which were identified prior to the completion of SDD. As of November 4, 2019, the program had 873 open deficiencies, 13 of which were designated Category I. This “technical debt,” especially the most significant deficiencies, should be addressed by the program to ensure the SDD baseline configuration of software and hardware is stable, prior to introducing a large number of new capabilities to the software in the new hardware configuration associated with Block 4. • After almost 2 years and four fielded software releases since completing SDD with Block 3F development in April 2018, 66 percent of the current open deficiencies were identified prior to SDD completion. The program has not been able to address more of these deficiencies for several reasons, including new discoveries with the fielded configurations, contractual problems, and limitations in software development and test capacity. • The current C2D2 process has not delivered new increments of capability at the pace originally planned. The program attempted to field three versions of Block 30RXX software since Block 3F, but was unable to deliver some of the planned 22 JSF • • • • capabilities and adversely affected other previously working capabilities. For example, some software changes to add capabilities or fix deficiencies introduced stability problems or adversely affected other functionality due to the integrated architecture of the avionics hardware, software, weapons, and mission data. Due to these inefficiencies, along with a large amount of planned new capabilities, DOT&E considers the program’s current Revision 13 schedule to be high risk. DOT&E assesses the MVP and “agile” process as high risk due to limited time to evaluate representative IDT/OT data before fielding the software. Testing will not be able to fully assess fielding configuration of the integrated aircraft, software, weapons, mission data, and ALIS capabilities prior to fielding. The aggressive 6-month development and fielding cycle limits time for adequate regression testing and has resulted in significant problems being discovered in the field. For these reasons, a separate (but currently unplanned) OT must be accomplished on the final integrated configuration of the air system prior to being fielded. Although the program plans a greater dependence on M&S in C2D2 than was used during SDD, including using JSE, no other significant change in the laboratories or simulation venues has occurred. The program has established internal processes to aid in the development and enhancement of M&S capabilities. However, it still needs to ensure adequate funding to develop and sustain a robust laboratory and simulation environment, along with adequate VV&A plans that include the use of data from representative open-air missions. These VV&A plans must not only provide accreditation for M&S capabilities used in system development, but also for the use of JSE in 30R07/08, 40R02/03, and future increments. Adequate M&S capabilities are currently not fully planned nor funded as part of the Block 4 development processes. Sustaining multiple hardware configurations of fielded aircraft (i.e., Block 2B, Block 3F, the new electronic warfare (EW) system starting in Lot 11, and eventually TR-3 configured aircraft beginning in Lot 15), while managing a developmental and operational test fleet with updated hardware to support the production of new lot aircraft, continues to be a challenge for the JPO and Services. The Services developed a tail-by-tail accounting of OT aircraft, but critical aircraft, instrumentation, and other test infrastructure modifications (e.g. USRL test capacity, JSE hardware upgrades) are currently not fully programmed and scheduled to support future OT. The cost of software sustainment and testing to support the aforementioned four hardware configurations of aircraft needs to be accurately assessed and programmed into future Service Program Objective Memorandum planning processes. As of the end of September 2019, 430 aircraft had been delivered to the U.S. Services, international partners, and foreign military sales. The program is sustaining six different versions of software to support these aircraft. Additional versions will be needed as the program adds hardware changes through Lot 14, FY19 DOD PROGRAMS at which time the program will have fielded approximately 1,000 aircraft. Static Structural and Durability Testing Activity • Teardown inspections of the F-35A full scale durability test article (AJ-1) were completed in July 2019 and correlations to the finite element models (FEM) are in progress. The FEM data are used to estimate the structural and durability performance of the original design structure. The program expects the F-35A Durability and Damage Tolerance report to be released in February 2020. • Teardown inspections of the original F-35B full scale durability test article (BH-1) were completed in October 2018. The program canceled the third lifetime testing of BH-1 due to the significant amount of discoveries, modifications, and repairs to bulkheads and other structures that caused the F-35B test article to no longer be representative of the wing-carry-through structure in production aircraft. The program secured funding and contracted to procure another F-35B ground test article, designated BH-2, which will have a redesigned wing-carry-through structure that is production representative of Lot 9 and later F-35B aircraft. • Disassembly and teardown of the F-35C durability test article (CJ-1) were completed in November 2019. Testing was stopped during the third lifetime testing in April 2018, following the discovery of more cracking in the Fuselage Station (FS) 518 Fairing Support Frame. The cracking had been discovered near the end of the second lifetime and required repairs before additional testing could proceed. After estimating the cost and time to repair or replace the FS 518 Fairing Support Frame, coupled with other structural parts that had existing damage (i.e., fuel floor segment, bulkheads FS 450, FS 496, FS 556, and front spar repair), the program determined that the third lifetime testing would be discontinued. Assessment • For all F-35 variants, structural and durability testing led to significant discoveries requiring repairs and modifications to production designs, some as late as Lot 12 aircraft, and retrofits to fielded aircraft. • Based on durability test data, there are several life-limited parts on early production F-35 aircraft which require mitigation. In order to mitigate these durability and damage tolerance shortfalls, the program plans to make modifications to these early production aircraft, including the use of laser shock peening to increase fatigue life for specific airframe parts, e.g., bulkheads. The JPO will also continue to use Individual Aircraft Tracking of actual usage to help the Services project changes in timing for required repairs and modifications, and to aid in Fleet Life Management. • For the F-35A and F-35C, expected service life will be determined from the durability and damage tolerance analyses, once completed. Although the program planned for a third lifetime of testing to accumulate data for life extension, if needed, the program has no plans to procure another F-35C ground test article. • Procuring and testing a production-representative F-35B ground test article will allow the program to certify the life of the design improvements. Once on contract, program plan dates will be finalized. • Despite the F-35 program’s FEM-based structural design, static and durability testing, and developmental flight testing, additional structural discoveries requiring repairs and modifications are occurring in the field. For example, the F-35A has gun-related structural problems and the F-35A/C are experiencing longeron (structural component) cracks. The effect on F-35 service life and the need for additional inspection requirements are still being determined. Operational Effectiveness Initial Operational Test and Evaluation (IOT&E) Activity • Although numerous pre-IOT&E events – including cold weather testing, lower-threat open-air missions, deployments to assess sortie generation rate capabilities, alert launches, and weapons events – were completed earlier in CY18, the program was not able to enter formal IOT&E until December 3, 2018. Delays in delivery of the final aircraft operational flight program software and associated mission data, as well as fleet inspections for and replacement of defective fuel pump tubes that had resulted in the crash of an F-35B, postponed the formal start of test. Following DOT&E approval, the JOTT flew the first formal IOT&E open-air mission trial on December 5, 2018. • The JOTT began open-air trials against threat laydowns represented by the RSEs in February 2019. In an attempt to meet schedule expectations, the JOTT flew these trials “at risk” without complete, successful dress-rehearsals to ensure all test range readiness deficiencies were fully addressed. Problems with AARI integration, range networks, RSE operator training and proficiency, test force proficiency, and RSE integration on the test range all contributed to a series of invalid trials being flown from February through March 2019. The JOTT then proposed, and DOT&E concurred, to stop the test missions against RSE-based threat laydowns and focus on other mission trials. Testing against RSE-based threat laydowns resumed in early June, following a focused effort that successfully addressed the series of problems seen in earlier trials. • The JOTT completed the comparison testing between the A-10 and F-35A, as directed by the FY17 National Defense Authorization Act, in March 2019. • In May 2019, DOT&E approved modifications to the test plan for conducting trials in the Defensive Counter Air (DCA) and the Air Interdiction (AI) combined with Destructive/ Suppression of Enemy Air Defense (D-SEAD) mission areas. • DOT&E approved additional changes and deletions of trials in August 2019 associated with the DCA and AI/D-SEAD mission areas, based on the sufficiency of data collected during testing to date. JSF 23 FY19 DOD PROGRAMS • In August 2019, the program began moving range equipment (RSEs) and support equipment from the NTTR to the PMSR in preparation for the remaining open-air trials. • On September 10, 2019, the JOTT completed open-air testing on NTTR. Open-air missions against the RSEs on the PMSR, along with some weapons events, remain and are planned to be completed in early CY20. • The JSE team continued development under NAVAIR management, and began verification activities to support the required IOT&E trials in JSE. Assessment • Delays in completing necessary readiness requirements prevented the start of formal IOT&E in September 2018 as the program had planned. Prior to the start of formal IOT&E, the program had to address a Category 1 deficiency associated with blanking of the cockpit displays, which required development and testing of another version of software. The program was also waiting for the completion of verified “Level 4” mission data and required aircraft modifications and flight clearances. Additionally, following the crash of an F-35B near Beaufort, South Carolina, on September 28, 2018, the entire F-35 fleet was grounded in October 2018 to inspect fuel pump tubes. A number of the OT aircraft required fuel tube replacements as discovered by the inspections, and added to the delay in starting formal IOT&E. • The JOTT made good progress in managing test execution throughout CY19. Delays in completing AARI integration in the F-35, RSE integration and operator training on the test ranges, and suitability problems that limited aircraft availability all affected schedule execution. • In spite of clear requirements for a simulation to complete IOT&E, the program did not manage the development of the JSE to be ready for JSE test trials in CY19, as originally planned. Completion of IOT&E and the report will occur following successful completion of the required IOT&E trials in the JSE, currently projected for September 2020. • Results of the F-35 IOT&E, to support a Full-Rate Production decision now scheduled for FY21, will be in the DOT&E IOT&E report. Joint Simulation Environment (JSE) Activity • The JSE is a man-in-the-loop, F-35 software-in-the-loop mission simulator that will be used to conduct IOT&E scenarios with modern threat types and threat densities, and laydowns that are not able to be replicated on the open-air ranges. Originally slated to be operational by the end of 2017 to support IOT&E spin-up and testing, the JSE encountered significant contractual and developmental delays and is now expected to be ready for IOT&E trials by the summer of 2020, after the completion of open-air IOT&E trials. • The JSE’s physical facilities (i.e., cockpits, visuals, and buildings) and synthetic environment (i.e., terrain, threat, and target digital models) are complete. • The JSE team demonstrated partial capabilities to the JOTT in December 2018 (threats only) and July 2019 (with F-35). The JSE verification and validation (V&V) process started in 24 JSF mid-2019 and initial results were positive. At the time of this report, integration of the F-35 In-A-Box model (which runs actual aircraft software, re-hosted on commercial workstation computers) and models of its weapons with the JSE was nearly complete and planned to undergo user acceptance in late 2019 and early 2020. • The JPO performed an independent review of the JSE schedule in May 2019, resulting in the movement of the expected readiness date for starting IOT&E trials from fall 2019 to July 2020. • The U.S. Air Force plans to replicate the JSE at Nellis AFB, Nevada, and Edwards AFB, California, extending its capabilities to include the integration of models of other U.S. aircraft and weapons. Assessment • The government-led JSE team made slow progress in early CY19 in completing integration of the F-35 In-A-Box model into the high-fidelity threat environment, both of which are likely to meet requirements for IOT&E. Progress improved later in the year and the JPO strengthened the V&V team with the tools and expertise to enable accreditation by the start of IOT&E trials. • During the development demonstrations in December 2018 and July 2019, the JOTT noted progress on threat fidelity, simulator operations and data collection, and facilities. Problems were noted in weapons, sensor functions, and overall JSE stability. The JSE team, working with Lockheed Martin, have corrected most of these problems, and the simulation will likely be ready for upcoming JOTT-led acceptance events in January 2020. • Following the schedule review, the JSE team was consistently meeting most planned timelines and appeared to be on a path to provide a VV&A simulator for IOT&E trials in the summer of 2020. • The IOT&E JSE V&V processes and consistent independent schedule reviews must be continued through Block 4 to ensure JSE will be available to support operational testing. • The additional U.S. Air Force JSE venues may be useful for additional Block 4 operational test activities if the VV&A process support their intended use. Gun Testing Activity • All three F-35 variants have a 25-mm gun. The F-35A gun is internal; the F-35B and F-35C each use an external gun pod. Differences in the outer mold-line fairing mounting make the gun pods unique to a specific variant (i.e., an F-35B gun pod cannot be mounted on an F-35C aircraft). • Units flying newer F-35A aircraft discovered cracks in the outer mold-line coatings and the underlying chine longeron skin, near the gun muzzle, after aircraft returned from flights when the gun was employed. Assessment • Based on F-35A gun testing to date, DOT&E considers the accuracy of the gun, as installed in the F-35A, to be unacceptable. F-35A gun accuracy during SDD failed to meet the contract specification. Investigations into the gun FY19 DOD PROGRAMS mounts of the F-35A revealed misalignments that result in muzzle alignment errors. As a result, the true alignment of each F-35A gun is not known, so the program is considering options to re-boresight and correct gun alignments. • The program has made mission systems software corrections to improve the stability of gun aiming cues. The program also made progress with changes to the gun installation, boresight processes, and hardware. However, testing to confirm the effectiveness of these changes was not yet complete. Until the new hardware and software changes are successfully tested and verified in operationally representative conditions, the F-35A internal gun system remains unacceptable. • Due to the recent cracking near the gun muzzle in newer F-35A aircraft, the U.S. Air Force has restricted the gun to combat use only for production Lot 9 and newer aircraft. • F-35B and F-35C air-to-ground accuracy results to date with the gun pod have been consistent and meet the contract specifications. The results do not show the accuracy errors of the internal F-35A gun. Mission Data Load (MDL) Development and Testing Activity • F-35 effectiveness relies on the MDL, which is a compilation of the mission data files (MDF) needed for operation of the sensors and other mission systems. The MDL works in conjunction with the avionics software and hardware to drive sensor search behaviors and provide target identification parameters. This enables the F-35 avionics to identify, correlate, and respond to sensor detections, such as threat and friendly radar signals. -- The contractor produces an initial set of MDLs for each software version to support preliminary DT. -- The USRL at Eglin AFB, Florida, creates, tests, and verifies operational MDLs – one for OT and training, and one for each potential major geographic area of operation, called an area of responsibility (AOR). The OT and fielded aircraft use the applicable USRL-generated MDLs for each AOR. • Testing of the USRL MDLs is an operational test activity, as arranged by the JPO after the program restructure in 2010, and consists of laboratory and flight testing on OT aircraft. Testing of the USRL MDL is ongoing as part of IOT&E and will be included in operational testing during C2D2. • As part of IOT&E, the USRL completed an Emergency Reprogramming Exercise (ERE) in CY19. This was the second of two Rapid Reprogramming Exercises (RRE) conducted as part of F-35 OT, the first being an Urgent Reprogramming Exercise (URE) conducted on Block 2B in 2016. The URE differed from the ERE in that the former was accomplished during normal business hours, but with the use of all available resources; the ERE was done around-the-clock until the MDL was produced and uploaded to the system used to electronically transmit MDLs to operational units. The ERE in CY19 evaluated the ability of the USRL, with its hardware and software tools, to respond to an emergency request to modify the mission data in response to a new threat or a change to an existing threat. Assessment • Because MDLs are software components essential to F-35 mission capability, the DOD must have a reprogramming lab that is capable of rapidly creating, testing, and optimizing MDLs, as well as verifying their functionality under stressing conditions representative of real-world scenarios. -- The USRL demonstrated the capability to create functioning MDLs for Block 3F and earlier blocks during SDD. However, the process is slow and the USRL still lacks adequate equipment to be able to test and optimize MDLs under conditions stressing enough to ensure adequate performance against current and future threats in combat. -- For example, the USRL lacks a sufficient number of high‑fidelity radio frequency signal generator channels, which are used to stimulate the F-35 EW system and functions of the radar, with simulated threat radar signals. This situation has improved as of the writing of this report, but additional improvements, above and beyond those currently planned, are required. Also, some of the USRL equipment lacks the ability to accurately pass the simulated signals to the F-35 sensors in a way that replicates open-air performance. -- In 2019, both USRL mission data test lines were upgraded from three to eight high-fidelity signal generator channels. Eight high-fidelity channels per line represents a substantial improvement, but is still far short of the 16-20 recommended in the JPO’s own 2014 gap analysis. -- Even with this upgrade, the USRL does not have enough signal generators to simulate a realistic, dense threat laydown with multiple modern surface-to-air missile threats and the supporting air defense system radars that make up the background signals. • The reprogramming lab must also be able to rapidly modify existing MDLs because continuing changes in the threats require new intelligence data. -- The mission data reprogramming hardware and software tools used by the USRL during SDD were cumbersome, requiring several months for the USRL to create, test, optimize, and verify a new MDL for each AOR. For this reason, effective rapid reprogramming capability was not demonstrated during SDD. -- This situation improved in 2018 with the delivery of a new Mission Data File Generation (MDFG) tool set from the contractor, but additional improvements are necessary for the tools to fully meet expectations. • Significant additional investments, beyond the current upgrades to the signal generator channels and MDFG tools, are required now for the USRL to support F-35 Block 4 MDL development. -- The Block 4 plan includes new avionics hardware for the aircraft, which will also be required in the USRL. Concurrency in development and production during SDD resulted in three fielded F-35 configurations that will continue to need support indefinitely (i.e., until a specific configuration is modified or retired), after the development JSF 25 FY19 DOD PROGRAMS program enters the Block 4 phase. During Block 4, the program will require the USRL, or an additional reprogramming lab, to have the capability to simultaneously create and test MDLs for the different avionics hardware and software configurations. These configurations include the fielded TR-2 processors and EW system for Block 3F, new EW equipment in Lot 11 and later aircraft, an improved display processor that may be added to TR-2, new TR-3 open-architecture processors to enable Block 4 capabilities, and other avionics for later increments in Block 4. Adequate plans for supporting all these configurations do not appear to be in place. -- In order to be ready to support the planned Block 4 capability development timeline, the Block 4 hardware upgrades for the USRL should have already been on contract. However, as of this report, the requirements for the Block 4 software integration lab and USRL have yet to be fully defined. The JPO must expeditiously complete the development of these requirements while ensuring adequate lab infrastructure to meet the aggressive development timelines of C2D2 and the operational requirements of the Block 4 F-35. -- Additionally, given the new C2D2 Minimum Viable Product (MVP) delivery process, a significant reduction in risk could be achieved if the program made delivery possible of a “Level 2” verified MDL that is compatible with the capabilities being tested during the 6-month IDT/ OT program requirement window. This would allow the new MDL to be flight tested and matured with the software during the IDT/OT process, and have a better chance of being ready for delivery and fielding as soon as IDT/OT is complete. This capability is not on contract nor being considered by the Program Office. Radar Signal Emulators (RSE) Activity: • In early CY19, the NTTR completed its acceptance of the last of 16 RSE delivered under the DOT&E-initiated Electronic Warfare Infrastructure Improvement Program (EWIIP). The RSEs were integrated into the larger test infrastructure used in F-35 IOT&E missions. • The RSEs are advanced, reprogrammable radar simulators that work in conjunction with AARI and other elements of range infrastructure to emulate the signals and the detection, tracking, and missile engagement capabilities of advanced air defense radars and surface-to-air missile systems. The RSEs and AARI enable the presentation of high-fidelity threat scenarios that could not be represented with existing legacy range assets. • Initial IOT&E missions on the NTTR revealed problems with AARI and RSE integration and range network connectivity, as well as white force and RSE operator proficiency (see IOT&E section above). IOT&E missions involving the RSEs were successfully completed between June and September 2019. These missions yielded many important insights into the capabilities of the Block 3F aircraft and weapons, along with 26 JSF the viability of current tactics against the threat scenarios tested. Specific results are classified. • The RSEs are now in the process of being moved and integrated at the PMSR in California, where they will support additional Block 3F IOT&E missions in the spring of 2020. Assessment • The integration of the RSEs on NTTR enabled testing of the F-35 in realistic scenarios versus modern threats during IOT&E. Once the movement of the RSEs to PMSR is complete, DOT&E expects they will enable threat‑representative testing there as well. The RSEs will continue to provide valuable training and tactics development against more modern threat laydowns than were previously available on the DOD test ranges. Operational Suitability Autonomic Logistics Information System (ALIS) Activity • The program completed fielding of ALIS 3.0.1.2 and incorporated a fix release, ALIS 3.0.1.3, into ALIS release 3.1.1 (described below). ALIS 3.0.1 content included a filtering function designed to reduce false alarms in the post‑flight fault codes reported to maintenance personnel, the next version of the Training Management System (version 2.0), and the ability to process propulsion data concurrently with aircraft data. • ALIS 3.0.1.3 included some usability improvements with more efficient screen configurations and faster report generation. • User feedback noted overall faster processing performance for some functions, such as processing propulsion system data from Portable Memory Devices, pilot debriefing, air vehicle data transfers, synchronization times between Portable Maintenance Aids (PMAs), and the Standard Operating Unit (SOU). Users also noted screen response times improved for some functions, but were slower in others compared to previous ALIS releases. • The program completed fielding of ALIS 3.1.1, which is another fix release that merged ALIS 3.0.1.3 with limited sovereign data management capability, to all U.S. operating locations and to partner nations and foreign customers. Sovereign data management allows foreign partners and military sales customers to block, delay, or pass through all structured data, including propulsion data, and gives the ability to filter certain parts of propulsion messages based on sovereign data requirements. • The program planned to begin releasing ALIS 3.5 to fielded units in October 2019, but actual release was delayed to January 2020 as of the writing of this report. ALIS 3.5 focuses on improved usage stability. Enhancements include the alignment of mission capable status across ALIS applications, correcting deficiencies in time accrual associated with Production Aircraft Inspection Reporting System (PAIRS) processing, and improvements in the Low Observable Health Assessment System. FY19 DOD PROGRAMS • The program identified deficiencies with an initial release of ALIS 3.5 tested in July 2019, an engineering release of ALIS 3.5 tested in August 2019, and developed fixes in a second engineering release. Testing of the second engineering release at the ORE and Integrated Test Force (ITF) in October 2019 demonstrated the fixes eliminated all major deficiencies identified in earlier versions of ALIS 3.5. As a result, the program fielded ALIS 3.5 to Nellis AFB, Nevada, for a 30-day sustainment demonstration and the Services and partner countries are able to transition to ALIS 3.5 at their discretion. • The program indicated that it plans to relocate the ORE to Hill AFB, Utah, after the ITF and ORE complete ALIS 3.5 testing. DOT&E does not yet know the timeline or details of how this will occur, nor if Edwards AFB, California, will remain a node on the ORE network. The program delivered two SOUs to Hill AFB and planned to link both to the ORE CPE and ALOU located in Fort Worth, Texas, via a Lockheed Martin network, but this configuration is not operationally representative. • The program was planning two service pack releases, ALIS 3.5.1 and ALIS 3.5.2, in late 2019. • The program’s plan for ALIS development previously included ALIS 3.6 and 3.7 releases with most of the remaining planned SDD content and necessary deficiency fixes. However the program decided in September 2019 to not develop and field these software versions as previously planned. Instead, the program announced it plans to release capabilities via smaller, more frequent service pack updates. The program has not released an updated schedule showing the decomposition of the planned ALIS 3.6/3.7 requirements, deficiency fixes, and the associated test and fielding plan. • For example, ALIS 3.6 was to include migration to Windows 10 and cybersecurity improvements, including fixes to cybersecurity deficiencies. DOT&E is not aware of how the program will incorporate these changes to support the many fielded systems. • The program is also planning a re-architecture of ALIS, frequently termed ALIS NEXT, through a combination of new applications and re-hosted software code from the current ALIS. The program undertook this planning while simultaneously supporting ALIS 3.1.1, preparing to release ALIS 3.5, and developing and testing the service packs that will follow. • ALIS NEXT will use a cloud-focused model and will be government owned and managed. • The U.S. Air Force Kessel Run office is working with the Program Office on a separate effort termed “Mad Hatter,” or DevOps, to demonstrate the streamlining of existing and new ALIS software through development, testing, and fielding on a nearly continual basis. This would allow rapid fielding of new applications and improvements to existing applications. DOT&E does not have the results of the four applications developed through the Mad Hatter effort and demonstrated by the Blended Operational Lightning Technician Aviation Maintenance Unit, which is part of the 57th Wing at Nellis AFB, Nevada. The four applications, which exist outside of ALIS and were based on ALIS 3.0.1.2 software code, are: -- Kronos: Assists in flying and maintenance scheduling -- Titan: Assists maintenance expediters in determining fleet status and in assigning tasks -- Athena: Allows section chiefs to determine training status of maintainers -- Monocle: Provides technical orders in a user-friendly manner Assessment • Although the program released several new versions of ALIS in 2019 that improved ALIS usability, these improvements did not eliminate the major problems in ALIS design and implementation and are unlikely to significantly reduce technical debt or improve the user experience. ALIS remains inefficient and cumbersome to use, still requires the use of numerous workarounds, retains problems with data accuracy and integrity, and requires excessive time from support personnel. As a result, it does not efficiently enable sortie generation and aircraft availability as intended. Users continue to lack confidence in ALIS functionality and stability. The program should expedite fixes to Electronic Equipment Logbook data as it is a major ALIS degrader, frequent source of user complaints, and a major ALIS administrator burden. • The program’s decision to not release ALIS 3.6 and 3.7, while not yet providing a road map to fielding of the capabilities and fixes previously planned for those releases, increases timeline uncertainty and schedule risk for corrections to ALIS deficiencies, particularly those associated with cybersecurity and deploying Windows 10. The program should develop plans to deliver the remaining planned SDD capabilities and necessary deficiency fixes. • In order for the program to achieve its goal of fielding smaller ALIS releases more frequently, it will need a facility that permits development and testing of software in a truly operational environment. The lack of a single test venue to do this currently hurts the program’s ability to improve software quality. Neither the ITF nor the ORE allow testing of the full range of ALIS capabilities, including the ability to replicate the large volume of data transfers of an operational unit. • It is unclear whether the program has dedicated sufficient resources to improving ALIS capabilities, while supporting innovative approaches, such as ALIS NEXT and Mad Hatter. It must also develop a plan for how these separate efforts will be integrated into ALIS while continuing to support fleet operations. • To enhance the ability to evaluate performance of future versions of ALIS, the program should develop and track appropriate metrics for ALIS. • The period of performance for Mad Hatter will end in late 2019. DOT&E does not know if additional funding is available to continue this effort. JSF 27 FY19 DOD PROGRAMS Cybersecurity Operational Testing Activity • The JOTT continued to accomplish testing to support IOT&E based on the cybersecurity strategy approved by DOT&E in February 2015. • The JOTT conducted a Cooperative Vulnerability and Penetration Assessment (CVPA) of the United States Reprogramming Laboratory in March 2019 with a test team from the 47th Cybersecurity Test Squadron (CTS) and an Adversarial Assessment (AA) of the USRL in 2019 using a test team from the 177 Information Aggressor Squadron. • From October 2018 to July 2019, the JOTT conducted a series of air vehicle cyber demonstrations to assess Identification Friend or Foe (IFF), Link 16 datalink, navigation systems, Software Data Load, and Weapons Interfaces. The JOTT intended to assess the Variable Message Format (VMF) digital radio at the same time as IFF and Link 16, but the VMF test tool was not operable for any of the test windows. The table below summarizes the planned JOTT air vehicle demonstrations. TABLE 1. PLANNED JOTT AIR VEHICLE DEMONSTRATIONS AV COMPONENT LOCATION IFF/Link 16 Chamber Test at Pax River OCT 2018 IFF/Link 16/VMF Chamber Test at Pax River 1 APR/MAY 2019 IFF/Link 16/VMF Chamber Test at Pax River 2 JUN 2019 IFF/Link 16/VMF Lab Test at Mission Systems Integration Lab (MSIL) in Fort Worth TBD IFF/Link 16/VMF Flight Test at Pax River TBD Navigation Lab Test at MSIL in Fort Worth JUL 2019 Navigation Ground Test at Edwards AFB TBD Weapons Interface MSIL in Fort Worth 1 JUL 2019 Weapons Interface MSIL in Fort Worth 2 JUL 2019 Software Data Load Vehicle Systems Integration Facility in Fort Worth FEB 2019 • Not all JSF cyber tests in 2019 were completed in accordance with their individual, DOT&E-approved test plans. -- The JOTT did not undertake any VMF testing due to unavailability of completed cyber test tools. -- The JOTT did not undertake the planned IFF, Link 16, and VMF laboratory test at the Lockheed Martin Fort Worth Mission Systems Integration Lab (MSIL), originally scheduled for May 2019, due to laboratory unavailability. The JOTT performed further validation of the VMF test tool in late October 2019 and will complete IFF/VMF/Link 16 testing in an appropriate venue in 2020. -- Lack of a suitable air vehicle test asset prevented the JOTT from undertaking the planned IFF, Link 16, and VMF flight test at Pax River, Maryland, originally scheduled for July 2019, as well as the planned Navigation Ground Test at Edwards AFB, California, originally scheduled for April 2019. However, the JOTT plans to conduct additional navigation system cyber testing in an anechoic chamber in September 2020. -- Weapons interface testing at the MSIL in June 2019 satisfied two of three requirements of the current weapons interface test plan, with the remaining event still to be rescheduled. • Throughout 2019, the JOTT continued to work with stakeholders across the DOD to identify relevant scenarios, qualified test personnel, and adequate resources for conducting cyber testing on air vehicle components and systems. • In 2019, the JPO conducted a Supply Chain Cyber Table Top (CTT). The CTT analyzed the potential threats to 28 JSF COMPLETED OR SCHEDULED two air vehicle systems, plus the possible consequences to F-35’s mission capability and suitability of a compromise of production or re-supply of select components within these systems. The JOTT provided significant input to and involvement in this CTT effort. Assessment • Cybersecurity testing to date during IOT&E continued to demonstrate that vulnerabilities identified during earlier testing periods still have not been remedied. • More testing is needed to assess the cybersecurity of the air vehicle. Actual on-aircraft or appropriate hardware- and software-in-the-loop facilities are imperative to enable operationally representative air vehicle cyber testing. • Testing of the JSF supply chain to date has not been adequate. Additional testing is needed to ensure the integrity of hardware components for initial production of air vehicles and ALIS components, plus resupply of replacement parts. The Supply Chain CTT conducted in 2019 can potentially provide focused future test scenarios to gain insight into the resilience of the F-35 supply chain, and effects of any compromise of components within it. • Cybersecurity testing to date identified vulnerabilities that must be addressed to ensure secure ALIS, Training System, USRL, and air vehicle operations. • According to the JPO, the air vehicle is capable of operating for up to 30 days without connectivity to ALIS via the SOU. In light of current cybersecurity threats and vulnerabilities, along with peer and near-peer threats to bases and communications, the F-35 program and Services should FY19 DOD PROGRAMS conduct testing of aircraft operations without access to the ALIS SOU for extended periods of time, with an objective of demonstrating the 30 days of operations. Availability, Reliability, and Maintainability Activity • The program continued to deliver aircraft to the U.S. Services, international partners, and foreign military sales participants throughout CY19 in production Lot 11. As of the end of September, 430 aircraft had been produced for the U.S. Services, international partners, and foreign military sales. These aircraft are in addition to the 13 aircraft dedicated to developmental testing. • The following assessments of fleet availability, reliability, and maintainability are based on sets of data collected from the operational and test units and provided by the JPO. The assessment of aircraft availability is based on data provided through the end of September 2019. Reliability and maintainability (R&M) assessments, with the exception of the Mean Flight Hours Between Maintenance Event (MFHBME), in this report are based on data covering the 12-month period ending June 13, 2019. Due to inconsistencies between the data from the June 2019 report compared to the February 2019 report, DOT&E did not consider the data from the June 2019 report for this metric to be reliable. Data for R&M include the records of all maintenance activity and undergo an adjudication process by the government and contractor teams, a process which creates a lag in publishing those data. The differences in data sources and processes create a disparity in dates for the analyses in this report. • In September 2018, the Secretary of Defense directed the Services to increase fighter mission capable (MC) rates to 80 percent by the end of FY19. The MC rate represents the percentage of unit-assigned aircraft capable of performing at least one defined mission, excluding those aircraft in depot status or undergoing major repairs. MC aircraft are either Full Mission Capable (FMC), meaning they can perform all missions assigned to the unit, or Partial Mission Capable (PMC), meaning they can fly at least one, but not all, missions. The MC rate is different than the availability rate, which is the number of aircraft capable of performing at least one mission divided by all aircraft assigned, including aircraft in depot status or undergoing major repairs. Assessment • The operational suitability of the F-35 fleet remains at a level below Service expectations. However, after several years of remaining stable or only moving within narrow bands, several key suitability metrics showed signs of slow improvement in CY19. • Aircraft availability is determined by measuring the percentage of time individual aircraft are in an “available” status, aggregated monthly over a reporting period. -- The program-set availability goal is 65 percent; the following fleet‑wide availability discussion uses data from the 12-month period ending September 2019. -- For this report, DOT&E is reporting availability rates only for the U.S. fleet, vice including international partner and foreign military sales aircraft, as was done in previous reports. • The average fleet-wide monthly availability rate for only the U.S. aircraft, for the 12 months ending September 2019, is below the target value of 65 percent. However, the DOT&E assessment of the trend shows evidence of slight overall improvement in U.S. fleet-wide availability during 2019. In particular, while the average monthly availability for the 12 months ending September 2019 was only a few percent higher than the average monthly availability for the 12 months ending September 2018, the F-35 fleet’s monthly availability was generally slowly increasing in 2019, and achieved historic program highs that approached the target availability rate. • The whole U.S. fleet can be broken down into three distinct sub-fleets: the combat-coded fleet of aircraft which are slated into units that can deploy for combat operations; the training fleet for new F-35 pilot accession; and the test fleet for operational testing and tactics development. The combat‑coded fleet represented roughly a third of the whole U.S. fleet over the period, and demonstrated significantly higher availability than the other two fleets. The combat‑coded fleet still fell short of the 65 percent monthly availability goal over the 12 months ending September 2019, but did achieve the goal each month for the last 3 months of FY19. • Aircraft that are not available are designated in one of three status categories: Not Mission Capable for Maintenance (NMC-M), Depot (in the depot for modifications or repairs beyond the capability of unit-level squadrons), and Not Mission Capable for Supply (NMC-S). -- The average monthly NMC-M and Depot rates were relatively stable, with little variability, and near program targets. -- The average monthly NMC-S rate was more variable, and was higher (i.e., worse) than program targets. The NMC-S rate showed the greatest improvement over the period, however, and this improvement was largely responsible for the corresponding improvement in fleet-wide availability. The program should continue to resource and develop alternate sources of repair (including organic repair) for current and projected NMC-S drivers. • The average monthly utilization rate measures flight hours per aircraft per month. The average utilization rate of flight hours per tail per month increased slightly over previous years, but remains below original Service beddown plans. -- Low utilization rates continue to prevent the Services from achieving their full programmed fly rates, which are the basis of flying hour projections and sustainment cost models. For the 12 months ending September 2019, the average monthly utilization rate for the whole U.S. fleet was 18.1 flight hours per tail per month for the F-35A, 15.3 for the F-35B, and 23.8 for the F-35C. This compares to Service bed-down plans from 2013, which expected F-35A and F-35C units to execute 25 flight hours per tail per month and F-35B units to execute 20 flight hours per tail per month to achieve Service goals. JSF 29 FY19 DOD PROGRAMS • DOT&E conducted a separate analysis of availability of the fleet of operational test aircraft, using data from the 10-month period beginning December 2018, when formal IOT&E started, through September 2019. This assessment accounts for the full complement of 23 U.S. and international partner aircraft assigned to the OT fleet at the end of September 2019 (eight F-35A, nine F-35B, and six F-35C). -- The average monthly availability rate for F-35 OT aircraft was below the planned 80 percent needed for efficient conduct of IOT&E. However, judicious maintenance planning, test range scheduling, and effective mission execution allowed the JOTT to execute trials at a quicker pace than planned for worst-case scenario projections. • No portion of the fleet, including the combat-coded fleet, was able to achieve and sustain the 80 percent MC rate goal set by former Secretary of Defense Mattis. However, individual units were able to achieve the 80 percent target for short periods during deployed operations. Similar to the trend in availability, the MC and FMC rates of the whole U.S. fleet improved slightly in 2019. FMC rates lagged the overall MC rates by a large margin, indicating low readiness for the mission sets requiring fully capable aircraft. All three variants achieved roughly similar MC rates, but significantly different FMC rates. The F-35A displayed the best FMC performance, while the F-35C fleet suffered from a particularly poor FMC rate; the F-35B’s FMC rate was roughly midway between the other two variants. F-35 Fleet Reliability • Aircraft reliability assessments include a variety of metrics, each characterizing a unique aspect of overall weapon system reliability. -- Mean Flight Hours Between Critical Failure (MFHBCF) includes all failures that render the aircraft unsafe to fly or would prevent the completion of a defined F-35 mission. -- Mean Flight Hours Between Removal (MFHBR) indicates the degree of necessary logistical support and is frequently used in determining associated costs. -- Mean Flight Hours Between Maintenance Event Unscheduled (MFHBME_Unsch) is a reliability metric for evaluating maintenance workload due to unplanned maintenance. -- Mean Flight Hours Between Failure, Design Controllable (MFHBF_DC) includes failures of components due to design flaws under the purview of the contractor. • The F-35 program developed reliability growth projection curves for each variant throughout the development period as a function of accumulated flight hours. These projections compare observed reliability with target numbers to meet the threshold requirement at maturity (200,000 total F-35 fleet flight hours, with a minimum of 50,000 flight hours per variant). In the program’s reliability growth plan, the target flight hour values were set at 75,000 flight hours each for the F-35A and F-35B, and 50,000 flight hours for the F-35C to establish the 200,000 flight hours of fleet maturity. The F-35A fleet reached 75,000 flight hours in July 2018 and had not reached ORD thresholds for reliability and maintainability at the time. DOT&E is continuing to track these metrics beyond the flight hours required for maturity of the F-35A fleet for reporting purposes. As of June 13, 2019, the date of the most recent set of reliability data available, the fleet and each variant accumulated the following flight hours, with the percentage of the associated hour count at maturity indicated: -- The complete F-35 fleet accumulated 170,453 flight hours, or 85 percent of its maturity value. -- The F-35A accumulated 102,821 hours, or over 137 percent of its target value in the reliability growth plan. -- The F-35B accumulated 45,161 hours, or 60 percent of its target value in the reliability growth plan. -- The F-35C accumulated 22,471 hours, or 45 percent of its target value in the reliability growth plan. • The program reports reliability and maintainability metrics for the three most recent months of data. This rolling 3-month window dampens month-to-month variability while providing a short enough period to distinguish current trends. • Table 2 shows the trend in each reliability metric by comparing values from June 2018 to those of June 2019 and whether the current value is on track to meet the requirement at maturity. TABLE 2. F-35 RELIABILITY METRICS (UP ARROW REPRESENTS IMPROVING TREND) Assessment as of June 30, 2018 MRHBCF (Hours) Flight Hours for ORD for JCS Threshold Cumulative Flight Hours F-35A 75,000 F-35B F-35C Variant MFHBR (Hours) MFHBME (hours)1 MFHBF_DC (Hours) ORD Threshold Change: June 2018 to June 2019 Meeting Interim Goal for ORD Threshold ORD Threshold Change: June 2018 to June 2019 Meeting Interim Goal for ORD Threshold ORD Threshold Change: June 2018 to June 2019 Meeting Interim Goal for ORD Threshold JCS Requirement Change: June 2018 to June 2019 Meeting Interim Goal for ORD Threshold 102,821 20 ↓ No 6.5 ↓ No 2.0 ↓ No 6.0 ↓ Yes 75,000 45,161 12 ↑ No 6.0 ↓ No 1.5 ↑ No 4.0 ↑ Yes 50,000 22,471 14 ↑ No 6.0 ↑ No 1.5 ↑ No 4.0 ↑ Yes 1. For MFHBME, DOT&E assessment is based on data through February 2019 vice June 2019 due to inconsistencies in data reports. • Between June 2018 and June 2019, three of the six ORD metrics increased in value, and three decreased. MFHBME decreased between June 2018 and February 2019 for 30 JSF the F-35A and increased for the F-35B and F-35C. Unlike previous reports, however, two of the three JSF JCS metrics increased, while one decreased, and all three were FY19 DOD PROGRAMS above interim goals. The improvement in MFHBF_DC reliability performance has still not translated into equally strong ORD reliability metric reliability performance, all of which fall short of their interim goals. Maintainability • The amount of time needed to repair aircraft and return them to flying status has changed little over the past year, and remains higher than the requirement for the system at maturity. The program assesses this time with several measures, including Mean Corrective Maintenance Time for Critical Failures (MCMTCF) and Mean Time To Repair (MTTR) for all unscheduled maintenance. Both measures include “active touch” labor time and cure times for coatings, sealants, paints, etc., but do not include logistics delay times, such as how long it takes to receive shipment of a replacement part. • The program reports maintainability metrics for the three most recent months of data. Table 3 shows the nominal change in each maintainability metric by comparing values from June 2018 to those of June 2019. • All mean repair times are longer, some up to more than twice as long, as their original ORD threshold values for maturity, reflecting a heavy maintenance burden on fielded units. • The JPO, after analyzing MTTR projections to maturity, acknowledged that the program would not meet the MTTR requirements defined in the ORD. The JPO sought and gained relief from the original MTTR requirements. The new values are 5.0 hours for both the F-35A and F-35C, and 6.4 hours for the F-35B. This will affect the ability to meet the ORD requirement for sortie generation rate, a Key Performance Parameter. TABLE 3. F-35 MAINTAINABILITY METRICS (DOWN ARROW REPRESENTS IMPROVING TREND) Assessment as of June 13, 2019 Variant Flight Hours for ORD Threshold MCMTCF (Hours) Cumulative Flight Hours MTTR (Hours) ORD Threshold Change: June 2018 to June 2019 Meeting Interim Goal for ORD Threshold ORD Threshold Change: June 2018 to June 2019 Meeting Interim Goal for ORD Threshold 4.0 ↓ No 2.5 ↓ No F-35A 75,000 93,356 F-35B 75,000 42,176 4.5 ↑ No 3.0 ↑ No F-35C 50,000 20,505 4.0 ↓ No 2.5 ↓ No Ship Integration • The Navy has started in-depth table top analyses of the logistics footprint for the first carrier air-wing deployment that will include the F-35C onboard a nuclear-powered aircraft carrier. These analyses show that the air wing with the F-35C incorporated will bring a larger logistical footprint than legacy air wings, which may extend the timelines required and increase the risk to conduct certain shipboard flight and resupply operations. Not all of the cited increase in footprint is directly related to the F-35C since the planned air wing includes additional numbers of other types of aircraft. The air wing which has incorporated the F-35C also replaces the C-2 Carrier Onboard Delivery (COD) logistical support aircraft with the CMV-22B, since the latter can internal carry the F-135 power module to resupply F-35C engine components. The Navy analyses make several recommendations pertinent to the F-35C, that are consistent with DOT&E observations from F-35 ship integration testing conducted to date. Specifically these recommendations include: -- The JPO and Navy continue to fund efforts to share Support Equipment among multiple different types of aircraft, often called multipath. Previous DOT&E reports have shown that fleet personnel believe the F-35 Support Equipment, much of which is peculiar to the F-35, is much larger than legacy aircraft Support Equipment and will complicate shipboard maintenance evolutions. -- T he JPO develop and provide environmental seals and covers for the F-135 power module when outside of its normal shipping pod, to ease transfer of un-podded power modules to and from the CMV-22B COD. Live Fire Test and Evaluation F-35 Vulnerability to Kinetic Threats Activity • In April 2018, Lockheed Martin delivered the F-35 Vulnerability Assessment Report summarizing the force protection and vulnerabilities of all three F-35 variants, and the F-35 Consolidated LFT&E Report, which summarizes the live fire test and analysis efforts supporting the vulnerability assessments. Assessment • For three of the four specification threats, the F-35 variants meet JSF contract specification requirements to enable safe ejection of the pilot in the event of an engagement. • For two of the four specification threats, the F-35A and F-35C variants meet JSF contract specification requirements to return safely to the Forward Line of Troops following an engagement. The F-35B met the requirements for only one of the four threats. • All three F-35 variants are less vulnerable to three of the four specification threats than the legacy F-16C aircraft, both for safe ejection and for return to Forward Line of Troops. JSF 31 FY19 DOD PROGRAMS • DOT&E will publish an independent evaluation of the vulnerabilities of the F-35 aircraft variants to expected and emerging threats in the report to support the Full-Rate Production decision scheduled for FY21. F-35 Vulnerability to Unconventional Threats Activity • As of FY19, the Naval Air Warfare Center Aircraft Division at Naval Air Station Pax River, Maryland, completed system‑level testing of F-35A and C variants, and limited testing of the F-35B, to evaluate tolerance to electromagnetic pulse (EMP) threats. • The program completed full-up system-level, chemical‑biological decontamination testing on BF-40 (a low-rate initial production F-35B aircraft) in February 2017. Assessment • Testing was done to the threat level defined in Military Standard 2169B. Follow-on, system-level tests of the F-35B, including a test series to evaluate Block 3F hardware and software changes, are anticipated. • In the event of a chemical or biological attack, specialized equipment not readily available to deployed units is capable of decontaminating the F-35. Additional work would be needed to develop an operational decontamination capability. • To assess the protection capability of the Gen II HMDS against chemical-biological agents, the JPO completed a comparison analysis of HMDS materials with those in an extensive DOD aerospace materials database. Compatibility testing of legacy protective ensembles and masks showed that the materials used in the protective equipment can survive exposure to chemical agents and decontamination materials and processes. The program plans similar analyses for the Gen III and Gen III Lite HMDS designs. While this assessment of material compatibilities provides some understanding of the force protection capability against chemical and biological agents, it does not demonstrate a process to decontaminate either HMDS. F-35 Gun Lethality Activity • From August through December 2017, during DT Weapons Delivery Accuracy testing, the Naval Air Warfare Center Weapons Division at Naval Air Weapons Station China Lake, California, completed air-to-ground flight lethality tests of three different 25-mm ammunitions: 1) Semi-Armor-Piercing High-Explosive Incendiary on the F-35B and F-35C only, 2) Armor-Piercing High-Explosive (APEX), and 3) Frangible Armor-Piercing on the F-35A only. Flight lethality tests included gun firings from all three F-35 variants against armored and technical vehicles, small boats, and plywood mannequins. Tests revealed deficiencies with the APEX fuze reliability for impacts into the ground. The manufacturer conducted follow-up testing on a new fuze design, but initial indications were that fuze reliability was not improved, and further APEX flights were grounded due to unexploded ordinance hazard range clean-up concerns. 32 JSF Assessment • The Air Force delivered two of three required draft reports to DOT&E covering ground and air-to-ground lethality tests spanning 2015-2018. DOT&E has provided the program with comments for revisions to satisfy DOT&E needs for the final lethality assessment. Recommendations • The program (i.e., JPO, Services, Lockheed Martin) should: 1. Fully fund RSE, JSE, and OABS upgrades to meet test adequacy requirements in time for planned test periods. 2. Continue to work with the Services to prioritize and correct the remaining Category 1 and 2 deficiencies currently not corrected to ensure the SDD baseline configuration of software and hardware is stable prior to introducing the large number of new capabilities to the software in the new hardware configuration planned in Block 4. 3. Expedite fixes to Electronic Equipment Logbook data as it is a major ALIS degrader, frequent source of user complaints, and a major ALIS administrator burden. 4. Quickly complete the development of the requirements for the Block 4 software integration lab and USRL while ensuring adequate lab infrastructure to meet the aggressive development timelines of C2D2 and the operational requirements of the Block 4 F-35. 5. In light of the recent decision to not complete planned ALIS 3.6 and 3.7 releases, develop plans to deliver the remaining planned SDD capabilities and necessary deficiency fixes. 6. Develop and track appropriate metrics for ALIS to evaluate performance of future versions of ALIS. 7. Conduct more in-depth cyber testing of the air vehicle, and provide a dedicated air vehicle cyber-test asset. 8. Correct program-wide deficiencies identified during cybersecurity testing in a timely manner. 9. In collaboration with the Services, conduct testing of aircraft operations without access to the ALIS SOU for extended periods of time, with the objective of 30 days of disconnected operations. 10. Continue to resource and develop alternate sources of repair (including organic repair) for current and projected NMC-S drivers. 11. Continue to investigate multi-use opportunities for Support Equipment so that F-35’s can share Support Equipment with legacy aircraft in order to reduce logistics footprints for shipboard deployments. 12. Develop environmental seals and covers for un-podded F-35 power modules to ease transfer of resupply and retrograde power modules between the CVN and the CMV-22B carrier-onboard-delivery aircraft.