FY18 DOD PROGRAMS F-35 Joint Strike Fighter (JSF) Executive Summary Programmatics • Block 3F Development - The program completed System Design and Development (SDD) flight testing in April 2018, but continued testing new modernization increments of software to address open deficiencies and improve performance. - The program and stakeholders reviewed open deficiencies between May and July, re-categorizing many of the 102 Category 1 deficiencies (as of May 2018) to Category 2, leaving 13 open Category 1 deficiencies for entry into IOT&E, which later became 15. • IOT&E Readiness - The program focused on preparations for IOT&E readiness throughout FY18. - The Defense Acquisition Executive certified the program as ready for entry into formal IOT&E, provided eight remaining readiness requirements are met prior to the start of for-score events. - DOT&E verified readiness and approved the F-35 IOT&E Test Plan on December 3, 2018. - The Joint Strike Fighter (JSF) Operational Test Team (JOTT) began formal IOT&E open-air testing in accordance with the plan on December 5, 2018. • Continuous Capability Development and Delivery (C2D2) - The JSF Program Office (JPO) and Lockheed Martin began to transition the development effort from delivering Block 3F capabilities in the SDD contract to a more rapid development, testing, and fielding cycle for additional capabilities in Block 4, and to address deficiencies carried over from SDD. - DOT&E considers the current C2D2 schedule to be high risk due to the large amount of planned capabilities to be delivered in 6-month increments. Operational Effectiveness • Operational Testing - The JOTT began conducting pre-IOT&E early test events for score in January 2018 with cold weather testing, followed by additional testing starting in April, including two-ship scenarios, deployments, and weapons testing. • Mission Data Load (MDL) Development and Testing - The U.S. Reprogramming Laboratory (USRL) demonstrated the capability to create functioning MDLs for Block 3F and earlier blocks during SDD; however, it still lacks adequate equipment to be able to fully test and optimize MDLs under stressing conditions to ensure adequate performance against current and future threats. - Significant additional investments, well beyond the current upgrades to the signal generator channels and reprogramming tools, are required now for the USRL to support F-35 Block 4 C2D2 MDL development. Operational Suitability • Autonomic Logistics Information System (ALIS) - The program completed fielding of ALIS 2.0.2.4 in early CY18 and focused on testing the next iteration of the software, version 3.0.1. - Two additional versions of ALIS 3.0.1 software were developed and tested – versions 3.0.1.1 and 3.0.1.2 – to address deficiencies before delivery to fielded units. • Cybersecurity Operational Testing - During CY18, the JOTT assessed ALIS version 3.0, F-35 training systems, and the ALIS-to-shipboard network interface onboard a nuclear powered aircraft carrier. - Cybersecurity testing in 2018 showed that some of the vulnerabilities identified during earlier testing periods still had not been remedied. - Limited cybersecurity testing of the air vehicle is planned during IOT&E; more testing will be needed. • Availability, Reliability, and Maintainability - There was no improving trend in fleet aircraft availability - Fleet-wide average availability is below program target value of 60 percent and well below planned 80 percent needed for efficient conduct of IOT&E. - The trend in fleet availability has been flat over the past 3 years; the program’s reliability improvement initiatives are still not translating into improved availability. - Reliability and maintainability metrics defined in the JSF Operational Requirements Document are not meeting interim goals needed to reach requirements at maturity. Live Fire Test and Evaluation (LFT&E) • In FY18, Lockheed Martin completed the Vulnerability Assessment Report and the Consolidated LFT&E Report. These reports do not include results from Electromagnetic Pulse (EMP) or gun lethality testing, which were not completed by the end of FY18. JSF 23 FY18 DOD PROGRAMS • DOT&E is reviewing the F-35 vulnerability reports and completing its own evaluation, which will be documented in the combined IOT&E and LFT&E report to be published prior to the Full-Rate Production decision, anticipated in FY20. • The JPO evaluated the chemical and biological agent protection and decontamination systems during dedicated full-up system-level testing. However, the test plan to assess the chemical and biological decontamination of pilot protective equipment is not adequate because the JPO does not plan to test the decontamination process for either the Generation (Gen) III or Gen III Lite Helmet-Mounted Display System (HMDS). • Air-to-ground flight lethality tests of three 25-mm round variants against armored and technical vehicles, small boats, and plywood mannequins were conducted at the Naval Air Warfare Center Weapons Division (NAWCWD) at NAWS China Lake, California, from August through December 2017. The rounds tested were the Projectile Gun Unit (PGU)-32/U Semi-Armor-Piercing High-Explosive Incendiary round, PGU-47/U Armor-Piercing High-Explosive Incendiary with Tracer round, and PGU-48/B Frangible Armor-Piercing round. The target damage results are classified. System • The F-35 JSF program is a tri-Service, multinational, single-seat, single-engine family of strike aircraft consisting of three variants: - F-35A Conventional Take-Off and Landing - F-35B Short Take-Off/Vertical-Landing - F-35C Aircraft Carrier Variant • The F-35 is designed to survive in an advanced threat environment (year 2015 and beyond). It is also designed to have improved lethality in this environment compared to legacy multi-role aircraft. Programmatics Block 3F Developmental Testing • Activity - The program completed SDD developmental flight testing on April 11, 2018, after nearly 10 years of flight testing. - At the completion of Block 3F developmental flight testing in April, the program had 941 open deficiencies – either in work or under investigation. These included 102 Category 1 deficiencies and 839 Category 2 deficiencies. - The Integrated Test Force (ITF) published their report on Block 3F testing in March 2018. The report documented numerous open deficiencies across the air system in the final version of Block 3F software, 18 of which were designated Category 1. The ITF recommended that the deficiencies be corrected, although the system could proceed into IOT&E. 24 JSF • Using an active electronically scanned array radar and other sensors, the F-35 with Block 3F or later software is intended to employ precision-guided weapons (e.g., GBU-12 Laser-Guided Bomb, GBU-31/32 JDAM, GBU-39 Small Diameter Bomb, Navy Joint Stand-Off Weapon version C1) and air-to-air missiles (e.g., AIM-120C Advanced Medium-Range Air-to-Air Missile (AMRAAM), AIM-9X infrared-guided, air-to-air missile) and a 25 mm Gun Automatic Unit (GAU)-22/A cannon. • The SDD program was designed to provide mission capability in three increments: - Block 1 (initial training; two increments were fielded: Block 1A and Block 1B) - Block 2 (advanced training in Block 2A and limited combat capability with Block 2B) - Block 3 (limited combat capability in Block 3i and full SDD warfighting capability in Block 3F) • The F-35 is under development by a partnership of countries: the United States, United Kingdom (UK), Italy, the Netherlands, Turkey, Canada, Australia, Denmark, and Norway. Mission • The Combatant Commander will employ units equipped with F-35 aircraft in joint operations to conduct a variety of missions during day or night, in all weather conditions, and in heavily defended areas. • The F-35 will be used to attack fixed and mobile land targets, surface units at sea, and air threats, including advanced aircraft and cruise missiles. Major Contractor Lockheed Martin, Aeronautics Company – Fort Worth, Texas - As of October 17, 2018, the JPO had collected data and verified performance to close out 475 of 536 (89 percent) contract specifications paragraphs. Additionally, 3,363 of 3,452 (97 percent) success criteria derived from the contract specifications had been completed. - The program continued to address documented deficiencies in the Block 3F software by developing and flight testing additional software versions, under the nomenclature of Block 30RXX, as part of planned modernization. Throughout CY18, the program developed and tested numerous iterations, including versions 30R00, 30R01, and 30R02, and associated “Quick Reaction Cycle” versions (e.g., 30R01.02) to correct deficiencies and improve performance. FY18 DOD PROGRAMS - The test centers at Edwards AFB, California, and Naval Air Station (NAS) Patuxent River, Maryland, made plans to transition test aircraft from Block 3F SDD to follow-on modernization. The status and configuration of the 18 developmental test aircraft used for SDD testing as of the end of September 2018 are as follows: 3 were retired, 2 were in storage, 5 were available for flight sciences testing, 5 were continuing missions systems testing, and 3 were returned to the Marine Corps and Navy as operational test aircraft. - The program and stakeholders reviewed open deficiency reports between May and July, re-categorizing many of the 102 Category 1 deficiencies (as of May 2018) to Category 2, leaving 13 open Category 1 deficiencies for entry into IOT&E, which later became 15. • Assessment - Although the program completed SDD flight testing in April, the test centers continued to work on Block 3F technical debt by addressing known deficiencies. The extent that the open deficiencies will affect combat capability will be assessed during IOT&E. Static Structural and Durability Testing • Activity - The F-35A full scale durability test article (AJ-1) completed the third lifetime of testing (one lifetime is 8,000 equivalent flight hours (EFH) on October 17, 2017. The test article was delivered to an inspection facility in June 2018, and is currently undergoing disassembly, inspections, and analysis. - The program suspended testing of the F-35B ground test article (BH-1) after completing the second lifetime of testing in February 2017. Due to the significant amount of modifications and repairs to bulkheads and other structures, the program declared the F-35B ground test article no longer representative of the wing-carry-through structure in production aircraft, deemed it inadequate for further testing, and canceled the testing of the third lifetime with BH-1. The program secured funding to procure another ground test article, which will be production-representative of Lot 9 and later F-35B aircraft built with a re-designed wing-carry-through structure, but to date does not have the procurement of the test article on contract. The program has not completed durability testing of the aircraft with the new wing-carry-through structure to date. - The F-35C durability test article (CJ-1) began third lifetime testing on April 4, 2017, and reached 18,792 EFH on April 12, 2018. Testing was stopped at that time following the discovery of more cracking in the Fuselage Station (FS) 518 Fairing Support Frame (cracking had been discovered at the end of the second lifetime), requiring repair before additional testing could proceed. After making an estimate for the cost and time to repair or replace the FS 518 Fairing Support Frame, coupled with the need to manage other structural parts that had existing damage (fuel floor segment, FS 450 bulkhead, FS 496 bulkhead, FS 556 bulkhead, and front spar repair) via scheduled inspections, the program determined that the third lifetime testing should be discontinued. The test article was removed from the test fixture in August 2018 and prepped for shipment to the tear down and inspection facility in September. Although the program planned for a third lifetime of testing to accumulate data for life extension, if needed, the program currently has no plans to procure another F-35C ground test article. • Assessment - For all variants, this testing has led to discoveries requiring repairs and modifications to production designs, some as late as Lot 12 aircraft, and retrofits to fielded aircraft. - Based on durability testing, the service life of early-production F-35B aircraft is well under the expected service life of 8,000 flight hours, and may be as low as 2,100 flight hours. Fleet F-35B aircraft are expected to start reaching their service life limit in CY26, based on design usage. The JPO will continue to use Individual Aircraft Tracking (IAT) of actual usage to help the Services project changes in timing for required repairs and modifications, and aid in Fleet Life Management. - For the F-35C, expected service life will be determined from the durability and damage tolerance analysis following tear down. IOT&E Readiness • Activity - The JPO, Lockheed Martin, and JOTT continued to make preparations for entry into formal IOT&E. - On August 24, 2018, DOT&E provided guidance in a memorandum to the test agencies on detailed requirements for formal entry into IOT&E. Specifically, to add clarity to the formal entrance criteria, the following items were listed as requirements for formal start: ▪ F-35 software version Block 30R02 with Level 4 (fully validated and verified) mission data files (MDF) ▪ ALIS software version 3.0 ▪ Air-to-Air Range Infrastructure (AARI) system with corrections planned for Block 30R02 software. - On October 2, 2018, the Defense Acquisition Executive certified the program as ready for entry into formal IOT&E provided eight remaining readiness requirements are met prior to the start of for-score events: ▪ A fully validated and verified mission data file for the Block 30R02.03 software ▪ U.S. Services airworthiness authorities provide flight clearances for each variant with the Block 30R02.03 software ▪ The program provides flight series data and joint technical data updated for the Block 30R02.03 software ▪ Full partner participation is authorized for the applicable portions of the IOT&E mission sets ▪ The last OT aircraft undergoing depot modifications – BF-18 – is delivered to Edwards AFB JSF 25 FY18 DOD PROGRAMS ▪ Accreditation of necessary models for use in IOT&E are completed or on track for use ▪ All unit-level modifications to the OT aircraft are complete, except those specifically waived or deferred by DOT&E ▪ AARI has been installed on aircraft BF-17, BF-18, and CF-8 (the last three U.S. OT aircraft to complete depot modifications). - DOT&E approved the IOT&E test plan on December 3, 2018, after verifying that the remaining readiness actions listed above had been met. • Assessment - Two additional factors caused readiness for the formal start of IOT&E to slip into early December. A Category 1 deficiency associated with blanking of the cockpit displays was discovered in Block 30R02.03 software, causing an additional software patch called 30R02.04 to be developed and tested prior to start of formal IOT&E. Additionally, a fleet-wide grounding in October 2018 to inspect and replace fuel pump tubes in a number of the OT aircraft added to the delay in readiness to start. Continuous Capability Development and Delivery (C2D2) • Activity - The JPO and Lockheed Martin began to transition the development effort from delivering Block 3F capabilities in the SDD contract to a more rapid development, testing, and fielding cycle for additional capabilities in Block 4 and to address deficiencies carried over from SDD. - The program’s plans for the Block 4 modernization are included in an updated F-35 acquisition strategy that was approved on October 16, 2018. ▪ These plans include lean test designs and agile development tenets. ▪ The developmental test effort will be government-led compared to the contractor-led approach used for SDD. ▪ The program plans to leverage a greater dependence on modeling and simulation than was used during SDD. - The program developed and began staffing a draft Test and Evaluation Master Plan (TEMP) to support Block 4 development activities. • Assessment - The current C2D2 schedule is high risk with the planned content of capabilities to be made available for delivery in 6-month increments. - Many of the lessons learned from SDD involving the amount of testing that can be done in laboratories and simulations, vice flight testing, could be applied to C2D2 planning. - The program needs to ensure adequate funding is available to support a robust laboratory and simulation environment and develop adequate verification, validation, and accreditation plans. - Sustaining multiple configurations of fielded aircraft (i.e., Block 2B, Block 3F, and the new electronic warfare (EW) system in Lot 11 and later aircraft) while managing a 26 JSF developmental test fleet with updated hardware to support the production of new lot aircraft will be a challenge for the JPO. - The cost of software sustainment for multiple configurations of aircraft needs to be adequately assessed. - The planned 6-month software release cycle does not align with the timelines of other increments of capability needed to support the entire JSF system (i.e., ALIS, mission data, training simulators, aircraft modifications). Other modern fighters (e.g., F/A-18, F-22) have historically taken much longer than 6 months – 2 and 3 years, respectively – to field new increments of capability. A more realistic C2D2 schedule with achievable content releases that includes adequate test infrastructure (labs, aircraft, and time) and modifications while aligning the other fielding requirements is necessary. - F-35 modernization is on OT&E oversight. DOT&E will review the content of each Block 4 increment and, if the increment contains significant new capabilities or new hardware, it will require a tailored formal OT&E. DOT&E routinely conducts “agile” OT for other programs, so each F-35 OT&E will be tailored to be as efficient as possible while maintaining test adequacy by leveraging integrated testing with developmental testing (DT) and focusing on evaluating the new capabilities and affected mission areas. Operational Effectiveness Operational Testing • Activity - DOT&E, in coordination with the JPO and the JOTT, approved execution of select for-score pre-IOT&E test activities, prior to satisfying all 47 TEMP readiness criteria for IOT&E, when the applicable readiness criteria were met and the testing could be adequately completed. ▪ Pre-IOT&E Increment 1: On January 18, 2018, DOT&E approved the JOTT to conduct planned cold weather testing that occurred from January 18 to February 2, 2018, at Eielson AFB, Alaska. The operational test squadrons deployed six F-35 aircraft, two of each variant, from Edwards AFB, California. The purpose of this for-score testing was to evaluate the suitability of the F-35 air system and evaluate alert launch timelines in the extreme cold weather environment. The deployment was one of six required by the F-35 IOT&E test design. ▪ Pre-IOT&E Increment 2: Following approval from DOT&E on March 30, 2018, the JOTT began for-score testing of limited two-ship mission scenarios with Block 3F (30R00) software and Level 2 MDFs. The scenarios included Close Air Support, Reconnaissance, Forward Air Controller-Airborne, Strike Coordination and Armed Reconnaissance, and Combat Search and Rescue, along with ship deployments and weapons delivery events. Some missions were re-flown by the FY18 DOD PROGRAMS • A-10 as part of the planned F-35A and A-10 comparison testing. ▪ The JOTT and the F-35A operational test squadrons deployed four F-35A OT aircraft from June 4 – 29, 2018, to Eglin AFB, Florida, to conduct Pre-IOT&E air-to-air missile Weapons Demonstration Events over the Gulf Coast test ranges. During the deployment, the test team completed six AIM-120 and six AIM-9X missile events, some with multiple shots, and all in accordance with the approved plan. In limited cases, DOT&E approved modifications to the mission profile when warranted. ▪ The JOTT, in coordination with VFA-125, the Navy’s west coast F-35C Fleet Replacement Squadron, deployed six aircraft aboard the USS Abraham Lincoln from August 18 – 31, 2018, to conduct shipboard operations and evaluate F-35C sortie generation rate (SGR) capabilities, per the IOT&E test plan. ▪ The test included participation of aircraft from Carrier Air Wing Seven, which provided an operationally representative flight deck environment. This was the first time the F-35C was integrated with the rest of a carrier air wing as it would during an operational deployment. ▪ The Navy approved the use of the F-35 Integrated Power Package (IPP) in the hangar bay for maintenance purposes, on an interim basis, just prior to the SGR testing onboard CVN 72. This approval will enable more efficient maintenance during deployments, increasing the options for providing electrical power and cooling air to aircraft undergoing maintenance. Squadrons will use temperature sensing devices to ensure that the IPP exhaust, which vents upwards on the F-35C, does not damage hangar bay overhead equipment, cabling, and structure while in use. ▪ The Navy finalized a design for the Closed Bay Fire Fighting Tool (CBFFT), and produced several examples to provision CVN 72’s crash and fire personnel prior to the SGR testing. The CBFFT will allow emergency responders to cut through the exterior of an F-35 aircraft carrying live internal ordnance and plug a water hose into the hole to provide ordnance cooling during a fire on the flight deck. ▪ The JOTT and the F-35A operational test squadron deployed four F-35A OT aircraft to Volk Field Air National Guard Base, Wisconsin, to evaluate sortie generation rate surge operations from September 10 – 16, 2018. Although the test plan called for six aircraft to deploy, two remained at Edwards AFB due to maintenance problems. Assessment - DOT&E will report the results of the pre-IOT&E test events following IOT&E. Gun Testing • Activity - All three F-35 variants have the GAU-22/A cannon. The F-35A gun is internal; the F-35B and F-35C each use an external gun pod. Differences in the outer mold-line fairing mounting make the gun pods unique to a specific variant (i.e., an F-35B gun pod cannot be mounted on an F-35C aircraft). - Through July 2018, 19 air-to-ground strafing missions had been completed to assess gun accuracy on the F-35A. Eighteen missions were flown with AF-31 and one mission with AF-80. Over 3,400 rounds were fired using a cross section of rounds, including PGU-23, PGU-47, and PGU-48. - Through July 2018, 13 air-to-ground strafing missions had been completed using the missionized gun pod; one on BF-15, one on BF-16, six on BF-17, and five on CF-08. Overall, 2,695 rounds were fired using PGU-23 and PGU-32 rounds, including some for assessing accuracy compliance. - Operational test pilots conducted live firings of the gun against airborne targets, including drones and towed banners, throughout CY18. These firings were often in combination with other weapon demonstration events, such as air-to-air missile employment events. • Assessment - Based on F-35A gun testing through September 2018, DOT&E currently considers the accuracy of the gun, as installed in the F-35A, to be unacceptable. - F-35A gun accuracy during SDD failed to meet the contract specification. Although software corrections were made to the F-35 mission systems software to improve the stability of gun aiming cues, no software or hardware corrections have yet been implemented to correct the gun accuracy errors. - Investigations into the gun mounts of the F-35A revealed misalignments that result in muzzle alignment errors. As a result, the true alignment of each F-35A gun is not known, so the program is considering options for re-boresighting and correcting gun alignments. - During air-to-air gun testing, F-35A operational test pilots received intermittent “unsafe gun” cockpit alerts while attempting gun attacks. These alerts occurred with two different aircraft; the root cause is under investigation. - F-35B and F-35C air-to-ground accuracy results to date with the gun pod have been consistent and meet the contract specifications. They do not show the accuracy errors of the internal gun on the F-35A. Mission Data Load (MDL) Development and Testing • Activity - F-35 effectiveness relies on the MDL, which is a compilation of the mission data files (MDF) needed for JSF 27 FY18 DOD PROGRAMS • 28 operation of the sensors and other mission systems. The MDL works in conjunction with the avionics software and hardware to drive sensor search behaviors to provide target identification parameters. This enables the F-35 avionics to identify, correlate, and respond to sensor detections, such as threat and friendly radar signals. ▪ The contractor produces an initial set of MDLs for each software version to support DT during SDD. ▪ The USRL at Eglin AFB, Florida, creates, tests, and verifies operational MDLs – one for OT and training, plus one for each potential major geographic area of operation, called an area of responsibility (AOR). OT aircraft and fielded aircraft use the applicable USRL-generated MDLs for each AOR. - The testing of the USRL MDLs is an operational test activity, as arranged by the JPO after the program restructure that occurred in 2010, and consists of laboratory and flight testing on OT aircraft. Assessment - Because MDLs are software components essential to F-35 mission capability, the Department must have a reprogramming lab that is capable of rapidly creating, testing, and optimizing MDLs, as well as verifying their functionality under stressing conditions representative of real-world scenarios. ▪ The USRL demonstrated the capability to create functioning MDLs for Block 3F and earlier blocks during SDD. However, it still lacks adequate equipment to be able to test and optimize MDLs under conditions stressing enough to ensure adequate performance against current and future threats in combat. ▪ The lab lacks a sufficient number of high-fidelity radio frequency signal generator channels, which are used to stimulate the F-35 EW system and functions of the radar, with simulated threat radar signals. This situation is improving as of the writing of this report, but additional improvements, above and beyond those currently planned, will be required. ▪ By late 2019, both USRL mission data test lines will have been upgraded from three to eight high-fidelity channels. Eight high-fidelity channels per line represents a substantial improvement, but is still far short of the 16-20 recommended in the JPO’s own 2014 gap analysis. ▪ Even when this upgrade is complete, the USRL will still not have enough signal generators to simulate a realistic, dense threat laydown with multiple modern surface-to-air missile threats and the supporting air defense system radars that make up the signal background in the laydown. - The reprogramming lab must also be able to rapidly modify existing MDLs when intelligence data changes. ▪ The mission data reprogramming hardware and software tools used by the USRL during SDD were cumbersome, requiring several months for the USRL to create, test, optimize, and verify a new MDL for each AOR. For this JSF reason, effective rapid reprogramming capability was not demonstrated during SDD. ▪ This situation recently improved with the delivery of a new Mission Data File Generation (MDFG) tool set from the contractor. How much improvement these tools will bring to MDL development timelines is yet to be determined, but initial indications are that the improvements will be significant. - Significant additional investments, well beyond the current upgrades to the signal generator channels and MDFG tools, are required now for the USRL to support F-35 Block 4 C2D2 MDL development. ▪ The C2D2 plan includes new avionics hardware. Concurrency in development and production during SDD resulted in multiple fielded F-35 configurations that will continue to need to be supported indefinitely (i.e., until a specific configuration is modified or retired), after the development program enters the C2D2 phase. During C2D2, the program will require the USRL, or an additional reprogramming lab, to have the capability to simultaneously create and test MDLs for different avionics hardware and software configurations. These different configurations include the fielded Technical Refresh 2 processors for Block 3F, new EW equipment in Lot 11 and later aircraft, an improved display processor, new Technical Refresh 3 open-architecture processors, and other avionics for later increments in C2D2. ▪ In order to be on a timeline that is fully aligned with the planned C2D2 capability development timeline, the C2D2 hardware upgrades for the USRL should have already been on contract. However, the requirements for the C2D2 software integration lab have yet to be fully defined. The JPO must expeditiously complete the development of these requirements while ensuring adequate lab infrastructure to meet the aggressive development timelines of C2D2 and the operational requirements of the Block 4 F-35. - As part of IOT&E, the USRL will complete an Urgent Reprogramming Exercise (URE). This test event will evaluate the ability of the USRL, with its hardware and software tools, to respond to an urgent request to modify the mission data in response to a new threat or a change to an existing threat. ▪ During a URE at the USRL in 2016, the total hours recorded were double the Air Force standard for rapidly reprogramming a mature system. The JOTT identified several key process problems, including the lack of necessary hardware, analysis tools that were not built for operational use, and missing capabilities, such as the ability to quickly determine ambiguities in the mission data. ▪ The JPO is working to correct these problems in order to bring the ability of the USRL to react to new threats up to the identified standards routinely achieved on legacy aircraft. A new Ambiguity Analysis Tool (AAT), FY18 DOD PROGRAMS originally developed to meet requirements set forth for the Australia-Canada-UK Reprogramming Lab (ACURL), was delivered to both the ACURL and the USRL. The initial version of the AAT has provided improvements in identifying and correcting mission data ambiguities. Enhancements to the AAT now in work promise to significantly speed up the mission data development process. - In addition to resolving the laboratory deficiencies above, the program will need to properly sustain the USRL to ensure a high state of readiness, particularly if the Services have an urgent reprogramming requirement, which could happen at any time for the fielded aircraft. To meet these tasks, the USRL will also need to maintain all necessary equipment in a functioning status with a high rate of availability, which will require a sufficient number of prime contractor Field Service Engineers to assist in maintenance and operation of the lab equipment, and adequate training for laboratory personnel. In addition, the USRL requires adequate technical data for lab equipment and enough spare parts and/or supply priority to quickly repair key components. Joint Simulation Environment (JSE) • Activity - The JSE is a man-in-the-loop, F-35 software-in-the-loop mission simulator intended to conduct IOT&E scenarios with modern threat types and densities that are not able to be replicated in open air. Originally slated to be operational by the end of 2017, first use of a fully functional simulator is now planned for the beginning of 2019 with accreditation later in 2019, near the end of planned IOT&E trials. - The JSE’s physical facilities (cockpits, visuals, and buildings) and synthetic environment (terrain, threat, and target models) are nearing completion and security accreditation. Integration of the F-35 and its weapons is planned for 1QFY19. The JSE verification and validation process has made progress, but the bulk of validation testing still remains for the first half of FY19. • Assessment - The government-led JSE team made good progress this year in getting the hardware developed and installed, which will likely meet requirements for IOT&E. - The planned schedules for JSE software development and accreditation support IOT&E, but there is some risk to software development (particularly F-35 model integration), which also affects verification and validation. Without the JSE, the IOT&E will be unable to adequately assess the F-35 against dense and modern threats that are not available for open-air testing, resulting in operational risk. Once the JSE completes development and accreditation, it should be an invaluable resource for follow-on F-35 testing and possibly for testing of other platforms. Radar Signal Emulators (RSE) • Activity - The Nevada Test and Training Range (NTTR) began accepting Radar Signal Emulators in late CY16 to support the DOT&E-initiated Electronic Warfare Infrastructure Improvement Program (EWIIP). As of October 10, 2018, 9 of 16 emulators had been accepted on the NTTR and had been used to conduct integration testing with the F-35 and other range test assets. - The RSEs will be used to provide operationally realistic threat laydowns for use in F-35 IOT&E. • Assessment - All 16 RSEs should complete acceptance testing and integration by the end of CY18 and will be used to emulate threats during IOT&E. - More detail on the background, development, and fielding of EWIIP can be found in the T&E Resources section of this report. Operational Suitability Autonomic Logistics Information System (ALIS) • Activity - The program completed fielding of ALIS 2.0.2.4 in early 2018. Feedback from operational users included: ▪ The Deployment Planning Tool did not work well or significantly improve the ease of deploying F-35 units. ▪ Life Limited Parts Management, which includes propulsion data integration and Production Aircraft Inspection Requirements (PAIRs), requires a great deal of time with manual workarounds by maintenance personnel. - The program rolled the capabilities planned for release in ALIS 2.0.2.5 into the next block of software – ALIS 3.0.1. ALIS 2.0.2.5 was intended to address deficiencies and usability problems, upgrade the browser to Internet Explorer 11, and include a filtering function to decrease false alarms in the Prognostic Health Management (PHM) System, referred to as Advanced Filter and Correlate (AFC). - The program focused on testing in preparation for fielding ALIS software version 3.0.1 throughout CY18. This version of ALIS software includes the following new major capabilities: ▪ Support for lightning protection. ▪ Low Observable Health Assessment System (LOHAS) improvements. ▪ Security enhancements. ▪ The first increment of the new Training Management System for tracking maintainer qualifications. ▪ Improvements to address technical debt and corrections to existing deficiencies. JSF 29 FY18 DOD PROGRAMS - The program conducted initial testing of ALIS 3.0.1 with field data between November 28, 2017, and January 7, 2018. ▪ Testing with developmental test aircraft occurred at the Air Force Test Center at Edwards AFB and NAS Patuxent River. ▪ The Operationally Representative Environment (ORE) at Edwards AFB was also used, which consists of production-representative ALIS hardware in a closed network and is designed for testing ALIS software using data downloaded from OT aircraft. The ORE also allows testing of ALIS propulsion capabilities as ALIS cannot support SDD propulsion systems. ▪ Because of limitations associated with the hardware versions of the ALIS equipment used to support the SDD aircraft and the ORE, the program could not conduct fully operationally representative testing of new ALIS software versions in either venue. - The initial report issued jointly by the test centers at Edwards AFB and NAS Patuxent River recommended that ALIS 3.0.1 continue development and testing before fielding. - After making several fixes, the program completed testing of ALIS 3.0.1.1 with field data at the same venues between April 3 and May 31, 2018, and recommended fielding of this release. Findings included: ▪ Updated software corrected the erroneous recording of air vehicle flight hours to components installed on a different air vehicle, a deficiency identified during ALIS 3.0.1 testing. ▪ Problems with existing ALIS 2.0.2.4 capabilities noted in ALIS 3.0.1 testing were largely resolved. ▪ PHM performance improved as ALIS 3.0.1.1 eliminated intermittent failures of PHM to auto-populate and display data during debrief. ▪ AFC reduced non-actionable Health Reporting Codes (HRC) and maintainer workload. ▪ Supply chain management data processing, data accessibility of Electronic Equipment Logbooks (EELs), which contain a virtual record of data for a specific part, and Anomaly Fault Resolution System reliability improved. ▪ Significant deficiencies in supporting aircraft parts records remained, including long-standing enterprise-wide problems with data quality. ▪ Documenting maintenance tasks in ALIS frequently takes more time than completing the maintenance action. ▪ The lack of accurate and complete data in ALIS continued to drive many workarounds. ▪ Deficiencies in the Deployment Planning Tool and in air vehicle data transfer functionality were not resolved in ALIS 3.0.1.1. Both require a high level of contractor support with frequent work stoppages, creating a heavy burden on support personnel time. - The program completed verification testing of ALIS 3.0.1.1 at Nellis AFB, Nevada, to evaluate some 30 JSF • capabilities, including LOHAS enhancements and lightning protection, which the program could not fully evaluate during prior testing. Following completion of this verification period, the program approved the release of ALIS 3.0.1.1 to operational test at Edwards AFB, which took place in August 2018. Concurrently, the program continued implementing fixes to ALIS 3.0.1.1 for the next software release, ALIS 3.0.1.2. The program conducted initial testing of ALIS 3.0.1.2 on SDD aircraft and at the ORE between June 9 and September 20, 2018, using five engineering releases. Initial testing was followed by verification testing at Nellis AFB beginning September 15, 2018. ALIS 3.0.1.2 does not deliver any new capabilities, focusing instead on delivering fixes to existing deficiencies. These fixes include: ▪ Improvements within ALIS reporting of the inert gas state of the aircraft fuel system for lightning protection. ▪ A propulsion data processing anomaly introduced in ALIS 3.0.1.1 was corrected. ▪ A deficiency introduced in ALIS 3.0.1.1 that caused some damage tracings to not translate properly into LOHAS, resulting in significant inaccuracies in LOHAS status beyond the scope of actual damage, was corrected. - The program installed ALIS 3.0.1.2 at the operational test sites at Edwards AFB beginning on September 25, 2018; it is expected to be the fielded version of ALIS that is currently being used during formal IOT&E. Assessment - ALIS is designed to bring efficiency to maintenance and flight operations, but it does not yet perform as intended. User feedback on ALIS deficiencies, some of which can have a significant effect on aircraft availability and sortie generation, fall into three major categories: ▪ Users must employ numerous workarounds due to data and functionality deficiencies. Most capabilities function as intended only with a high level of manual effort by ALIS administrators and maintenance personnel. Manual workarounds are often needed to complete tasks designed to be automated. Configuration management of ALIS software and data products remains complex and time-consuming. ▪ Users must deal with pervasive problems with data integrity and completeness on a daily basis. Maintainers frequently have to manually enter missing or incorrect EEL data, which accompany spare parts, so they can be accepted and tracked by an ALIS Standard Operating Unit (SOU) at the squadron and installed on an aircraft. Fixing data in complex EELs, which represent an assembly such as ejection seats, requires a great deal of time from ALIS administrators. EELs problems have many sources, including vendors who have not complied with guidance on creating EELs; a lack of standardization among suppliers, contractors, and field locations for updating EELs; and a lack of automation in the EEL process. Problems with EELs are a top-5 FY18 DOD PROGRAMS - - - - - Not Mission Capable (NMC) maintenance driver and a top-10 propulsion degrader for the U.S. Air Force. ▪ Users lack confidence in some ALIS functionality. For example, the problems noted above have resulted in users maintaining separate databases to track life usage in case PAIRs erroneously generates incorrect data. Users reference the external database created to determine the correct values. The timeline for correcting ALIS deficiencies is typically excessive, causing workarounds to remain in place for extended periods. For example, ALIS incorrectly reports the status of aircraft as NMC in the Squadron Health Management application based on HRCs (faults). Meanwhile, a separate application – Customer Maintenance Management System, which relies on the Mission Essential Function List (MEFL) – reports the same aircraft as mission capable. A logistics test and evaluation report for ALIS version 1.0.3A3 in December 2012 first noted this problem, yet it remains today in ALIS 3.0.1.2. ▪ Many open deficiencies were not resolved during SDD and will continue to negatively affect aircraft availability and SGR. ▪ During SDD, the program repeatedly demonstrated that attempting major software releases with large increments of ALIS capability resulted in delays and deferring capability. The program also did not allocate sufficient resources to simultaneously develop new required capabilities and reduce technical debt. Smaller, more frequent releases would allow the program to field new capabilities and fixes and receive frequent user feedback to plan for future improvements, which the program plans to do in C2D2. The program has completed several deployments to established bases and to austere locations and ships. In each location, the complexities of ALIS have caused a variety of information technology problems that delay the unit’s ability to start generating sorties. Often, the timeframe to start flight operation is longer than that with legacy aircraft. The program plans to release an updated version of ALIS software (ALIS 3.1) to the international partners and foreign military sales customers that includes country-unique data (a.k.a. sovereign data) management within ALIS beginning in January 2019 . The program plans an additional major release of ALIS software, version 3.5, scheduled for fielding in mid-2019, during IOT&E. ALIS 3.5 will be a stabilization release, since it is intended to address a large amount of technical debt, meet cybersecurity threshold requirements – including the use of internet protocols, improving LOHAS, and providing an initial centralized capability for ALIS administration. The program plans to complete ALIS 3.5 with SDD funds. The program currently plans two additional releases, ALIS 3.6 and 3.7, to provide additional stabilization and improved sortie generation capabilities. ▪ ALIS 3.6, scheduled for release in mid-2020, is planned to include Windows 10, additional cybersecurity enhancements, improved air vehicle data transfers between SOUs, and a decentralized maintenance capability, which would allow deployments without a full suite of ALIS hardware. The program also plans to replace obsolescent hardware with the rollout and fielding of the ALIS 3.6 software. ▪ The goal of ALIS 3.7, planned for release in mid-2021, is improved mission support by adding capability to the Training Management System, improved spare parts support for deployments, support for partial squadron deployments, corrosion management, and ALIS support for helmets and other pilot flight equipment. ▪ Because EELs is a top degrader, the program is working on high-priority corrective actions. However, per the JPO, the software capabilities planned for ALIS 3.7 will not address the root causes of the enterprise issues. This is an excessive delay for needed fixes. - The release plan for ALIS 3.5 through 3.7 shows the program is moving toward a pace of one major software release per year with fielding of service packs between major releases. The program has demonstrated that it has difficulty fielding large increments in ALIS capability. While this movement toward more agile software development is positive, the JPO will need to provide sufficient resources for this effort. - The use of ALIS across the F-35 enterprise would improve data integrity as contractors and vendors would be required to adhere to EELs requirements earlier in production and sustainment. ▪ Lockheed Martin did not use ALIS in its production facilities until recently, adding an SOU to the factory floor in March 2018, shortly before propulsion system installation, to improve data quality. ▪ Because data problems are frequently found when new aircraft arrive at operational locations, Lockheed Martin plans to begin using an SOU on the Fort Worth flight line in early 2019 to support aircraft before delivery. ▪ While the addition of SOUs to the production line is a positive step in addressing data problems, the program will not extract maximum benefit from this effort unless ALIS is fully integrated into production facilities. ▪ Vetting the data accompanying spare parts provided by suppliers in an SOU before allowing delivery to field units will reduce EELs deficiencies. - Assessment of the testing regimen for ALIS. ▪ The program still relies heavily on the results of laboratory testing of ALIS software, which does not resemble operational conditions in several ways, including the limited amount of data processed and external connections. ▪ After the problems found during ALIS 2.0.2.4 testing and fielding, the program moved toward heavier use of ALIS testing facilities at Edwards AFB. However, these test venues do not permit testing of the full range JSF 31 FY18 DOD PROGRAMS of ALIS capabilities. A single ALIS test venue would increase test efficiency and support more timely fielding of ALIS software to operational units. In the meantime, the program uses an operational assessment process at Nellis AFB to evaluate ALIS software releases before deployment to the rest of the fleet. ▪ The current, non-operationally representative method of testing ALIS releases leads to delays in finding and fixing deficiencies, often after the new software is fielded. ▪ Differences in laboratory testing and fleet personnel procedures show that fleet personnel use ALIS differently than the laboratory testers. Developmental testing, particularly laboratory-based testing, should include a variety of personnel from different Services and experience levels to increase the chances of finding problems early. ▪ ALIS testing, architecture, operations, and fielding each absorb a disproportionate amount of time, manpower, and funding. The program is developing automated testing capabilities that are being accelerated in an attempt to improve lab testing speed and quality. Cybersecurity Operational Testing • Activity - The JOTT continued to accomplish testing based on the cybersecurity strategy approved by DOT&E in February 2015. The JOTT assessed F-35 training systems, the ALIS-to-shipboard network interface onboard a nuclear-powered aircraft carrier (CVN) with ALIS 2.0.2, and ALIS version 3.0. - The JOTT tested ALIS 3.0 at all three levels of operation: ▪ Autonomic Logistics Operating Unit (ALOU) ▪ Central Point of Entry (CPE) ▪ Squadron Kit (SQK), composed of the SOU, the Mission Planning and Support Boundary, and the Low Observable Maintenance Boundary - In September 2018, the JOTT conducted Cooperative Vulnerability and Penetration Assessments (CVPAs) of ALIS 3.0.1.1 using National Security Agency-certified cybersecurity test organizations and personnel: ▪ The Air Force’s 346 Test Squadron assessed the sole ALOU at Lockheed Martin, Fort Worth, Texas. ▪ The Air Force’s 47 Cyber Test Squadron (CTS) assessed the sole U.S. CPE at Eglin AFB, Florida, and the SQK at Edwards AFB, California. - In October 2018, the JOTT conducted Adversarial Assessments (AAs) of the next iteration of ALIS 3.0 software – version 3.0.1.2 – with the assistance of National Security Agency-certified Red Teams. ▪ The Marine Corps Red Team (MCRT) assessed the ALOU. ▪ The Air Force’s 57 Information Assurance Squadron (IAS) assessed the CPE. 32 JSF ▪ The Air Force’s 177 IAS assessed the SQK at Edwards AFB, California. - The ALIS 3.0 AA also included a limited Enterprise Assessment of the boundaries and interfaces between the ALOU, CPE, and SOU; Lockheed Martin Red Team testing of the Lockheed Martin Internal network, with observation by U.S. Government cyber test personnel; and a preliminary investigation into the cybersecurity posture of the supply chain for components of the SQK. - The JOTT tested the three different network environments present at the Academic Training Center at Eglin AFB, Florida: ▪ The Unclassified Operating Environment (UOE), consisting of unclassified classroom and training resources. ▪ The Classified Operating Environment (COE), consisting of classified classroom and training resources. ▪ The Full Mission Simulator (FMS), consisting of pilot training stations for rehearsing mission tasks in a simulated cockpit. - In February through April 2018, the JOTT conducted CVPAs of the UOE, COE, and FMS respectively in partnership with the 47 CTS. - In April 2018, the JOTT conducted AAs of the UOE and COE utilizing the 57 IAS. - In July 2018, the JOTT conducted an AA of the FMS with the assistance of the 177 IAS. - In August 2018, the JOTT conducted an AA onboard the USS Abraham Lincoln of the network interface between a deployed SQK in the ALIS 2.0.2 configuration and the ship’s Consolidated Afloat Networks and Enterprise Services internal network. The MCRT also facilitated the test. - All JSF cyber tests in 2018 were completed in accordance with their individual, DOT&E-approved test plans. - Throughout 2018, the JOTT continued to work with stakeholders across the DOD to identify relevant scenarios, qualified test personnel, and adequate resources for conducting cyber testing on air vehicle components and systems. - The JOTT expects to conduct a CVPA and AA of the USRL in early 2019, as well as several cyber demonstrations involving air vehicle components and sub-systems. • Assessment - Cybersecurity testing in 2018 showed that some of the vulnerabilities identified during earlier testing periods still had not been remedied. - More testing is needed to assess the cybersecurity of the air vehicle. Actual on-aircraft or appropriate hardwareand software-in-the-loop facilities are necessary to enable operationally representative air vehicle cyber testing. - Testing of the JSF supply chain to date has not been adequate. Additional testing is needed to ensure the FY18 DOD PROGRAMS integrity of hardware components for initial production of air vehicles and ALIS components, plus resupply of replacement parts. - Testing to date has identified vulnerabilities that must be addressed to ensure secure ALIS operations. - According to the JPO, the air vehicle is capable of operating for up to 30 days without connectivity to ALIS. In light of current cybersecurity threats and vulnerabilities, along with peer and near-peer threats to bases and communications, the F-35 program and Services should conduct testing of aircraft operations without access to ALIS for extended periods of time. Availability, Reliability, and Maintainability • Activity - The program continued to deliver aircraft to the U.S. Services, international partners, and foreign military sales throughout CY18 in production Lot 10. As of the end of September, 323 operational aircraft had been produced for the U.S. Services, international partners, and foreign military sales. These aircraft are in addition to the 13 aircraft dedicated to developmental testing. - As of the end of June, the U.S. fleet of F-35s had accumulated 126,136 flight hours - The following assessment of fleet availability, reliability, and maintainability is based on sets of data collected from the operational and test units and provided by the JPO. The assessment of aircraft availability is based on data provided through the end of August 2018. Reliability and maintainability assessments in this report are based on data covering the 12-month period ending June 30, 2018. Data for reliability and maintainability include the records of all maintenance activity and undergo an adjudication process by the government and contractor teams, a process which creates a lag in publishing those data. The variety of data sources and processes are the reasons the data have different dates and appear to be delayed. • Assessment - The operational suitability of the F-35 fleet remains at a level below Service expectations. Similar to the 2017 DOT&E report, most suitability metrics remained nearly the same throughout 2018 or moved only within narrow bands. - Aircraft availability is determined by measuring the percentage of time individual aircraft are in an “available” status, aggregated monthly over a reporting period. ▪ The program-set availability goal is modest at 60 percent, and the fleet-wide availability discussion uses data from the 12-month period ending August 2018. ▪ For this report, DOT&E is reporting availability rates only for the U.S. fleet, vice including international partner and foreign military sales aircraft, as was done in previous reports. - The fleet-wide monthly availability rate for only the U.S. aircraft, for the 12 months ending August 2018, is below the target value of 60 percent. The DOT&E assessment of the trend shows no evidence of improvement in U.S. fleet wide availability during 2018 . - Aircraft that are not available are designated in one of three status categories: Not Mission Capable for Maintenance (NMC-M), Depot (in the depot for modifications or repairs beyond the capability of unit-level squadrons), and Not Mission Capable for Supply (NMC-S). ▪ The average monthly NMC-M and Depot rates were relatively stable, with little variability, and near program targets. ▪ The average monthly NMC-S rate was more variable, and was higher (i.e., worse) than program targets . ▪ The average monthly utilization rate measures flight hours per aircraft per month. The average utilization rate of flight hours per tail per month increased slightly over previous years, but remains below original Service bed down plans. ▪ The low utilization rates continue to prevent the Services from achieving their programmed fly rates, which are the basis of flying hour projections and sustainment cost models. As of June 30, 2018, the fleet had flown 126,136 hours. This amounted to 83 percent of an early 2017 “modeled achievable” projection of 152,445 flight hours by the end of June, 2018. Similarly, for the 12 months ending April 2018, the U.S. Services had contracted for 42,836 flight hours, but the U.S. F-35 fleet logged only 33,365 hours, or 78 percent of the contracted amount over this period. - A separate analysis of availability of the OT-instrumented fleet, using data from the 12-month period ending August 2018, is important to consider now that formal IOT&E is underway. The numbers below account for the full complement of 23 U.S. and international partner aircraft assigned to the OT fleet at the end of August 2018 (8 F-35A, 9 F-35B, and 6 F-35C). ▪ The average monthly availability rate for F-35 OT aircraft was below the planned 80 percent needed for efficient conduct of IOT&E. The low availability during this period is partly explained by the fact that the aircraft of the OT fleet spent over a quarter of the time in depot modifications to bring them up to the Lot 9 production-representative standard configuration, as required prior to the start of IOT&E, with some DOT&E-approved modification deferrals. ▪ Availability of the OT fleet will remain a challenge for the efficient conduct and timely completion of IOT&E. Although the necessary modifications have been completed on the OT aircraft and formal testing has started, mission capable aircraft will need to be available at a high rate to complete the open-air test trials as scheduled. JSF 33 FY18 DOD PROGRAMS the contractor, such as the inability to withstand loads encountered in normal operation. - The F-35 program developed reliability growth projection curves for each variant throughout the development period as a function of accumulated flight hours. These projections compare observed reliability with target numbers to meet the threshold requirement at maturity (200,000 total F-35 fleet flight hours, made up of 75,000 flight hours each for the F-35A and F-35B, and 50,000 flight hours for the F-35C). As of June 30, 2018, the date of the most recent set of reliability data available, the fleet and each variant accumulated the following flight hours, with the percentage of the associated hour count at maturity indicated as well: ▪ The complete F-35 fleet accumulated 126,136 flight hours, or 61 percent of its maturity value. ▪ The F-35A accumulated 74,758 hours, or over 99 percent of its maturity value. ▪ The F-35B accumulated 35,076 hours, or 47 percent of its maturity value. ▪ The F-35C accumulated 16,302 hours, or 33 percent of its maturity value. - The program reports reliability and maintainability metrics for the 3 most recent months of data. This rolling 3-month window dampens month-to-month variability while providing a short enough period to distinguish current trends. - Table 1 shows the trend in each reliability metric by comparing values from May 2017 to those of June 2018 and whether the current value is on track to meet the requirement at maturity. F-35 Fleet Reliability - Aircraft reliability assessments include a variety of metrics, each characterizing a unique aspect of overall weapon system reliability. ▪ Mean Flight Hours Between Critical Failure (MFHBCF) includes all failures that render the aircraft unsafe to fly, along with any equipment failures that would prevent the completion of a defined F-35 mission. It includes failures discovered in the air and on the ground. ▪ Mean Flight Hours Between Removal (MFHBR) indicates the degree of necessary logistical support and is frequently used in determining associated costs. It includes any removal of an item from the aircraft for replacement. Not all removals are failures; some removed items are later determined to have not failed when tested at the repair site, and other components can be removed due to excessive signs of wear before a failure, such as worn tires. ▪ Mean Flight Hours Between Maintenance Event Unscheduled (MFHBME_Unsch) is a reliability metric for evaluating maintenance workload due to unplanned maintenance. Maintenance events are either scheduled (e.g., inspections or planned part replacements) or unscheduled (e.g., failure remedies, troubleshooting, replacing worn parts such as tires). MFHBME_Unsch is an indicator of aircraft reliability and must meet the Operational Requirements Document (ORD) requirement. ▪ Mean Flight Hours Between Failure, Design Controllable (MFHBF_DC) includes failures of components due to design flaws under the purview of TABLE 1. F-35 RELIABILITY METRICS (UP ARROW REPRESENTS IMPROVING TREND) Assessment as of June 30, 2018 MRHBCF (Hours) Flight Hours for ORD for JCS Threshold Cumulative Flight Hours F-35A 75,000 F-35B F-35C Variant MFHBR (Hours) MFHBF_DC (Hours) ORD Threshold Change: May 2017 to June 2018 Meeting Interim Goal for ORD Threshold ORD Threshold Change: May 2017 to June 2018 Meeting Interim Goal for ORD Threshold ORD Threshold Change: May 2017 to June 2018 Meeting Interim Goal for ORD Threshold JCS Requirement Change: May 2017 to June 2018 Meeting Interim Goal for ORD Threshold 74,758 20 ↑ No 6.5 ↑ No 2.0 No Change No 6.0 ↑ Yes 75,000 35,076 12 ↑ No 6.0 ↑ No 1.5 ↑ No 4.0 ↑ Yes 50,000 16,302 14 ↓ No 6.0 ↓ No 1.5 ↑ No 4.0 ↑ Yes - Between May 2017 and June 2018, six of the nine ORD metrics increased in value, often marginally, two decreased marginally, and one remained the same. Consistent with previous reports, the three JSF Contract Specification (JCS) metrics continued to show the strongest growth and, in all cases, were above their specifications for the 3 months ending June 2018. This strong MFHBF_DC growth has still not translated into equally strong growth 34 MFHBME (Hours) JSF for the ORD reliability metrics, all of which fall short of their interim goals. - More in-depth reliability growth analyses conducted by DOT&E show that the ORD reliability metrics are growing, albeit slowly, especially for F-35B and F-35C MFHBCF. Also, for the majority of the metrics, reliability grew markedly more slowly after the release of the Block 2B flight envelope than before. Based on these FY18 DOD PROGRAMS ▪ MCMTCF measures active maintenance time to correct only the subset of failures that prevent the F-35 from being able to perform a specific mission. It indicates the average time for maintainers to return an aircraft from NMC to MC status. ▪ MTTR measures the average active maintenance time for all unscheduled maintenance actions. It is a general indicator of the ease and timeliness of repair. - The program reports maintainability metrics for the 3 most recent months of data. Table 2 shows the nominal change in each maintainability metric by comparing values from May 2017 to those of June 2018, and whether the current value is on track to meet the requirement at maturity. ▪ All mean repair times are longer, some up to more than twice as long, as their ORD threshold values for maturity, reflecting a heavy maintenance burden on fielded units. - The JPO, after analyzing MTTR projections to maturity, acknowledged that the program would not meet the MTTR requirements defined in the ORD. The JPO is seeking relief from the original MTTR requirements and has proposed new values of 5.0 hours for both the F-35A and F-35C, and 6.4 hours for the F-35B. This will affect the ability to meet the ORD requirement for SGR, a Key Performance Parameter. analyses, none of the ORD metrics are predicted to meet their requirements by their individual variant maturity milestones. - In addition to reporting the MFHBCF values above, the JPO adopted a second, alternative approach for reporting MFHBCF in 2017 that only counts critical failures that take 8 hours or more to remedy. This approach presumably supports modeling of SGR, a Key Performance Parameter in the ORD. ▪ DOT&E continues to disagree with this approach because failures that take less than 8 hours to remedy will likely still affect SGR, especially during a combat sortie surge. Also, it is not consistent with the widely accepted definition of the MFHBCF measure. Maintainability - The amount of time needed to repair aircraft and return them to flying status has changed little over the past year, and remains higher than the requirement for the system at maturity. The program assesses this time with several measures, including Mean Corrective Maintenance Time for Critical Failures (MCMTCF) and Mean Time To Repair (MTTR) for all unscheduled maintenance. Both measures include “active touch” labor time and cure times for coatings, sealants, paints, etc., but do not include logistics delay times, such as how long it takes to receive shipment of a replacement part. TABLE 2. F-35 MAINTAINABILITY METRICS (DOWN ARROW REPRESENTS IMPROVING TREND) Assessment as of June 30, 2018 MCMTCF (Hours) Variant Flight Hours for ORD Threshold Cumulative Flight Hours MTTR (Hours) ORD Threshold Change: May 2017 to June 2018 Meeting Interim Goal for ORD Threshold ORD Threshold Change: May 2017 to June 2018 Meeting Interim Goal for ORD Threshold F-35A 75,000 74,758 4.0 ↓ No 2.5 ↓ No F-35B 75,000 35,076 4.5 ↑ No 3.0 ↑ No F-35C 50,000 16,302 4.0 ↓ No 2.5 ↓ No Live Fire Test and Evaluation F-35 Vulnerability to Kinetic Threats • Activity - In April 2018, Lockheed Martin delivered the F-35 Vulnerability Assessment Report summarizing the force protection and vulnerabilities of all three F-35 variants, and the F-35 Consolidated LFT&E Report, which summarizes the live fire test and analysis efforts supporting the vulnerability assessments. • Assessment - The assessments conclude the following: ▪ For three of the four specification threats, the F-35 variants meet JSF contract specification requirements to enable safe ejection of the pilot in the event of an engagement. ▪ For two of the four specification threats, the F-35A and F-35C variants meet JSF contract specification requirements to return safely to the Forward Line of Troops (FLOT) following an engagement. The F-35B met the requirements for only one of the four threats. ▪ All three F-35 variants are less vulnerable to three of the four specification threats than the legacy F-16C aircraft, both for safe ejection and for return to FLOT. - DOT&E will publish an independent evaluation of the vulnerabilities of the F-35 aircraft variants to expected JSF 35 FY18 DOD PROGRAMS and emerging threats in the report to support the Full-Rate Production decision scheduled for FY20. F-35 Vulnerability to Unconventional Threats • Activity - As of FY17, the Naval Air Warfare Center Aircraft Division at NAS Patuxent River, Maryland, completed full-up system-level testing of F-35A and C variants, and limited testing of the F-35B, to evaluate tolerance to electromagnetic pulse threats. - The program completed full-up, system-level, chemical-biological decontamination testing on BF-40 (a low-rate initial production F-35B aircraft) in February 2017. • Assessment - Testing was done to the threat level defined in Military Standard 2169B. Follow-on, full-up, system-level tests of the F-35B, including a test series to evaluate Block 3F hardware and software changes, are ongoing. - In the event of a chemical or biological attack, the equipment is capable of decontaminating the F-35. Additional work would be needed to develop an operational decontamination capability. ▪ To assess the protection capability of the Generation (Gen) II Helmet-Mounted Display System (HMDS) against chemical-biological agents, the JPO completed a comparison analysis of HMDS materials with those in an extensive DOD aerospace materials database. Compatibility testing of legacy protective ensembles and masks showed that the materials used in the protective equipment can survive exposure to chemical agents and decontamination materials and processes. The program plans similar analyses for the Gen III and Gen III Lite HMDS designs. While this assessment of material compatibilities provides some understanding of the force protection capability against chemical and biological agents, it does not demonstrate the process required to decontaminate either HMDS. F-35 Gun Lethality • Activity - From August through December 2017, during DT Weapons Delivery Accuracy testing, the Naval Air Warfare Center Weapons Division at Naval Air Weapons Station China Lake completed air-to-ground flight lethality tests of three different 25 mm ammunitions including the PGU-32/U Semi-Armor-Piercing High-Explosive Incendiary round, PGU-47/U Armor-Piercing High-Explosive Incendiary with Tracer round, and PGU-48/B Frangible Armor-Piercing round. Flight lethality tests included gun firings from all three F-35 variants against armored and technical vehicles, small boats, and plywood manikins. Tests revealed deficiencies with the Armor-Piercing High-Explosive round’s fuze reliability for impacts into 36 JSF the ground. Nammo, the Norwegian manufacturer, is conducting testing to further modify the fuze design and increase reliability. • Assessment - The weapon-target-pairing lethal effects are currently being analyzed by DOT&E. Recommendations: • The program should: 1. Continue to work with the Services to prioritize and correct the remaining Category 1 and 2 deficiencies discovered during SDD. 2. Apply lessons learned from SDD and other programs for scoping the amount of C2D2 testing that can be done in laboratories and simulations, compared with the need for flight testing. 3. Reassess the C2D2 plan to ensure adequate test infrastructure (labs, aircraft, and time) is provided and modifications are aligned with other fielding requirements. 4. Assess the annual cost of software sustainment. 5. Determine the cause of the accuracy problems with the F-35A gun firing and implement a solution for increasing gun accuracy for the fielded aircraft. 6. Develop a consolidated and adequate ALIS test venue to ensure ALIS capabilities are fully tested prior to fielding to operational units 7. Conduct a study to determine the optimum balance of additional spare parts procurement versus adding depot capacity to repair spare parts, in order to decrease the percentage of NMC aircraft waiting for spare parts. 8. Continue implementing measures to improve fleet availability. 9. Make actual aircraft or appropriate hardware- and software-in-the-loop facilities available to enable operationally representative air vehicle cyber testing. 10.Continue conducting periodic rounds of cybersecurity testing and correcting open cyber deficiencies. 11. Continue testing the integrity and security of the JSF supply chain, expanding on initial testing conducted in 2018. • The JPO should: 1. Complete contracting actions to procure a second F-35B ground test article in order to complete at least two lifetimes of structural durability testing to validate the wing-carrythrough structure. 2. Fund and contract for the 16-20 recommended signal generators called for in the JPO’s own 2014 gap analysis study. 3. Fund and contract for the necessary hardware upgrades to the USRL to support Block 4 development and testing.