OFFICE OF THE SECRETARY OF DEFENSE
1700 DEFENSE PENTAGON
WASHINGTON. DC 20301-1700

 

OCT 1 4 21115

OPERATIONAL TEST
AND EVALUATION

MEMORANDUM FOR UNDER SECRETARY OF DEFENSE FOR ACQUISITION,
TECHNOLOGY AND LOGISTICS

SUBJECT: Concerns Regarding Progress and Readiness of the Joint Strike Fighter (J SF)
Program for Initial Operational Test and Evaluation 

The purpose of this memorandum is to document my continuing concerns regarding
progress in the -35 JSF program as you prepare to conduct the upcoming Defense Acquisition
Board review. In a memorandum dated August 9, 2016, I identi?ed concerns to you, the
Secretary of the Air Force and the Chief of Staff of the Air Force that, in spite of the recent
Initial Operational Capability (10C) declaration by the US. Air Force, achieving full Block 3F
combat capability is actually at substantial risk. The primary concerns were that the program
appeared to be prematurely ending System Development and Demonstration (SDD) and was not
taking the necessary steps to be ready for which will be conducted using realistic
combat missions fully consistent with our war plans and threat assessments. The program?s
limited progress since the memorandum continues to indicate clearly the program will not be
able to deliver the full Operational Requirements Document (ORD)-required combat capability
within the planned remaining SDD schedule. The reasons I have reached this conclusion include
the following:

0 Continued schedule delays. According to the program?s baseline mission systems
software and capability release schedule, the planned release to ?ight test of Block
3FR6 mission systems software has slipped from February 2016 to December 2016,
10 months later than originally planned. This delay was caused in part by the need
for multiple additional ?Quick Reaction Capability? (QRC) software builds of Block
3FR5 to enable weapons testing to proceed and to reduce stability problems.
However, since the program was funded to the baseline schedule, this 10-month delay
in Block 3FR6 software indicates strongly that the program has shortfalls in funding
and time to complete the planned testing of the remaining set of full Block 3F
capabilities and necessary ?xes. Moreover, releasing Block 3FR6 in December is
another 3-month delay to the program?s more recent estimate that this version of
Block 3F software would be released to ?ight test in September.

0 Need to complete all planned and agreed-to developmental testing (D T). The
program?s continued cost- and schedule-driven plan to truncate planned DT points
and prematurely close-out SDD would shift significant risk to OT and the warfighter.
This ill-advised action would also discard carefully planned build-up test content in
the Test and Evaluation Master Plan (TEMP) that was not included as an optional
throwaway, but rather was content the Program Executive Of?cer formally agreed
was required when he signed the TEMP. The program?s plan to ?quarantine? build-
up test points that were in the Joint Test Plan (J TP) and planned to be ?own by the

a



test centers, skip ahead to complex mission effectiveness test points, and then delete
the build-up test points as ?no longer required? will only delay problem discoveries
and increase the risk to as well as to the men and women who will use the F-
35 in combat. Additionally, the program will need to allocate test points not in its
current plans for characterization, root cause investigations, and correction of a large
number of the open de?ciencies and technical debt described later in this
memorandum. The completion of the planned baseline test points from the Block 3F
TP, along with correction or mitigation of signi?cant de?ciencies, is necessary to
ensure full Block 3F capabilities are adequately tested and veri?ed before operational
test and, more importantly, before they are ?elded for use in combat.

Insuf?cient progress in -35A, 17-353 and -35C ?ight sciences testing. Although
progress has been made in all variants, each is behind in planned test point
completion for the year, as shown in the table below (data as of the end of
September).

 

 

 

 

 

 

Variant Planned Points Planned Points for
thruSep 30,2016 thruSep 30,2016 CY16
F-35A 1322 1080 1364
F-35B 1593 1580 2119
F-35C 1441 1354 1906

 

 

 

 

Insuf?cient progress in F-35 mission systems testing. As of the end of September,
the program had only accomplished 2,069 mission systems test points against the goal
of 3,1 89 and the plan of 3,709 for the year. Despite falling farther behind and
carrying a signi?cant number of open de?ciencies, the program has decided to
terminate testing of Block 3F so?ware as scheduled in CY17 due to inadequate
funding to complete the planned testing in the JT P. As a result of this decision and
ongoing software delays, the program has deleted two full software releases from
their mission systems schedule, removing Block 3FR8 and replacing 3FR7 with
additional contingency QRC software builds of 3F R6, which will now be the last full
developmental software release. The outcome of these decisions is that the remaining
number of software releases to complete Block 3F development is currently
insuf?cient to support adequate testing to identify and correct de?ciencies prior to
and use in combat. Although the 3FR6 release in late 2016 is planned to
have full Block 3F capabilities, some of those capabilities will be tested for the ?rst
time in that release and will certainly not be mature enough to be effective without
additional testing and the necessary additional time and resources. In particular,
additional builds of software to characterize and correct de?ciencies, each of which
will also require regression testing to verify ?xes, will be needed. These problems
are exacerbated by the proposal to quarantine test points described above. Despite
these delays, and the fact that some of the ?full? Block 3F capabilities are just
beginning ?ight test or have not yet started gun accuracy testing), the program
still plans to terminate ?ight testing as scheduled in early 2017 and ?nalize Block 3F.

0 Insuf?cient time and resources to conduct all required weapons delivery accuracy

(WDA) events. The program completed a surge of weapons test events in August and
is analyzing the results. While some of the events appear to have been successful,
several WDAs unsurprisingly had signi?cant issues that either required control room
intervention or the employment of the weapon was likely unsuccess?il. Despite
making some progress, the program still has not completed the full set of planned test
events for Block 3F weapons in the TEMP, with 13 WDAs remaining, excluding the
multiple gun scoring events, which must also be completed. Due to the limited time
and funding remaining in SDD, the program has prioritized completing testing of new
and de?cient Block 3F mission systems capabilities over completing the remaining
WDAs. While completion of Block 3F mission systems is necessary, the WDAs are
also an integral part of successfully completing required development and adequate
testing of full Block 3F capabilities. Each of the planned WDA events is an essential
end-to-end test of the full ?re-control chain. Conducting all of the WDAs is the only
way to discover problems that otherwise will be realized in operational test and/or
combat. For example, one of the recent AIM-120 missile WDA events required
control room intervention to direct the pilot when to launch, as there were no shoot
cues or launch zone indications displayed to the pilot due to an outdated AIM-120
missile attack model within the mission systems so?ware. Due to their importance
and the distinct differences among them, all of the planned WDA events must be
completed during otherwise, these events will have to be completed before or
during which will delay discovery of de?ciencies and the completion of
while adding to its cost.

Insuf?cient progress in gun testing. Planned gun testing continues to fall farther
behind as the program works through design de?ciencies, test discoveries, and the
resulting modi?cations to the test aircraft. Despite the limited time remaining in
SDD, the program still has not completed initial ?ight sciences testing of the -35B
gun pod, started ground testing of the F-35C gun pod, or attempted an aimed gunshot
using the Helmet Mounted DiSplay System (HMDS) on any variant. Based on
discoveries during F-35A ?ight sciences gun testing, required changes to vehicle
systems software are being added to Block 3FR6 to attempt to mitigate yaw induced
by the gun ?ring in the F-3 5A, as well as expected pitching moments when the gun
pod is ?red under the F-3SB and F-35C this adds further to the substantial burden of
problems 3FR6 is supposed to correct. The ?rst ?ight testing of a properly modi?ed
F-3 5A gun from a mission systems aircra? with 3F software, aimed by the Gen 
HMDS, was planned to start in October but will likely not begin until 2017 due to
continued delays.

Ineffective operational performance. The performance of earlier Block 3F versions
during DT to date shows signi?cant operational shortfalls. An assessment, based on
OT pilot observations of DT missions, of the operational utility of Block 3F R5 .03
software to support planned missions, including Close Air Support,
Destruction/Suppression of Enemy Air Defenses, Offensive and Defense Counter-
Air, Air Interdiction, and Surface Warfare, rated each of the mission areas ?red? and

unacceptable overall, with signi?cant de?ciencies in capabilities and/or performance
shortfalls.

Numerous remaining de?ciencies and technical debt. The program?s recent
decision to eliminate two full so?ware builds and delete TEMP- and TP-required
testing due to software schedule slips and funding shortages is inadequate to address
the large number of signi?cant open De?ciency Reports (DRs) remaining in SDD.
This plan assumes no further signi?cant discoveries in however, even in the
unlikely event no additional discoveries are made, the program is running out of time
and budget to properly test and verify the required ?xes for the existing DRs. The
program currently has 146 Category 1 and 1,033 Category 2 ?active? Open DRs,
along with 16 new DRs, since the last de?ciency review board on September 26,
2016. Of the 1,179 DRs, there are 528 that are being categorized as ?Open Under
Investigation? (OUIN) and 385 categorized as ?Open Awaiting Fix Veri?cation?
(OAF V). All of the 385 OAFV DRs require ?ight test activity by the Integrated Test
Force (ITF), and a large percentage of the OUIN will need ?ight test points to gather
root cause data. None of these test points are currently allocated or accounted for in
the ITF ?ight test priority. The scope of unaccounted-for DRs and the program?s
intention to terminate ?ight testing early demonstrate clearly the need for additional
resources to complete SDD.

Shortfalls in the Autonomic Logistics Information System (ALIS). The program
continues to experience delays in the development and ?elding of ALIS.

The latest version of ALIS in development version 2.0.2 was planned to be
delivered by August 2016, as the Air Force had expected it to be ?elded prior to
their declaration of Initial Operational Capability (IOC), but it has yet to
successfully complete testing and likely will not be ?elded until early 2017. The
key additional capabilities in ALIS 2.0.2 include propulsion integration, which
will allow uniformed maintenance personnel to download and process engine data
with the rest of the aircraft data in ALIS following ?ight. Currently, the
propulsion data must be processed separately by Pratt Whitney ?eld service
representatives.

0 Delays in ALIS 2.0.2 deve10pment have also delayed the deve10pment of ALIS
3.0, the planned ?nal release of ALIS software for SDD. Because of these
cascading delays and additional emerging service and partner requirements,
including critical security enhancements, the program adjusted development and
?elding of remaining capabilities and has moved content out of ALIS 3.0 into
post-SDD releases. The cumulative effect of these deferrals and unresolved
de?ciencies on suitability will be evaluated during 

Inconsistencies between contract specifications and the 0RD. The program has
accepted numerous changes or deferrals to contract speci?cations, while not receiving
formal relief from, or changes to, the associated requirements in the 0RD. As an
example, the program of?ce, in coordination with the Services, determined that the

speci?cation requirements for gun accuracy could not be met with the new
ammunition planned to be used, the Frangible Armor Piercing (FAP) round for the -
35A and the Semi-Armor Piercing High Explosive Incendiary-Tracer (SAPHEI-T)
round for the F-3 5B and F-35C. The program of?ce completed a speci?cation
change to the contract to delete the old requirement for gun accuracy and lethality,
but did not add the new planned speci?cation values nor complete any requirements
changes for the 0RD. As a result, the program now apparently has no contract
speci?cations for either air-to-air or air-to- ground lethality and engagement
performance; however, the program still has approved air-to-ground 0RD criteria that
have not been adjusted or changed, which are not possible to achieve due to the
change in ammunition. The SF stakeholders, including the Services and Joint Sta??,
should immediately conduct a requirements review of the 0RD versus the contract
speci?cations to identify documentation or performance shortfalls as the program
closes out SDD.

Inadequate preparations for The program of?ce and some other SF
stakeholders have proposed a ?phase-start? for based on the assumption that
the modi?cation schedule for the ?eet of OT aircraft will provide some aircraft earlier
with which testing could begin. Besides the modi?cations to the OT aircraft being
substantially late to need to start (see immediately below), the full Block 3F
?ight envelope and weapons clearances, along with a veri?ed Block 3F mission data
?le, will not be available before May 2018, according to the program?s most recent
schedule estimates. will not approve a ?phased start? for that
violates the spin-up and test entrance criteria, as outlined in the TEMP (list of criteria
attached), which was signed and approved by the -35 stakeholders, including the

SF Program Executive Of?cer. (Note that these criteria include a detailed and
de?nitive de?nition of the agreed composition of full Block 3F combat capability.)
This includes the requirement for all 18 U.S. OT aircraft and the partner OT aircra?
to be in the Block 3F production-representative con?guration. The full ?eet of OT
aircraft, with the full Block 3F capabilities including enve10pe and weapons, is
required for the ef?cient and effective execution of Spin-up mission rehearsals and for
successful execution of the complex plan, which includes four-ship and eight-
ship test trial missions. These are common-sense, long-agreed-to criteria that must be
satis?ed to conduct a realistic and rigorous test of the Block 3F capabilities that will
actually be ?elded so that our war?ghters will know what the aircraft truly can and
cannot do in combat the inviolate reason for the test.

0 Late plans for modi?cation of OT aircraft. The TEMP requirement to provide
production-representative Block 3F OT aircraft for has been well known
for more than seven years; however, the program has not adequately planned nor
contracted for the necessary modi?cations, including the Technical Refresh 2
(TRZ) processor upgrades. This failure to develop an adequate plan for providing
modi?ed OT aircraft does not relieve the program of the spin-up and test
entrance criteria. Late discovery of issues during development such as those
requiring the extensive modi?cations to provide an operational gun system or the
ability to carry the AIM-9X missile throughout the employment envelope on the

F-3SC - are continuing and should be expected for a program as complicated as
the SF that is experiencing signi?cant development and testing delays.

However, these issues must still be addressed with modi?cations to the OT
aircraft. Expecting to allow to start without a full complement
of fully production representative aircraft, as agreed to and documented for years,
is a recipe for a failed test, especially in light of the aircraft availability issues
mentioned later. Failure to meet the TEMP entrance criteria means not only that
the program is unready for Operational test it means SF is not ready for combat
and, therefore, certainly not ready for a Block Multi-Year) Buy or full-rate
production.

0 Inadequate aircraft availability (A VA). Although AVA is not an entrance
criteria, if the program is only able to achieve and sustain its goal of 60 percent
AVA, the length and cost of will increase signi?cantly because the
expected combat-ready availability of 80 percent was planned for in the TEMP
and is needed to ef?ciently accomplish the open-air mission trials with the
number of aircraft planned for The ?eet of operational test aircraft,
currently consisting of 8 F-35A and 7 F-3 SB aircraft, averaged an AVA of
approximately 50 percent over the last 6 months (through the end of September),
as shown in the table below. Although better than average AVA of all of
the Lot 3 through Lot 5 aircraft from which the OT aircraft were produced this
is well short of the 60 percent objective and not adequate to support the ?ight rate
of test trials planned for The table below also shows the maximum and
minimum average AVA over the last 6 month period, for reference, and
indicates the wider variance in the OT ?eet, as would be expected from a smaller
sample size. Over the same six-month period there has been no readily
discernable trend of increasing or decreasing availability for any of the groups of
aircraft, supporting the assertion that availability has ?at-lined and will not
improve signi?cantly prior to the start of 

 

 

 

 

 

 

 

 

 

Aircraft Avegge Maximum Minimum
OT (8 NC) 51.2% 64.5% 39.8%
F-3SB OT (7 50.4% 64.2% 34.6%
Lots 3 thru 5 (76 NC) 44.5% 49.0% 40.8%

 

0 Insufficient progress in air-to-air range instrumentation (AARI). AARI has not
yet been tested in the F-35. In fact, the required DT of AARI has not yet been
planned. Despite the limited time remaining in SDD, the AARI DT must be
completed in time to support a conection cycle so this TEMP-required
system is ready in time to support and not delay 

Inadequate Fusion Simulation Model (FSM). Corrections to this model, which
is currently too unrealistic to be used for are required and must be put on
contract to ensure FSM can support requirements.

 Inadequate Virtual Threat Insertion (VTI). The task of adding missing threats
required for to the VTl-associated reference table must also be put on
contract as soon as possible. This will ensure threat messages from AARI for
required threats can be recognized and displayed by FSM on the F-35 cockpit
displays during 

Inadequate United States Reprogramming Lab (USRL). Upgrading the USRL to
the necessary Block 3F configuration is late to need to enable the USRL to begin the
development of Block 3F mission data ?les the latest projection is that the
USRL will not be able to start building basic Block 3F MDFs until February 2017.
However, because of the inadequate tools provided to the USRL and the complexity
of the MDFs, the USRL estimates that it will take approximately 15 months to create,
optimize and validate the MDF for Also, because the program failed to
order the required signal generators, the Block 3F MDFs will not be optimized
against several ?elded threats of signi?cant concern. The inadequately equipped
USRL increases the likelihood of failure in operational test, and, m0re importantly, in
combat.

Substantial Risks to Follow 0n Modernization (F oM). Despite the signi?cant
ongoing challenges with F-35 development listed above, including the certainty of
additional problem discoveries, the pr0posed modernization schedule is not
executable. Even with the signi?cant ongoing SDD delays and problems delivering
full Block 3F capabilities, the program still plans to award contracts to start
simultaneous development of Blocks 4.1 and 4.2 in 2018, well prior to completion of
(and possibly before it has even started for the reasons detailed above), and
therefore lacking understanding of the inevitable problems it will reveal. Also, the
proposed aggressive modernization plan and overlapping schedule for Block 4
increments do not depict adequate schedule and resources for formal operational
testing. In addition, due to the cost and complexity of the proposed additional
capabilities in Block 4, sufficient test resources, including enough test aircraft, will
need to be available. Furthermore, because of program concurrency resulting in the
?elding of multiple con?gurations, different avionics processors) additional
con?gurations of test ?eet aircraft will be needed. For example, enhancements and
?xes of mission systems software for aircra? with TR2 processors will be needed
while capabilities are developed and tested simultaneously for aircraft with new open-
architecture Technical Re??esh 3 (TR3) processors. Due to the hundreds of aircraft
that will already have been produced, the program and Services will be sustaining
aircraft with TR2 processors with versions of Block 4 software for 10 to 15 years
before all aircraft can be modi?ed to the TR3 con?guration.

For all the reasons stated above and described in my previous memoranda, the F-35
program clearly lacks suf?cient time and resources to deliver full combat capability and be ready
for operational testing and deployment to combat if it is unwisely constrained to operate within
its currently planned budget and schedule. The program should now be provided the additional
resources required to deliver full Block 3F combat capability; i. e. to complete all the testing
(including regression) needed to rectify a substantial number of existing critical de?ciencies as

well as the new de?ciencies that will inevitably be discovered during the remaining Block 3F
testing.

Failure to adequately finish SDD will result in far greater risks and costs than completing
it now. First, since the program clearly will not be able to start in August 2017, as
indicated in their program of record, the program?s plan to draw down staf?ng and test
infrastructure in to close out SDD would occur at a time when the program should be
aggressively using the full capacity of the current test resources and experienced personnel to
complete testing, address de?ciencies, and ensure full Block 3F capability is delivered and ready
for and combat. Second, if the program continues with plans to close out SDD
prematurely, it will carry the high risk of failing and having to repeat the approximately $300-
million operational test, and failing for many years to provide the full combat capability Block
3F has long been meant and claimed to provide. Third, the unresolved technical debt will spill
into where it will take longer to ?x and cost more to address than if recti?ed now. Finally,
the combination of un?nished SDD work and the likely follow-on Operational test would
signi?cantly delay. and increase the cost of, achieving the important capabilities which are
urgently needed to counter current and emerging threats.

I therefore recommend very strongly that the program be restructured now and provided
the additional resources it clearly requires to deliver its long-planned and sorely needed full

Block 3F combat capability.
Michael Gilmore

Director

Attachment:
As stated

cc:
Members of the Defense Acquisition Board