PRE-DECISIONAL – FOR INTERNAL IRS W&I USE ONLY – NOT FOR PUBLIC RELEASE

Internal Revenue Service (IRS)
Wage & Investment (W&I)
Sponsor: IRS
Dept. No.: P181
Contract No.: TIRNO-99-D-00005
Project No.: 2032H5-18-F-00593

The views, opinions and/or findings
contained in this report are those of The
MITRE Corporation and should not be
construed as an official government position,
policy, or decision, unless designated by
other documentation.

IRS Free File Program

Independent Assessment of the
Free File Program

For Internal MITRE Use. This document was
prepared for authorized distribution only. It
has not been approved for public release

This technical data was produced for the
U.S. Government under Contract Number
TIRNO-99-D-00005, and is subject to
Federal Acquisition Regulation Clause
52.227-14, Rights in Data—General, Alt. II,
III and IV (DEC 2007) [Reference
27.409(a)].

Free File Program Assessment Report
Executive Summary
October 3, 2019

No other use other than that granted to the
U.S. Government, or to those acting on
behalf of the U.S. Government under that
Clause is authorized without the express
written permission of The MITRE
Corporation.
For further information, please contact The
MITRE Corporation, Contracts Management
Office, 7515 Colshire Drive, McLean, VA
22102-7539, (703) 983-6000.
©2019 The MITRE Corporation.
All rights reserved.

McLean, VA

PRE-DECISIONAL – FOR INTERNAL IRS W&I USE ONLY – NOT FOR PUBLIC RELEASE

 PRE-DECISIONAL – FOR INTERNAL IRS W&I USE ONLY – NOT FOR PUBLIC RELEASE

This page intentionally left blank for two-sided printing.

PRE-DECISIONAL – FOR INTERNAL IRS W&I USE ONLY – NOT FOR PUBLIC RELEASE
ii

 PRE-DECISIONAL – FOR INTERNAL IRS W&I USE ONLY – NOT FOR PUBLIC RELEASE

Executive Summary
In January 2019, the Internal Revenue Service (IRS) Free File program (“Free File” or
“the program”) opened for its 17th filing season. Established in 2002, the IRS Free File program
makes free online tax preparation and electronic filing available to 70 percent of the taxpaying
population; for Filing Season (FS) 2019, this equated to taxpayers with an adjusted gross income
(AGI) of $66,000 or less. The program is made possible through a partnership between the IRS
and the tax software industry. The partnership is governed through a mutually negotiated
Memorandum of Understanding (MOU) between the IRS and the Free File Alliance (FFA)1—a
coalition of individual private-sector tax preparation companies2—that has evolved over the
program’s lifespan to address issues as they have arisen.
While cumulatively the program has resulted in more than 53 million free returns e-filed and an
estimated $1.6 billion in savings to taxpayers3, the Free File program has also faced criticism in
recent years as questions have been raised by Congress and the press about low annual
participation rates and assertions of inadequate oversight, deceptive business practices, and
misaligned roles for government and industry. This year’s launch came amid criticisms of the
program and speculation that industry partners were not acting in the best interest of taxpayers.
In response to these assertions as well as recent oversight activities and congressional inquiries,
IRS leadership determined that an independent assessment was necessary to ensure the continued
operations and integrity of the IRS Free File program. Because the MITRE Corporation operates
the Service’s federally funded research and development center (FFRDC), the IRS engaged
MITRE to conduct an independent and objective assessment of the IRS Free File program in its
current state, providing recommendations for improvements to the program, and by extension to
the MOU that governs the program.

Methods
Beginning in June 2019, MITRE established several workstreams for addressing each of the
targeted assessment areas reflected in Figure ES-1. Taken together, these workstreams produced
findings and recommendations related to Free File member compliance with the MOU and
potential enhancements to the program. Highlights of the assessment approach include:

1

In 2014, the Free File Alliance officially changed the name of the organization to Free File, Inc. (FFI) doing business as the
Free File Alliance. While FFI is the listed name on all agreements and MOUs beginning in 2014, the two names are used
interchangeably. The IRS references the Alliance on IRS.gov “Free File: About the Free File Alliance,” and the organization has
not changed the name of its own website: https://freefilealliance.org/. This document will use the term Free File Alliance, the
Alliance, or FFA except when FFI is used in a specific context or quote.
2 Industry participation in the Free File program is voluntary. As stipulated in the MOU, there is a process for applying for
membership to the Free File Alliance and companies may leave the program at their discretion. Also outlined in the MOU, IRS
and FFA leadership can remove members from the program if there are certain MOU violations. Through FS 2019, the Free File
Alliance included 12 member companies; currently there are only 11 participating members of the program. As of July 2019,
Drake [1040.com] chose to withdraw from the program and will not be a part of it going forward. Since they were a part of the
program for FS 2019, they are still included in this assessment. The decision to leave came from the member itself and to
MITRE’s knowledge were not related to any MOU infractions.
3 Internal Revenue Service, "IRS, Free File Alliance announce changes to improve program; Improved taxpayer options available
for 2019 Free File program," Internal Revenue Service, 2 November 2018. [Online]. Available:
https://www.irs.gov/newsroom/irs-free-file-alliance-announce-changes-to-improve-program-improved-taxpayer-optionsavailable-for-2019-free-file-program. [Accessed 2 July 2019].

PRE-DECISIONAL – FOR INTERNAL IRS W&I USE ONLY – NOT FOR PUBLIC RELEASE
iii

 PRE-DECISIONAL – FOR INTERNAL IRS W&I USE ONLY – NOT FOR PUBLIC RELEASE

• Current Environmental Scan. MITRE conducted an environmental scan to better
understand the Free File program in terms of historical context, taxpayer participation, the
role of IRS and industry in its public-private partnership (PPP), the economics of “free” as
it pertains to Free File, and recent interest in the program.
• Free File Program Analysis. MITRE conducted stakeholder interviews—including IRS
leadership—and performed document reviews to assess the IRS Free File program’s
alignment to objectives, performance, and the IRS management and oversight functions.
• FFA Member Organization Interviews (“site visits”). MITRE conducted interviews
with representatives from each Free File member organization to collect information on
their interactions and agreements with the IRS regarding the Free File program and
applicable business processes, practices, and techniques for how each Free File member
determines if a taxpayer qualifies for Free File. Each of the interviews followed a similar
approach: standard interview protocols, targeted questions (both related to the IRS Free
File program as well as the MOU Article 4, Standards of Practice) and in-person or phone
interviews with participants from each Free File member organizations (selected by the
member company’s leadership).
• Web Forensics Analyses. MITRE performed an analysis to determine if any coding
within the member’s website influences search engine routing. In addition, MITRE
analyzed web search query traffic metrics for internet searches that signaled an intent to
find free tax filing services, based on a representative sample of organic (unpaid) search
keywords for free tax filing. This included a click-through analysis of IRS.gov site usage
metrics to investigate where taxpayers went from the IRS Free File web portal. This
analysis was also used to determine how often taxpayers searched for free tax filing
services (especially during tax filing season) using two primary search engines, Google
and Bing; to what site(s) these users went following their searches; and the differences in
search engine results.
• Taxpayer Journey. MITRE conducted an expert review of the IRS Free File portal as
well as each individual Free File member offer. This review was twofold in its objective:
first, MITRE evaluated the 12 member sites for compliance with MOU requirements
(including an initial landing page analysis); second, MITRE evaluated the 12 member sites
and the Free File-related pages on IRS.gov against recognized software user interface
usability principles (called heuristics) in order to identify potential usability problem
areas. For both parts, MITRE set-up user accounts and ran scenarios to test the sites in a
systematic, exhaustive fashion. MITRE’s usability assessment used both industryrecognized criteria (heuristics) as well as MITRE’s interpretation of how MOU
requirements translate into usability criteria and whether sites are optimized for mobile
devices.
• Taxpayer Usability Testing. MITRE conducted a study to test the usability of the IRS
Free File portal and assess taxpayers’ actual experience in filing tax returns using three
randomly selected Free File software offerings. MITRE conducted in-person usability
testing sessions with 29 taxpayers who filed their taxes electronically using Free File
software in prior year filings. During the testing, taxpayers were observed as they
completed fictitious tax scenarios using the IRS Free File portal and Free File software. In
PRE-DECISIONAL – FOR INTERNAL IRS W&I USE ONLY – NOT FOR PUBLIC RELEASE
iv

 PRE-DECISIONAL – FOR INTERNAL IRS W&I USE ONLY – NOT FOR PUBLIC RELEASE

addition to data on the usability of the Free File services, MITRE collected and analyzed
data on the frequency of errors committed, taxpayer satisfaction, and experience (via a
brief questionnaire), and taxpayers’ perceptions of the Free File program.
• Legal Analysis. MITRE’s General Counsel, in coordination with outside counsel,
conducted a review of the MOU’s terms and conditions to determine recommended
revisions to the MOU to more clearly define the parties' responsibilities to enhance and
promote the IRS Free File program, address any shortcomings in the MOU language as it
relates to the prohibition against unfair methods of competition, and provide enforcement
capability via liability clauses if the MOU's terms and conditions are breached by a
Free File member organization.
• Recommendations. Based on the data gathered from the workstreams, MITRE developed
recommendations for overall improvements to the Free File program, potential
enhancements to the MOU, and improvements to usability of IRS.gov and some aspects of
member offerings.
Figure ES-1 shows how each of these workstreams collectively contributed to addressing
targeted assessment areas (assessment objectives) including: documenting the current Free File
program environment, assessing Free File program management and oversight, assessing FFA
member compliance with the MOU, assessing taxpayer experience with respect to the program,
and ultimately drafting recommendations for the program and MOU. Outputs from some
workstreams contributed to more than one assessment area.

Figure ES-1. High-Level Approach to Free File Program Assessment (Aligned to Assessment Objectives).

PRE-DECISIONAL – FOR INTERNAL IRS W&I USE ONLY – NOT FOR PUBLIC RELEASE
v

 PRE-DECISIONAL – FOR INTERNAL IRS W&I USE ONLY – NOT FOR PUBLIC RELEASE

Addressing Public Perceptions
Throughout the assessment, MITRE considered the perspectives of various media, oversight, and
advisory organizations: the recent ProPublica articles on the Free File program, the 2018 Internal
Revenue Service Advisory Council (IRSAC) Annual Report, and the 2018 National Taxpayer
Advocate Annual Report to Congress4. Three primary themes emerged from these external
sources:
1. Member companies use deceptive strategies to steer taxpayers away from using the Free
File program services towards fee-based services.
2. The IRS Free File Program Office does not provide adequate oversight.
3. Low usage rates signal that the Free File program is not working.
However, MITRE’s findings from this assessment determined that while there may be an
element of truth to each of the criticisms, the data indicate the issues are much more nuanced and
don’t tell the full story.
MITRE conducted the assessment of the Free File program in its current state—the tax software
industry providing free tax preparation and filing services under a MOU with the IRS. It was not
in the scope of MITRE’s effort to address the underlying political issue that is the genesis of
much of the controversy surrounding the Free File program—whether the IRS should create its
own free tax preparation and filing program or continue to rely on the tax software industry to
provide these services. While MITRE recognizes that government services provided to its
citizens are continually evolving, and that Congress may determine in the future that the IRS
should offer that service (and provide it with the appropriate funding to do so), for the purposes
of this study, MITRE assumes that industry will continue to be the entity that provides free tax
return preparation and filing offerings to taxpayers. All of MITRE’s recommendations in this
report are based on that assumption.
Search Engine Avoidance
The issue that seems to have created the most publicity and was at the center of much of the
congressional interest is the practice of coding that caused members’ Free File landing pages to
be excluded from an organic search through Google or Bing search engines. During FS 2019,
five of the 12 FFA members did engage in a practice that excludes their company’s Free File
landing page in organic searches. When questioned about it, most members contended this keeps
them in compliance with the MOU (language in the MOU preamble) and in alignment with IRS
marketing that states Free File is “accessible only through IRS.gov.”5 MITRE did not take a
position on the practice—determination of whether this is a deceptive business practice is not
within MITRE’s legal purview. The perception points to the need for the IRS to take a position
on the practice and incorporate new language into the MOU. MITRE also conducted an in-depth
analysis of what the results of organic searches provided.

4

Internal Revenue Service, "Internal Revenue Service Advisory Council Public Report, Publication 5316, Catalog Number
71824A," Internal Revenue Service, Washington, D.C., 2018.; Taxpayer Advocate Service, "National Taxpayer Advocate Annual
Report to Congress," Taxpayer Advocate Service, Washington, D.C., 2018.; Series of ProPublica articles from March 20, 2017 to
June 5, 2019, available at https://www.propublica.org
5 Internal Revenue Service, "Changes strengthen program; Free File can also help navigate new tax law provisions," Internal
Revenue Service, 11 january 2019. [Online]. Available: https://www.irs.gov/newsroom/irs-free-file-opens-today-in-advance-oftax-season. [Accessed 10 September 2019].

PRE-DECISIONAL – FOR INTERNAL IRS W&I USE ONLY – NOT FOR PUBLIC RELEASE
vi

 PRE-DECISIONAL – FOR INTERNAL IRS W&I USE ONLY – NOT FOR PUBLIC RELEASE

What’s been said …

Member companies use deceptive
strategies to steer taxpayers away from
using the Free File program services
towards paid commercial services

What the data show…
•

Five of the 12 members used a coding device6 to keep
their Free File landing page out of organic searches.

•

Based on a representative sample of organic (unpaid)
search keywords for free tax filing, the highest
number of searches went to IRS.gov (>780,000). The
top member landing page had 698,000 searches. The
next highest was less than 180,000 searches. The
total unpaid traffic to the 12 members was 1.8
million.

•

Paid searches resulted in the highest amount of
traffic (using the same representative sample of
keywords as above), more than 10 million total to the
12 member sites combined, none to IRS.gov (IRS does
not pay for advertising).

•

IRS.gov Free File landing page received 2.8 million
entrances7 (the landing page was the first page a user
saw when arriving on IRS.gov) during FS 2019.

•

Four of the five members who engaged in the
practice had increases in Free Filing in FS 2019, and
one remained relatively flat. Five of the seven who
did not engage in the practice had increases in Free
Filing, and two had decreases.

•

Number of taxpayers who received free filing outside
of Free File through free commercial offerings is not
publicly available. However, according to the FFA, the
current members provided a total of more than 17
million of these free commercial filings in FS 20198.

(ProPublica).

6

“<META NAME="ROBOTS" CONTENT="NOINDEX">meta
Because the IRS does not employ any tracking codes once a user leaves the site, there is no way to know how many of these 2.8
million entrances became Free File users.
8
The actual number of free offerings may be higher. One company who was a member in 2019 did not provide a number.
7

PRE-DECISIONAL – FOR INTERNAL IRS W&I USE ONLY – NOT FOR PUBLIC RELEASE
vii

 PRE-DECISIONAL – FOR INTERNAL IRS W&I USE ONLY – NOT FOR PUBLIC RELEASE

Table ES-1 depicts the effects of organic versus paid searches for the top 108 non-branded high
value keyword search terms that drove traffic to Free File member landing pages. This data was
derived from website traffic information downloaded from SEMrush9, a digital market software
as a service company.
Table ES-1. Comparing Paid Search Traffic to Organic Search Traffic9.

The data in Table ES-1 shows that for the organic searches, more traffic went to the IRS.gov
Free File landing page than to any one of the individual member sites’ non-Free File landing
pages. However, the most traffic, as would be expected, went to paid advertising. Seven of the
12 Free File members bought ads for keywords related to free tax filing that directed traffic to
the landing pages for their non-Free File free tax filing software offering, resulting in five times
more traffic than organic searches. The data used for this analysis (not shown in this table)
indicated that no member had paid advertising for their Free File landing pages. This is not
unexpected, as the MOU specifically assigns advertising responsibility to the IRS. However,
during the FFA member interviews, one member stated that their company does employ
advertising practices for the Free File program10.
Program Oversight
The perception that members were undertaking potentially deceptive practices that resulted in
vulnerable taxpayers paying for tax services–compounded by the program’s low participation
rates–led to criticism that the IRS Free File Program Office was not providing adequate oversight
of member behavior. What this critique overlooks is the fact that the Free File program operates
as a public-private partnership and, as such, program oversight is a mutual, collaborative effort

9

SEMrush Inc., "All-in-one Marketing Toolkit," SEMrush Inc., 2019. [Online]. Available: https://www.semrush.com/. [Accessed
week of 27 May 2019].
10 During MITRE interviews with representatives of FFA member organizations, one member indicated that their company pays
for advertising of the Free File program through a number of methods. However, the interview did not include detailed discussion
of advertising practices specific to key word search. For this analysis, MITRE looked specifically at ads purchased for key words
related to free tax filing. The results of this analysis indicated that no purchased ads directed users to the landing pages of Free
File offers.

PRE-DECISIONAL – FOR INTERNAL IRS W&I USE ONLY – NOT FOR PUBLIC RELEASE
viii

 PRE-DECISIONAL – FOR INTERNAL IRS W&I USE ONLY – NOT FOR PUBLIC RELEASE

with the industry, represented by the Free File Alliance. While there is room for improvement,
based on assessment findings, MITRE considers the IRS’s current compliance processes
adequate and effective to support the integrity of the program. In part, because of this longstanding partnership with FFA and continuous assessment and improvement of the program as it
has evolved.

What’s been said …

What the data show…
•

The IRS Free File Program Office does
not provide adequate oversight (IRSAC,
Taxpayer Advocate Service [TAS],
members of Congress).

•

•
•

Because the Free File program is a PPP, and not a
paid contractual relationship, it must work
collaboratively with industry on oversight of the
program.
IRS program office conducts member website
reviews, evaluates member offer coverage and
monitors member acceptance rates, while working
closely with FFA to address, document, and resolve
any identified or potential compliance breaches.
IRS’s existing agency governance of the Free File
program effectively allows the program to fulfill the
IRS’s currently articulated program goals.
While the program office does not have updated,
measurable objectives for internal agency
performance, its shared oversight of member
compliance strictly follows the provisions of the MOU
and meets joint program objectives.

Taxpayer Participation in Free File Program
Finally, what has led many to question the overall effectiveness of the Free File program is the
participation numbers themselves. In 2018, fewer than 3 million filers used the Free File program
to file their tax year 2017 returns out of nearly 104 million taxpayers that meet the eligibility
requirements for the program. However, these numbers don’t tell the entire story and take the
issues of taxpayer choice and taxpayer behavior out of the equation. The Free File program will
generally appeal to those taxpayers that prefer a “do-it-yourself” (DIY) method of tax
preparation and filing. Looking retrospectively at the eligible population from FS 2018 and how
those taxpayers chose to file can help the IRS better understand their target population for Free
File.

PRE-DECISIONAL – FOR INTERNAL IRS W&I USE ONLY – NOT FOR PUBLIC RELEASE
ix

 PRE-DECISIONAL – FOR INTERNAL IRS W&I USE ONLY – NOT FOR PUBLIC RELEASE

What’s been said…

What the data show…
•

Low usage rates signal that the Free
File program is not working (TAS,
ProPublica, members of Congress).

•

•

•
•

•
•

While 103.8 million taxpayers were technically
eligible to use Free File in FS2018, half of those used a
paid preparer, taking them out of the do-it-yourself
(DIY) tax preparation pool in that year.
Another 9.8 million taxpayers in the DIY pool received
refund anticipation checks (RACs) or refund
anticipation loans (RALs), taking them out of the
eligible pool.
Though paper filers are a potential target population
for Free File, the 8.8 million DIY taxpayers who filed
on paper removed themselves from the 2018
candidate pool.
Three million taxpayers went to Volunteer Income
Tax Assistance (VITA) sites, removing them from the
candidate pool.
The actual number of filers using DIY software and
eligible to use Free File in FS 2018 was about 30
million (see Figure ES-2). Of those, about 9 percent
used Free File.
Of those 30 million, a significant percentage (17.7
million for FS 2019 according to FFA) received free
commercial filing from the current members.
The VITA program (the bricks and mortar assisted
free equivalent of Free File) had approximately 3
million taxpayers use their service VITA has
thousands of volunteer preparers, and more than
2,000 partner organizations who conduct outreach
for their 11,000 sites nationwide.

As shown in Figure ES-2, once the number of taxpayers that choose to file via a different
method—e.g., with a paid preparer or through a Volunteer Income Tax Assistance (VITA)—or
elects to receive their tax refund immediately through a refund anticipation loan (RAL) or a
refund anticipation check (RAC)—are removed due to their preferences, the actual pool of
candidates for the Free File program from 2018 is about 30 million taxpayers. A significant
percentage of those receive free filing outside the Free File program. Though the numbers for
free commercial filings are unknown for tax year (TY) 2017, a study by the FFA in 2015 showed
it to be 19 million for that year from the 14 members, and the FFA attested to 17.7 million for
TY 2018 from 11 current members.
PRE-DECISIONAL – FOR INTERNAL IRS W&I USE ONLY – NOT FOR PUBLIC RELEASE
x

 PRE-DECISIONAL – FOR INTERNAL IRS W&I USE ONLY – NOT FOR PUBLIC RELEASE

103,778,000 Eligibile Taxpayers - TY 2017
Potential Add'l Free
Filers, 27,684,000

free commercial
offerings

Paid Preparers
RAC/RAL self-prepared
Paid Preparers,
51,860,000

Free Filers, 2,649,000

Paper filers - self-prepared
VITA/TCE
Free Filers

VITA/TCE, 3,085,000

Potential Add'l Free Filers
Paper filers - selfprepared, 8,750,000
RAC/RAL self-prepared,
9,750,000

Figure ES-2. Break-down of 103,778,000 Taxpayer Eligible for Free File (TY2017).

It is outside the scope of this assessment for MITRE to determine what the proper percentage of
Free File taxpayers should or could be. Without further analysis and research into taxpayer
behavior it is impractical to make that type of determination. Presumably if awareness was at
peak saturation, participation would be greater than it is now. However, there are many
competing free offers in the market that even taxpayers who are aware of Free File may gravitate
to. The IRS’s ability to affect the Free File members’ behavior in terms of their commercial
offerings, or taxpayers’ behavior in terms of how they choose to file their return, is limited. What
the IRS does have control over is what and how much they do to ensure program awareness.

Free File Member Assessment Results – MOU Compliance
Recent attention and criticisms notwithstanding, the Free File program has operated for 17 years
under the governance of an MOU—an agreement that all members must abide by or be removed
from the program. The IRS asked MITRE to conduct a fact-finding review of each member to
determine whether members were in compliance with the most recent version of the MOU at the
time of the assessment.
To assess MOU compliance, MITRE drew findings from IRS interviews and document reviews,
interviews with Alliance leadership and an Alliance-contracted reviewer, and fact-finding
interviews with representatives from each of the Free File member organizations. In addition,
MITRE incorporated the results of the hands-on taxpayer journey reviews of member Free File
landing pages and websites, as well as results from the web forensics analyses where applicable.
Overall, based on the various assessment workstreams, all Free File members were compliant
with the MOU, in particular with Article 4, Standards of Practice.
PRE-DECISIONAL – FOR INTERNAL IRS W&I USE ONLY – NOT FOR PUBLIC RELEASE
xi

 PRE-DECISIONAL – FOR INTERNAL IRS W&I USE ONLY – NOT FOR PUBLIC RELEASE

Article 4 outlines level of service, software and website functionality, security, and disclosure of
forms and schedules. In addition, it includes taxpayer service options and other requirements
such as innovation. MITRE found two instances of MOU non-compliance and an inconclusive
finding for nine requirements for some or all members. There were also two requirements that
were not applicable at the time of review for several members.
The two non-compliance instances were:
• One member was non-compliant with a sentence in MOU 4.15.4. The MOU states that
“Members must clearly list their free customer options … on the Member’s Free File
Landing Page (or such page must have a clear and prominent link to such disclosures
directly from this page).” Based on MITRE’s landing page analysis, the link was not
“clear and prominent” on their landing page11.
• One member was non-compliant with MOU 4.32.4. The MOU requires members to
“communicate not less than once annually via email with their taxpayer customers who
used Free File services and completed their returns through Free File in the immediately
preceding tax year.” The representative from the member acknowledged they were
supposed to initiate an email to their prior tax season Free File taxpayers and self-attested
that they did not provide the required email prior to the tax season.
While MITRE deems these to be non-compliant issues, none rise to the level of severity or
concern regarding the integrity of the Free File program.

Taxpayer Experience Analysis
In addition to assessing compliance with MOU requirements, MITRE also conducted research
activities to understand the taxpayer experience using the Free File program. These research
activities were designed to represent the perspective of the end user of the Free File system—the
taxpayer. As such, MITRE identified pain points and areas for improvement to make the
experience easier, clearer, and more accurate for taxpayers. This included an expert review of
IRS.gov and the 12 member sites for potential usability issues, and usability testing interviews
with actual taxpayers.
Taxpayer Journey Analysis
MITRE conducted a taxpayer journey analysis to evaluate potential usability issues with the Free
File member systems and Free File-related web pages. The taxpayer journey analysis also
evaluated member landing pages for compliance with the MOU, as described above. The rest of
the analysis was focused on the taxpayer experience from a usability perspective and should not
be interpreted as evidence for or against member compliance with the MOU.
The taxpayer journey analysis used both established industry criteria and MOU requirements
related to system usability and user expectations for system functionality as anchor points for the
analysis. Specifically, MITRE used the following sets of criteria to conduct the review of the 12
member sites and IRS.gov’s Free File-related web pages:

11

As of the date of this report, the member has updated their Free File landing page.

PRE-DECISIONAL – FOR INTERNAL IRS W&I USE ONLY – NOT FOR PUBLIC RELEASE
xii

 PRE-DECISIONAL – FOR INTERNAL IRS W&I USE ONLY – NOT FOR PUBLIC RELEASE

• Nielsen’s 10 usability heuristics12. These 10 criteria, called “heuristics,” are widely
recognized measures of system usability. Taken together, the 10 heuristics construct a
measure of user experience and include factors such as “match between system and the
real world,” “error prevention,” “flexibility and ease of use,” and “consistency and
standards.”
• MOU requirements, from a usability perspective (user experience [UX] MOU
Adherence). MITRE examined MOU requirements pertaining to the user experience and
constructed usability criteria from these MOU requirements. These criteria assess user
interface elements closely associated with MOU requirements and included factors such as
“understands offerings available” and “transparency around ineligibility.” MITRE’s
assessment against these criteria are not included in the evaluation of MOU compliance
presented in Section 4.0.
• Mobile experience. MITRE evaluated the extent to which the examined sites were
optimized for mobile device use.
For all three of these groups of criteria, the objective of the evaluation was to identify potential
usability issues and prioritize them via a 1-5 (1 = worst, 5 = best) severity score.
MITRE’s analysis found that, while the Free File members adhere to many UX best practices as
measured by Nielsen’s usability heuristics, they could improve the most in “error prevention”
and “match between system and real world” heuristics. With respect to examining the sites in
terms of a usability interpretation of MOU requirements, the area where members have the most
need for improvement was “transparency around ineligibility.”
All of the scores from the review are rolled up into a dashboard that shows a quick view of the
summary scores for user experience, UX MOU adherence, and mobile experience, with a more
detailed breakout of the scores from the heuristic review by each heuristic and grouping of MOU
requirements (see Section 5.1). Figure ES-3 shows the summary dashboard from the taxpayer
journey heuristic analysis, and aggregates Nielsen’s 10 heuristics under “user experience” issues
related to the usability in the MOU under “UX MOU adherence,” and whether or not the site is
mobile-optimized under “mobile experience.” The 12 member sites are shown in the columns by
number and the heuristic dimensions are displayed down the rows.

12

Nielsen’s 10 Usability Heuristics: Nielsen, Jakob. (1994). Heuristic evaluation. In Nielsen, J., and Mack, R.L. (Eds.), Usability
Inspection Methods, John Wiley & Sons, New York, NY

PRE-DECISIONAL – FOR INTERNAL IRS W&I USE ONLY – NOT FOR PUBLIC RELEASE
xiii

 PRE-DECISIONAL – FOR INTERNAL IRS W&I USE ONLY – NOT FOR PUBLIC RELEASE

Figure ES-3. Heuristic Evaluation Dashboard Summary.

The top two highest-rated members both had exceptional user experience, intuitive mobile
experiences, and followed the MOU’s guidance related to taxpayer experience and usability.
Other members might consider mirroring the transparency Member 10 has around Free File
eligibility, which lets the user know quickly whether or not they are eligible to proceed through
the Free File process.
Two members received the lowest ratings for the usability of their user interfaces. Three
members did not have mobile-optimized sites, which would make it very difficult for users to file
their taxes from a mobile device.
Taxpayer Usability Analysis
MITRE conducted the taxpayer usability study via in-person interviews with 29 taxpayers who
previously used Free File from selected zip codes in Chicago. To select an interview location,
MITRE reviewed the density of TY 2018 Free Filers in multiple clusters of zip codes in U.S.
metropolitan areas; Chicago had a relatively high volume of Free Filers, which allowed MITRE
to send invitation letters to potential participants and elicit a sufficient interview sample. MITRE
randomly selected a subset of Chicago Free Filers to receive invitation letters to participate.
The objectives of these interviews were to understand how taxpayers locate the Free File
program options; understand how taxpayers select a software provider; capture usability issues
with IRS websites related to the Free File program; capture usability issues with specific vendor
platforms; and understand taxpayers’ previous experiences with the Free File program. During
the sessions, participants completed fictitious scenarios to evaluate the experiences of completing
three tasks: 1) locating the Free File program online; 2) selecting a Free File software offer,
including with the software lookup tool; and 3) completing a tax return. MITRE randomly
selected three Free File member software offers to test with taxpayers. Each participant filed a
PRE-DECISIONAL – FOR INTERNAL IRS W&I USE ONLY – NOT FOR PUBLIC RELEASE
xiv

 PRE-DECISIONAL – FOR INTERNAL IRS W&I USE ONLY – NOT FOR PUBLIC RELEASE

fictitious return with one of the three selected offers. Taxpayers used mocked-up information
(e.g., fictitious Form W-2s and other documentation) to complete the tasks.
MITRE identified and reported all usability issues observed by the research team with each of
the three Free File usability tasks and provided suggested solutions for addressing each of these
issues individually. Drawing on the data collected from the usability tasks, analysis of qualitative
data gathered during debriefing and with semi-structured interview questions, the satisfaction
and user experience questionnaires, and metrics on taxpayer behaviors and selections during the
tasks, MITRE identified the several high-level findings. When it comes to navigating to the Free
File program and selecting a member offer, taxpayers:
• Lacked understanding of what the Free File program is and its relationship to the IRS
• Struggled to find the Free File landing page, including through IRS.gov
• Experienced difficulties and confusion with selecting a software offering, particularly
when evaluating Earned Income Tax Credit (EITC) eligibility
• Expressed that they were overwhelmed with choices and information
• Used strategies beyond eligibility criteria to select software offers
With respect to using member-specific Free File program tax preparation software, common
usability issues included trouble finding where to report adjustments to income, difficulty
choosing between filing statuses, and trouble navigating different systems’ “flow.”
Overall, taxpayer ratings of ease and satisfaction varied by task, vendor, and taxpayer
demographic characteristics.

Findings and Recommendations
Building upon the analysis from the various workstreams in this assessment, MITRE provides
the following findings and recommendations for improvements to the Free File program and to
the Free File MOU that governs the program. The findings and recommendations are grouped by
key themes that emerged through the execution of the MITRE assessment. The recommendations
include actions that the IRS may take to improve the integrity and effectiveness of the Free File
program.
Where appropriate, MITRE provides workstream-specific recommendations the body of this
assessment.

PRE-DECISIONAL – FOR INTERNAL IRS W&I USE ONLY – NOT FOR PUBLIC RELEASE
xv

 PRE-DECISIONAL – FOR INTERNAL IRS W&I USE ONLY – NOT FOR PUBLIC RELEASE

1. Objectives of the Free File program have not been updated since the
program’s inception.
Findings

Recommendations

The IRS and FFA have not revisited the
objectives of the program since its
inception. The objectives (reflected in the
MOU) are outdated and don’t reflect
leadership’s current view of program
success.

MITRE recommends that IRS leadership, in
collaboration with the FFA, define new program
objectives and create metrics to determine the
Program Office’s performance against those
objectives. Recognizing that the IRS and FFA
will need to negotiate any formal changes to
joint program objectives, MITRE recommends
The program objective of providing the
that IRS develop internal objectives and metrics
venue for free tax filing for 70 percent of
that reflect its own priorities. While agreement
the population has been met. The e-filing
between the IRS and FFA would be the ideal,
objective has been met. No other metrics
currently exist by which the program office translating priorities into objectives and metrics
should not be contingent on that agreement.
can measure its performance.
For example, according to IRS leadership, its
focus for free file is less about target number of
participants, and more about maximizing
awareness to ensure people recognize it is a
choice.
Potential metrics could include:
•
•
•
•
•

Number of companies participating,
potentially targeting a specific market
share
Actual number of Free File taxpayer
candidate pool/participants (e.g., xx% of
the DIY community minus RALs/RACs)
Metrics associated with compliance (e.g.,
offering parameters, etc.)
Awareness numbers (see additional
recommendations)
Web analysis metrics (e.g., number of
clicks on the Free File link on the
IRS.gov home page, percentage of select
keyword searches that bring taxpayers to
the IRS.gov landing page, etc.)

PRE-DECISIONAL – FOR INTERNAL IRS W&I USE ONLY – NOT FOR PUBLIC RELEASE
xvi

 PRE-DECISIONAL – FOR INTERNAL IRS W&I USE ONLY – NOT FOR PUBLIC RELEASE

2. If Free File objectives are revised to focus on taxpayer awareness, more
needs to be known about the target population and taxpayer behavior.
Findings

Recommendations

The IRS has collected demographics data
on users of Free File and migration data of
those users, but they do not have
demographics data of the potential pool of
Free File users. This data would be helpful
in targeting outreach. (see Section 3.3)

MITRE recommends the IRS conduct data
analysis of the demographics of the population
who are prime candidates for Free File but not
using it.

The total population pool of 103 million
taxpayers who are eligible may not
represent an accurate picture of the pool of
taxpayers who are candidates for Free File.

MITRE recommends the IRS conduct a taxpayer
behavior study to better understand the factors
involved in a taxpayer’s choice of filing
methods. Understanding taxpayer motivation
Taxpayers make choices that take them out and choices could help determine what would be
an acceptable Free File number, if a number is
of the candidate pool. About 9 percent of
needed at all. This aligns with the IRS-related
taxpayers who have not taken themselves
out of the candidate pool by (for example) priorities of the internal MITRE research
using a paid preparer or choosing a RAC or program and will be proposed for a future
research project.
RAL, use Free File (not taking into
consideration those who choose free
commercial offerings) There is no
consensus on what would be an acceptable
number.
While each individual member receives
customer feedback in various ways for
their own Free File product, the IRS has
not conducted a customer survey since
2009.

MITRE recommends the IRS conduct a
customer survey of Free File users specifically to
the IRS experience of Free File (not the software
they filed their return through) to use as a
baseline and create an online survey for users to
capture continuous feedback.

The IRS has no access to data that would
indicate whether a DIY filer paid for a
return or received a free commercial
offering. As the issue is whether a lowincome taxpayer has to pay for filing,
knowing how many taxpayers receive free
filings either through Free File or a
commercial product would give the IRS a
better understanding of their target
markets.

MITRE recommends, that in the spirit of
transparency, the IRS propose the industry to use
an indicator on free commercial filings
Or
MITRE recommends the IRS request that FFA
provide them the aggregate number of taxpayers
who file their taxes using free commercial
products each year. When developing metrics,
this could help with understanding a more

PRE-DECISIONAL – FOR INTERNAL IRS W&I USE ONLY – NOT FOR PUBLIC RELEASE
xvii

 PRE-DECISIONAL – FOR INTERNAL IRS W&I USE ONLY – NOT FOR PUBLIC RELEASE

accurate picture of the population of the
Free File candidate pool.
Leadership acknowledged that the number
of returns submitted through Free File have
not increased in the same way that e-file
numbers have and articulated a goal of
increased awareness. The last awareness
study was conducted in 2011.

MITRE recommends the IRS commission an
awareness study to baseline taxpayer awareness
of Free File, specifically of targeted
demographics determined in the prior
recommendation.

Paid advertising yields results. In MITRE’s
web analysis of those searching for free tax
filing, an estimated 85 percent of the traffic
followed a paid search result to member
sites, while the other 15 percent of visits to
member sites came from an organic search
result. Searchers were more than five times
likely to click on a paid search result.

MITRE recommends the IRS conduct a costbenefit analysis to determine whether to
purchase paid advertising to reach taxpayers
searching for free tax filing. We recognize the
IRS may have to seek statutory authority and
budget-specific funding for such advertising if it
was deemed beneficial.

MITRE further recommends, based on the
results of this study—and other recommended
research—the IRS conduct a targeted awareness
campaign to the desired Free File demographics,
and include other IRS-sponsored venues such as
low-income tax clinics (LITCs) and VITA sites,
including IRS’ own website.

3. IRS Free File program is a long-standing public-private partnership, but
there is a lack of public awareness about what that means to the program.
Findings

Recommendations

Free File program is a partnership between
the IRS and industry that requires
cooperation to achieve success. This
partnership requires a balance that serves
the interests of the government and
taxpayers, but also creates a value to the
for-profit industry that provides the
service.

MITRE recommends that a health assessment of
the PPP become part of the overall metrics for
the Free File program, using a generally
accepted framework provided by external
sources, such as that used by MITRE in this
assessment.

The IRS Free File program could benefit
from providing clarity and scope around
the PPP, both internally and externally.
The purpose and scope of PPPs are not
common, and awareness is key to success.

MITRE further recommends that the IRS
Free File Program Office incorporate talking
points addressing the unique nature of the PPP in
terms of joint governance and the value a PPP
provides to all partners as central themes when
addressing critics or publicly discussing the
program to help address this lack of awareness.

PRE-DECISIONAL – FOR INTERNAL IRS W&I USE ONLY – NOT FOR PUBLIC RELEASE
xviii

 PRE-DECISIONAL – FOR INTERNAL IRS W&I USE ONLY – NOT FOR PUBLIC RELEASE

4. Rules that were put into the MOU to promote fairness among FFA
members and promote choices for taxpayers translate into unclear
presentation of offers and confusion for taxpayers.
Findings

Recommendations

The FFA is obligated to provide Free File
to eligible taxpayers through individual
commercial sites such that, when taken in
aggregate these services are offered to the
lowest 70 percent of the taxpayer
population, calculated using AGI.
Moreover, the MOU requires that each
company must offer services to at least 10
percent of total eligible taxpayers but to no
more than 50 percent of taxpayers (“10/50
rule”)—a requirement set to level the
playing field for smaller providers.

MITRE recommends the IRS conduct an
analysis/study of the 10/50 rule. This could
include a confidential market study of the
members to help understand the economic
incentive to assist in preserving competition
amongst the members and promoting taxpayer
economic welfare; a small pilot to determine the
business impact of removing the rule; a
consideration of whether differing rules for the
two large companies would benefit or create
inequity; and/or an in-depth data analysis of how
the different offerings have impacted members’
free file numbers.

While the 10/50 rule has been (and may
continue to be) beneficial to FFA members
to ensure equity, it’s possible that it has
outlived its usefulness. MITRE did not
have adequate time, nor was it within
scope, to do an analysis of how the 10/50
rule impacts Free Filed returns. However,
from a taxpayer perspective, the different
company offerings create confusion as a
taxpayer navigates the various choices.

Even if the rule is not changed, MITRE
recommends simplifying the display of
members’ offering to reduce taxpayer confusion.

5. Usability activities—both with taxpayers and by usability experts—
provide insight into how taxpayers experience various aspects of the
Free File program.
Findings

Recommendations

MITRE conducted a complete heuristic
assessment of the Free File website and
members websites, and a usability
assessment with Free File users. Both
assessments found that improvements

A complete list of recommendations to improve
usability and the user experience on IRS.gov is
provided in the Taxpayer Experience Results
Report (Appendix G) and taxpayer journey
supplemental materials (Appendix F). While
usability-specific, these recommendations

PRE-DECISIONAL – FOR INTERNAL IRS W&I USE ONLY – NOT FOR PUBLIC RELEASE
xix

 PRE-DECISIONAL – FOR INTERNAL IRS W&I USE ONLY – NOT FOR PUBLIC RELEASE

could be made to enhance the user
experience.

complement the program-level recommendations
above and as such may be repetitive.
MITRE recommends that IRS:
• Standardize how Free File is referenced on
IRS platforms and communications
• Take steps to increase ease of understanding
eligibility requirements
• Ensure that taxpayers understand that
determining eligibility for a software offer is
ultimately their responsibility
• Recommend best practices to increase
usability of members’ tax preparation
software systems
• Conduct research on the usability taxpayer
access and use of the Free File system
• Conduct research on Free File awareness,
attending to different demographic groups’
awareness
• Eliminate or change the EITC eligibility
question in the lookup tool and on the
software offers page
o If the IRS stated on the Free File site that
all taxpayers eligible for EITC are
eligible for Free File, the question could
be removed from the tool. MITRE
recognizes this would be a topic for
negotiation between the IRS and FFA
because of how that could impact some
companies’ appearance in the tool.
o If the IRS chooses to leave EITC in the
lookup, MITRE recommends linking the
question to the EITC eligibility tool, but
ONLY after conducting usability testing
on the EITC eligibility tool and
addressing the results.

PRE-DECISIONAL – FOR INTERNAL IRS W&I USE ONLY – NOT FOR PUBLIC RELEASE
xx

 PRE-DECISIONAL – FOR INTERNAL IRS W&I USE ONLY – NOT FOR PUBLIC RELEASE

6. The MOU is an evolving document that is added to with each new
negotiation, resulting in a piecemeal agreement that is out of date in some
areas.
Findings

Recommendations

The MOU generally has provisions added
with each revision, but few are taken out.
Much of the same language is used, not
only in the objectives, but also in the
technology portion, that has become
outdated (e.g., text CAPTCHA).

MITRE recommends that the IRS and the
Alliance jointly determine new, measurable
objectives for the Free File program, and draft a
new MOU, pulling and using provisions from
the current MOU that are relevant, discarding
provisions that are not, and creating new
provisions based on the current environment.

All but two of the members’ Free File
offerings are the same as their commercial
offerings. The security/privacy rules of
Free File are the same as e-File, and all
Free File members are also members of the
Security Summit where the IRS works
collaboratively with the industry to
improve security.

MITRE recommends simplifying the MOU in its
security and privacy provisions by simply
requiring members to adhere to the eFile/Security Summit requirements and
recommendations.

MITRE’s review of the individual
members’ processes for applying the
Free File indicator (without looking at the
actual coding) indicated most members
used the same process that should place the
indicator only on Free Filed returns.
However, that is not always the case.

MITRE recommends adding language to the
MOU to specify conditions under which the
indicator is applied to ensure quality control.

In MITRE’s data analysis of returns
marked with the Free File indicator,
several thousand returns from one vendor
indicated the presence of a RAC. The
vendor noted that the IRS informed them
of this irregularity, and they discovered a
coding error had mistakenly placed the
indicator on returns that had not been Free
Filed, which was subsequently corrected.
Another vendor also had several hundred
returns with the Free File indicator on
returns with AGI of more than $66,000.

Though not necessarily dependent on the MOU,
MITRE also recommends the IRS develop a
quality control method that would allow the IRS
to validate returns with the Free File indicator
were actually Free Filed.

PRE-DECISIONAL – FOR INTERNAL IRS W&I USE ONLY – NOT FOR PUBLIC RELEASE
xxi

 PRE-DECISIONAL – FOR INTERNAL IRS W&I USE ONLY – NOT FOR PUBLIC RELEASE

7. Search exclusion techniques are the latest new issue for consideration
when entering into the next round of MOU negotiations.
Findings

Recommendations

Five of the companies used a NOINDEX
code to exclude their company’s Free File
landing page from organic searches. They
reported that their belief was that this kept
them compliant with the MOU. However,
while the MOU states the IRS will provide
links to their site, it does not prohibit
taxpayers from accessing their sites
directly. IRS press releases state that
taxpayers can ONLY access Free File sites
through IRS.gov.

MITRE recommends that if the IRS wants
taxpayers to go to Free File sites only through
IRS.gov, the MOU should reflect that
specifically as a provision, as opposed to the
current reference in the preamble.
MITRE further recommends the IRS determine
their position on the use of NOINDEX and add a
provision to the MOU addressing that use.

PRE-DECISIONAL – FOR INTERNAL IRS W&I USE ONLY – NOT FOR PUBLIC RELEASE
xxii