United States Government Accountability Office Report to Congressional Committees February 2020 AVIATION SECURITY TSA Updated Screener Training to Address Risks, but Should Enhance Processes to Monitor Compliance GAO-20-219 February 2020 AVIATION SECURITY TSA Updated Screener Training to Address Risks, but Should Enhance Processes to Monitor Compliance Highlights of GAO-20-219, a report to congressional committees Why GAO Did This Study What GAO Found TSA is responsible for screening millions of airline passengers and their baggage each day at the nation’s commercial airports for items that could threaten aircraft and passengers. In carrying out its mission, TSA requires its screener workforce to complete various trainings on screening procedures and technologies. TSA updated its security screening procedures and technologies in recent years to address risks identified through covert tests in 2015 and reports of emerging threats. Since 2015, the Department of Homeland Security’s (DHS) Transportation Security Administration (TSA) developed and updated screener training to address potential risks to commercial airports identified through covert testing and reports on emerging threats. From May 2015 through June 2019, TSA identified 62 potential risks that warranted review for a potential change in training. TSA made training changes in response to 56 of the identified risks— affecting 40 different training courses. TSA also responded to risks by developing or updating job aids or briefings for screeners. TSA uses established models for developing, updating, and evaluating its screener training. The figure below shows TSA’s process for updating and evaluating its screener training, in accordance with a training development model that is widely accepted and used across the federal government. The TSA Modernization Act of 2018 included a provision for GAO to examine Transportation Security Administration’s (TSA) Screener Training Development Process the effectiveness of TSA’s updated screener training. This report addresses: (1) changes TSA made to screener training since 2015; (2) how TSA updates and evaluates screener training; and (3) how TSA ensures screener compliance with training requirements. GAO analyzed TSA documentation on training development, compliance monitoring, and a non-generalizable sample of six recently updated training courses—selected to reflect a range of training types and topics. GAO also reviewed TSA data on airport screener training compliance rates from fiscal years 2016 through 2018, and interviewed TSA officials. What GAO Recommends GAO is making two recommendations, including that TSA (1) document its process for monitoring screener training compliance and (2) monitor screener compliance data across fiscal years. DHS concurred with the recommendations. TSA relies on an online database to monitor screener compliance in completing required training at the nation’s commercial airports. However, TSA has not documented its process for monitoring screener training compliance, including for analyzing compliance data and reporting and addressing instances of noncompliance at airports. Moreover, while TSA monitors airport compliance rates in a given year, it does not analyze the data across fiscal years for potential trends in noncompliance by individual airports that may warrant corrective action at the headquarters level. GAO found that in fiscal years 2016 and 2017, screeners at 435 commercial airports met TSA’s 90 percent target compliance rate, while in 2018, five airports had compliance rates well below this target, dropping 15 to 26 percentage points from the prior year. TSA officials stated they were unaware of this development. By documenting its screener training compliance monitoring process and monitoring screener training compliance data across fiscal years, TSA would be better positioned to ensure it is aware of potential noncompliance trends warranting corrective action at the headquarters level. View GAO-20-219. For more information, contact William Russell at (202) 512-8777 or russellw@gao.gov. ______________________________________ United States Government Accountability Office Contents Letter 1 Background TSA Revised Screener Training to Address Risks Identified through Covert Tests and Emerging Threats TSA Uses Established Models for Updating and Evaluating TSO Training and Has Followed Leading Practices TSA Monitors Training Compliance, but its Process Does Not Look for Trends Across Fiscal Years and is Not Fully Documented Conclusions Recommendations for Executive Action Agency Comments 5 8 10 13 17 18 18 Appendix I Comments from the Department of Homeland Security 21 Appendix II GAO Contact and Staff Acknowledgments 24 Table 1: Presence of Selected Leading Practice Attributes for Training Development in TSA’s Documented Training Development Processes for Transportation Security Officers 11 Figure 1: Transportation Security Administration (TSA) Training and Development Office’s Process for Developing and Updating Transportation Security Officer (TSO) Training 10 Table Figure Page i GAO-20-219 Abbreviations ADDIE DHS FSD OPM Security Operations TSA TSO T&D Analysis, Design, Development, Implementation, and Evaluation Department of Homeland Security Federal Security Director Office of Personnel Management TSA Security Operations Transportation Security Administration Transportation Security Officer TSA Training and Development This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. Page ii GAO-20-219 Letter 441 G St. N.W. Washington, DC 20548 February 13, 2020 Congressional Committees The screening of airport passengers and their belongings is a critical component in securing our nation’s civil aviation system. Since the terrorist attacks of September 11, 2001, the Transportation Security Administration (TSA) has been responsible for screening airline passengers and their carry-on and checked baggage for prohibited items that could pose a threat to aircraft and passengers. According to TSA, in fiscal year 2018, Transportation Security Officers (TSOs) screened more than 804 million passengers at approximately 440 commercial airports across the United States. 1 TSOs are to follow standard operating procedures that guide their screening processes and utilize technologies, such as advanced imaging technology or walk-through metal detectors to screen passengers, and x-ray and explosives detection technologies to screen their belongings. TSA has a variety of training and development programs to help ensure the TSO workforce has the skills and expertise needed to fulfill security responsibilities. In April and May of 2015, the Department of Homeland Security (DHS) Office of Inspector General conducted covert tests that raised questions regarding the effectiveness of aspects of TSA’s passenger screening process. 2 In response, in June 2015, the secretary 1For purposes of this report, a commercial airport is any airport in the United States that operates pursuant to a TSA-approved security program in accordance with 49 C.F.R. pt. 1542 and at which TSA performs or oversees the performance of screening services. Additionally, for the purposes of this report, the term TSO includes private sector screeners employed by companies contracted with TSA to provide screening services at airports participating in TSA’s Screening Partnership Program. See 49 U.S.C. § 44920. TSA classifies commercial airports in the United States into one of five categories (X, I, II, III, and IV) based on various factors, such as the total number of takeoffs and landings annually and other special security considerations. In general, category X airports have the largest number of passenger boardings and Category IV airports have the smallest. 2Department of Homeland Security Office of Inspector General, Covert Testing of the TSA’s Passenger Screening Technologies and Processes at Airport Security Checkpoints, OIG-15-150 (Sept. 22, 2015). The DHS Office of Inspector General conducted covert testing at airport checkpoints from April through May 2015. These covert tests consisted of carrying metallic (inoperable handguns) and nonmetallic (simulated explosive) test items concealed on the body through TSA passenger security checkpoints and into the sterile areas of the airports in the DHS Office of Inspector General’s sample. Page 1 GAO-20-219 of DHS directed TSA to take a number of actions to address the risks identified in the testing, including further training for all TSOs. The 2018 TSA Modernization Act required that TSA establish a training program for new TSOs located at the TSA Academy. 3 The Act also required TSA to establish, not later than 180 days after enactment, recurrent training that covers updates to TSO screening procedures and technologies, including, in response to weaknesses identified in covert tests at airports; methods to detect false or fraudulent travel documents; and training on emerging threats. 4 Further, the Act states that the recurrent training shall include internal controls for monitoring and documenting compliance of TSOs with the requirements. 5 The TSA Modernization Act includes a provision for us to examine the effectiveness of the new and updated security screening personnel training. 6 This report (1) describes changes TSA has made to its screener training to reflect updates to its screening procedures and technologies since 2015; (2) examines how TSA updates and evaluates its screener training programs, and the extent to which these processes align with leading practices; and (3) assesses how TSA ensures screener compliance with training requirements. To describe changes TSA has made to screener training to reflect updates to its screening procedures and technologies in recent years, we reviewed TSA training program documentation, including its National Training Plan. 7 We also reviewed information on TSA’s efforts to change its TSO training from May 2015 to June 2019 in response to updates to screening procedures and technologies, methods to detect false or fraudulent travel documents, and emerging threats, such as standard operating procedures and training curriculum. We also identified specific TSO training changes TSA’s Training and Development office (T&D) 3The TSA Modernization Act was enacted as part of the Federal Aviation Administration Reauthorization Act of 2018 on October 5, 2018. Pub. L. No. 115-254, div. K, tit. I, 132 Stat. 3186, 3542-3647 (2018); see also 49 U.S.C. §44935(l)(1). 449 5§ U.S.C. § 44935(l)(2)(A). 44935(l)(2)(B). 6Pub. L. No. 115-254, div. K, tit. I, § 1948(b), 132 at 3588. 7TSA’s National Training Plan specifies annual training requirements, including the core curriculum for TSOs, and the classes and hours they must complete for the year. Page 2 GAO-20-219 made from May 2015 through June 2019 in response to DHS Office of Inspector General covert test results and emerging threat information. We selected six of the 40 TSO courses that T&D updated during this time to review specific curriculum changes. We selected the six courses to provide a range of course type and topic. 8 We then compared the course materials from the six courses with the risks identified by covert tests and reports of emerging threats, as well as standard operating procedure documents, to examine how the courses aligned with the risks and documented procedures. Findings from our review of the selected courses cannot be generalized to all TSO courses, but provide illustrative examples of how TSA updated curriculum in response to covert testing results and emerging threat information. We also interviewed T&D officials to obtain insight into the changes TSA made to its TSO training programs. To examine how TSA updates and evaluates its training programs and the extent TSA’s processes for doing so align with leading practices, we reviewed TSA documentation, such as TSA Management Directive 1900.14, Handbook, Training Standards, which outlines the process T&D is to follow when developing and updating its training programs. We then compared TSA’s process against selected leading practices that we have identified in our prior work for training development in the federal government. 9 For our analysis, we selected eight leading practice attributes that we identified in TSA’s Management Directive and our prior work. 10 With respect to TSA’s training evaluation process, we examined TSA’s use of the Kirkpatrick Model—a training evaluation model endorsed 8We selected these courses to reflect a range of criteria, including courses that were recurring; were non-recurring but created in response to a specific event; covered one of the three primary areas of the TSO job (e.g. passenger and carry-on baggage at the checkpoint, and checked baggage), or covered emerging threats. 9GAO. Human Capital: A Guide for Assessing Strategic Training and Development Efforts in the Federal Government, GAO-04-546G (Washington, D.C.: March 2004). In this report, we identified the leading practices through consultations with government officials and experts in the private sector, academia, and nonprofit organizations; examinations of laws and regulations related to training and development in the federal government; and reviewing literature on training and development issues, including previous GAO products on a range of human capital topics. 10Our prior work identified four broad, interrelated components to the training development process: (1) planning/front-end analysis, (2) design/development, (3) implementation, and (4) evaluation. In conducting our work, we selected the leading practice attributes to ensure that we assessed the TSA training and development process across all four components of the training development process. Page 3 GAO-20-219 by the Office of Personnel Management (OPM)—as a means to assess TSO training. To do this, we examined TSA documentation related to the evaluation steps of its curriculum development process, including the applicable evaluations for the six recently updated courses described above. We also interviewed T&D officials about their processes to update, develop, and evaluate training. To assess how TSA ensures TSO compliance with training requirements, we reviewed TSA documentation on its process for monitoring training compliance, such as management directives, and assessed this process against Standards for Internal Control in the Federal Government— specifically those related to monitoring. 11 We also collected and analyzed data on TSO training compliance rates from TSA’s Online Learning Center database for fiscal years 2016 through 2018 (the most recent complete year of data available at the time of our review). We analyzed the data to determine trends in the annual training compliance rates for each of 435 commercial airports that reported training compliance rates during this timeframe. We assessed the reliability of these data by reviewing TSA documentation related to the database, such as the user manual, and interviewing knowledgeable officials about its use and limitations. We also conducted electronic testing to check for missing values and outliers and followed-up with TSA officials on the issues we found. We determined the data to be sufficiently reliable for purposes of determining trends in compliance rates. Additionally, we interviewed TSA headquarters officials from T&D and Security Operations, as well as a non-generalizable sample of seven Federal Security Directors (FSDs) who oversee operations at 47 commercial airports, to understand their roles ensuring training compliance. 12 We conducted this performance audit from May 2019 to February 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our 11GAO, Standards for Internal Control in the Federal Government, GAO-14-704G (Washington, D.C.: September 2014). 12To determine the FSDs to interview, we selected one FSD for each of TSA’s seven regions to account for geographical differences. We also took into account the airport’s size (as defined by TSA’s five categories), and historical trends on training compliance rates based on TSA’s training compliance data to ensure our sample included a range of airport sizes and training compliance rates. Findings from our discussions with FSDs cannot be used to make inferences about the practices of all FSDs, but provide illustrative examples of how FSDs approach their role in monitoring training compliance at the airports they oversee. Page 4 GAO-20-219 findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Background TSOs and Agency Roles for Their Training TSA is the primary federal agency responsible for implementing and overseeing the security of the nation’s civil aviation system and, in general, is responsible for ensuring that all passengers and belongings transported by passenger aircraft to, from, within, or overflying the United States are adequately screened. Over 43,000 TSOs stationed across the nation’s approximately 440 commercial airports are responsible for inspecting individuals and belongings to deter and prevent passengers from bringing prohibited items on board an aircraft or into the airport sterile area. 13 Within TSA, two offices—T&D and Security Operations—are to work together to manage TSOs and ensure their training is current and relevant. T&D is responsible for developing initial and ongoing training curricula for TSOs based in part on TSA’s standard operating procedures that govern how TSOs screen passengers and baggage. 14 Security Operations is responsible for allocating TSO staff to airports, and scheduling TSO work hours and training availability. Within Security Operations, FSDs are responsible for overseeing security operations at the nation’s commercial airports, many overseeing multiple airports within a specific geographic area. FSDs report to one of three executive directors, who in turn are responsible for annually assessing FSD performance, including oversight of TSO training. TSO Training Requirements TSA’s screener training is comprised of a compendium of courses that includes basic training for initial hires, recurrent training, remedial training, 13In general, the sterile area is an area of an airport that provides passengers access to boarding aircraft and to which access is controlled through the screening of persons and property. 14In February 2019, TSA moved responsibility for developing and updating standard operating procedures from Security Operations to its Requirements, Capabilities, and Analysis office. Page 5 GAO-20-219 and return-to-duty training. 15 The National Training Plan specifies annual training requirements and contains the core curriculum for TSOs, including the classes and hours required for TSOs to complete. In accordance with the Aviation and Transportation Security Act, screeners must complete a minimum of 40 hours of classroom instruction and 60 hours of on-the-job training, and must successfully complete an on-thejob training examination. 16 Until 2016, new TSOs completed these training requirements at or near their home airports through the New Hire Training Program. In January 2016, TSA centralized this training under the TSO Basic Training Program at the TSA Academy in Glynco, Georgia. 17 Further, in August 2018, TSA launched the first phases of TSO Career Progression, in which new hire screeners receive local training and gain experience in a limited number of screening functions before advancing to the next stage of training at the TSA Academy, roughly around the four-month mark. In 2015, in response to the DHS Office of Inspector General covert test findings that highlighted areas of concern in the passenger screening process, TSA implemented a TSO re-training effort, beginning with a nationwide training called “Mission Essentials—Threat Mitigation.” According to TSA, this training provided the opportunity for the TSO 15For screeners employed by private contractors under TSA’s Screening Partnership Program, the training requirements are the same as for screeners employed by TSA, including the requirement to attend the TSO Basic Training Program. At airports with private screeners, TSA continues to be responsible for overseeing airport security operations and ensuring the contractors provide effective and efficient screening operations in a manner consistent with law, regulation, and other TSA requirements. The Screening Partnership Program’s contractor responsibilities include recruiting, assessing, and training screening personnel to provide security screening functions in accordance with TSA regulations, policies, and procedures. As of July 2019, 22 commercial airports across the country used the program’s contractors for passenger and baggage screening. 16See Pub. L. No. 107-71, § 111, 115 Stat. 597 (codified at 49 U.S.C. § 44935(g)(2)). Additionally, TSOs may not use any security screening device or equipment in the scope of that individuals’ employment unless the individual has been trained on that device or equipment and has successfully completed a test on the use of the device or equipment. 49 U.S.C. § 44935(g)(3). 17For more information on TSA’s TSO Basic Training Program, see GAO, AVIATION SECURITY: Basic Training Program for Transportation Security Officers Would Benefit from Performance Goals and Measures. GAO-18-552, (Washington, D.C.: July 26, 2018). In this report, among other things, we found that TSA had implemented a training evaluation model but had not yet established specific program goals and performance measures to assess TSO Basic Training. We recommended that TSA establish specific goals and performance measures for the TSO Basic Training program and TSA has done so. Page 6 GAO-20-219 workforce to become familiar with the threat information that underlies TSA’s use of checkpoint technologies and operational procedures to mitigate risks. Federal Training Evaluation Requirements and Training Evaluation Models In 2009, OPM developed and published regulations that require agencies to evaluate training programs annually. 18 According to the regulations, these training evaluations are to help agencies determine how well such plans and programs contribute to mission accomplishment and meet organizational performance goals. One commonly accepted training evaluation model, endorsed by OPM and commonly used in the federal government to evaluate training, is known as the Kirkpatrick model. The Kirkpatrick model consists of a four-level approach for soliciting feedback from training course participants and evaluating the impact the training had on individual development, among other things. 19 The following describes what each level within the Kirkpatrick model is to accomplish: • Level 1: The first level measures the training participants’ reaction to, and satisfaction with, the training program. A level 1 evaluation could take the form of a course survey that a participant fills out immediately after completing the training. • Level 2: The second level measures the extent to which learning has occurred because of the training effort. A level 2 evaluation could take the form of a written exam that a participant takes during the course. • Level 3: The third level measures how training affects changes in behavior on the job. Such an evaluation could take the form of a survey sent to participants several months after they have completed the training to follow up on the impact of the training on the job. • Level 4: The fourth level measures the impact of the training program on the agency’s mission or organizational results. Such an evaluation could take the form of comparing operational data before, and after, a training modification. 18See 5 C.F.R. § 410.202. 19Donald L. Kirkpatrick, Evaluating Training Programs: The Four Levels, third edition (San Francisco, California: Berrett-Koehler Publishers, Inc.) 2012. The fourth level is sometimes split into two levels with the fifth level representing a comparison of costs and benefits quantified in dollars. Page 7 GAO-20-219 TSA Revised Screener Training to Address Risks Identified through Covert Tests and Emerging Threats Since 2015, TSA’s T&D has developed and updated TSO training programs in response to findings from covert tests and reporting on emerging threats that identified risks to aviation security. 20 T&D uses an online database to track results from covert tests and reporting on emerging threats, and any changes to training that T&D makes as a result. According to T&D data from May 2015 through June 2019, T&D officials reviewed 62 risks that warranted a review for a potential change to training, and 56 of the risks led officials to make training changes across its TSO curriculum. 21 Overall, T&D made changes affecting 40 different training courses. Based on our review of TSO training curriculum from May 2015 through June 2019, we found that changes T&D made to its TSO training took many forms. In some cases, T&D changed training to place additional emphasis on a certain aspect of a current standard operating procedure or provide context on the importance of following it. For example, in 2019, T&D updated its instructor-led course—”Mission Essentials: Resolution Tools and Procedures”—to address covert tests where TSOs failed to detect simulated explosive devices hidden in bags or concealed on individuals at checkpoints. The training included a review of methods terrorists may use to plan and carry out attacks in order to emphasize the importance of following the standard operating procedure. The updated training also included leading practices for searching belongings and a discussion of issues that may affect a TSO’s ability to detect threat items hidden in belongings or on individuals. In fiscal year 2019, T&D also updated instructor-led courses on its explosives detection system for checked baggage to respond to covert test findings that TSOs failed to detect certain simulated explosive devices. The updated training included images of simulated explosives hidden in checked bags that replicated scenarios similar to the failed covert tests. In other cases, T&D developed TSO training in response to new or updated standard operating procedures for using technologies. T&D officials said that for this type of TSO training, they wait until TSA’s Requirements, Capabilities, and Analysis office updates or establishes new standard operating procedures for using new technologies and then 20A risk may include a finding that a certain type of simulated explosive device was not detected during a covert test or that an attack in another country may have included methods that could be used against U.S. airports. 21The remaining six risks that did not lead to a change in training were reviewed by T&D officials who concluded that a training response was not needed to address the risks. Page 8 GAO-20-219 develops training based on the revisions. For example, T&D developed TSO training to cover the differences between a prior and updated version of the standard operating procedure for screening passengers and belongings at security checkpoints. T&D included curriculum to cover the major changes in the standard operating procedure and incorporated additional training to address a covert test in which TSOs failed to detect a simulated explosive device at a screening checkpoint. T&D also developed training for TSOs who check passenger IDs and travel documents. The training focused on updates to the standard operating procedure and included procedures specific to the 2005 REAL ID Act, which TSA will fully implement in 2020. 22 Additionally, T&D incorporated this new training to address covert tests that had found issues with identifying false or fraudulent travel documents. In addition to updating or developing new training involving instructor-led courses, TSA responded to identified risks by developing or updating job aids or briefings for TSOs. For example, TSA developed the “It’s Not the Container” briefing in 2017 to address risks highlighted by an attempted attack in Australia and included tactics used to conceal explosives in benign items. 23 The briefing provided best practices for using screening technologies to identify concealed explosives, which aligns with current standard operating procedures. T&D also developed the “Electronics vs. Electrical Devices Job Aid”—covering how TSOs should handle the devices at checkpoints—which instructors circulated during classroom training and provided to TSOs at the screening checkpoints. 22In 2005, after the National Commission on Terrorist Attacks Upon the United States recommended enhanced security for licenses, Congress passed the REAL ID Act of 2005, which, among other provisions, sets minimum national standards for driver’s license security including procedures for states to follow when verifying the identity of license applicants. Pub. L. No. 109-13, div. B, 199 Stat. 231, 302. Title II of the REAL ID Act addresses driver’s license security and is codified at 49 U.S.C. § 30301 note. 23In July 2017, a terrorist group shipped partially-assembled components of a bomb to Australia with plans to detonate the assembled device on a passenger flight. Page 9 GAO-20-219 TSA Uses Established Models for Updating and Evaluating TSO Training and Has Followed Leading Practices TSA uses established models and processes for updating and evaluating TSO training, and these processes follow leading practices for training and evaluation development. TSA updates its trainings using a training development process that can be segmented into five broad, interrelated elements, and is typically referred to as the ADDIE model. The elements include (1) analysis, (2) design, (3) development, (4) implementation, and (5) evaluation. In our prior work, we have found that these five elements of the ADDIE model help to produce a strategic approach to federal agencies’ training and development efforts. 24 See figure 1 for how T&D aligns its training development process with the ADDIE model. Figure 1: Transportation Security Administration (TSA) Training and Development Office’s Process for Developing and Updating Transportation Security Officer (TSO) Training 24GAO-04-546G Page 10 GAO-20-219 T&D’s guidance and our prior work on federal agency training development identify various leading practice attributes for developing training. Such attributes include that the training development process: (1) is formal and based on industry recognized standards; (2) provides the ability to update training based on changing conditions and, if necessary, quickly; (3) includes mechanisms to ensure programs provide training that addresses identified needs; (4) ensures measures of effectiveness are included in training program; (5) prevents duplication of effort and allows for consistent message; (6) allows for stakeholder feedback; (7) provides for continuous evaluation of effort; and (8) includes mechanisms to ensure training programs are evaluated. 25 We found that T&D’s training development process incorporates all of the identified leading practice attributes, as shown in table 1. Table 1: Presence of Selected Leading Practice Attributes for Training Development in TSA’s Documented Training Development Processes for Transportation Security Officers Selected leading practices for training development Standard Development Process Formal process based on industry recognized standard. Ability to update training based on changing conditions and, if necessary, quickly. Mechanisms to ensure programs provide training that addresses identified needs.    Measures of effectiveness are included in training program. Prevention of duplication of effort and allows for consistent message. Process allows for stakeholder feedback. Process for evaluation of effort. Mechanisms to ensure training programs are evaluated.      Source: GAO Analysis of Transportation Security Administration (TSA) information. GAO-20-219 Legend: : Attribute Met X: Attribute Not Met Two examples of TSA’s implementation of selected leading practice attributes are that T&D (1) has methods for updating training quickly, if needed, and (2) has mechanisms to ensure TSO training is evaluated. Specifically: 25For our analysis, we selected leading practice attributes relevant to training development that were identified in TSA’s Management Directive No. 1900.14 Training Standards and our prior work in Human Capital: A Guide for Assessing Strategic Training and Development Efforts in the Federal Government (GAO-04-546G). Page 11 GAO-20-219 Methods to quickly update training. In alignment with the leading practice that agencies should have a process to enable quick updates to training to respond to changing conditions, T&D has alternative processes to develop and deliver training to TSOs faster than the approximately 6 months its standard process takes to develop or revise training. For example, in 2018, T&D formalized a set of alternative processes to rapidly develop and deliver training to TSOs. One such alternative is for T&D to use its Rapid Response process, which allows for a response time to the field of 72 hours. Additional options include the Rapid Update/Revision or Rapid Development (Priority Training) processes to allow for a new training to be issued in approximately 30 days. T&D officials said that the rapid development processes are used when an issue, such as an emerging threat, requires a response in days or weeks. T&D’s guidance outlines situations when these processes are appropriate for use and provides checklists to help T&D personnel follow key steps. Mechanisms to help ensure evaluations of training effectiveness. T&D has mechanisms for ensuring it evaluates the effectiveness of its TSO training programs. In particular, T&D uses the Kirkpatrick model to evaluate its training and, according to its policy, all of its courses are to be evaluated at Level 1 of the model, which measures training participants’ reaction to, and satisfaction with, the training. 26 T&D is also to plan course evaluations for each training during the curriculum development process, determine the formal review cycle, and include it in the curriculum development paperwork. According to its policy, T&D must complete a curriculum review at least once every 5 years, but may do so at shorter intervals. During the curriculum review, T&D examines the training to confirm the content is valid with respect to the applicable listing of tasks and competencies, current law, policy, procedures, and equipment. As a part of this process, T&D assesses participant evaluations to determine whether changes to TSO training are needed. As of October 2019, T&D’s efforts to evaluate new or updated TSO training made from May 2015 through June 2019 are in line with its policy. For example, T&D officials said they updated participant evaluations for TSO training they changed during this time period to address risks identified by covert testing and reports on emerging threats. These 26TSA Management Directive No 1900.14 Training Standards (May 11, 2018) states that all government-funded, resourced TSA training must be evaluated at a Level 1. This policy excludes shift briefings, information circulars, and general information sharing. Additionally, TSA Management Directive 1900.14, Handbook, Training Standards (May 11, 2018) states that Level 1 evaluations are optional for non-recurring training events. Page 12 GAO-20-219 officials told us that they had not yet formally analyzed the results of the evaluations. This progress is in line with T&D policy, which requires a review of each training every 5 years. We verified this by obtaining evaluations T&D collected for the six selected sample courses we reviewed. T&D provided us level 1 survey responses it had collected that measure training participants’ reaction to, and satisfaction with, the training programs for four of the courses. 27 T&D implemented the four courses from calendar years 2015 to 2019. Based on those dates and T&D policy, T&D should complete curriculum reviews for the courses between 2020 and 2024. TSA Monitors Training Compliance, but Its Process Does Not Look for Trends across Fiscal Years and Is Not Fully Documented TSA relies on a database that both field and headquarters staff use to monitor TSO training compliance. According to TSA policy, TSA documents and maintains the training status of all TSOs across approximately 440 commercial airports through its Online Learning Center database. Within the database, TSA records training completion in three ways: 1. TSOs self-certify they completed the training activity, such as reading a policy; 2. A training staff member at a commercial airport will record training completion on behalf of a TSO for instructor-led courses and on-thejob training; 3. The database automatically records completion for training actions, such as online training. After recording training completion, the database calculates the percentage of TSOs at a given airport who are on pace for completing their required annual training. According to TSA guidance, the agency has set its annual TSO target compliance rate at 90 percent per commercial airport. 28 While TSA has guidance outlining roles and responsibilities for training oversight at a high level, TSA headquarters and field officials told us their processes for monitoring training compliance—including analyzing training compliance data, reporting their results, and taking action to 27For the two remaining courses in our sample, a T&D official stated that the courses were briefings and were not required to have evaluations as per T&D policy. 28According to TSA guidance, an airport’s TSO training compliance rate would be calculated by dividing the number of officers who are on pace with the National Training Plan monthly goal by the total number of officers at the airport. Page 13 GAO-20-219 address the results—were not documented. Below are descriptions of these roles and responsibilities at the field and headquarters levels, based on what officials from each level told us. TSA personnel in the field have various responsibilities for overseeing training compliance: • FSDs. FSDs, who oversee operations at one or more airports, have the primary responsibility for ensuring that TSOs within the airports they oversee have fulfilled their training requirements. FSDs are assessed on training compliance among TSOs at their respective airports during their annual performance reviews. All seven FSDs we interviewed said they use the Online Learning Center database to verify that TSOs are on track for meeting their training requirements. Further, these FSDs said they meet regularly with their on-site training staff to discuss how training is going and whether TSOs are at risk of not meeting their training requirements. • Executive Directors. Executive Directors oversee the FSDs who work within their respective portfolios and discuss training compliance with the FSDs during their annual performance review. 29 To monitor FSDs’ efforts, Executive Directors also review data from TSA’s Online Learning Center database on TSO training compliance for airports within their area of responsibility. According to an Executive Director we spoke with, if an Executive Director notices that TSO training compliance rates for an airport whose FSD they oversee are lower than the 90 percent compliance target, he or she may reach out to the FSD to obtain information on the causes and discuss an action plan to improve training compliance. TSA personnel at headquarters also have various responsibilities for overseeing training compliance: • T&D. T&D officials said that on a monthly basis they analyze TSO training compliance data from TSA’s Online Learning Center database to identify how TSOs nationwide are meeting requirements and whether there may be trends that indicate a need for changes to training during the fiscal year. For example, officials told us that in fiscal year 2019 they noticed that airports were generally behind in 29TSA’s three Executive Directors oversee commercial airports based on the airport’s size. One Executive Director oversees large airports (larger category X and I), another medium airports (smaller category X and I), and the other small airports (category II-IV). Page 14 GAO-20-219 meeting annual training requirements and determined this was due to the effects of the federal government shutdown. 30 In response, they stated they adjusted the duration of some training courses to shorten the amount of time it would take TSOs to complete the training within the remainder of the fiscal year. • Security Operations. Security Operations tracks individual airport progress toward meeting TSA’s annual 90 percent compliance target. Security Operations officials said they receive and review monthly training compliance reports from T&D. They are responsible for analyzing the data to monitor whether airports are on pace toward meeting the annual TSO training compliance target. For example, TSA has set the required training completion pace goal at 8.3 percent per month for each commercial airport—-so that by maintaining the pace, by the end of the fiscal year, TSOs at each airport will have completed their required annual training. Officials told us that if they identify instances where an airport’s overall TSO training compliance rate for a given month is below this goal during the course of a fiscal year, they will reach out to the FSD responsible. They will provide the FSD a point of contact at a comparable airport with higher compliance rates to share best practices for addressing the issue. While TSA headquarters officials from Security Operations and T&D are responsible for analyzing and addressing TSO training compliance, they focus on monthly airport progress toward the 90 percent TSO training target, rather than annual changes in compliance rates. In particular, they do not look back at prior year airport compliance data to assess whether airports did not meet the compliance target across fiscal years, and whether they require corrective action at the headquarters level. However, we reviewed annual TSO training compliance data across fiscal years for each of the 435 commercial airports that reported data from fiscal years 2016 through 2018. We found that while all airports met TSA’s 90 percent training compliance target in fiscal years 2016 and 2017, the compliance rates for five airports dropped well below 90 30A federal government shutdown took place from December 22, 2018 to January 25, 2019. Page 15 GAO-20-219 percent in 2018. 31 These five airports’ TSO compliance rates dropped 15 to 26 percentage points from their reported compliance rate in 2017. 32 T&D and Security Operations headquarters officials said they were not aware that five airports had not met TSA’s TSO training compliance target in fiscal year 2018, nor the causes for it. Headquarters officials said that they did not identify this development because their focus is on monthly nationwide trends, rather than instances of noncompliance at individual airports across fiscal years, which field officials would be responsible for addressing. However, unlike headquarters officials, field officials do not have the visibility to identify if or when such noncompliance may be occurring across other commercial airports; and whether it may indicate a broader issue. For example, the five airports whose TSO compliance rates dropped significantly between fiscal years 2017 and 2018 varied by size and location. As a result, FSDs and Executive Directors would generally not have been aware that other airports experienced noncompliance or been in a position to determine whether the noncompliance was due to related reasons. Based on TSA’s process, TSA headquarters officials from T&D and Security Operations are best positioned to identify training compliance trends and their causes when they occur, as they have visibility into training compliance data across the agency in a way that field officials do not. Headquarters officials from T&D and Security Operations told us the fieldlevel processes for overseeing training compliance are not documented because TSA has intentionally given field officials the flexibility to manage TSO workload and training to meet the individual needs of their airports. They said they did not document their processes at the headquarters level because they already understood what to do and were not required to document the analysis results. However, the headquarters officials said there may be a benefit to documenting the headquarters process to ensure consistency in how they carry out the process in the event of attrition. 31According to TSA officials, some airports may not report training compliance data each year because the airport may become de-federalized or the airport may be open seasonally and operated by TSOs that are based from a different airport. 32According to TSA officials, there are a number of factors that may impact their reported training compliance rate, such as having TSOs that are out on extended leave or staffing shortages that prevent TSOs from being able to leave the checkpoints to complete training. Page 16 GAO-20-219 Standards for Internal Control in the Federal Government calls for agencies to develop and maintain documentation of their internal control system. 33 This documentation allows management to retain organizational knowledge and communicate that knowledge to external parties. This documentation of controls is also evidence that controls are identified, can be communicated to those responsible for their performance, and can be monitored and evaluated by the entity. Moreover, internal control standards state that internal control monitoring should generally be designed to ensure that ongoing monitoring occurs in the course of normal operations to ensure that known weaknesses are resolved. By documenting its headquarters process for monitoring TSO training compliance—including its process for analyzing monthly training compliance data, the results of its analyses, and actions taken in response—TSA could better ensure its headquarters staff are aware of their responsibilities for overseeing TSO training compliance and consistently carry these responsibilities out as staff change over time. Additionally, by monitoring for instances of TSO noncompliance at individual airports across fiscal years in its analysis of training compliance data, TSA headquarters would be better positioned to determine whether they constitute a trend warranting corrective action at the headquarters level. Conclusions TSOs’ ability to perform their duties effectively in screening passengers and their belongings is crucial to the security of the nation’s aviation system. While TSA has made updates to its TSO training programs to address risks identified in covert testing, additional actions could improve its processes for monitoring TSO training compliance so that the agency can identify and address any potential training issues. In particular, by documenting its process for monitoring TSO training compliance— including those for analyzing monthly training compliance data, reporting the results of its monitoring efforts, and taking action to address potential issues—TSA could help ensure that all of the various officials responsible for monitoring training compliance, including new staff over time, understand the process and can consistently implement it. Further, by monitoring for instances of airport TSO non-compliance across fiscal years in its analysis of training compliance data, TSA would be better positioned to ensure that it is aware of potential trends so it may 33GAO-14-704G Page 17 GAO-20-219 determine whether corrective action at the headquarters level is warranted. Recommendations for Executive Action We are making the following two recommendations to TSA: The TSA Administrator should direct T&D and Security Operations to document their processes for monitoring TSO training compliance— including those for analyzing training compliance data, reporting the results from their analysis, and actions taken to address the results. (Recommendation 1) The TSA Administrator should direct T&D and Security Operations to monitor for instances of TSO non-compliance by individual commercial airports across fiscal years that could potentially warrant corrective action at the headquarters level. (Recommendation 2) Agency Comments We provided a draft of our report to DHS for review and comment. In its comments, reproduced in appendix I, DHS concurred with both of our recommendations. DHS also provided technical comments, which we incorporated as appropriate. With respect to our first recommendation that TSA document its process for monitoring TSO training compliance, DHS stated that, among other things, Security Operations will collaborate with T&D to develop and maintain an internal control mechanism that will document responsibilities at the field and headquarters level for monitoring TSO training completion compliance, and actions taken to address the results. With respect to our second recommendation that TSA monitor for instances of TSO noncompliance by individual commercial airports across fiscal years, DHS stated that T&D and Security Operations will begin monitoring trends in non-compliance at individual airports and for specific courses. Further, T&D has developed an internal website to share its findings with Security Operations through monthly compliance reports. Page 18 GAO-20-219 We are sending this report to the appropriate congressional committees and to the acting Secretary of Homeland Security. In addition, this report is available at no charge on the GAO website at http://gao.gov. If you or your staff members have any questions about this report, please contact me at (202) 512-8777 or russellw@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix II. W. William Russell Director, Homeland Security and Justice Page 19 GAO-20-219 List of Committees The Honorable Roger Wicker Chairman The Honorable Maria Cantwell Ranking Member Committee on Commerce, Science, & Transportation United States Senate The Honorable Ron Johnson Chairman The Honorable Gary C. Peters Ranking Member Committee on Homeland Security and Governmental Affairs United States Senate The Honorable Bennie Thompson Chairman The Honorable Mike Rogers Ranking Member Committee on Homeland Security House of Representatives Page 20 GAO-20-219 Appendix I: Comments from the Department of Homeland Security Appendix I: Comments from the Department of Homeland Security Page 21 GAO-20-219 Appendix I: Comments from the Department of Homeland Security Again, thank you for the opportunity to review and comment on this draft report. Please feel free to contact me if you have any questions. We look forward to working with you again in the future. Sincerely, Vivi)? H. CIA, CFE Departmental Liaison Of?ce Attachment Page 22 GAO-20-219 Appendix I: Comments from the Department of Homeland Security Attachment: Management Response to Recommendations Contained in GAO-20-219 GAO recommended that the TSA Administrator direct Training and Development and Security Operations (SO) to: Recommendation 1: Document their processes for monitoring TSO training compliance?including those for analyzing training compliance data, reporting the results from their analysis, and actions taken to address the results. ReSponse: Concur. will identify, investigate, and assess trends in training completions by coordinating with SO to jointly determine root causes for training de?ciencies and develop solutions for identi?ed de?ciencies. will record the identi?ed training de?ciencies, root causes, and solutions in a training compliance database, and will provide updates to SO leadership on compliance with training requirements. More speci?cally, will identify courses that have upcoming due dates or are of signi?cant importance due to threat, administration priority, or other need. This will be accomplished through a routine review/spot check of National Training Plan completion rates, looking for anomalies such as inaccurate training records and data mismatches. and SO will also conduct a review of compliance ?ndings and results between Training Operations Division and SO Domestic Aviation Operations. Finally, SO, in collaboration with will develop and maintain a control mechaniSm that will document responsibilities at the ?eld and HQ level for monitoring TSO training completion compliance and actions taken to address the results. Estimated Completion Date (ECD): June 30, 2020. Recommendation 2: Monitor for instances of TSO noncompliance by individual commercial airports across ?scal years that could potentially warrant corrective action at the headquarters level. Response: Concur. Concurrently with the implementation of Recommendation I, will record instances of non-compliance and, in collaboration with SO, begin monitoring trends at individual airports and with speci?c courses. On December 30, 2019, developed an iShare training compliaHCe reporting repository and is testing various data pull/reporting options for incorporating it into an internal standard operating procedure. Once reporting options are incorporated, will share compliance reports with SO. ECD: June 30, 2020. Page 23 GAO-20-219 Appendix II: GAO Contact and Staff Acknowledgments Appendix II: GAO Contact and Staff Acknowledgments GAO Contact William Russell (202) 512-8777 or russellw@gao.gov Staff Acknowledgments In addition to the contact named above, Jason Berman (Assistant Director), Julia Vieweg (Analyst-in-Charge), Benjamin Crossley, Elizabeth Dretsch, Michael Dworman, Eric Hauswirth, Susan Hsu, Tom Lombardi, and Heidi Nielson made key contributions to this report. (103566) Page 24 GAO-20-219 GAO’s Mission The Government Accountability Office, the audit, evaluation, and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO’s commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony The fastest and easiest way to obtain copies of GAO documents at no cost is through our website. Each weekday afternoon, GAO posts on its website newly released reports, testimony, and correspondence. You can also subscribe to GAO’s email updates to receive notification of newly posted products. Order by Phone The price of each GAO publication reflects GAO’s actual cost of production and distribution and depends on the number of pages in the publication and whether the publication is printed in color or black and white. Pricing and ordering information is posted on GAO’s website, https://www.gao.gov/ordering.htm. Place orders by calling (202) 512-6000, toll free (866) 801-7077, or TDD (202) 512-2537. Orders may be paid for using American Express, Discover Card, MasterCard, Visa, check, or money order. Call for additional information. Connect with GAO Connect with GAO on Facebook, Flickr, Twitter, and YouTube. Subscribe to our RSS Feeds or Email Updates. Listen to our Podcasts. Visit GAO on the web at https://www.gao.gov. To Report Fraud, Waste, and Abuse in Federal Programs Contact FraudNet: Website: https://www.gao.gov/fraudnet/fraudnet.htm Automated answering system: (800) 424-5454 or (202) 512-7700 Congressional Relations Orice Williams Brown, Managing Director, WilliamsO@gao.gov, (202) 512-4400, U.S. Government Accountability Office, 441 G Street NW, Room 7125, Washington, DC 20548 Public Affairs Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800 U.S. Government Accountability Office, 441 G Street NW, Room 7149 Washington, DC 20548 Strategic Planning and External Liaison James-Christian Blockwood, Managing Director, spel@gao.gov, (202) 512-4707 U.S. Government Accountability Office, 441 G Street NW, Room 7814, Washington, DC 20548 Please Print on Recycled Paper.