DHS' Pilots for Social Media Screening Need Increased Rigor to Ensure Scalability and Long-term Success (Redacted) February 27, 2017 OIG-17-40 FOR OFFICIAL USE ONLY DHS OIG HIGHLIGHTS DHS’ Pilots for Social Media Screening Need Increased Rigor to Ensure Scalability and Long-term Success February 27, 2017 Why We Did This Inspection Following the December 2015 terrorist attack in San Bernardino, California, Congress raised concerns about the use of social media by terrorist groups and requested that the Department of Homeland Security expand social media background checks. We conducted this inspection to review DHS’ social media pilot programs. What We Recommend What We Found DHS has established a task force for using social media to screen applicants for immigration benefits. In connection with that effort, U.S. Citizenship and Immigration Services (USCIS) began pilots to expand social media screening of immigration applicants. Additionally, Immigration and Customs Enforcement (ICE) independently began a pilot to use social media screening during the visa issuance process. However, these pilots, on which DHS plans to base future department-wide use of social media screening, lack criteria for measuring performance to ensure they meet their objectives. Although the pilots include some objectives, such as determining the effectiveness of an automated search tool and assessing data collection and dissemination procedures, it is not clear DHS is measuring and evaluating the pilots’ results to determine how well they are performing against set criteria. Absent measurement criteria, the pilots may provide limited information for planning and implementing an effective, department-wide future social media screening program. DHS Response I&A, on behalf of itself, USCIS, and ICE, concurred with the recommendation and has begun taking corrective actions. We made one recommendation to help ensure DHS develops an effective social media screening program. For Further Information: Contact our Office of Public Affairs at (202) 254-4100, or email us at DHS-OIG.OfficePublicAffairs@oig.dhs.gov www.oig.dhs.gov FOR OFFICIAL USE ONLY OIG-17-40 FOR OFFICIAL USE ONLY OFFICE OF INSPECTOR GENERAL Department of Homeland Security Washington , DC 20528 / www.oig.dhs .gov February 27 , 2017 MEMORANDUM FOR: David J. Glawe Acting Under Secretary Office of Intelligenoe and Analysis Lori Scia1abba Acting Director U.S. Citizenship and Immigration Services Thomas D. Homan ___ Acting Director U.S. Immigration and Customs Enforcement FROM: John Roth ~k~h_ Inspector General SUBJECT: DHS' Pilots for Social Media Screening Need Increased Rigor to Ensure Scalability and Long-term Success ­ For Officia1. Use Omy For your action is our final report, DHS' Pilots for Social Media Screening Need Increased Rigor to En.sure Scalability and Long-term Success. We incorporated the formal! comments provided by your office. The report contains one recommendation aimed at improving DHS' social media scrieening pilots. The Office of Intelligence and Analysis il~ concurred with tlte recommendation. Based on information provided in the response to the draft :repon, we consider the recommendation open and resolved. Once the recommendation has been fully implemented, plea5e submit a formal doseout letter to us within 30 days so that we may close the recommendation. The memorandum should be accompanied by evidence of completion of agreed­ upon corrective actions. Please send your response or closure request to OIGinspectionsFollov.n..ipg)oig.dhs.gov. -- -· -- ·-· Consistent \vi.th our responsibility under the Inspector General Act. we will provide copies of our report to congressional committees with oversight and appropriation responsibility over the Department of Homeland Security. We will post a redacted version of the report on our website. Please call me with any questions, or your staJf may amt.act Laurel Rim.on, Acting Ac;sistant Inspector General for Inspections and Evaluations or Angela Garvin. Deputy Assistant Inspector General, at f202) 254-4100. www.oig.dhs.gov OIG- 17-40 FOR OFFICIPaL USE ONLY FOR OFFICIAL USE ONLY OFFICE OF INSPECTOR GENERAL Department of Homeland Security Background Following the December 2, 2015 terrorist attack in San Bernardino, California, lawmakers became increasingly concerned about the use of social media by terrorist groups. On December 15, 2015, Senator Jeanne Shaheen and 24 other Senators sent a letter to the Secretary of Homeland Security requesting that, as soon as possible, DHS expand social media background checks, focused on possible connections to terrorist activity, to screening for visa determinations for visitors and immigrants. In response, the then DHS Secretary and Deputy Secretary asked the Under Secretary for Intelligence and Analysis (I&A) to lead a task force to review the Department’s current use of social media and identify options to optimize its use across DHS. The task force comprises senior representatives and staff from DHS, U.S. Citizenship and Immigration Services (USCIS), U.S. Immigration and Customs Enforcement (ICE), U.S. Customs and Border Protection, the Transportation Security Administration, and DHS oversight offices including the Office of General Counsel, Privacy Office, and Office for Civil Rights and Civil Liberties, which have a vested interest in DHS’ use of social media. According to the task force’s implementation plan, the first priority was to conduct a pilot at USCIS to test systematic screening of applicants for immigration benefits. 1 USCIS had used social media in a limited capacity but had no experience using it as a large-scale screening tool. In December 2015, USCIS started screening the social media accounts of a limited number of applying for status, using both manual and automated screening. The task force intended to use the December 2015 pilot and lessons learned from other DHS components’ use of social media to develop policies and processes for standardized use of social media department-wide. Additionally, to expand social media screening across all DHS components, the task force recommended, and the Secretary approved, creation of a DHS Social Media Center of Excellence (COE). 2 (Appendix B contains details about the COE.) In April 2016, USCIS expanded social media screening, testing another automated tool together with manual screening. In 1 USCIS grants immigration benefits to people who meet the eligibility requirements to stay in the United States temporarily or permanently. These benefits include granting U.S. citizenship to eligible individuals, authorizing individuals to reside permanently in the United States, and authorizing aliens to work in the United States. 2 During our review it was called the “Center of Excellence”; however, the name was later changed to “Shared Social Media Screening Service”. We refer to it as the “Center of Excellence” in this report. www.oig.dhs.gov 1 OIG-17-40 FOR OFFICIAL USE ONLY FOR OFFICIAL USE ONLY OFFICE OF INSPECTOR GENERAL Department of Homeland Security August 2016, ICE began using a different automated tool to screen the social media of nonimmigrant visa holders .3 Results of Inspection DHS has established a task force for using social media to screen applicants for immigration benefits. In connection with that effort, USCIS began pilots to expand social media screening of immigration applicants. Additionally, ICE independently began a pilot to use social media screening during the visa issuance process. However, these pilots, on which DHS plans to base future department-wide use of social media screening, lack criteria for measuring performance to ensure they meet their objectives. Although the pilots include some objectives, such as determining the effectiveness of an automated search tool and assessing data collection and dissemination procedures, it is not clear DHS is measuring and evaluating the pilots’ results to determine how well they are performing. Absent measurement criteria, the pilots will be of limited use in planning and implementing an effective, department-wide future social media screening program. The Government Accountability Office (GAO) has issued best practices for an effective pilot phase of a program. According to GAO, “The pilot phase allows for a check on whether program operations occur as expected.” 4 GAO has also stated that pilot programs should have well-defined, clear, and measurable objectives; criteria or standards for determining pilot performance; and a plan to track the pilot’s performance and evaluate the final results. 5 Following these best practices can increase the rigor of the pilots to ensure scalability and longterm success. USCIS started a pilot in December 2015 to screen the social media accounts of and applicants for status. The pilot’s objective was to examine the feasibility of using social media screening with an automated search tool called and determine whether useful information for adjudicating refugee applications could be obtained. 6 Although the pilot had an objective, it did not define what would constitute a successful outcome, nor did it identify metrics against which to benchmark its findings. For example, the pilot discovered that individuals had confirmed social media accounts; likely had a social media account; individuals had unconfirmed American embassies and consulates issue nonimmigrant visas authorizing temporary visits to the United States for individuals who do not intend to become permanent residents. 4 GAO-12-208G, January 2012 5 GAO-09-45, November 2008 6 , a social media analytics tool, covers a large number of social media platforms, has access to third-party information providers, and can access web-based information. www.oig.dhs.gov 2 OIG-17-40 3 FOR OFFICIAL USE ONLY FOR OFFICIAL USE ONLY OFFICE OF INSPECTOR GENERAL Department of Homeland Security accounts; and individuals had no identified social media account. USCIS concluded that the individuals had a minimal presence on social media platforms accessible through . Because the pilot did not have metrics to measure success, it is difficult to conclude whether finding individuals with confirmed social media accounts constitutes success. The purpose of another USCIS pilot, which began in April 2016, was to expand screening to an additional , predominately , applicants through use of the Defense Advanced Research Projects Agency (DARPA) tool. 7 The applicants were asked to voluntarily give their social media user names. 8 USCIS then screened the user names against using the tool; USCIS also manually screened the user names against . USCIS assessed identified accounts to determine whether the refugees were linked to derogatory social media information that could impact their eligibility for immigration benefits or admissibility into the United States. Using the tool and manual screening, USCIS identified individuals with confirmed social media accounts and individuals with unconfirmed accounts. In reviewing the pilot, USCIS concluded that the tool was not a viable option for automated social media screening and that manual review was more effective at identifying accounts. USCIS based its conclusion on the tool’s low “match confidence.” Because the resulting accounts identified by the tool did not always match up with the applicants, officers had to manually check the results. However, USCIS did not establish match benchmarks for the tool, so it does not know what level of match confidence would signify success or failure. In August 2016, ICE independently began a pilot to screen the social media activity of nonimmigrant visa applicants from . The intent of this pilot is to evaluate the capability of to conduct initial screening of social media activity during visa application and continue social media monitoring during the .9 According to the pilot plan, this approach will complement existing background DARPA is a social media screening platform that screens against . National Security Council staff requested USCIS and the Department of State, which also participated, to pilot the elicitation of the applicants’ social media identifiers. USCIS is to access social media platforms by DHS Privacy regulations and using government accounts on social media platforms may introduce additional risk to government systems. DHS and its components have of identifying social media account for individuals that may not be appropriate or viable for large scale screening of applicants for immigration benefits. 9 is a web search tool that specializes in social media data exploitation by analyzing social media data and funneling it into actionable information. www.oig.dhs.gov 3 OIG-17-40 7 8 FOR OFFICIAL USE ONLY FOR OFFICIAL USE ONLY OFFICE OF INSPECTOR GENERAL Department of Homeland Security checks conducted in conjunction with the Department of State and help identify potential derogatory information not found in Government databases. Although ICE has identified several measurable criteria for data that will be collected during the pilot, it has not articulated a plan to use these metrics to evaluate performance and determine overall program effectiveness. In our opinion, a clearly defined performance evaluation is essential to determining whether, how, and when to integrate pilot activities into overall efforts and will enable the Department to implement a more effective social media screening program. DHS and the task force are planning department-wide use of social media for screening based on these pilots. Because components are not measuring their pilots against clear success criteria, DHS may not be able to make informed decisions when it designs its social media screening program. The task force needs to better manage ongoing efforts to pilot social media screening to help ensure an effective and successful social media screening program in the future. Recommendation We recommend that the Under Secretary of Intelligence and Analysis coordinate with USCIS and ICE to develop and implement a plan to evaluate the performance of social media screening pilots that includes features such as well-defined, clear, and measurable objectives and standards for determining pilot performance to help ensure DHS develops an effective social media screening program. Management Comments and OIG Analysis I&A responded on behalf of itself, ICE, and USCIS. The components concurred with our recommendation and provided comments to the draft report. A summary of the management response and our analysis follows. We have included a copy of the management comments in their entirety in appendix A. I&A also provided technical comments to our report, which we incorporated, as appropriate. I&A’s Response to the Recommendation: I&A concurred with the recommendation and stated that DHS has taken steps to improve its social media screening pilots by implementing a four-pronged approach that measures performance to develop consistent benchmarks and continue improving performance to ensure rigor and scalability for long-term success. This approach includes using qualitative and quantitative criteria for measuring tool performance; collecting and analyzing comprehensive www.oig.dhs.gov 4 FOR OFFICIAL USE ONLY OIG-17-40 FOR OFFICIAL USE ONLY OFFICE OF INSPECTOR GENERAL Department of Homeland Security performance metrics of ongoing research and development pilots; reporting project milestones to the task force; and reporting select metrics measuring pilot performance in a weekly task force agenda. Additionally, DHS will continue to identify and refine new and existing metrics to measure social media screening pilots’ performance, to include identifying benchmarks for the various tools tested. OIG Analysis: I&A’s planned actions are partially responsive to this recommendation. This recommendation is resolved, but will remain open until I&A provides documentation of well-defined, clear, and measurable objectives and standards for determining pilot performance. I&A’s response indicated that a new social media screening pilot was scheduled to begin in January 2017. I&A should take this opportunity to develop and implement GAO’s best practices for an effective pilot program to ensure scalability and long-term success of the pilot. I&A also emphasized that, at the launch of the task force, neither the private sector nor the U.S. Government possessed the capabilities for large-scale social media screening. OIG recognizes that DHS has made strides in identifying and assessing social media screening technology and conducting pilots that will enable the development of such capabilities. The Homeland Security Systems Engineering & Development Institute report, Social Media Analytics Capability Testing: Independent Assessment, referenced in I&A’s response, provides evidence of some of the progress that has been made in leveraging technology to meet the Department’s needs. The report was not received by OIG until January 6, 2017, and therefore was not included in our analysis. We note that this independent assessment includes many metrics that may be useful in establishing benchmarks and determining performance of future pilot programs. The assessment included seven steps in conducting evaluations for quantitative analysis: (1) define the purpose of the evaluation; (2) elaborate a task model; (3) define top-level quality characteristics; (4) produce system requirements; (5) define metrics to measure requirements; (6) define techniques to measure metrics; and (7) carry out and interpret the evaluation. This framework was followed for the independent assessment and should be continued with testing pilots. Objective, Scope, and Methodology DHS OIG was established by the Homeland Security Act of 2002 (Public Law 107−296) by amendment to the Inspector General Act of 1978. The initial objective of this inspection was to evaluate DHS’ progress in developing social media screening policies, procedures, and capabilities. www.oig.dhs.gov 5 FOR OFFICIAL USE ONLY OIG-17-40 FOR OFFICIAL USE ONLY OFFICE OF INSPECTOR GENERAL Department of Homeland Security However, upon discovering the lack of performance measures in USCIS’ and ICE’s pilots for social media screening for immigration benefits, we adjusted our scope and reviewed DHS’ social media task force and DHS’ use of pilots to help create a social media screening program. Although we reviewed DHS policies relevant to social media screening, including DHS Instruction Number 110–01-001, Privacy Policy for Operational Use of Social Media, we could not evaluate specific policies and procedures for the pilots because they had not been written. Likewise, we reviewed the task force’s concept of operations for the role of the COE, but could not evaluate the COE at this time because it has not been set up or funded. We interviewed DHS staff from the Office of Policy, USCIS’ Fraud Detection and National Security Directorate, ICE’s Visa Security Program and Homeland Security Investigations, Science and Technology Directorate, Privacy Office, Office for Civil Rights and Civil Liberties, and the Chair of the Social Media task force to determine their roles in social media screening. We conducted this inspection between July and October 2016 pursuant to the Inspector General Act of 1978, as amended, and according to the Quality Standards for Inspection and Evaluation issued by the Council of the Inspectors General on Integrity and Efficiency. The Office of Inspections and Evaluations major contributors to this report are William McCarron, Chief Inspector; Natalie Fussell Enclade, Lead Inspector; Carie Mellies, Lead Inspector; Stephen Farrell, Inspector; Adam Robinson, Inspector; James Alver, Inspector; Kelly Herberger, Communications Analyst; and Elizabeth Kingma, Independent Referencer. www.oig.dhs.gov 6 FOR OFFICIAL USE ONLY OIG-17-40 FOR OFFICIAL USE ONLY OFFICE OF INSPECTOR GENERAL Department of Homeland Security Appendix A I&A’s Comments to the Draft Report www.oig.dhs.gov 7 FOR OFFICIAL USE ONLY OIG-17-40 FOR OFFICIAL USE ONLY OFFICE OF INSPECTOR GENERAL Department of Homeland Security www.oig.dhs.gov 8 FOR OFFICIAL USE ONLY OIG-17-40 FOR OFFICIAL USE ONLY OFFICE OF INSPECTOR GENERAL Department of Homeland Security www.oig.dhs.gov 9 FOR OFFICIAL USE ONLY OIG-17-40 FOR OFFICIAL USE ONLY OFFICE OF INSPECTOR GENERAL Department of Homeland Security www.oig.dhs.gov 10 FOR OFFICIAL USE ONLY OIG-17-40 FOR OFFICIAL USE ONLY OFFICE OF INSPECTOR GENERAL Department of Homeland Security Appendix B Center of Excellence The COE concept of operations envisions the center conducting social media activities on behalf of components that lack resources or expertise and supporting component-level social media screening programs. The COE will also identify new social media tools and opportunities, test and evaluate new and emerging technologies, and identify best practices for the appropriate sharing and collaborative use of social media within DHS and with external partners. The COE will set standards for social media use in relevant DHS operations while ensuring privacy and civil rights and civil liberties protections. The task force also developed a proposal for the creation of a social media board and social media council to replace the existing task force and oversee the future governance of DHS’ social media use and the COE. According to the proposed charter, the social media board will be a senior-level independent group that will ensure awareness of social media use and capabilities among DHS leadership and control DHS’ social media use programs. The social media council will ensure working-level communication for social media use and capabilities, develop new policies and procedures, oversee performance management, and promote information sharing. www.oig.dhs.gov 11 FOR OFFICIAL USE ONLY OIG-17-40 FOR OFFICIAL USE ONLY OFFICE OF INSPECTOR GENERAL Department of Homeland Security Appendix C Report Distribution Department of Homeland Security Secretary Deputy Secretary Chief of Staff Deputy Chiefs of Staff General Counsel Executive Secretary Director, GAO/OIG Liaison Office Assistant Secretary for Office of Policy Assistant Secretary for Office of Public Affairs Assistant Secretary for Office of Legislative Affairs Office of Intelligence and Analysis Audit Liaison USCIS Audit Liaison ICE Audit Liaison S&T Audit Liaison Office of Management and Budget Chief, Homeland Security Branch DHS OIG Budget Examiner Congress Congressional Oversight and Appropriations Committees www.oig.dhs.gov 12 FOR OFFICIAL USE ONLY OIG-17-40 ADDITIONAL INFORMATION AND COPIES To view this and any of our other reports, please visit our website at: www.oig.dhs.gov. For further information or questions, please contact Office of Inspector General Public Affairs at: DHS-OIG.OfficePublicAffairs@oig.dhs.gov. Follow us on Twitter at: @dhsoig. OIG HOTLINE To report fraud, waste, or abuse, visit our website at www.oig.dhs.gov and click on the red "Hotline" tab. If you cannot access our website, call our hotline at (800) 323-8603, fax our hotline at (202) 254-4297, or write to us at: Department of Homeland Security Office of Inspector General, Mail Stop 0305 Attention: Hotline 245 Murray Drive, SW Washington, DC 20528-0305