CLEARVIEW Al Accuracy Test Report OCTOBER 2019 Conducted Independently By: Meade MW ML Hon. Jonathan Nicholas Cassimatls, Aaron M. Renn REPORT SUMMARY In October 2019, the undersigned Panel conducted an independent accuracy test of Clearview AI, a new image-matching technology that functions as an Internet search engine for faces. The test was undertaken in order to measure Clearview’s performance in terms of accuracy across all demographic groups. For the purposes of this analysis, the Panel used the same basic methodology used by the American Civil Liberties Union (ACLU) in its July 2018 accuracy test of Amazon’s Rekognition technology. The ACLU’s approach entailed comparing photographs of all 535 members of the U.S. House of Representatives and Senate against a database of 25,000 arrest photos. The test resulted in 28 members of Congress being incorrectly matched to arrestees from the photo database. It should be noted, however, that the ACLU ran its test using Rekognition’s 80% ‘default’ confidence setting. (The program advises setting the confidence interval at 95% for law enforcement applications.) Even so, the test was highly publicized and might serve to give the general impression that facial recognition technology is inaccurate and/or biased. With those important concerns in mind, the Panel conducted the same test of Clearview. Along with analyzing all 535 members of Congress, the Panel also analyzed all 119 members of the California State Legislature and 180 members of the Texas State Legislature, for good measure. The test compared the headshots from all three legislative bodies against Clearview’s proprietary database of 2.8 billion images (112,000 times the size of the database used by the ACLU). The Panel determined that Clearview rated 100% accurate, producing instant and accurate matches for every one of the 834 federal and state legislators in the test cohort. TEST CONCLUSION: The Independent Review Panel determined that Clearview rated 100% accurate, producing instant and accurate matches for every photo image in the test. Accuracy was consistent across all racial & demographic groups. WHAT IS CLEARVIEW AI? Clearview is a facial-image-matching software system that operates as an Internet search engine for faces. Clearview has indexed the publicly available Internet to create a database of images containing approximately 2.8 billion faces. With a traditional search engine, users search by typing in search terms. With Clearview, users search by uploading an image containing the face to be searched. If Clearview detects a face in the photo, it matches the face against the images in its database, returning any images containing a face that matches. (If the submitted image contains multiple faces, such as in a group photo, Clearview provides the user with a choice of which face to search for.) The matched face is displayed, along with the hyperlink to the website where the image was found. It is important to note that Clearview only matches faces in images. It does not attempt to determine any characteristics of the person such as sex, age, or race. It only searches for images from the Internet in its database with matching faces. ACLU TEST In 2018 the American Civil Liberties Union (ACLU) conducted a well-publicized test of Amazon’s facial recognition software, Rekognition.1 The ACLU used Rekognition to build a database of 25,000 arrest photos. The legal advocacy group then ran a search against that database using pictures of the members of Congress. In the ACLU test, Rekognition incorrectly matched 28 members of Congress, three in the Senate and 25 in the House of Representatives — and “the false matchers were disproportionately people of color.” The ACLU used the default settings in Rekognition for the test. However, that tool’s default confidence interval is only 80%. Amazon recommends setting the confidence interface to 95% for uses involving law enforcement and public safety use. https://www.aclu.org/blog/privacy-technology/surveillance-technologies/amazons-facerecognition-falsely-matched-28 1 CLEARVIEW TEST The Clearview test was designed and conducted along the same lines as the ACLU test in order to evaluate the accuracy of the Clearview system. To make the test more expansive, in addition to the 535 members of Congress, the test also conducted face searches for every member of the legislatures of the two largest states, California and Texas. The final list included a total of 834 legislators: • • • • • • 100 US Senators 435 US House Members 79 California State Assembly Members (one seat was vacant) 40 California State Senators 149 Texas State House Members (one seat was vacant) 31 Texas State Senators Also, instead of searching only 25,000 images, the test searched Clearview’s entire database of 2.8 billion. Unlike Amazon’s Rekognition, Clearview does not allow the user to set the confidence level, but instead is fixed at 99.6%. Publicly available images of the legislators were processed through Clearview automatically using the Clearview Application Programming Interface (API). Use of the API, versus manual entry, ensured reproducibility and limited the possibility of human error. For each individual in the test, the two top-ranked matches returned from Clearview’s 2.8 billion image database were compared with the submitted image. Results were reviewed by the three members of the Panel for their determination as to whether the matches were accurate. The evaluation of the accuracy of each match was determined visually and/or by review of the webpage from which the matched photo was originally taken. In some cases, the originating site is no longer available or no longer contains the image. And in some cases, a cached version of the file was used for comparison. No incorrect matches were found. All returned photos contained the person whose photo was originally submitted. Note: In the case of one member of the Texas State House of Representatives, one member’s photo did return matches that included arrest photos. That is because the individual had, in fact, been arrested. PANEL BIOGRAPHIES The Honorable Jonathan Lippman Judge Lippman served as Chief Judge of the State of New York from 2009 to 2015. During his tenure, Chief Judge Lippman authored landmark decisions addressing constitutional, statutory and common law issues that reshaped the major aspects of of New York law and the contours of NY State government. In the process, he promoted equal access to justice in New York and around the country, and established permanent funding streams for civil legal services. His work included: • Making New York the first state in the nation to require 50 hours of law-related pro bono work prior to bar admission and established the Pro Bono Scholars and Poverty Justice Solutions Programs to help alleviate the crisis in civil legal services • Strengthening the State's indigent criminal defense system • Addressing the systemic causes of wrongful convictions • Creating Human Trafficking Courts across New York State • Reforming New York’s juvenile justice, bail and pre-trial justice systems Judge Lippman received the 2018 William H. Rehnquist Award for Judicial Excellence and the 2016 American Bar Association’s John Marshall Award for judicial excellence, integrity, fairness and professional ethics. The New York Times said Judge Lippman altered the legal profession in New York by using “his authority to promote an ideal of lawyering as a public service.” As Chair of the Independent Commission on New York City Criminal Justice and Incarceration Reform, Judge Lippman drafted the blueprint for the closing of the Rikers Island Jail and the conversion of the City’s troubled jail system to a network of community-based facilities. Dr. Nicholas Cassimatis, Ph.D. Dr. Cassimatis has worked in Artificial Intelligence (AI) his entire career. He is currently the founder of Unitary Labs, a startup that makes software thousands of times faster to build. Previously, he served as the Chief of Samsung's North American AI Research. He was the founder of SkyPhrase, which created a technology that understood more complex natural language with greater precision than had ever before been previously possible.  In 2013, SkyPhrase was acquired by Yahoo, where he was the head of the Deep Natural Language Processing team.  Dr. Cassimatis founded SkyPhrase while he was on the faculty of the Cognitive Science and Computer Science Departments at the Rensselaer Polytechnic Institute. While there, he founded the Human-Level Intelligence Laboratory and led its research into learning, cognitive architectures, reasoning, knowledge representation, and computational linguistics. He was a National Research Council Postdoctoral Associate at the Naval Research Laboratory, where he conducted research in robotics and cognitive architecture. Dr. Cassimatis received his doctorate and undergraduate degrees from the Massachusetts Institute of Technology, and his masters degree from Stanford University. Aaron M. Renn Aaron Renn is a Senior Fellow at the Manhattan Institute, a contributing editor of City Journal, and an economic development columnist for Governing Magazine. He focuses on ways to help America’s cities thrive in an ever-more complex, competitive, globalized, and diverse twenty-first century. During Renn’s 15-year career in management and technology consulting, he served as a partner at Accenture, where he led the development and testing of multiple software systems for major corporations and directed multimillion-dollar, global-technology implementations. He also developed his own online software company ,Telestrian, which provided urban data analytics and mapping. He has contributed to The Guardian, Forbes.com, and numerous other publications. His perspectives on urban issues are regularly cited in the New York Times, Washington Post, Time, The Economist, Daily Telegraph, and other international media. Renn holds a B.S. from Indiana University, where he coauthored an early socialnetworking platform in 1991. He has created several widely used, open-source software packages, including the only program for recovering data from corrupted gzip backups. In 1998, Renn launched one of the nation’s first blogs, the Weekly Breakdown, to cover the Chicago Transit Authority. ###