The AI Now Institute at New York University’s Testimony at the European Parliament 
LIBE Committee Public Hearing on “Artificial Intelligence in Criminal Law and Its Use 
by the Police and Judicial Authorities in Criminal Matters,” February 20, 2020. 
 
Thank you, Mr. Chairman and members of this committee, for the opportunity to speak 
at today’s hearing. My name is Andrea Nill Sánchez, and I am the Executive Director of 
the AI Now Institute at New York University in the United States. AI Now is the first 
university research institute dedicated to studying the social implications of artificial 
intelligence. I am also a lawyer by training who worked on immigration, labor, and 
criminal justice issues for over a decade.  
 
My testimony today will focus on the risks and harms associated with predictive 
policing systems. Despite what the term may suggest, predictive policing is neither 
magic nor a precise science that allows us to see into the future. Instead, predictive 
policing refers to fallible systems that use algorithms to analyze available data and aim 
to produce a forecasted probability of where a crime may occur, who might commit it, 
or who could be a victim.  
 
Left unchecked, the proliferation of predictive policing risks replicating and amplifying 
patterns of corrupt, illegal, and unethical conduct linked to legacies of discrimination 
that plague law enforcement agencies across the globe. 
 
There are three overarching concerns that arise with the use of predictive policing.  
 
First, the culture of secrecy that defines law enforcement and the AI industry alike 
leaves the public in the dark about who is using predictive policing tools, how the 
technology works, and what its effect has been.1 
1

Few if any governments provide the public with sufficient notice or an opportunity to learn about these 
systems and raise concerns before they are deployed. See, e.g., Patrick Williams and Eric Kind, 
“Date-Driven Policing: The Hardwiring of Discriminatory Policing Practices across Europe,” the European 
Network Against Racism (ENAR)​ (​ November 2019): 24, 
https://www.enar-eu.org/IMG/pdf/data-driven-profiling-web-final.pdf​.“There is no central record on the 
number of police forces using predictive policing tools in any country in Europe, and consequently there is 

1 

  
Second, predictive policing primarily relies on inherently subjective police data, which 
reflects police practices and policies—not actual crime rates. Law enforcement 
exercises enormous discretion in how it carries out its work and collects data, including 
the crimes and criminals it overlooks.2 Notably, predictive policing tends to ignore 
white-collar crimes that are comparatively under-investigated—despite a strong 
probability that they are more common.3 Predictive policing also fails to account for the 
absence of crime data; many populations, particularly undocumented immigrants in the 
US, tend to underreport crime.4 
 

little available information about the types of crimes these tools are being applied to, the companies 
behind the tools, or continuous assessment of whether they are effective,” Williams and Kind write. When 
the public has been aware of the use of predictive policing systems, a consequent backlash has led to 
greater internal scrutiny and investigations, resulting in the systems being halted altogether. See Mark 
Puente and Richard Winton, “LAPD’s Data-Driven Culture under Scrutiny amid Scandal over Fake Gang 
Identifications,” ​Los Angeles Times​, January 21, 2020, 
https://www.latimes.com/california/story/2020-01-21/lapd-measured-the-number-of-gang-members-itsmetro-officers-interviewed​; see also Cory Doctorow, “Chicago PD’s Predictive Policing Tool Has Been 
Shut Down after 8 Years of Catastrophically Bad Results,” Boing Boing, January 25, 2020,  
https://boingboing.net/2020/01/25/robo-racism.html​. 
2
See Rashida Richardson, Jason Schultz, and Kate Crawford, “Dirty Data, Bad Predictions: How Civil 
Rights Violations Impact Police Data, Predictive Policing Systems, and Justice,” ​New York University Law 
Review​ 94 (February 13, 2019): 201, 
https://www.nyulawreview.org/wp-content/uploads/2019/04/NYULawReview-94-Richardson-Schultz-Cra
wford.pdf​.  
3
“Dirty Data,” 218–219. Studies estimate that approximately 49 percent of businesses and 25 percent of 
households have been victims of white-collar crimes, compared to a 1.06 percent prevalence rate for 
violent crimes and a 7.37 percent prevalence rate for property crime. For these figures, see Rachel E. 
Morgan and Grace Kena, “Criminal Victimization, 2016: Revised,” US Department of Justice, Bureau of 
Justice Statistics, October 2018, h
​ ttps://www.bjs.gov/content/pub/pdf/cv16re.pdf​; Rodney Huff, 
Christian Desilets, and John Kane, “2010 National Public Survey on White Collar Crime,” , National White 
Collar Crime Center, 2010, 
https://www.nw3c.org/docs/research/2010-national-public-survey-on-white-collar-crime.pdf​; Didier 
Lavion, “Pulling Fraud Out of the Shadows: Global Economic Crime and Fraud Survey 2018,” PwC, 2018, , 
https://www.pwc.com/gx/en/forensics/global-economic-crime-and-fraud-survey-2018.pdf​; and Gerald 
Cliff and April Wall-Parker, “Statistical Analysis of White-Collar 
Crime,” Criminology and Criminal Justice, Oxford Research Encyclopedias (Apr. 2017), 
https://doi.org/10.1093/acrefore/9780190264079.013.267​. 
4
See, e.g., Cora Engelbrecht, “Fewer Immigrants Are Reporting Domestic Abuse. Police Blame Fear of 
Deportation,” N
​ ew York Times​, June 3, 2018, 
https://www.nytimes.com/2018/06/03/us/immigrants-houston-domestic-violence.html​; see also Lindsey 
Bever, “Hispanics ‘Are Going Further into the Shadows’ amid Chilling Immigration Debate, Police Say,” 
Washington Post​, May 12, 2017, 
https://www.washingtonpost.com/news/post-nation/wp/2017/05/12/immigration-debate-might-be-havi
ng-a-chilling-effect-on-crime-reporting-in-hispanic-communities-police-say/​; and see ACLU, “Freezing Out 
Justice: How Immigration Arrests at Courthouses Are Undermining the Justice System,” 2018, 
https://www.aclu.org/issues/immigrants-rights/ice-and-border-patrol-abuses/freezing-out-justice​. 

2

 Finally, there are currently no known methods of mitigating or correcting the biases this 
data introduces into the system’s predictions, so when predictive policing systems rely 
on police data tainted by illegitimate police practices, there is a high risk that the system 
will perpetuate these problems. 
 
In a recent study, my colleagues at the AI Now Institute examined 13 US police 
jurisdictions that had engaged in illegal, corrupt, or biased practices and subsequently 
built or acquired predictive policing systems.5 Specifically, my colleagues found that in 
nine of those jurisdictions, there was a high risk that the system’s predictions reflected 
the biases embedded in the data. 
 
One of the most egregious examples was the Chicago Police Department (CPD), which 
has a notorious, decades-long history of documented corrupt, biased, and abusive 
conduct that has disproportionately affected Black and Latino residents.6 Starting in 
2012, the CPD began using a predictive policing tool to list and rank individuals at risk of 
becoming a victim or offender of a violent crime. 
 
The fact that the list reflected the same demographic targets of unlawful and biased 
police practices was no coincidence.7 Our researchers concluded that the CPD’s 
discriminatory practices generated “dirty data” that the city’s predictive policing system 
directly ingested, creating an unacceptably high risk that the technology was reinforcing 
and amplifying deeply ingrained biases and harms. By relying on such biased policing, 
predictive policing effectively put innocent people who were wrongfully stopped and 
arrested on a Strategic Subject List, thereby reflecting and—when acted 
upon—perpetuating the CPD’s harmful practices. 
 
Following increased public scrutiny8 and community activism, the CPD quietly ended its 
use of the predictive policing tool this past January. It turns out that the tool not only 

5

“Dirty Data,” 197. 
“Dirty Data,” 206–207.  
7
“Dirty Data,” 209. “​The SSL data also revealed that fifty-six percent of Black men under the age of thirty 
in Chicago have a risk score on the SSL, and this is the same demographic that has been 
disproportionately affected by CPD’s unlawful and biased practices identified in the Department of 
Justice and ACLU reports.” 
8
Brianna Posadas, “How Strategic Is Chicago’s “Strategic Subjects List”? Upturn Investigates,” Medium, 
June 22, 2017, 
https://medium.com/equal-future/how-strategic-is-chicagos-strategic-subjects-list-upturn-investigates-9e
5b4b235a7c​. 
6

3

 risked calcifying discriminatory and unlawful police practices; it also failed to reduce 
violence.9  
 
Even police departments that have not been found to engage in discriminatory methods 
may still unwittingly incorporate biased data into their own predictive policing systems 
due to the common practice of sharing data across jurisdictions.10  
 
The AI Now Institute’s researchers found evidence of this risk by examining the 
practices of the the Maricopa County Sheriff’s Office (MCSO), located approximately 
200 kilometers away from the US-Mexico border. The MCSO’s Sheriff Joe Arpaio 
fashioned his department into a “freelance immigration-enforcement agency,”11 but this 
was mostly a pretext to engage in discriminatory law enforcement practices under the 
guise of immigration control—a pretext that is reportedly common in Europe as well.12  
 
According to the US Department of Justice, the MCSO exhibited “a pervasive culture of 
discriminatory bias against Latinos,” including illegal discriminatory stops, retaliation, 
and reduction of policing services to the local Latino community,13 all culminating in a 
federal court holding Sheriff Arpaio in criminal contempt for defying orders to stop 
targeting Latinos.14 
 
Although there was no evidence that the MCSO used predictive policing tools at the 
time of our study, four cities within Maricopa County that shared data with the MCSO 
were either actively using predictive policing or had previously participated in a pilot that 
may have relied on MCSO data.15   
 
9

Jeremy Gorner and Annie Sweeney, “For Years Chicago Police Rated the Risk of Tens of Thousands 
Being Caught Up in Violence. That Controversial Effort Has Quietly Been Ended,” ​Chicago Tribune​, January 
24, 2020, 
https://www.chicagotribune.com/news/criminal-justice/ct-chicago-police-strategic-subject-list-ended-20
200125-spn4kjmrxrh4tmktdjckhtox4i-story.html​. 
10
“Dirty Data,” 225. 
11
William Finnegan, “Sheriff Joe,” N
​ ew Yorker​, July 13, 2009, 
https://www.newyorker.com/magazine/2009/07/20/sheriff-joe​. 
12
See “Data-Driven Policing,” 9; and see Council of Europe: Commissioner for Human Rights, 
“Criminalisation of Migration in Europe: Human Rights Implications,” February 2010, 
https://www.refworld.org/docid/4b6a9fef2.html​. 
13
See Thomas E. Perez, Assistant Attorney General, US Department of Justice (Civil Rights Division) to 
Bill Montgomery, County Attorney, Maricopa County, December 15, 2011, 
https://www.justice.gov/sites/default/files/crt/legacy/2011/12/15/mcso_findletter_12-15-11.pdf. 
14
Richard Pérez-Peña, “Former Arizona Sheriff Joe Arpaio Is Convicted of Criminal Contempt,” N
​ ew York 
Times​, July 31, 2017, 
https://www.nytimes.com/2017/07/31/us/sheriff-joe-arpaio-convicted-arizona.html​. 
15
“Dirty Data,” 216.

4

 Meanwhile, any predictive policing software that relies on data tainted by the targeting 
of immigrant communities will deliver particularly skewed results. It obscures a 
well-documented fact: multiple studies both in the US and Europe have found that 
immigrants as a group are equally or even less likely to commit crimes than their 
native-born counterparts.16  
 
Life and liberty are at stake with predictive policing systems, making government 
oversight and community input essential. As a first step, agencies considering using 
predictive policing tools should undertake Algorithmic Impact Assessments that include 
the following: (1) a self-assessment evaluating the system’s potential impacts on 
fairness, justice, and bias; (2) a meaningful external review process; (3) public notice 
and comment; and (4) enhanced due process mechanisms to challenge unfair, biased, 
or other harmful effects.17 Law enforcement should also conduct a racial-equity impact 
assessment that specifically examines how different racial and ethnic groups will be 
affected, including identifying mitigating solutions.18 
 
But impact assessments, though necessary, are just a stopgap. 
 
Ultimately, predictive policing systems and the data they process are the offspring of an 
unjust world. While the United States’ criminal justice system is a vestige of slavery and 
centuries of racism against Black and Brown people, discriminatory policing is endemic 
across the globe, including in Europe. Civil society groups have repeatedly raised 
concerns about routine ethnic profiling in Europe,19 with certain ethnic groups 

16

See Walter Ewing, Daniel E. Martínez, and Rubén G. Rumbaut, “The Criminalization of Immigration in the 
United States,” American Immigration Council, July 13, 2015, 
https://www.americanimmigrationcouncil.org/research/criminalization-immigration-united-states​; Anna 
Flagg, “Is There a Connection between Undocumented Immigrants and Crime?” Marshall Project, May 13, 
2019, 
https://www.themarshallproject.org/2019/05/13/is-there-a-connection-between-undocumented-immigra
nts-and-crime​; and Brian Bell, Francesco Fasani, and Stephen Machin, “Crime and Immigration: Evidence 
from Large Immigrant Waves,” MIT Press Journals, September 30, 2013, 
https://doi.org/10.1162/REST_a_00337​.  
17
Dillon Reisman et al., “Algorithmic Impact Assessments: A Practical Framework for Public Agency 
Accountability,” AI Now Institute, April 2018, h
​ ttps://ainowinstitute.org/aiareport2018.pdf​. 
18
Rashida Richardson, ed., “Confronting Black Boxes: A Shadow Report of the New York City Automated 
Decision System Task Force,” AI Now Institute (December 4, 2019) 42, 
https://ainowinstitute.org/ads-shadowreport-2019.pdf​. 
19
“Ethnic Profiling in the European Union: Pervasive, Ineffective, and Discriminatory,” Open Society 
Justice Initiative, 2009, 
https://www.justiceinitiative.org/uploads/8cef0d30-2833-40fd-b80b-9efb17c6de41/profiling_20090526.p
df​. 

5

 consistently reporting experiences of overpolicing, according to the European Network 
Against Racism.20  
 
While predictive policing tools can exacerbate structural bias and discrimination, only 
humans can dismantle the systems of oppression that technology reflects and 
empower the communities that have borne the unfair burden of being wrongfully 
suspected, stopped, arrested, and feared. Predictive policing systems will never be safe 
or just until the criminal justice system they’re built on is reformed.  
 
Thank you.  

20

“Data-Driven Policing,” ​https://www.enar-eu.org/IMG/pdf/data-driven-profiling-web-final.pdf​. 

6