November 2019 New York City Automated Decision Systems Task Force Report AUTOMATED 0 DECISION SYSTEMS Understanding this report This report is a product of the New York City Automated Decision Systems (ADS) Task Force. In accordance with the mandates of Local Law 49 of 2018, the Task Force includes representatives from both the City and external organizations, and has prepared this report and recommendations contained herein for the Mayor of New York City and the Speaker of the New York City Council. Inclusion of any material in the report does not represent an endorsement of such material by the City of New York or any individual City agencies. The ADS Task Force’s work and recommendations are advisory in nature and, as set forth in Local Law 49, nothing “shall require [the City or its agencies’] compliance with the task force’s recommendations or disclosure of any information where such disclosure would violate local, state, or federal law, interfere with a law enforcement investigation or operations, compromise public health or safety, or that would result in the disclosure of proprietary information.” Table of Contents 2 Letter from the Chairs 3 Executive Summary 4 Chapter 1: Acknowledgments 6 Chapter 2: Members of the Automated Decision Systems Task Force 12 Chapter 3: Background 15 Chapter 4: Process 17 Chapter 5: Recommendations 26 Chapter 6: Challenges 30 Chapter 7: Unresolved and Related Topics for Consideration 31 Appendix I: Summary of Recommendations 32 Appendix II: Text of Local Law 49 of 2018 NOVEMBER 2019   1   Letter from the Chairs It has been a great honor to serve as chairs of the Automated Decision Systems Task Force. We went into this process expecting to have smart, in-depth conversations about a whole new frontier of government work. We didn’t have models to look to or a guidebook to follow, but we expected the members to raise new questions, delve into tough topics, and approach this issue with creativity, expertise, and rigor. And we couldn’t be more proud of the Task Force for rising to—and exceeding—those expectations. It has been an incredible privilege to sit down with this group of experienced, thoughtful experts and talk about a topic that hasn’t been talked about in government enough yet, and to look at issues new and old through a lens that many New Yorkers have never had to consider: What do the values of equity, transparency, and accountability that are already embedded in our work mean in this context? How do we make sure that the technologies that can help improve the lives of those who rely on local government services are being used in an ethical manner and do not have unintended consequences that are unfair or harmful? There aren’t easy answers to these questions, but in this report you’ll find an immense amount of deliberative thinking that advances the conversation on how government can better protect the people it serves when using these automated tools. We are bringing forward a set of consensus-based recommendations to guide the Administration, which we are proud to say took a great deal of time, effort, energy, and deliberation to develop. An issue this complicated and new deserves nothing less. Jeff Thamkittikasem Chair, ADS Task Force Director, Mayor’s Office of Operations Brittny Saunders Co-Chair, ADS Task Force Deputy Commissioner for Strategic Initiatives, NYC Commission on Human Rights Kelly Jin Co-Chair, ADS Task Force NYC Chief Analytics Officer Director, Mayor’s Office of Data Analytics 2   NEW YORK CITY Automated Decision Systems Task Force Report Executive Summary Automated Decision Systems (ADS) are tools or systems that New York City agencies and offices may use to help them make decisions about services and resources provided to New Yorkers. These tools use data, algorithms, and, in some cases, machine learning or other artificial intelligence (AI) to aid in decision-making. In May of last year, Mayor Bill de Blasio convened the Automated Decision Systems Task Force, as required by Local Law 49 of 2018. Since then, the members and chairs of the Task Force have worked together to develop recommendations that will provide a framework for the use of and policy around ADS. Local Law 49 required that a mayoral Task Force provide recommendations on six different topics related to the use of ADS by City agencies. During the past 18 months, we have worked diligently toward the common goal of providing meaningful recommendations to the Mayor and City Council. In this report, we outline how we met the requirements of Local Law 49 through rigorous research, a months-long public engagement process, and collaborative deliberation. Additionally, our report explains how we plan to continue pushing forward the work of managing ADS in New York City beyond our tenure as a Task Force. This report is the culmination of the ADS Task Force’s work together, in which we provide our recommendations to the Mayor and City Council, and important background about our process, the challenges we faced, and more. The recommendations we present in this document center around three key themes: 1. Building capacity for an equitable, effective, and responsible approach to the City’s ADS. 2. Broadening public discussion on ADS. 3. Formalizing ADS management functions. While our recommendations reflect our agreement about the need for ADS management in New York City, we did not reach consensus on every potentially relevant issue, and there were certain topics beyond the scope of our mandate and timeframe as a Task Force. We outline these topics following our recommendations. It is our hope that this report will help drive meaningful work around the local government’s use of ADS and related tools and systems in the future, and will support the Administration as it delivers services equitably and effectively for millions of New Yorkers. NOVEMBER 2019   3   Chapter 1: Acknowledgments This report would not have been possible without the dedicated work of many people across New York City and beyond the five boroughs. First, the Task Force would like to thank the members of the public who provided their insights throughout this process. Without the written feedback via email or our online form, and the insights gleaned from our public forums and community sessions, the Task Force would not have had the knowledge necessary to provide meaningful recommendations that could positively impact the lives of all New Yorkers. The Task Force would also like to acknowledge the many experts who came forward and provided us with their insights. Understanding how ADS are used to support government decision-making is a complex undertaking, and the insights of policymakers, nonprofit organizations, advocacy groups, community leaders, technology experts, and others are necessary to help create clarity on the best ways to use and develop these tools. In particular, we wish to acknowledge the expert panelists who spoke at our public forums1: †† Aaron Pallas, Professor of Sociology and Education, Teachers College, Columbia University †† Andrew Nicklin, Futurist-At-Large, Johns Hopkins University Center for Government Excellence †† Chancey Fleet, Fellow, Data & Society and Assistive Technology Coordinator, Andrew Heiskell Braille and Talking Book Library, New York Public Library †† Ginger Zielinskie, President & CEO, Benefits Data Trust †† Janai Nelson, Associate Director-Counsel, NAACP Legal Defense and Educational Fund, Inc. †† Natalie Evans Harris, Co-Founder and Head of Strategic Initiatives, BrightHive †† Rumman Chowdhury, Senior Principal, Global Lead of Ethical Artificial Intelligence, Accenture †† Sarah Kaufman, Associate Director, New York University Rudin Center for Transportation The Task Force would also like to thank the many organizations that submitted comments to our public forums: †† AI Now Institute †† Legal Aid Society †† Luminosity †† NAACP Legal Defense and Educational Fund, Inc. †† National Domestic Worker Alliance †† Surveillance Technology Oversight Project 1 Titles for the individuals listed here reflect positions held in April and May 2019, when the public forums were held. 4   NEW YORK CITY Automated Decision Systems Task Force Report Michael Appleton/Mayoral Photography Office, NYC †† Brennan Center for Justice The Task Force would also like to thank the organizations that provided event space or otherwise supported our public engagement sessions: New York Law School, the Queens Public Library, Hostos Community College, the NYC Department of Youth & Community Development, the Atlantic Plaza Towers Tenant Association, and the Arden Heights Boulevard Jewish Center. And we would like to thank representatives of President Barack Obama’s Task Force on 21st Century Policing, who provided us with a model that helped inform the Task Force’s public engagement. We would especially like to thank Mayor Bill de Blasio for convening and appointing our Task Force, and for providing us with the first-in-the-nation opportunity to tackle this challenging and timely topic. We are excited to have New York City lead the way on innovative solutions in public policy. The Task Force would like to thank the New York City Council for introducing and passing Local Law 49 of 2018, which catalyzed this group’s important work. In particular, we would like to thank the Local Law’s original lead sponsor former Council Member James Vacca, Council Member and current Chair of the Committee on Technology Robert Holden, and Council Member and former Chair of the Committee on Technology Peter Koo. As technologies continue to develop and we find new and better ways to provide New Yorkers with the services they need, the continued partnership between the Mayor and the City Council will be important for driving the work forward. The Task Force is grateful to the many City staff at the Mayor’s Office of Operations (Operations), the Mayor’s Office of Data Analytics (MODA), the City Commission on Human Rights (CCHR), and the Mayor’s Office of Information Privacy (MOIP) for their collective advising, research, and administrative support that included coordinating meeting logistics, conducting interviews, and planning our public forums and community sessions. This administrative work allowed the Task Force to set its sights on the bigger picture and develop meaningful recommendations. We would also like to thank the New York City agencies and offices that provided insights or informational briefings, including: †† City Commission on Human Rights †† Department of Education †† Department of Transportation †† Economic Development Corporation †† Fire Department †† Mayor’s Office of Contract Services †† Mayor’s Office of Data Analytics †† Mayor’s Office for Economic Opportunity †† Mayor’s Office of Information Privacy/Chief Privacy Officer †† Mayor’s Office of Operations †† Office of Administrative Trials and Hearings †† Police Department The Task Force would also like to thank former Task Force member Judith Germano and former Task Force Co-Chair Emily W. Newman. Finally, the Chairs would like to thank everyone who participated on the Task Force itself. The challenging work we’ve done together is valuable and will help guide policy surrounding ADS for years to come. NOVEMBER 2019   5   Chapter 2: Members of the Automated Decision Systems Task Force In convening the Task Force, the City recognized the importance of including members with experience working with a range of communities to advance equity, expertise in computer science and technology, and familiarity with creating and implementing public policy. We understood that this diversity of perspectives and expertise would be essential to developing meaningful recommendations that would not be limited to this report, but rather would have the potential for implementation. Ultimately, Task Force members represented a wide array of City agencies and organizations. City agencies directly named to the Task Force were: †† Administration for Children’s Services †† Department of Education †† Department of Social Services (Human Resources Administration/Department of Homeless Services) †† Department of Transportation †† Mayor’s Office of Criminal Justice †† Police Department Members representing the Mayor’s Office of Operations, the Mayor’s Office of Data Analytics, and the City Commission on Human Rights served as chairs. The following individuals served in leadership roles on the Task Force: Jeff Thamkittikasem Director, Mayor’s Office of Operations Jeff Thamkittikasem is the Director of the Mayor’s Office of Operations, an office dedicated to enhancing government efficiency and effectiveness. Previously, Thamkittikasem served for four years as Chief of Staff at the New York City Department of Correction (DOC). During his time at DOC, Thamkittikasem led many progressive reforms, including dramatically reducing the use of punitive segregation and increasing incarcerated individuals’ access to educational and job-preparation programming in New York City jails. Prior to joining the Administration and returning to public service, Thamkittikasem co-founded and served as Managing Partner and Vice President of Sentinel Strategy and Policy Consulting, where he provided strategic and operational guidance to a wide-ranging array of clients and developed a strong cohort of data analysis experts. The firm served public and private-sector clients in the US and abroad. Previously, Thamkittikasem has served as senior advisor and Chief of Staff for US Customs and Border Protection, where he directed various emergency response events. His public service began in New York City, where he worked for the NYC City Council. Thamkittikasem holds a Master’s of Public Administration degree from Columbia University’s School of International and Public Affairs, a Master’s degree in Sociology from Stanford University, and Bachelor’s degrees in Political Science and International Relations from Stanford University. Brittny Saunders Deputy Commissioner for Strategic Initiatives, NYC Commission on Human Rights Brittny Saunders is Deputy Commissioner for Strategic Initiatives at the NYC Commission on Human Rights. At the Commission, Brittny oversees projects related to racial justice, the intersection between the City Human Rights Law and emerging technologies, and other topics. Prior to joining the Commission, Brittny worked in the Office of the Mayor’s Counsel where she focused on issues related to broadband access and human rights. Before joining the Mayor’s Office, Brittny worked for the Center for Popular Democracy, where, as Supervising Attorney for Immigrant Rights and Racial Justice, she successfully pushed for creation of a program to provide free legal representation for detained individuals facing deportation, and the Center for Social Inclusion (now “Race Forward”), where she advocated on broadband access, transportation equity, and disaster and emergency policies from a racial justice perspective. Brittny holds a BA magna cum laude in sociology from Harvard College, a Masters in education from Fordham University, and a JD cum laude from Harvard Law School. 6   NEW YORK CITY Automated Decision Systems Task Force Report Kelly X. Jin NYC Chief Analytics Officer and Director, Mayor’s Office of Data Analytics Kelly X. Jin is the Chief Analytics Officer for the City of New York and Director of the Mayor’s Office of Data Analytics (MODA). She brings over a decade of diverse public service and consulting experience to MODA. She pairs a broad range of analytics experience in both the public and private sectors with a deep understanding of policy and strategy for federal, state, and local governments. Prior to the Chief Analytics Officer role, Kelly served as Director at the Laura and John Arnold Foundation, where she worked with public servants to identify data-driven investments to support effective state and local government policy making and operations priorities. Previously, Kelly worked in the White House Office of Science and Technology Policy as a policy advisor to the US Chief Technology Officer and Chief Data Scientist, advising on strategy for local jurisdictions. She also built and co-led the City of Boston’s analytics team and served as Citywide Analytics Manager. Her earlier career included advisory roles in the US General Services Administration, White House Domestic Policy Council, and as a management consultant. Kelly holds a BA in economics from the University of Pennsylvania. The following individuals were appointed to the Task Force by Mayor Bill de Blasio in May of 2018 and contributed to this report: Solon Barocas Assistant Professor, Cornell University and Principal Researcher, New York City Lab of Microsoft Research Solon Barocas is a Principal Researcher in the New York City Lab of Microsoft Research and an Assistant Professor in the Department of Information Science at Cornell University. He is also a Faculty Associate at the Berkman Klein Center for Internet & Society at Harvard University. His research explores ethical and policy issues in artificial intelligence, particularly fairness in machine learning, methods for bringing accountability to automated decision-making, and the privacy implications of inference. Previously, he was a Postdoctoral Researcher at Microsoft Research and the Center for Information Technology Policy at Princeton University. He is a New York native and earned his doctorate from New York University. Shelby Chestnut National Organizing and Policy Strategist, Transgender Law Center Shelby Chestnut is a National Organizing and Policy Strategist at the Transgender Law Center. Chestnut served as the Director of Community Organizing and Public Advocacy at the New York City Anti-Violence Project (AVP) for five years prior to joining Transgender Law Center. At AVP, Chestnut worked at the city, state, and national level advancing the rights and protections of LGBTQ survivors of violence. For over a decade, Chestnut has been organizing with LGBTQ people, people of color, and low income communities to address violence, promote access to resources, and affect local policy change that is for and by the people most impacted by oppression. Chestnut is a gender non-conforming, two spirit, mixed race organizer who has called Brooklyn their home for the past seven years, but always draws on their Montana roots for country sensibility and dry sense of humor. Khalil A. Cumberbatch Chief Strategist, New Yorkers United for Justice Khalil A. Cumberbatch currently serves as Chief Strategist at New Yorkers United for Justice, a coalition of broad and diverse organizations whose goal is to pass criminal justice reform legislation in New York State. Previously, he served as Associate Vice President of Policy at the Fortune Society, a reentry organization whose goal is to build people and not prisons. He also previously served as Manager of Training at JustLeadershipUSA, a national nonprofit dedicated to cutting the US correctional population in half by year 2030. He is also a lecturer at the Columbia University School of Social Work. In December 2014, after being held for five months in immigration detention, Khalil was one of two recipients to receive an Executive Pardon from NYS Governor Andrew Cuomo to prevent his deportation from the United States. NOVEMBER 2019   7   Howard Friedman, Esq. General Counsel, NYC Department of Education Howard Friedman is the General Counsel of the New York City Department of Education (DOE). As the chief legal advisor for DOE, he oversees the provision of legal services, counsels the Chancellor, and focuses on the development and implementation of new initiatives and the revision of existing education policy. Prior to joining DOE, Friedman served as the Chief of the Contracts and Real Estate Division of the New York City Law Department, where he counseled City agencies and the Mayor’s Office on transactional matters and special projects. Friedman joined the Contracts and Real Estate Division in 1998, became Deputy Chief of the Division in 2004, and was promoted to Chief in 2015. Friedman began working for the city in 1996 with its Loft Board. Prior to joining the New York City Law Department, Friedman worked as an attorney for the Legal Aid Society, where he first worked in the Criminal Appeals Bureau, and then in the Civil Division serving the Harlem neighborhood. He is a 1985 cum laude graduate of Harvard Law School. Daniel Hafetz, Esq. Special Counsel to the First Deputy Commissioner, NYC Department of Social Services Dan Hafetz is Special Counsel to the Department of Social Services’ First Deputy Commissioner. The Department of Social Services encompasses the Human Resources Administration and the Department of Homeless Services. At DSS, Dan assists the DSS First Deputy Commissioner in managing operations, advises on strategic policy and runs special projects. Prior to joining DSS, Dan was Senior Advisor to the General Counsel at the New York City Housing Authority where he led initiatives in criminal justice, reentry, public safety, and eviction, and advised on other matters in law and policy. Dan also served as Counsel to the Committee on Health at the New York City Council where he helped enact landmark legislation related to transgender rights, smoking and e-cigarettes. Dan was a Skadden Fellow in the Community and Economic Development unit of Brooklyn Legal Services Corporation A in East New York. Before entering the law, Dan was a middle and high school social studies teacher in Brooklyn through the New York City Teaching Fellows. Tanya Meisenholder Deputy Commissioner, Equity and Inclusion, New York City Police Department Tanya Meisenholder is the Deputy Commissioner, Equity and Inclusion at the New York City Police Department. In this role, she serves as a senior advisor in the development, implementation, coordination and evaluation of major department policies and procedures, which impact all aspects of Police Department operations. Commissioner Meisenholder holds a Ph.D. in Criminal Justice from the State University of New York at Albany. Afaf Nasher, Esq. Executive Director, Council on American-Islamic Relations – New York Afaf Nasher currently serves as the Executive Director for the New York Chapter of the Council on American Islamic Relations, CAIR-NY. Prior to accepting the role of Executive Director, she served as Board President for the organization. Before shifting her focus to civil rights advocacy, she worked as an associate for the Law Firm of Rossi and Crowley, LLP, performing work in commercial litigation. Nasher continues to serve as a volunteer with several religious and secular organizations in various capacities. Her involvement with CAIR–NY stems from an enthusiasm to challenge discrimination in all its forms, promote positive activism, and foster an understanding of the Muslim American identity. Nasher obtained her Juris Doctor from St. John’s University School of Law and has a Bachelor of Science in Criminal Justice. Michael Replogle Deputy Commissioner for Policy, NYC Department of Transportation Michael Replogle is a globally recognized expert on sustainable transportation who has served since 2015 as Deputy Commissioner for Policy for the New York City Department of Transportation. At the Department he is responsible for strategic planning and guides new and emerging programs. He has helped shape the City’s highly successful Vision Zero traffic safety initiatives, new mobility strategies such as bike-sharing, car-sharing, and for-hire vehicle policy, as well as parking and freight initiatives. He founded and for many years led the Institute for Transportation 8   NEW YORK CITY Automated Decision Systems Task Force Report and Development Policy and the Partnership on Sustainable Low Carbon Transportation, which have both advanced sustainable transportation worldwide. Through those groups, he mobilized a $175 billion 10-year commitment from eight multilateral development banks towards more sustainable transport at the Rio+20 conference, with annual reporting. He spent 17 years as Transportation Director for the Environmental Defense Fund, helping cities reduce car dependence. For a decade he helped guide Montgomery County, Maryland’s growth and comprehensive planning policies. He concurrently co-founded and led the Bikes Not Bombs campaign, which sent 10,000 bikes to teachers and health workers in Nicaragua and capitalized a bicycle assembly industry there. Trained as a civil engineer and sociologist, he brings to the Task Force an interest in algorithmic governance in transportation, particularly focused on improving curb and road management for safety and congestion relief. He holds BSE and MSE degrees in Civil and Urban Engineering and a BA in Sociology, all from the University of Pennsylvania. Jennifer Rodgers, Esq. Former Executive Director, Center for the Advancement of Public Integrity at Columbia Law School and Lecturer-in-Law, Columbia Law School Jennifer Rodgers is a Lecturer-in-Law at Columbia Law School. From 2013-2018, Jennifer served as the Executive Director of the Center for the Advancement of Public Integrity at Columbia Law School (CAPI), which works to improve the capacity of public offices to identify, deter, and combat corruption. Prior to joining CAPI, Jennifer worked for 13 years at the United States Attorney’s Office for the Southern District of New York, where she served in numerous capacities, including as a Deputy Chief Appellate Attorney, the Chief of the Organized Crime Unit, and a Chief of the General Crimes Unit. Jennifer writes extensively on and speaks all over the world about anti-corruption efforts and best practices and criminal justice matters, and chairs the New York City Bar Association’s Government Ethics and State Affairs Committee. Julie Samuels, Esq. Executive Director, Tech:NYC Julie Samuels is the founder and Executive Director of Tech:NYC, an organization representing New York’s fast growing, entrepreneurial tech industry. Before that she was Executive Director at Engine, a nationwide nonprofit focused on technology entrepreneurship and advocacy, where she remains a member of the Board. She previously worked at the Electronic Frontier Foundation, where she was a senior staff attorney and the Mark Cuban Chair to Eliminate Stupid Patents. Before joining EFF, Julie litigated IP and entertainment cases. Prior to becoming a lawyer, Julie spent time as a legislative assistant at the Media Coalition in New York, as an assistant editor at the National Journal in Washington DC, and she worked at the National Center for Supercomputing Applications (NCSA) in Champaign, IL. Julie earned her JD from Vanderbilt University and her BS in Journalism from the University of Illinois at Urbana-Champaign.  Susan Sommer, Esq. General Counsel, Mayor’s Office of Criminal Justice Susan Sommer joined the Mayor’s Office of Criminal Justice (MOCJ) in 2018, where she has led the Mayor’s Task Force on Cannabis Legislation. As the Chief Legal Officer for MOCJ, she oversees the office’s legal work and participates in policy development and implementation of strategies to make the City’s criminal justice system smaller, safer, and fairer. Prior to joining MOCJ, she was the Director of Constitutional Litigation at Lambda Legal Defense and Education Fund, a national civil rights organization advocating for those who are LGBT and living with HIV. At Lambda Legal, she led its efforts to achieve marriage equality and parenting rights for LGBT New Yorkers and was Lambda Legal’s lead counsel in Obergefell v. Hodges, winning the right to marry for same-sex couples nationwide. Sommer supervised attorneys and staff and worked on the full range of the organization’s litigation, public policy, and advocacy issues, including supervision of the Youth in Out-of-Home Care Project and the Transgender Rights Project. Sommer also litigated cases involving the criminal justice system and the civil rights of LGBT police officers. Prior to joining Lambda Legal, she was a partner at Lankler Siffert & Wohl, a New York criminal defense and civil litigation firm, and earlier worked as an associate at Davis Polk & Wardwell. Sommer clerked for Federal District Court Judge William Schwarzer (N.D. Cal.), and received a BA from Yale College and a JD from Yale Law School. NOVEMBER 2019   9   Vincent Southerland, Esq. Executive Director, Center on Race, Equality, and the Law, NYU Law School Vincent M. Southerland joined the Center on Race, Inequality, and the Law at NYU Law School as its inaugural Executive Director in February 2017. He has dedicated his career to advancing racial justice and civil rights. Vincent joined NYU Law after serving as an Assistant Federal Public Defender with the Federal Defenders for the Southern District of New York since 2015. Prior to his time at the Federal Defenders, Vincent spent seven years at the NAACP Legal Defense and Educational Fund, Inc. (LDF), where he was a Senior Counsel. While at LDF, he engaged in litigation and advocacy at the intersection of race and criminal justice, including the successful representation of death-sentenced prisoners across the American South and juveniles sentenced to life imprisonment without parole. He also led LDF’s advocacy efforts around race and policing, and was lead counsel in school desegregation and as employment discrimination matters. Vincent previously served as a staff attorney at The Bronx Defenders and an E. Barrett Prettyman Fellow at Georgetown University Law Center. He began his career as a law clerk to the Honorable Theodore McKee, Judge of the United States Court of Appeals for the Third Circuit, and the Honorable Louis H. Pollak, of the United States District Court for the Eastern District of Pennsylvania. Vincent holds an LLM from Georgetown University Law Center, received his JD from Temple University School of Law and his BA from the University of Connecticut. Julia Stoyanovich Assistant Professor of Computer Science and Engineering, Assistant Professor of Data Science, NYU Julia Stoyanovich is an Assistant Professor of Computer Science and Engineering at the Tandon School of Engineering, and an Assistant Professor of Data Science at the Center for Data Science at New York University. She is a recipient of an NSF CAREER award and of an NSF/CRA CI Fellowship. Julia’s research focuses on responsible data management and analysis practices on operationalizing fairness, diversity, transparency, and data protection in all stages of the data acquisition and processing lifecycle. Prof. Stoyanovich holds MS and Ph.D. degrees in Computer Science from Columbia University, and a BS in Computer Science and in Mathematics and Statistics from the University of Massachusetts Amherst. Andrew White Deputy Commissioner for Policy and Planning, NYC Administration for Children’s Services Andrew White is Deputy Commissioner for Policy, Planning & Measurement at the New York City Administration for Children’s Services (ACS). The division is responsible for bringing knowledge to practice across ACS programs through research and analytics, policy and program development, workforce development, provider agency monitoring and evaluation, and quality management. From 1998 to 2014, White directed the Center for New York City Affairs, an applied policy research center at The New School, where he served on the graduate faculty in urban policy. He is co-founder of the journal Child Welfare Watch, founder of the Center for an Urban Future, and former editor of City Limits, a magazine serving the community development and human services sectors in New York City. Meredith Whittaker Co-Founder and Co-Director, AI Now Institute at NYU; Distinguished Research Scientist at NYU; and Founder of Google’s Open Research Group Meredith Whittaker is a Distinguished Research Scientist at New York University, Co-founder and Co-director of the AI Now Institute, and the founder of Google’s Open Research Group. She has over a decade of experience working in industry, leading product and engineering teams. She co-founded M-Lab, a globally distributed network measurement system that provides the world’s largest source of open data on internet performance. She has also worked extensively on issues of data validation and privacy. She has advised the White House, the FCC, the City of New York, the European Parliament, and many other governments and civil society organizations on artificial intelligence, internet policy, measurement, privacy, and security. She is the co-founder and co-director of the AI Now Institute at NYU, which is a leading university institute dedicated to researching the social implications of artificial intelligence and related technologies. 10   NEW YORK CITY Automated Decision Systems Task Force Report Maya D. Wiley, Esq. Senior Vice President for Social Justice, The New School and Co-Director, Digital Equity Laboratory at The New School Maya Wiley is a nationally renowned expert on racial justice and equity. She has litigated and lobbied the US Congress and developed programs to transform structural racism in the US and in South Africa. Wiley is currently the Senior Vice President for Social Justice at The New School and serves as the Henry Cohen Professor of Public & Urban Policy at The New School’s Milano School of International Affairs, Management & Urban Policy. She is an expert on digital equity and is the founder and co-director of The New School’s Digital Equity Laboratory. She is also the Co-Chair of the New York City Department of Education’s School Diversity Working Group, formulating recommendations on school desegregation. She is the former Chair of the New York City Civilian Complaint Review Board (CCRB), the independent oversight agency on police misconduct by officers in the New York Police Department and formerly served as Counsel to the Mayor of the City of New York from 2014-2016. As the Mayor’s chief legal advisor and a member of his Senior Cabinet, Wiley was placed at the helm of the Mayor’s commitment to expanding affordable broadband access across New York City, advancing civil and human rights and gender equity, and increasing the effectiveness of the City’s support for Minority/Women Owned Business Enterprises (M/WBEs). During her tenure, she also served as the Mayor’s liaison to the Mayor’s Advisory Committee on the Judiciary. Before her position with the de Blasio Administration, Wiley was the Founder and President of the Center for Social Inclusion. She has also worked for the Open Society Foundation in the US and in South Africa, the NAACP Legal Defense & Educational Fund, Inc., the American Civil Liberties Union and US Attorney’s Office for the Southern District of New York. Wiley appears regularly on MSNBC and has written numerous opinion editorials for major news outlets, including The Guardian, Time magazine, Essence.com, Fast Company, and the New York Daily News. In 2016, Good Housekeeping magazine honored Wiley as one of its 50 Over 50. City and State magazine named Wiley one of the 100 most powerful people in New York City in 2014 and in 2015. In 2011, Wiley was named one of “20 Leading Black Women Social Activists Advocating Change” by TheRoot.com and a “Moves Power Woman” in 2009 by the same magazine. Wiley holds a JD from Columbia University School of Law and a BA in psychology from Dartmouth College. She resides in Brooklyn with her two daughters and her partner. Jeannette M. Wing Avanessians Director, Data Science Institute and Professor of Computer Science, Columbia University Jeannette M. Wing is the Avanessians Director of the Data Science Institute and Professor of Computer Science at Columbia University. From 2013 to 2017, she was a Corporate Vice President of Microsoft Research. She is an Adjunct Professor of Computer Science at Carnegie Mellon where she twice served as the Head of the Computer Science Department and had been on the faculty since 1985. From 2007–2010, she was the Assistant Director of the Computer and Information Science and Engineering Directorate at the National Science Foundation. She received her SB, SM, and PhD degrees in Computer Science, all from the Massachusetts Institute of Technology. She is a Fellow of the American Academy of Arts and Sciences, American Association for the Advancement of Science, the Association for Computing Machinery (ACM), and the Institute of Electrical and Electronic Engineers (IEEE). She received the CRA Distinguished Service Award in 2011 and the ACM Distinguished Service Award in 2014. Her current research interests are in trustworthy AI and privacy-preserving technologies. NOVEMBER 2019   11   Chapter 3: Background The City of New York serves a population of more than 8.5 million people across five boroughs. The City’s numerous agencies work to deliver an enormous range of services to all New Yorkers. Advances in technology, when leveraged to help process and analyze data, can bring tremendous potential to cities like New York in developing equitable and efficient approaches to delivering these services. Every day, New York City agencies and offices regularly use technology in the course of their business operations to accurately, quickly, and effectively deliver hundreds of services and perform innumerable functions that support and help New Yorkers and the City’s many visitors. Every agency has a unique mission, set of responsibilities, and problems to solve, and each of them uses data and technology tools and systems to carry out the day-to-day work of serving New Yorkers. Many of these technological solutions play an active role in helping City employees make decisions. For example, any given City agency will have many different types of databases, and many City employees use software programs such as Microsoft Excel on a daily basis to record, analyze, or organize data. Some of those solutions may be what many people consider to be “Automated Decision Systems,” or “ADS.” For this report, we will rely on the definition of an ADS that was provided by Local Law 49 of 2018 (LL49): The term “automated decision system” means computerized implementations of algorithms, including those derived from machine learning or other data processing or artificial intelligence techniques, which are used to make or assist in making decisions.2 The use of ADS is increasing both within New York City government and in municipalities across the United States, as cities continue to see value in delivering services more quickly and effectively to residents who depend on them, streamlining decision-making processes, expanding their abilities to help their residents, and attempting to identify and remove any human bias from their work. We know that certain decisions, whether made by a human or a computer, carry with them a risk of certain implicit or explicit biases. Furthermore, new technologies and systems—while providing new efficiencies—might introduce issues that may take some time for organizations to understand and correct. It is important to understand how the City uses ADS so that it can better understand these issues and risks, and use ADS most effectively and responsibly to improve service delivery and the ways that constituents interact with local government. As a provider—and in many cases the exclusive provider—of essential services, the City of New York has a unique responsibility to find solutions to new problems while continuing to provide the services upon which New Yorkers depend. And given the critically important services that the City provides and the diverse populations and communities to which it delivers them, the City must consider equity concerns in the decisions it makes. This is not only the case when weighing the use of new technical tools and systems, but also if or when relying on manual procedures or outdated legacy systems that may impair the effective and efficient delivery of services. LL49 requires that the Mayor establish and convene a task force to examine how the City uses ADS in decisionmaking, and provide specific recommendations related to the following: †† Criteria for identifying which agency automated decision systems should be subject to one or more of the procedures recommended by such task force…; †† Development and implementation of a procedure through which a person affected by a decision concerning a rule, policy or action implemented by the City, where such decision was made by or with the assistance of an agency automated decision system, may request and receive an explanation of such decision and the basis therefor; †† Development and implementation of a procedure that may be used by the City to determine whether an agency automated decision system disproportionately impacts persons based upon age, race, creed, color, religion, national origin, gender, disability, marital status, partnership status, caregiver status, sexual orientation, alienage or citizenship status; 2 Local Law 49 of 2018. https://legistar.council.nyc.gov/LegislationDetail.aspx?ID=3137815&GUID=437A6A6D-62E1-47E2-9C42461253F9C6D0. 12   NEW YORK CITY Automated Decision Systems Task Force Report †† Development and implementation of a procedure for addressing instances in which a person is harmed by an agency automated decision system if any such system is found to disproportionately impact persons based upon a category described in [the above paragraph]; †† Development and implementation of a process for making information publicly available that, for each agency automated decision system, will allow the public to meaningfully assess how such system functions and is used by the City, including making technical information about such system publicly available where appropriate; and, †† The feasibility of the development and implementation of a procedure for archiving agency automated decision systems, data used to determine predictive relationships among data for such systems and input data for such systems, provided that this need not include agency automated decision systems that ceased being used by the City before the effective date of this local law. We interpret these requirements to address how the City uses ADS, how ADS are managed, how information about them is retained, and what happens when the public asks about, or has a concern about, a specific ADS. To comply with these mandates, the Mayor established the ADS Task Force in May 2018. Since then, we have met regularly to discuss these topics and develop recommendations. The Task Force is composed of representatives with relevant expertise both from City agencies and organizations external to the City. A list of Task Force members is included in Chapter 2. As the use of ADS to support government decision-making is continuously evolving, our conversations have been rich and forward-looking. Having experts who bring perspectives from outside the City in addition to subject matter expertise from within City government has been critical to ensuring that we were able to consider the opportunities and risks associated with using ADS in government, as well as existing and potential City governance structures and practices to effectively and responsibly manage ADS. Ed Reed/Mayoral Photography Office, NYC The Task Force also recognizes that the recommendations put forth in this document may be of interest to other municipalities that are also grappling with how to manage the use of ADS, to ensure they are used fairly and responsibly, and in a way that best serves all residents. As such, while these recommendations were designed specifically for New York City, it is the Task Force’s hope that the values and strategies underlying them could help inform other US cities and beyond. NOVEMBER 2019   13   How New York City Works Any government action takes place within a framework of federal, state, and local laws, institutions with the authority to perform oversight, and the many policies and procedures that fulfill both legal mandates and the priorities of government officials. For these reasons, addressing the requirements of LL49 requires a broad consideration of existing laws and bodies. This report does not seek to fully document the complexities—and certainly not the scope and scale—of our local government structure and its operations, but there are a few concepts that were important to consider as we fulfilled the charge of LL49. †† Agencies make countless decisions daily, from simple decisions about which supplies to purchase, to policy decisions with citywide impact. Many decisions they make are subject to various types of internal review. Before a decision is made, it may undergo multiple layers of approvals. But as we were considering what tools or systems may fit the criteria to be considered an ADS, we found that it is not always clear what constitutes a “decision.” Moreover, there may be sequences of decisions associated with the actions that an ADS supports. There is not a single answer to these questions, which means that our recommendations needed to account for the complexity of agency decisionmaking processes. †† Agencies already have existing pathways for interaction with members of the public. These pathways serve diverse purposes, function with different levels of formality, and allow for varying degrees of engagement. Actions taken by the City that have direct impact on individuals—such as issuing a ticket or a summons— generally have formalized processes to find out more information and for appeal, such as a hearing at the Office of Administrative Trials and Hearings (OATH) for a traffic violation or an “Article 78” court proceeding to further challenge an agency decision. Agencies must interpret and reconcile often complex legal requirements—and sometimes competing policy objectives—in determining when and how to use ADS and particularly how to avoid potential unintended harm to any individual or protected group. We must consider the effects of these existing frameworks when considering the question of how the public should request information about ADS, or report or pursue a remedy for an instance of harm. †† Agencies have certain processes and control systems in place to help ensure that they are making decisions that best serve the public interest, even if these existing frameworks were not originally created to manage ADS specifically. Similarly, agencies already implement various functions and procedures to manage the development, implementation, and usage of information technology tools, including consulting with a number of existing City entities and officials, and obtaining any necessary approvals. These systems exist to ensure agencies make decisions fairly, equitably, and responsibly, while also recognizing the principle of government discretion to act in the public interest. In looking at developing guidelines specifically related to ADS, we had to consider existing agency processes and protocols to ensure coordination and avoid duplication of effort. 14   NEW YORK CITY Automated Decision Systems Task Force Report Chapter 4: Process LL49 posed a unique opportunity to the Task Force it created. The questions presented by the Law required a novel approach to viewing agency practices because they came at a time when the field of ADS in both the public and private domains, with the attendant management responsibilities, was experiencing continuous evolution and innovation. This required us to put into place a rigorous process for managing the Task Force, with resources dedicated to research and ample time spent debating fundamental principles of good governance. Even before the official launch of the Task Force itself, the opportunity to catalyze a public conversation on ADS through implementation of LL49 was clear. Early on, Task Force members recognized that insights from members of the public impacted by ADS would be essential to inform deliberations, and thus proceeded to develop and implement a robust public engagement plan with this in mind. Our process was also iterative, shifting in the face of diverse challenges, which we will detail in a later section of this report. Throughout our time together, we leaned on different sets of resources, including external support and varying approaches to organizing our deliberative materials. Task Force Meetings The Task Force held over a dozen formal meetings for all members. At these key meetings, members discussed and provided direction for the Task Force’s programmatic and research agenda, deliberated over recommendations submitted to the Task Force, and determined their final recommendations. Outside of these formal meetings, we held several dozen working meetings on related topics, including, but not limited to, briefings from select City agencies regarding their operations and practices, planning sessions for the Task Force’s public forums and community sessions, and informational interviews with individual members. AT TA N NH The Queens Public Library at Flushing MA Because public engagement was of utmost importance to our process, the Task Force also held seven public meetings. In conversation with former officials with the US Department of Justice’s Office of Community Oriented Policing Services, the Task Force decided to model its public engagement on President Obama’s Task Force on 21st Century Policing. After a series of planning New York sessions within our Task Force, we held two public Law School forums, with expert panelists selected by Task Force members and an opportunity for members of the public to submit comments or speak directly to the Task Force. BRONX Hostos Community College QUEENS Department of Youth & Community Development’s Democracy Corp Day of Action Atlantic Plaza Towers Tenant Association The Task Force held public forums and community sessions across all five boroughs. BROOKLYN STATEN ISLAND Arden Heights Blvd Jewish Center NOVEMBER 2019   15   These forums, which were free and open to the public, were livestreamed, with video and transcripts of the events posted to our website. Moreover, the Task Force held six additional community sessions across all five boroughs, thanks to outreach conducted by Task Force members and chairs, in partnership with: †† The Atlantic Plaza Towers Tenant Association †† The Queens Public Library at Flushing †† Hostos Community College †† The Department of Youth & Community Development’s Democracy Corp Day of Action (two sessions held) †† The Arden Heights Boulevard Jewish Center These smaller-group sessions were aimed at providing a space for individual community members to share their perspectives and lived experiences with the Task Force in a more intimate setting. Chronology of Task Force Work The Task Force’s work was carried out in the following phases: In May 2018, the Task Force members were appointed by Mayor de Blasio and the Task Force was officially convened. Research Task Force members discussed, and conducted research into, the uses of ADS, both in and outside of New York City. Task Force members also received briefings on relevant City agency operations, including the City’s administrative justice system (from the Office of Administrative Trials and Hearings), procurement processes (from the Mayor’s Office of Contract Services), privacy protocols (from the Chief Privacy Officer), and enforcement of the New York City Human Rights Law (by the Commission on Human Rights Law Enforcement Bureau). Task Force members provided their unique expertise through informational interviews, conducted by City support staff and distributed to the full Task Force. Starting in late April 2019 with our first public forum, the Task Force shifted its focus to public engagement and the collection of discrete recommendations. All recommendations provided by invited panelists, submitted by independent organizations, and provided directly from members of the public were compiled and placed before the Task Force for review and deliberation. Recommendations from Task Force members, sourced through individual interviews, were also collected for member deliberations and review. Deliberations As concrete recommendations were collected through our public sessions, the Task Force began meeting to review these submissions and deliberate over potential topics and themes for the Task Force’s final report. All told, the Task Force collected more than 400 recommendations from the public, advocates, experts, and individual members. Through these deliberations, the Task Force identified areas of potential consensus that would serve as the foundation of its final recommendations. 16   NEW YORK CITY Automated Decision Systems Task Force Report Michael Appleton/Mayoral Photography Office, NYC Public Engagement and Recommendation Development Chapter 5: Recommendations Background Conversations about ADS are often challenging and complex. At the same time, understanding the workings of government—particularly the interactions between different levels of government and their institutions—can be equally complex. Not surprisingly, as we tackled conversations about how City government does and should use ADS, we found our conversations extending beyond the enumerated requirements of the Local Law. This presented us with an opportunity to think more broadly about government use of technology in supporting policymakers’ decision-making, considering the realities of how government currently operates in practice and the concerns and priorities of the public. As Task Force members, we used our conversations to examine not only the types of new citywide and agency practices that could be envisioned to help ensure ADS are being used responsibly by the City and its agencies, but also the likely effects that the introduction of new structures and practices would have on existing government operations and legal frameworks. Through this approach—iteratively identifying possible practices and then testing them against the realities of government structures and processes—we carefully solidified recommendations that set forth our consensus around striving to ensure fair and effective use of ADS by City agencies now and into the future. Throughout these recommendations, we use the term “ADS management” to refer to the overall collection of practices involved in administering how agency ADS tools and systems do and should operate, many of which are subsequently described in detail under particular recommendations. As noted above, because some of our recommendations are organized differently than the explicit mandates of LL49, we indicate for each recommendation which of the specific statutory requirements apply, using the shorthand listed in the table below. For clarity, we also indicate where recommendations are more general in nature. A summary of which mandates each recommendation meets can be found in Appendix I. Mandate Shorthand 3a. C riteria for identifying which agency automated decision systems should be subject to one or more of the procedures recommended by such task force 3a Criteria 3b. D evelopment and implementation of a procedure through which a person affected by a decision concerning a rule, policy or action implemented by the City, where such decision was made by or with the assistance of an agency automated decision system, may request and receive an explanation of such decision and the basis therefor 3b Explanation 3c. D evelopment and implementation of a procedure that may be used by the City to determine whether an agency automated decision system disproportionately impacts persons based upon age, race, creed, color, religion, national origin, gender, disability, marital status, partnership status, caregiver status, sexual orientation, alienage or citizenship status 3c Impact Determination 3d. D evelopment and implementation of a procedure for addressing instances in which a person is harmed by an agency automated decision system if any such system is found to disproportionately impact persons based upon a category described in subparagraph (c) 3d Impact Address 3e. D evelopment and implementation of a process for making information publicly available that, for each agency automated decision system, will allow the public to meaningfully assess how such system functions and is used by the City, including making technical information about such system publicly available where appropriate 3e Available Information 3f. T he feasibility of the development and implementation of a procedure for archiving agency automated decision systems, data used to determine predictive relationships among data for such systems and input data for such systems, provided that this need not include agency automated decision systems that ceased being used by the City before the effective date of this local law 3f Archiving NOVEMBER 2019   17   Our recommendations are organized into three sections, representing the three principal themes of ADS management: †† The first section, which includes three recommendations, focuses on management capacity: identifying the need for a centralized, citywide structure for some ADS management functionality, establishing frameworks for ADS management practices generally, and strengthening related agency capabilities. †† The second section, which includes two recommendations, focuses on public involvement in ADS: emphasizing public education opportunities and creating pathways for engagement. †† The third section, which includes three recommendations, focuses on operations management: guiding practices on information sharing, channeling public inquiry, and enabling assessment. We also want to emphasize that our recommendations should be read through the lens of certain key principles that we believe come from the spirit of the Local Law. These principles have governed our work as a Task Force, and as our recommendations show, we believe that they should be incorporated into management effortsand the continued development of best practices as the City carries this important work forward: †† Using ADS where they promote innovation and efficiency in service delivery. †† Promoting fairness, equity, accountability, and transparency in the use of ADS. †† Reducing potential for harm across the lifespan of ADS. As Chapter 6 lays out, diverse viewpoints among Task Force members were surfaced during the development of these recommendations. The recommendations as they appear here reflect our effort to identify key principles where we reached consensus. In some cases, reaching consensus meant adopting more generalized recommendations over alternatives with greater specificity. Where relevant, these recommendations include annotations with references to other parts of our report where some of these issues are discussed in more detail. Section 1: Build capacity for an effective, fair, and responsible approach to the City’s ADS Enhancing the City’s existing ability to effectively manage ADS can further encourage innovative advances in the delivery of services by the City. Used responsibly, ADS can help unlock solutions for the benefit of the people of our City, making agencies more effective. The recommendations in this section describe the proposed means to strengthen this capacity by establishing new citywide practices and expanding existing agency know-how. Given the broad array of tools or systems that may be defined as ADS, these recommendations should be read to apply variably depending on priority and relevance of existing tools and systems. 1.1 Develop and centralize resources within City government that can guide policy and assist agencies in the development, implementation, and use of ADS. While each agency has a specific mandate, mission, and services to deliver, baseline principles of management and best practices around using ADS should be standardized citywide. 1.1.1 Establish a centralized ADS Organizational Structure within City government. The City should institutionalize an Organizational Structure within City government that would serve as a centralized resource for guiding agency management of ADS and carrying out citywide management functions, as outlined in subsequent recommendations. The Organizational Structure should take on the following: †† Developing citywide policies and best practices to assist agency management of the development, implementation, and use of ADS broadly (Recommendation 1.1.2); †† Developing guidelines for which ADS tools and systems rise to a level of priority for management (Recommendation 1.2.1); †† Providing consultative support to agencies in the development and implementation of ADS, and encouraging agencies to seek out that support (Recommendation 1.3); †† Receiving and incorporating input from the public, external experts, and agencies in the development of ADS policies and protocols, as appropriate and consistent with relevant laws and regulations (Recommendation 2.2); 18   NEW YORK CITY Automated Decision Systems Task Force Report †† Receiving and reviewing information from agencies about ADS and recommending courses of action to resolve issues based on this review (Recommendation 3.1); and †† Developing a method for determining whether a disproportionate impact or harm occurred or could occur (Recommendation 3.3). The Organizational Structure should have sufficient staff, funding, and resources to support these tasks, and should have expertise, or access to expertise, in the fields of technology, data science, law, City government and agency operations, and community engagement. [LL Requirements: 3a Criteria, 3b Explanation, 3c Impact Determination, 3d Impact Address, 3e Available Information] 1.1.2 Incorporate key principles of fairness, transparency, innovation and efficiency, and accountability to help guide responsible City agency use and management of ADS. The Organizational Structure should not only embrace the principles we outline above in the Background section, but should also seek to better integrate these principles as City management practices mature and knowledge bases expand. [LL Requirements: General Recommendation] 1.1.3 Advise agencies on compliance with laws or regulations that may affect their use of ADS. The Organizational Structure should work with other relevant City legal, technology, and policy experts to develop and provide agencies with guidance on the effects of existing or new legislation or regulations that could impact their ADS tools or systems (see Recommendations 1.3.3 and 3.2.2). [LL Requirements: General Recommendation] 1.2 Adopt a phased approach to developing and institutionalizing agency and citywide ADS management practices. There is currently no comprehensive, citywide management framework that specifically addresses how City agencies use ADS. The City should focus on establishing ADS management practices over time, considering available research, resources, operational feasibility, and input from City and other experts in the field. A phased approach would allow the City to prioritize the most important concerns for agencies and the public first, while not overwhelming government operations with a spate of new or infeasible requirements for which guidance or best practices either do not yet exist or have not yet been tested. The City can adopt phased-in management of ADS through two steps: 1.2.1 Create a framework for identifying ADS that should be prioritized. The ADS Task Force found the broad statutory definition of ADS provided in LL49 difficult to work with as a practical matter, as its breadth made it hard to clearly identify which tools and systems could, in fact, be considered an ADS. This broad-sweeping definition also implicated a potentially unmanageable number of tools and systems, including some that simply perform ministerial functions that could be subject to further review without a compelling reason to do so. The limitations of this definition thus posed challenges for our work in developing the mandated recommendations, as well as for envisioning what ADS management will look like in the future. To resolve this challenge and facilitate the identification of ADS without refining the definition, the Organizational Structure should develop prioritization guidelines, in which key concepts or principles are put in place to inform agencies about the characteristics of technical tools or systems that are most important to direct their focus. Agencies can, in turn, use these guidelines to identify the technical tools or systems they use which have the highest-priority characteristics. Once prioritization protocols are established, agencies can then begin to examine their highest-priority tools or systems first, moving on to other priority tools or systems as resources permit. The development of these guidelines by the Organizational Structure should include quality assurance opportunities to ensure that the highest-priority systems are being correctly identified. NOVEMBER 2019   19   The considerations and principles upon which the Organizational Structure should develop its guidelines for a prioritization framework should minimally include3: †† General Descriptive Characteristics †† The type of task the tool or system performs (e.g., one that collects and analyzes private/sensitive data versus an email, word processing task, or tabulation or basic mathematical formulas). †† Whether the tool or system aids in making a decision or is only exploratory or pre-decisional. †† The use of machine learning or other artificial intelligence techniques. †† The use of personally identifying information. †† Explainability †† The complexity of the task the tool or system performs. †† The “explainability” of the tool or system (e.g., the ease with which someone can describe components of the tool or system to someone else). †† Cost-Benefit †† The urgency of the need for the tool or system and the downsides of a delay in, or not implementing the tool or system. †† The benefits, efficiencies, or cost-savings resulting from the tool or system. †† The actual or expected operational impact of using the tool or system, including effects on labor resources. †† Impact †† Any existing City or agency review or approval procedures relating to outputs resulting from use of the tool or system (e.g., the triggering of manual review by designated agency personnel in the event of a particular output from use of the tool or system). †† The nature and magnitude of the potential impacts (e.g., upon personal liberty or financial interests). †† The duration or permanence of the potential impacts. †† Any types of known or potential risks, including disproportionate impact or bias, cybersecurity, privacy, or restricted access by individuals to key information. [LL Requirements: 3a Criteria] 1.2.2 Incorporate flexibility into management processes. The technology underpinning ADS is constantly evolving, as may be the laws and regulations governing them. The guidelines that are applicable for agencies today may no longer be applicable within a year. Rather than creating a static framework for managing ADS, the City should employ a flexible approach to ADS management that allows for iteration and change, and focus on establishing management practices that are both appropriate for the current tools and systems in use and adaptable for the future. To do so, all guidelines and processes developed by the City via the Organizational Structure should contain guidance on reviewing and updating those guidelines. [LL Requirements: General Recommendation] 3 See Chapter 6, Level of Specificity of Recommendations. 20   NEW YORK CITY Automated Decision Systems Task Force Report 1.3 Strengthen the capacity of City agencies to develop and use ADS. City agencies must have increased access to the expertise needed to identify and continue to use ADS fairly and effectively, and the resources to carry out the various tasks and requirements outlined in these recommendations. To better prepare agencies for using and managing ADS, the City should: 1.3.1 Provide agencies with sufficient funding and staffing for ADS management. Some agencies will need more resources in managing their ADS than others, due to the number and complexity of relevant systems in use, size of the staff, breadth of technical expertise, policy area, nature of their operations, and other factors. Every agency should have access to the appropriate staff and civil service titles (including technical expertise) and funding it needs to responsibly manage ADS. [LL Requirements: General Recommendation] 1.3.2 Consider agency expertise in developing centralized policy regarding ADS management. The Organizational Structure should solicit and consider agency expertise when developing centralized, citywide ADS guidelines to help ensure that agency perspectives on feasibility, operations, and resources are understood. Agencies are experts in their own policy domains. Agency experience with ADS in their policy area can provide critical perspective to help shape ADS management citywide, particularly in understanding the legal and regulatory frameworks relating to such policies. [LL Requirements: General Recommendation] 1.3.3 Create best practices on ADS, including ADS procurement, data retention, and data sharing, to serve as a resource for agencies. The Organizational Structure should create best practices on ADS development and use, including on topics such as ADS procurement and ADS data retention, consistent with relevant laws, regulations, and City policies. As these best practices are compiled, agencies should be encouraged to refer to them early and frequently in their ADS development, procurements, and implementation processes. The Organizational Structure should provide guidance to agencies in developing ADS tools and systems, as appropriate, particularly where the ADS has been designated as high-priority. [LL Requirements: 3f Archiving] 1.3.4 Educate agency staff on ADS and how to communicate about ADS with the public. The Organizational Structure should identify relevant types of staff who should be familiarized with the general principles of ADS and train them on the technical aspects of ADS, how concepts of fairness, accountability and transparency relate to ADS management, and how to communicate and educate the public about ADS in use by the City. Similarly, working with agencies, the Organizational Structure should identify internal governance practices and procedures around the development and use of tools such as ADS, where educational material around ADS can be integrated. [LL Requirements: General Recommendation] 1.3.5 Enable the input of expertise external to the City to help support the City’s work of ADS management. The City should further pursue avenues by which experts external to the City can support the work of ADS management, recognizing the substantial administrative challenges expected in managing ADS and the evolving nature of the field. [LL Requirements: General Recommendation] NOVEMBER 2019   21   Section 2: Broaden public discussion on ADS The City should call upon the knowledge and insights of the many individuals and groups interacting with those ADS that impact the public with respect to issues of fairness and equity, including impacted communities and experts on specific technical elements. To facilitate this, the Organizational Structure should provide opportunities for the public to learn about ADS that impact the public and to inform policy-making around ADS. 2.1 Facilitate public education about ADS. The Organizational Structure should take the lead in educating the public about ADS, as public engagement cannot take place without basic shared knowledge and awareness. This should include publicly accessible, digital platforms that provide resources to help users understand ADS, and include published information about actual ADS. 2.1.1 Create a visible, accessible presence for ADS management. The Organizational Structure should be visible and easily accessible to the public since it will serve as the citywide resource for centralized and agency ADS management. There should be a publicly accessible and digital platform (or platforms) that the public can view to learn more about ADS and ADS management (see Recommendation 2.1.2), and to contact the City to ask questions about ADS generally or about specific tools or systems. [LL Requirements: 3b Explanation, 3e Available Information] 2.1.2 Develop educational materials that explain important concepts about ADS and ADS management in plain language. The Organizational Structure should develop and publish materials in plain language that explain what ADS are, why some ADS are relevant to issues of fairness and equity, what makes some ADS higher priority than others, and the key elements of the City’s management of ADS. The latter should include clear explanations of the Organizational Structure and its responsibilities, explanations of how agency and citywide management works, and information about the relevant laws, regulations, and City policies relating to the use of ADS. [LL Requirements: 3e Available Information] 2.1.3 Help individual City residents request additional information from agencies about ADS. Existing, public-facing agency contact points and legal mechanisms such as the New York State Freedom of Information Law (FOIL) already allow individuals to obtain information or request records from City agencies for government actions, including those that involve ADS. Agencies should improve or, where appropriate, supplement existing agency or citywide informational mechanisms to learn about government actions involving ADS by, to the extent feasible or permissible, developing and making available contact points, including any hotlines, ‘contact us’ pages, FAQs, or other standard methods. Where possible, the Organizational Structure should centralize this public information on its website or otherwise help the public navigate to the appropriate outlet. [LL Requirements: 3b Explanation] 2.1.4 Report to the public on overall ADS management. The Organizational Structure should create and publish, where legally permissible, key performance indicators for its work in relation to the City’s ADS tools and systems, outcomes of any ADS assessments, emerging best practices in ADS management, and other relevant information. [LL Requirements: General Recommendation] 2.2 Engage the public in ongoing work around ADS. Public engagement opportunities are a critical element of good government and participatory democracy and should happen around ADS as well. There should be opportunities for the public to provide feedback on, or otherwise engage in, discussions around agency ADS. In order to facilitate such engagement, the City should do the following: 22   NEW YORK CITY Automated Decision Systems Task Force Report 2.2.1 Provide an opportunity for public input to the Organizational Structure’s guidelines for agencies on ADS management. The Organizational Structure should provide an opportunity for public commentary into the process of developing identification/prioritization guidelines, information-sharing guidelines, and performance management and assessment guidelines (see Recommendation 1.3.5). [LL Requirements: General Recommendation] 2.2.2 Involve impacted communities in discussions about specific ADS. Agencies should engage with relevant communities during the course of developing or procuring an ADS if they do not already do so, where legally permitted, and where not contrary to the need for the efficient operation of government. Where engagement is not feasible during development, agencies should identify opportunities to solicit feedback from communities during the course of ADS usage. To support agencies in community engagement, the Organizational Structure should prepare guidelines and best practices to encourage deeper and more frequent community involvement. [LL Requirements: 3e Available Information] Section 3: Formalize ADS management functions A key aspect of fair and effective ADS management is identifying where additional resources or practices may be necessary to enhance existing management structures. ADS management will require the development of protocols for reporting and publishing certain ADS information, where legally appropriate, ensuring comprehensive and practical responses to individual inquiries, and assessing execution of agency tools or systems. 3.1 Establish a framework for agency reporting and publishing of information related to ADS. The Organizational Structure should develop protocols for agency reporting and publishing of certain information related to ADS, informed by the principle of promoting transparency and through a framework of prioritization. To facilitate the management of ADS on a citywide level and to foster public discourse, agencies should, when possible given relevant legal and security considerations, report certain information about highest-priority ADS to the Organizational Structure, and, when possible, make certain information about the ADS publicly available. 3.1.1 Identify highest-priority tools. Agencies should use the guidelines developed by the Organizational Structure (see Recommendation 1.2.1) to identify any highest-priority tools they use. [LL Requirements: 3a Criteria] 3.1.2 Establish reporting standards for information related to ADS. The City should require that certain information about agency ADS be compiled at the agency level and, when possible given relevant legal and security considerations, reported centrally to the Organizational Structure. Toward this goal, the Organizational Structure should develop guidelines for agencies about what information should be reported, based on appropriate legal review, and the reporting frequency. These guidelines should be developed based on prioritization, meaning highest-priority tools or systems should, where possible, have additional elements of information reported to the Organizational Structure. Reporting guidelines should be developed in accordance with privacy, security, and other legal requirements, and should emphasize plain language and explainability to the greatest extent possible. Reportable information may include: †† Descriptive information (e.g., system name and purpose, policy context, and background on development); †† Technical information (e.g., model choices, descriptions of data types used, choice of technology or platform, and descriptions of source code); and †† Assessment information (e.g., outcomes of any pre-deployment assessments, monitoring plans, and post-deployment assessment plans). [LL Requirements: 3e Available Information] NOVEMBER 2019   23   3.1.3 Publish agency reported information about ADS where legally permissible. Following an internal City and agency process for legal and security review and approval, reports compiled by agencies on ADS should be made available on a platform maintained by the Organizational Structure. Publishing should begin with the highest-priority tools, eventually expanding to other higherpriority tools with appropriate timing, resources, and legal review. The platform should have clearly delineated version-control protocols and archiving for older versions of reports. [LL Requirements: 3e Available Information, 3f Archiving] 3.1.4 Aid compliance with laws and regulations. To aid compliance with all relevant laws and regulations, any of the guidelines used to inform agencies about the sharing of ADS data should be developed with the assistance of relevant City agencies and officials with expertise in cybersecurity, law, information privacy, record retention, FOIL, procurement, human rights law, existing oversight and governance structures, and other policy areas that may affect the maintenance and/or disclosure of City records or information. [LL Requirements: General recommendation] 3.2 Incorporate information about ADS specifically, where relevant, into processes for public inquiry about or challenge to City agency decisions. There are a number of ways by which individuals can challenge or make inquiries regarding a specific decision made by a City agency. Often, the first step is reaching out to the agency. Depending on the decision in question, the agency may have a formal process for addressing challenges or a mechanism for submitting questions. However, these existing processes may not currently integrate relevant information related to ADS. 3.2.1 Integrate general ADS information into preexisting inquiry response channels. In cases where an agency has an internal process or mechanism for members of the public to inquire about or question an agency decision, the agency should ensure that ADS-related inquiries can be appropriately addressed through such channels (see Recommendation 1.3.3). To support this work, the Organizational Structure should work with agencies to develop guidelines on how to identify inquiries about the ADS tools and systems specifically, as well as guidelines on incorporating relevant ADS information into responses (see Recommendation 3.2.2). [LL Requirements: 3b Explanation, 3d Impact Address] 3.2.2 Provide guidelines to agencies on how to respond to and document specific public inquiries and challenges. The Organizational Structure should develop guidelines that agencies can use to inform the ways in which they respond to inquiries when those inquiries relate to the use of an ADS. Those guidelines should help agencies identify the scope of information that could be publicly shared about an ADS, in accordance with applicable law, how to internally evaluate specific inquires and challenges related to technical features of ADS, and how members of the public may resolve or challenge a decision with the agency if appropriate. [LL Requirements: 3b Explanation, 3d Impact Address] 3.2.3 Create a single point of contact in the City for individuals to submit questions or comments about specific ADS decisions. In addition to existing processes and mechanisms described above, the Organizational Structure should create and maintain an accessible, public, online portal through which individuals can submit questions or comments to agencies pertaining to decisions made by ADS (see Recommendation 2.1.1). The portal should include further information on ADS generally and informational resources on entities and means through which someone may seek remediation or redress. [LL Requirements: 3b Explanation, 3d Impact Address] 24   NEW YORK CITY Automated Decision Systems Task Force Report 3.3 Create an internal City process for assessing specific ADS for any risk of disproportionate impact to any individual or group on the basis of protected characteristics. Disproportionate impacts may not always be apparent and may require in-depth research in order to identify them. The Organizational Structure should develop a process for conducting an assessment of ADS that includes guidelines on which systems should be subject to review, the elements of the assessment, the relevant agency and City parties to conduct such reviews, and appropriate timelines for the assessment, depending on facts and circumstances. This should include a process for escalation of urgent matters involving actual or suspected harm to an individual or community in relation to use of an ADS. This process should include the following: 3.3.1 Provide guidelines on which ADS should be subject to review. The Organizational Structure should provide agencies with guidelines on the types of highest-priority ADS that should be reviewed by the agency in consultation with relevant City officials, based on prioritization framework guidelines (see Recommendation 1.2.1). [LL Requirements: 3a Criteria, 3c Impact Determination] 3.3.2 Develop options for protocols for assessment. In order to assess a tool or system for disproportionate impact, the Organizational Structure should develop protocols that should be used by agencies to carry out internal ADS assessments. At a minimum, the options of available protocols should guide agencies in identifying differences in outcomes and differences in error for the populations that could be impacted. The options for protocols should also include ways to assess technical and policy execution. [LL Requirements: 3c Impact Determination] 3.3.3 Provide opportunities for impacted communities and others from outside the City to provide input. Agencies, with guidelines from the Organizational Structure, should, where legally permissible and operationally feasible, provide opportunities for impacted communities and outside experts to communicate questions and comments about specific ADS tools and systems. This may include direct, affirmative outreach by agencies to impacted communities and opportunities to submit comment, among other approaches. Additionally, as appropriate given legal and security considerations, agencies may bring in outside experts to conduct parts of the ADS assessment and may also request the Organizational Structure help conduct parts of the assessment, as necessary (see Recommendation 1.3.5). [LL Requirements: 3c Impact Determination] 3.3.4 Develop a process for responding to instances of negative disproportionate impact on the basis of protected characteristics. The Organizational Structure should develop protocols to respond to instances where an assessment of an ADS indicates that there may be an unintended or unjustifiable disproportionate impact or harm upon any individual, group, or community. These protocols should include the prompt convening of appropriate City officials with the relevant agency personnel for the development of action plans and guidance on minimizing or eliminating the effects of such impacts as appropriate. [LL Requirements: 3c Impact Determination] NOVEMBER 2019   25   Chapter 6: Challenges The Automated Decision Systems Task Force is the first of its kind, so we started our work in largely uncharted territory. In hopes that the work of this Task Force will help set the stage for future work around ADS, we are outlining specific challenges the group faced and potential resolutions. It is our hope that this information about the challenges that arose during our own process will help not only the City prospectively, but also any other entities that wish to build upon or learn from our work to do so more effectively. Definitions: What is an Automated Decision System? As mentioned above, LL49 defines ADS as “computerized implementations of algorithms, including those derived from machine learning or other data processing or artificial intelligence techniques, which are used to make or assist in making decisions.” It became clear from the Task Force’s initial meetings that this definition posed challenges for our work. The definition in LL49 is quite broad and arguably includes tools as general as internet searches or spreadsheets. Many agency operations are performed on computers using software, and many of those computerized operations work due to the algorithms defined by software and process data as a matter of necessity. The ADS definition provided in the Local Law allows for the idea that these computerized tools, even if they were simple spreadsheets or internet searches, could be ADS, simply because they are computerized and guide decision-making. For example, staff entering data into a spreadsheet, then using a summation formula, then viewing the outputs of that formula, might conceivably—by a literal interpretation of the Local Law definition—be using an ADS. At the same time, sophisticated predictive models developed with machine learning techniques would also be considered ADS. Task Force members had different perspectives on the potential impact of such a broad definition. Some members believed it was better that definition continue to be broadly inclusive of possible tools and systems, and that limiting the scope of the definition could exclude tools or systems that might be considered relevant for additional agency assessment. Other members believed that including such a wide array of tools or systems could divert the City’s attention from the specific types of technical tools and systems that the law was intended to address, and also that too expansive a definition would render the recommendations infeasible to implement. Beyond the definition provided in LL49, we considered other possible definitions of ADS over many months, and some members advocated for specific alternatives. As a group, however, we did not reach a consensus on another definition. Instead, in order to move forward with our work, we turned our attention to our first requirement in LL49: criteria to determine which systems should be subject to our other recommendations. This requirement enabled us to focus on identifying the characteristics of systems or tools that we believe are important. As our conversations about criteria unfolded, it was clear that we needed to discern not only the characteristics that may make a system or tool qualify as an ADS, but also those that would allow us to decide which of those likely ADS should be subject to greater consideration—or “higher priority”—than others. Prioritization, as our recommendations make clear, is critical to ensure that the public is able to better understand important tools used by the City, while enabling agencies to carry out their regular duties in the most efficient and effective ways possible. We provide a number of the characteristics for prioritization that we believe to be most relevant in the recommendations. 26   NEW YORK CITY Automated Decision Systems Task Force Report S AD HIG H ME T EN EM G ZE UTERI D SYSTE P M MS CO ANT FOR V E L E MA TR O NA R O I I TY A N PR T W DS LO O I R R P ITY UM I A D R O I I T Y PR DS AD S, BU A framework for identifying and prioritizing ADS To identify relevant ADS, the Task Force proposes not a single definition, but rather a series of questions or criteria that can evolve over time that both identify relevant systems or tools and prioritize them. As mentioned earlier, ADS exist within a broader universe of the many technological solutions and information technology tools used by government. In the diagram above, the dotted line reflects the uncertainty around defining precisely what is—and is not—an ADS. The smaller number of higher-priority ADS reflect the idea that only certain systems are the most important and merit more attention from the City and the general public. Some of the characteristics that emerged as considerations in prioritizing ADS include the complexity of the tool or system, how explainable a given tool or system is, how robust existing review or approval processes are, how personally identifying information is used, a cost-benefit analysis of the use of the tool versus a nonautomated process, and the nature and magnitude of the potential impacts (e.g., with respect to liberty or financial outcomes). To confirm the feasibility of such an approach, the Task Force reviewed various questions and discussed means of administering such tools, and came to the conclusion that developing such a framework would require more time and examination, and could vary by agency due to differences in mission or service delivery. Recommendation 1.2 in Chapter 5 provides further information about the proposed framework. NOVEMBER 2019   27   Level of Specificity of Recommendations Because the use of ADS in government is a developing area of practice, it was important to ensure that our recommendations would apply to a wide range of agency missions and a potentially wide range of tools or systems— including those that may not even exist yet. A common refrain was to not focus on just recommendations tailored for today, but those that would help guide the City into the future. However, as a Task Force, we debated about the level of specificity with which we should write our recommendations, with some members asserting that more specific ideas and tools should be captured within our report. In particular, some members believed that we could have provided more specificity related to the framework for prioritization. Additionally, some members favored reporting a highly detailed set of data elements related to ADS tools and systems, including a rationale for their use, technical features, and evaluations. As a group, there were diverging views on these issues. Other members expressed concern that inclusion of too much detail risked projecting more certainty than was reasonable given our limited capacity to lay out the whole set of potential implications of those details. Ultimately, we chose to take an approach that focused on the structures of governance, existing legal and policy frameworks, and feasibility of operationalization. This allowed our recommendations to reflect where we reached consensus among the diverse perspectives represented on the Task Force, and to have the greatest potential for effective implementation. Level of Authority As noted in Chapter 3, the use of ADS by agencies takes place against a complex framework of emerging technologies and existing law and policy. We had many debates about what functions the Organizational Structure should ideally manage and often had even more challenging debates about what functions the Organizational Structure could lawfully perform. While some members wanted to recommend greater authority for the Organizational Structure, including enforcement and compliance powers, other members disagreed. Given the continued divergence of perspectives among Task Force members, our recommendations on this issue reflect those proposed Organizational Structure functions and duties about which we could achieve consensus. Privacy and Data Security As a Task Force, we faced challenges in working to develop recommendations for protocols that promoted public transparency in government use of ADS, considering the City’s duties to protect the privacy of the personally identifying information of its residents, public safety, the security of City infrastructure and technical assets, and certain proprietary and other sensitive types of City information. We grappled with finding the right balance between emphasizing opportunities to share information publicly about City tools, systems, and processes, while ensuring that any relevant legal, security, and privacy risks were accounted for; this entailed understanding the appropriate scope of disclosure, data access and use permissions and restrictions, and internal legal review procedures. We expect that this challenge will continue after our Task Force concludes. Issues of privacy and data security also came up as part of our Task Force process. Some members strongly believed that we needed to review some, if not all, examples of ADS currently in use by the City as part of our deliberative work in forming recommendations. Others believed that reviewing examples posed challenges, both in terms of being able to identify tools or systems in use that met the broad definition of “ADS” provided in the Local Law, as well as ensuring that reviewing examples with Task Force members would be permissible given any legal, privacy, proprietary, and security considerations relevant to specific technical tools and systems. Ultimately, we were able to develop a set of protocols for legal review and obtained approvals for and did reviewed four specific agency examples from the Department of Transportation, the Police Department, the Department of Education, and the Fire Department. During our meetings, we often specifically discussed the challenges that emerge when government agencies procure services or materials related to ADS from private entities. Most members agreed that proprietary restrictions can make certain key ADS information less available to the public, and given the complexities of procurement within government, that this important topic requires further review beyond the Task Force process. 28   NEW YORK CITY Automated Decision Systems Task Force Report An Ever-Evolving Field The field of ADS is constantly evolving. New systems are developed all the time, and older systems are able to be improved upon. There is no industry standard for what defines an ADS, and there are few, if any, existing best practices around their governance. Because this field is still evolving, the Task Force aimed to make its recommendations specific enough to be implementable, while still broad enough to allow for shifts in technology, policy, or regulation of such systems. The work of this Task Force is intended to be a starting point—not an end—to the important discussions around ADS. Logistics Our Task Force had 17 members and three cochairs, all of whom served on the Task Force in a fully voluntary capacity, on top of their roles as leaders in their organizations, agencies, and fields. The logistics of coordinating work for 20 people proved challenging. To keep us on task, City staff, in addition to convening the full Task Force, held many smaller meetings with groups of members to discuss specific aspects of the work, and to provide each member with individualized resources to ensure their commentary and contributions were integrated into the full Task Force’s deliberations. Ed Reed/Mayoral Photography Office, NYC Additionally, on a Task Force as diverse as ours, there were varying degrees of expertise on a wide array of relevant topics, including technology, human rights, law, policy, and city operations. While this rich diversity of expertise ultimately proved to be an asset, it also gave rise to some consistent challenges, not least of which was related to the absence of a shared language on ADS. As a result, many of our discussions required time to educate each other about our specific domains of expertise, which, while sometimes time consuming, helped us reach recommendations that took into account many considerations from various perspectives and fields. This diversity, we believe, led to stronger recommendations and insights that will help guide the City’s policies around ADS for years to come. Finally, our community sessions yielded some important lessons related to engaging the public on the topic of ADS. We found that the communities and organizations with which we most wanted to engage were often facing a multitude of competing demands, and the topic of ADS was not always included in their most pressing priorities. We did learn, however, that conversations about ADS were most engaging and productive when grounded in policy areas to which they considered themselves to be most closely connected. The public forums and community sessions completed as part of the Task Force process represent an important first effort to foster a broader public conversation about government use of ADS, and this is a process that will be ongoing. As such, these lessons should be carried forward as part of the recommended Organizational Structure’s work. NOVEMBER 2019   29   Chapter 7: Unresolved and Related Topics for Consideration While this Task Force made recommendations that provide direction for the City’s management of ADS, there are many considerations related to the use of complex automated systems in government that were not discussed at length by the Task Force or that we ultimately decided not to issue recommendations about. Topics fell under this category for a number of reasons. For example, the topic may not have been within the scope of the Task Force’s mandate, the Task Force may not have had the expertise or information required to make a conclusion about the topic, or the topic may already have clearly fallen under the purview of an existing City entity. This section identifies these topics related to ADS that were largely unaddressed in the Task Force’s recommendations but merit the City’s consideration in future discussions about government use of ADS. Archiving LL49 required the Task Force to make recommendations around the archiving of ADS. Through our work, we came to understand the many challenges involved in this task. For example, ADS take very different forms and are made up of various components, including software, data, algorithms, the resulting outcomes, and more. This raises questions about what parts of an ADS could feasibly be archived, the purpose of such archiving, the length of archiving, the form in which they should be archived in, and what cost in time, resources, and effort this process could take if required of every system. In the recommendations, we provided guidelines by which agencies may document the methodology behind these systems and descriptions of the business case for which they were designed. We also call for version control policies for information about ADS that is made public. Types of Systems or Tools Our Task Force was charged with developing recommendations for the use of ADS broadly. However, during the course of our work, conversations periodically focused on particular types of systems or tools, either to illustrate applications of concepts that were under discussion or to emphasize particular principles that were informing our deliberations. For example, systems or tools that collect and/or rely on biometric data were sometimes discussed specifically. Ultimately, we chose not to emphasize any specific types of systems or tools within our recommendations, to ensure applicability for the wide range of current technology and expectations of new systems and technology in the future. We believe, however, that beyond the work of our Task Force, understanding particular features of specific types of systems will be important for ongoing management of ADS, and the equity implications of such tools and systems. State and Federal Governance and Use of ADS The Task Force’s mandate was to make recommendations and issue a report about ADS use and governance by City government, ADS is a topic that has also garnered significant attention at other levels of government. In the future, it will be important in managing ADS within New York City to monitor the national discourse and policymaking surrounding ADS closely, and keep pace with evolving best practices and laws around their development and use. Private Sector Use of ADS Many of the most innovative uses of automated decision-making are happening in the field of business analytics, as businesses seek to lower their costs and become more responsive to changing circumstances. It was not within the scope of this Task Force to consider uses of ADS in the private sector, but private sector ADS can pose similar risks to those the Task Force were asked to consider in City operations. The question of oversight of applications of such technologies in private housing, for example, emerged at our second Public Engagement Session and through our community session with members of the Atlantic Plaza Towers Tenants Association. Proposed legislation in other levels of government seeks to regulate these systems, such as the proposed Algorithmic Accountability Act in the US Congress, which is focused on preventing and remediating harm related to discriminatory decisions made by private sector ADS in areas such as housing. Locally, the City’s Economic Development Corporation is undertaking an AI for Good Initiative to begin laying out the principles and best practices of responsible, ethical use of AI in the private sector. Education of Non-City-Government Entities While we strongly recommend a robust public education campaign about ADS be conducted by the Organizational Structure, it is likely the general public is not the only audience for whom additional knowledge about ADS would be useful. Other governmental entities outside of New York City’s executive branch (such as courts, legislatures, and other authorities) could benefit from educational campaigns that could be led by the proposed Organizational Structure. 30   NEW YORK CITY Automated Decision Systems Task Force Report Appendix I: Summary of Recommendations Recommendation General 3a 3b Criteria Explanation 3c 3d Impact Impact Determination Address 3e Available Information 3f Archiving 1.1 x 1.1.1 1.1.2 1.1.3 x x x x x x 1.2 x 1.2.1 1.2.2 x 1.3 1.3.1 1.3.2 x x x 1.3.3 1.3.4 1.3.5 x x 2.1 x 2.1.1 x x 2.1.2 x 2.1.3 2.1.4 x 2.2 2.2.1 x x 2.2.2 3.1 x 3.1.1 x x 3.1.2 3.1.3 3.1.4 x x 3.2 x x x 3.2.1 3.2.2 3.2.3 x x x 3.3 3.3.1 3.3.2 3.3.3 3.3.4 x x x x x NOVEMBER 2019   31   Appendix II: Text of Local Law 49 of 2018 By Council Members Vacca, Rosenthal, Johnson, Salamanca, Gentile, Cornegy, Williams, Kallos and Menchaca A Local Law in relation to automated decision systems used by agencies Be it enacted by the Council as follows: Section 1. a. For purposes of this local law: Agency. The term “agency” means an agency, as defined in section 1-112 of the administrative code of the city of New York, the head of which is appointed by the mayor. Automated decision system. The term “automated decision system” means computerized implementations of algorithms, including those derived from machine learning or other data processing or artificial intelligence techniques, which are used to make or assist in making decisions. Automated decision system, agency. The term “agency automated decision system” means an automated decision system used by an agency to make or assist in making decisions concerning rules, policies or actions implemented that impact the public. Charitable corporation. The term “charitable corporation” shall have the meaning ascribed to such term by section 102 of the not-for-profit corporation law. 1. No later than 120 days after the effective date of this local law, the mayor or a designee thereof shall convene an automated decision systems task force. 2. Such task force and the chair thereof shall be appointed by the mayor or a designee thereof and shall include, but need not be limited to, persons with expertise in the areas of fairness, accountability and transparency relating to automated decision systems and persons affiliated with charitable corporations that represent persons in the city affected by agency automated decision systems, provided that nothing herein shall prohibit the mayor, the designee thereof or the chair from limiting participation in or attendance at meetings of such task force that may involve consideration of information that, if disclosed, would violate local, state or federal law, interfere with a law enforcement investigation or operations, compromise public health or safety or result in the disclosure of proprietary information. 3. No later than 18 months after such task force is established, it shall electronically submit to the mayor and the speaker of the council a report that shall include, at a minimum, recommendations on: (a) Criteria for identifying which agency automated decision systems should be subject to one or more of the procedures recommended by such task force pursuant to this paragraph; (b) D evelopment and implementation of a procedure through which a person affected by a decision concerning a rule, policy or action implemented by the city, where such decision was made by or with the assistance of an agency automated decision system, may request and receive an explanation of such decision and the basis therefor; (c) D evelopment and implementation of a procedure that may be used by the city to determine whether an agency automated decision system disproportionately impacts persons based upon age, race, creed, color, religion, national origin, gender, disability, marital status, partnership status, caregiver status, sexual orientation, alienage or citizenship status; 32   NEW YORK CITY Automated Decision Systems Task Force Report (d) D evelopment and implementation of a procedure for addressing instances in which a person is harmed by an agency automated decision system if any such system is found to disproportionately impact persons based upon a category described in subparagraph (c); (e) Development and implementation of a process for making information publicly available that, for each agency automated decision system, will allow the public to meaningfully assess how such system functions and is used by the city, including making technical information about such system publicly available where appropriate; and (f) The feasibility of the development and implementation of a procedure for archiving agency automated decision systems, data used to determine predictive relationships among data for such systems and input data for such systems, provided that this need not include agency automated decision systems that ceased being used by the city before the effective date of this local law. 4. Such task force shall dissolve 60 days after submission of the report required by paragraph 3. 5. The mayor shall, no later than 10 days after receipt of the report required by paragraph 3, make such report publicly available online through the city’s website. 6. Nothing herein shall require compliance with the task force’s recommendations or disclosure of any information where such disclosure would violate local, state, or federal law, interfere with a law enforcement investigation or operations, compromise public health or safety, or that would result in the disclosure of proprietary information. § 2. This local law takes effect immediately. NOVEMBER 2019   33   AUTOMATED DECISION SYSTEMS TASK FORCE