THE NATIONAL ACADEMIES PRESS This PDF is available at http://nap.edu/25086 SHARE     Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation (2018) DETAILS 129 pages 8.5 x 11 PAPERBACK ISBN 978-0-309-47491-7 DOI 10.17226/25086 CONTRIBUTORS GET THIS BOOK Committee to Review Advances Made to the IRIS Process; Board on Environmental Studies and Toxicology; Division on Earth and Life Studies; National Academies of Sciences, Engineering, and Medicine FIND RELATED TITLES SUGGESTED CITATION National Academies of Sciences, Engineering, and Medicine 2018. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation. Washington, DC: The National Academies Press. https://doi.org/10.17226/25086. Visit the National Academies Press at NAP.edu and login or register to get: – Access to free PDF downloads of thousands of scientific reports – 10% off the price of print titles – Email or social media notifications of new titles related to your interests – Special offers and discounts   Distribution, posting, or copying of this PDF is strictly prohibited without written permission of the National Academies Press. (Request Permission) Unless otherwise indicated, all materials in this PDF are copyrighted by the National Academy of Sciences. Copyright © National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Committee to Review Advances Made to the IRIS Process Board on Environmental Studies and Toxicology Division on Earth and Life Studies A Consensus Study Report of Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation THE NATIONAL ACADEMIES PRESS 500 Fifth Street, NW Washington, DC 20001 This project was supported by Contract EP-C-14-005, TO#008 between the National Academies of Sciences, Engineering, and Medicine and the US Environmental Protection Agency. Any opinions, findings, conclusions, or recommendations expressed in this publication are those of the authors and do not necessarily reflect the view of any organization or agency that provided support for this project. Digital Object Identifier: https://doi.org/10.17226/25086 Copyright 2018 by the National Academy of Sciences. All rights reserved. Printed in the United States of America Suggested citation: National Academies of Sciences, Engineering, and Medicine. 2018. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation. Washington, DC: The National Academies Press. doi: https://doi.org/10.17226/25086. Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation The National Academy of Sciences was established in 1863 by an Act of Congress, signed by President Lincoln, as a private, nongovernmental institution to advise the nation on issues related to science and technology. Members are elected by their peers for outstanding contributions to research. Dr. Marcia McNutt is president. The National Academy of Engineering was established in 1964 under the charter of the National Academy of Sciences to bring the practices of engineering to advising the nation. Members are elected by their peers for extraordinary contributions to engineering. Dr. C. D. Mote, Jr., is president. The National Academy of Medicine (formerly the Institute of Medicine) was established in 1970 under the charter of the National Academy of Sciences to advise the nation on medical and health issues. Members are elected by their peers for distinguished contributions to medicine and health. Dr. Victor J. Dzau is president. The three Academies work together as the National Academies of Sciences, Engineering, and Medicine to provide independent, objective analysis and advice to the nation and conduct other activities to solve complex problems and inform public policy decisions. The National Academies also encourage education and research, recognize outstanding contributions to knowledge, and increase public understanding in matters of science, engineering, and medicine. Learn more about the National Academies of Sciences, Engineering, and Medicine at www.national academies.org. Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Consensus Study Reports published by the National Academies of Sciences, Engineering, and Medicine document the evidence-based consensus on the study’s statement of task by an authoring committee of experts. Reports typically include findings, conclusions, and recommendations based on information gathered by the committee and the committee’s deliberations. Each report has been subjected to a rigorous and independent peer-review process and it represents the position of the National Academies on the statement of task. Proceedings published by the National Academies of Sciences, Engineering, and Medicine chronicle the presentations and discussions at a workshop, symposium, or other event convened by the National Academies. The statements and opinions contained in proceedings are those of the participants and are not endorsed by other participants, the planning committee, or the National Academies. For information about other products and activities of the National Academies, please visit www.nationalacademies.org/about/whatwedo. Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation COMMITTEE TO REVIEW ADVANCES MADE TO THE IRIS PROCESS Members JONATHAN M. SAMET (Chair), Colorado School of Public Health, Denver, CO RICHARD A. CORLEY (retired), Pacific Northwest National Laboratories, Richland, WA GEORGE DASTON, Proctor & Gamble Company, Cincinnati, OH DAVID C. DORMAN, North Carolina State University, Raleigh, NC RUSS B. HAUSER, Harvard T.H. Chan School of Public Health, Boston, MA KAREN A. ROBINSON, Johns Hopkins University, Baltimore, MD RICHARD P. SCHEINES, Carnegie Mellon University, Pittsburgh, PA LAUREN ZEISE, CalEPA Office of Environmental Health Hazard Assessment, Berkeley, CA YILIANG ZHU, University of New Mexico, Albuquerque, NM Staff ELLEN K. MANTUS, Project Director RADIAH ROSE-CRAWFORD, Manager, Editorial Projects SUZANNE THILENIUS, Administrative Coordinator MIRSADA KARALIC-LONCAREVIC, Manager, Technical Information Center JESSICA WOLFMAN, Senior Program Assistant Sponsor US ENVIRONMENTAL PROTECTION AGENCY v Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation BOARD ON ENVIRONMENTAL STUDIES AND TOXICOLOGY Members WILLIAM H. FARLAND (Chair), Colorado State University, Fort Collins, CO RICHARD A. BECKER, American Chemistry Council, Washington, DC E. WILLIAM COLGLAZIER, American Association for the Advancement of Science, Washington, DC DOMINIC M. DITORO, University of Delaware, Newark, DE DAVID C. DORMAN, North Carolina State University, Raleigh, NC CHARLES T. DRISCOLL, JR., Syracuse University, Syracuse, NY ANNE FAIRBROTHER, Exponent, Inc., Philomath, OR GEORGE GRAY, The George Washington University, Washington, DC STEVEN P. HAMBURG, Environmental Defense Fund, New York, NY ROBERT A. HIATT, University of California, San Francisco, CA SAMUEL KACEW, University of Ottawa, Ontario, Canada H. SCOTT MATTHEWS, Carnegie Mellon University, Pittsburgh, PA ROBERT PERCIASEPE, Center for Climate and Energy Solutions, Arlington, VA R. CRAIG POSTLEWAITE, US Department of Defense, Burke, VA MARK A. RATNER, Northwestern University, Evanston, IL JOAN B. ROSE, Michigan State University, East Lansing, MI GINA M. SOLOMON, California Environmental Protection Agency, Sacramento, CA ROBERT M. SUSSMAN, Sussman and Associates, Washington, DC DEBORAH L. SWACKHAMER, University of Minnesota, St. Paul, MN PETER S. THORNE, University of Iowa, Iowa City, IA Staff TERESA A. FRYBERGER, Director ELLEN K. MANTUS, Scholar and Director of Risk Assessment RAYMOND A. WASSEL, Scholar and Director of Environmental Studies SUSAN N.J. MARTEL, Senior Program Officer for Toxicology ELIZABETH BOYLE, Program Officer TAMARA DAWSON, Program Associate BERNIDEAN WILLIAMS-SMITH, Financial Associate SUZANNE THILENIUS, Administrative Coordinator JESSICA WOLFMAN, Senior Program Assistant vi Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Acknowledgments This Consensus Study Report was reviewed in draft form by persons chosen for their diverse perspectives and technical expertise. The purpose of this independent review is to provide candid and critical comments that will assist the National Academies of Sciences, Engineering, and Medicine in making each published report as sound as possible and to ensure that it meets institutional standards of quality, objectivity, evidence, and responsiveness to the study charge. The review comments and draft manuscript remain confidential to protect the integrity of the deliberative process. We thank the following for their review of this report: David Eaton, University of Washington David Savitz, Brown University Joyce Tsuji, Exponent Harvey Clewell, Ramboll Lisa Bero, University of Sidney Nancy Reid, University of Toronto Although the reviewers listed above provided many constructive comments and suggestions, they were not asked to endorse the conclusions or recommendations of this report, nor did they see the final draft before its release. The review of the report was overseen by Mark Cullen, Stanford University, who was responsible for making certain that an independent examination of the report was carried out in accordance with institutional procedures and that all review comments were carefully considered. Responsibility for the final content rests entirely with the authoring committee and the National Academies. The committee gratefully acknowledges the staff of the US Environmental Protection Agency, especially Tina Bahadori and Kristina Thayer, for their presentations to the committee during open sessions. The committee is also grateful for the assistance of Norman Grossblatt who served as the report editor. vii Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Contents SUMMARY ......................................................................................................................................................... 1 1 INTRODUCTION........................................................................................................................................ 2 The Integrated Risk Information System and Previous National Academies Reports, 2 The Committee, Its Task, and Its Approach, 3 Organization of the Report, 4 References, 4 2 RESPONSES TO NATIONAL ACADEMIES RECOMMENDATIONS .............................................. 5 General Process Issues, 5 Systematic Review: Problem Formulation, Protocol Development, and Evidence Identification and Evaluation, 6 Evidence Integration, 8 Derivation of Toxicity Values, 10 Concluding Remarks, 12 References, 12 APPENDIXES A BIOGRAPHIC INFORMATION ON THE COMMITTEE TO REVIEW ADVANCES MADE TO THE IRIS PROCESS .......................................................................................................... 14 B WORKSHOP AGENDA ........................................................................................................................ 17 C PRESENTATIONS BY US ENVIRONMENTAL PROTECTION AGENCY .................................. 20 D POSTERS BY US ENVIRONMENTAL PROTECTION AGENCY ................................................. 99 E FINDINGS REGARDING INDIVIDUAL RECOMMENDATIONS .............................................. 110 BOXES AND FIGURE BOXES 1-1 2-1 Statement of Task, 4 Environmental Protection Agency Description of Its Portfolio Approach, 6 FIGURE 1-1 The IRIS process in the context of systematic review, 3 ix Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Summary Over the past several years, the US Environmental Protection Agency (EPA) has been transforming the procedures of its Integrated Risk Information System (IRIS), a program that produces hazard and dose‒response assessments of environmental chemicals and derives toxicity values that can be used to estimate risks posed by exposures to them. The transformation was initiated after suggestions for program reforms were provided in a 2011 report from the National Academies of Sciences, Engineering, and Medicine that reviewed a draft IRIS assessment of formaldehyde. In 2014, the National Academies released a report that reviewed the IRIS program and evaluated the changes implemented in it since the 2011 report. Although it provided many recommendations, the 2014 report concluded that “substantial improvements in the IRIS process have been made, and it is clear that EPA has embraced and is acting on the [National Academies] recommendations.” Since 2014, new leadership of EPA’s National Center for Environmental Assessment (NCEA) and IRIS program has instituted even more substantive changes in the IRIS program in response to the recommendations in the 2014 report. Given the new direction of the IRIS program, EPA asked the National Academies to review the agency’s progress toward addressing the past recommendations. Accordingly, the National Academies convened the Committee to Review Advances Made to the IRIS Process. The present committee heard presentations, reviewed posters, and received demonstrations of toolkits and databases from EPA over the course of a 1.5-day workshop, and it reviewed recent IRIS work products. This brief report provides the committee’s general findings regarding EPA’s progress (Chapter 2) and specific findings regarding changes made in response to individual recommendations from the 2014 report (Appendix E). Overall, the committee was impressed with the changes being instituted in the IRIS program since the 2014 report. The committee views the transformation of the IRIS program as a work in progress, recognizes that this review assesses one moment in time in a still-evolving program, and acknowledges that the IRIS program will (and should) continue to evolve as it adapts and applies new scientific approaches and knowledge. The change in NCEA and IRIS leadership has led to substantive reforms, and there is strong evidence that systematic review methods are being developed and implemented and that there is a commitment to use systematic-review methods to conduct IRIS assessments. Although the committee offers some refinements and identifies a few possibilities for further development in Chapter 2, its overall conclusion is that EPA has been responsive and has made substantial progress in implementing National Academies recommendations. 1 Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation 1 Introduction For many years, the National Academies of Sciences, Engineering, and Medicine has been asked to review assessments produced by the Integrated Risk Information System (IRIS) of the US Environmental Protection Agency (EPA). The reviews have consistently provided recommendations for revisions of specific assessments, but the National Academies committee that was tasked with reviewing the draft IRIS assessment of formaldehyde also suggested changes to improve the IRIS program itself, if EPA chose to do so. Since release of that committee’s report (NRC 2011), the IRIS program has been undergoing substantive changes. In 2014, another National Academies committee reviewed the changes in the IRIS program and provided an overall favorable assessment, noting that it was reviewing a work in progress (NRC 2014). In light of a change in leadership and continued revisions of the IRIS program, EPA asked the National Academies to review changes since 2014 and to determine whether they have been responsive to the recommendations in past National Academies reports. In response to EPA’s request, the National Academies convened the Committee to Review Advances Made to the IRIS Process, which prepared this brief report. THE INTEGRATED RISK INFORMATION SYSTEM AND PREVIOUS NATIONAL ACADEMIES REPORTS Given problems in several IRIS assessments noted by previous National Academies committees (see, for example, NRC 2006, 2010, 2011) and specific issues encountered in the formaldehyde assessment, the committee that evaluated the formaldehyde assessment provided a roadmap for reframing the development of IRIS assessments (Chapter 7, NRC 2011). The roadmap did not provide detailed guidance but rather suggestions for creating a more systematic and transparent IRIS process, if EPA chose to go forward with reforming the process. Congress directed EPA to respond to and incorporate the recommendations and suggestions provided in Chapter 7 of the 2011 National Academies report (House Report 112-151; Public Law 112-74). EPA indicated that the agency was committed to responding to National Academies recommendations and improving the IRIS program and began to make substantive changes. In a 2012 report to Congress, EPA highlighted its intended changes, such as a new document structure with a preamble that describes general methods for evidence identification, evidence evaluation, and derivation of toxicity values; new systematic approaches for data analysis; and expanded efforts for stakeholder engagement (EPA 2012; NRC 2014). EPA also noted that it had formed the Chemical Assessment Advisory Committee under the auspices of its Scientific Advisory Board to advise the agency on specific assessments and broader program issues. To ensure that EPA was responding adequately to National Academies recommendations, Congress asked the National Academies to review the changes that EPA was implementing. In 2014, the National Academies released the report Review of EPA’s Integrated Risk Information System (IRIS) Process (NRC 2014), which evaluated the changes that were being implemented in the IRIS program and assessed whether they were responsive to the recommendations and suggestions made in Chapter 7 of the 2011 report. The 2014 report concluded that “substantial improvements in the IRIS process have been made, and it is clear that EPA has embraced and is acting on the recommendations in the…formaldehyde report.” It urged EPA to adopt systematic review practices, framed the IRIS process 2 Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Introduction in the context of systematic review (see Figure 1-1), and provided specific recommendations on each step of the process (NRC 2014). Since release of the 2014 report, substantive efforts have been made to incorporate systematic review into the IRIS process, and EPA has now asked the National Academies to assess its progress. THE COMMITTEE, ITS TASK, AND ITS APPROACH The committee that was convened to address EPA’s request included expertise in toxicology, epidemiology, risk assessment, statistics, modeling, evidence integration, and systematic review; see Appendix A for biographic information on the committee. The verbatim statement of the committee’s task is provided in Box 1-1. As noted, the committee was asked to assess the changes that have been (or that are planned to be) implemented by EPA in response to National Academies recommendations. It is important to note that the committee was not asked to evaluate the overall value of the IRIS program, to recommend where IRIS should be located within the agency, or to review any specific chemical assessment. The committee was also not tasked with conducting a comprehensive review of the IRIS program; rather, it was asked to evaluate whether the current trajectory of the program agrees with past recommendations of the National Academies. To address its task, the committee held a 1.5-day workshop during which EPA presented its progress to the committee. Multiple opportunities for stakeholder input were provided. Appendix B provides the workshop agenda. The committee reviewed EPA presentations (Appendix C), posters (Appendix D), recently released materials (EPA 2017, 2018a,b; Orme-Zavaleta 2018 ), and all materials submitted by stakeholders. To fulfill its task of evaluating EPA’s progress in implementing past National Academies recommendations, the committee decided to focus its attention primarily on recommendations made in the report Review of EPA’s Integrated Risk Information System (IRIS) Process (NRC 2014). Although the report Review of the Environmental Protection Agency’s Draft IRIS Assessment of Formaldehyde (NRC 2011) provided general suggestions for reforming the IRIS program, it primarily made recommendations specifically for revising the draft assessment of formaldehyde. It is important to note that the 2011 committee was not tasked with an extensive review of the IRIS program. The 2014 report considered the general suggestions provided in the 2011 report, reviewed the IRIS program specifically, and made numerous recommendations directed at the program. Therefore, the present committee considered the 2014 report as expanding on the suggestions provided in the 2011 report and thus evaluated EPA’s progress in addressing each recommendation in the 2014 report. Scoping Human Problem Formulation Develop Protocols for Systematic Reviews Animal Human Identify Evidence Mechanistic Animal Human Evaluate Studies Mechanistic Animal Integrate Evidence Mechanistic Systematic Reviews DoseResponse Assessment and Derivation of Toxicity Values Broad Literature Search FIGURE 1-1 The IRIS process in the context of systematic review. Source: NRC (2014). 3 Copyright National Academy of Sciences. All rights reserved. Hazard Identification Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Progress Toward Transforming the Integrated Risk Information System Program: A 2018 Evaluation BOX 1-1 Statement of Task An ad hoc committee under the auspices of the National Academies of Sciences, Engineering, and Medicine will assess changes that have been implemented or plan to be implemented by the U.S. Environmental Protection Agency (EPA) for its Integrated Risk Information System (IRIS) in response to recommendations made in previous NRC reports, such as Review of EPA’s Integrated Risk Information System (IRIS) Process and Review of the Environmental Protection Agency’s Draft IRIS Assessment of Formaldehyde. The committee will base its assessment on EPA presentations and interactive sessions during a 1.5 day workshop at which multiple opportunities will be provided for stakeholder input. ORGANIZATION OF THE REPORT The present report is organized into two chapters and five appendixes. Chapter 2 presents the committee’s overall findings regarding advances made in the IRIS process. Appendix A provides biographic information on the committee. Appendixes B, C, and D provide the workshop agenda, EPA presentations made during the workshop, and EPA poster presentations, respectively. Appendix E details the committee’s findings concerning individual recommendations in the report Review of EPA’s Integrated Risk Information System (IRIS) Process (NRC 2014). REFERENCES EPA (US Environmental Protection Agency). 2012. EPA’s Integrated Risk Information Program, Progress Report and Report to Congress. Office of Research and Development, U.S. Environmental Protection Agency [online]. Available: https://www.epa.gov/sites/production/files/2015-06/documents/iris_report_to_congress_2012.pdf [accessed February 27, 2018]. EPA. 2017. IRIS Assessment Plan for Chloroform [CASRN 67-66-3]. EPA/635/R-17/330. Integrated Risk Information System, National Center for Environmental Assessment, Office of Research and Development, U.S. Environmental Protection Agency, Washington, DC. EPA. 2018a. Systematic Review Protocol for the IRIS Chloroform Assessment (Inhalation) [CASRN 67-66-3]. EPA/635/R-17/486. Integrated Risk Information System, National Center for Environmental Assessment, Office of Research and Development, U.S. Environmental Protection Agency, Washington, DC. EPA. 2018b. IRIS Assessment Plan for Uranium (Oral Reference Dose) (Scoping and Problem Formulation Materials) [CASRN 7440-61-1]. EPA/635/R-17/787. Integrated Risk Information System, National Center for Environmental Assessment, Office of Research and Development, U.S. Environmental Protection Agency, Washington, DC. NRC (National Research Council). 2006. Health Risks from Dioxin and Related Compounds: Evaluation of the EPA Reassessment. Washington, DC: The National Academies Press. NRC. 2010. Review of the Environmental Protection Agency’s Draft IRIS Assessment of Tetrachloroethylene. Washington, DC: The National Academies Press. NRC. 2011. Review of the Environmental Protection Agency’s Draft IRIS Assessment of Formaldehyde. Washington, DC: The National Academies Press. NRC. 2014. Review of EPA’s Integrated Risk Information System (IRIS) Process. Washington, DC: The National Academies Press. Orme-Zavaleta, J. 2018. Response to the Request for Correction (RFC). Letter to Robert Holden, Liskow & Lewis, New Orleans, LA, from Jennifer Orme-Zavaleta, Principal Deputy Assistant Administrator for Science, Office of Research and Development, Washington, DC, January 25, 2018; Attachment 1. EPA Response to the Denka Performance Elastomers (DPE) Request for Correction of the Toxicological Review of Chloroprene (CAS No. 126-99-8) In Support of Summary Information on the Integrated Risk Information System (IRIS); Attachment 2. Systematic Review of Chloroprene [CASRN 126-99-80] Studies Published Since 2010 IRIS Assessment to Support Consideration of the Denka Request for Correction (RFC). January 2018 [online]. Available: https://www.epa.gov/sites/production/files/2018-01/documents/epa_repsonse_to_mr._holdren_jan_ 25_2018_complete.pdf [accessed February 9, 2018]. 4 Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation   2 Responses to National Academies Recommendations Over the course of a 1.5-day workshop, the US Environmental Protection Agency (EPA) made presentations to the committee on changes that are transforming the Integrated Risk Information System (IRIS) process. The committee used that information and recently released IRIS documents to judge the extent to which EPA has adequately addressed recommendations made in previous National Academies reports, primarily Review of EPA’s Integrated Risk Information System (IRIS) Process (NRC 2014).1 The committee’s overall comments are provided below; findings regarding individual recommendations are in Appendix E. GENERAL PROCESS ISSUES The 2014 report (Chapter 2 in NRC 2014) offered recommendations related to the IRIS process and evaluated EPA’s progress in implementing the suggestions made in Review of the Environmental Protection Agency’s Draft IRIS Assessment of Formaldehyde (NRC 2011).2 Above all, the 2014 report commented on the need to continue to sustain the evolution of the program’s procedures and to consider how EPA will do so in the context of continually advancing scientific methods. In the 4 years since the 2014 recommendations, the IRIS program clearly has maintained a trajectory of change that has accelerated under the new leadership of the EPA National Center for Environmental Assessment (NCEA) and the IRIS program. The committee was impressed by the scope of changes that have been or are being implemented and by the engagement of scientists throughout NCEA, EPA more broadly, other federal agencies, and academe to effect change. Such engagement is appropriate inasmuch as funding for the use of external contractors has diminished, and there is expertise in relevant fields throughout the agency. Supervisory and communication strategies are in place, and formal quality management has been implemented. The committee notes that EPA will need to ensure that quality management extends to activities that are conducted by people who are outside the IRIS program. Changes in some of the critical elements of the overall IRIS process are still in progress. The 2014 committee was given an incomplete draft of the handbook; the handbook is intended to provide guidance on the IRIS process. The 2014 committee recommended completion of that handbook; the present committee was not given a draft of the handbook. EPA indicated that the handbook is still in development and is “being updated to reflect Agency input, evolving IRIS practices as systematic-review approaches are tested through implementation, and public comment received on chemical-specific protocols” (Slide 22, Appendix C). Public release is anticipated in 2018. The handbook is expected to provide critical guidance for the development of IRIS assessments, and the present committee urges that high priority be given to its completion, peer review, and release. Reference to it will facilitate transparency on the approach for specific IRIS assessments. In the absence of a final version of the handbook, EPA is describing its approach for the reviews in its protocol documents, and this practice provides transparency into the assessment process while the handbook is being completed. The committee notes, however, that the handbook should not become a final, fixed set of guidelines but rather should be a document that evolves over time. 1 Referred to hereafter as the 2014 report. The committee that produced that report is referred to as the 2014 committee. 2 Referred to hereafter as the 2011 report. 5 Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Progress Toward Transforming the Integrated Risk Information System Program: A 2018 Evaluation The 2014 committee also commented on the need to incorporate input from various stakeholders— including industry, academe, and nongovernment organizations—at appropriate points in the process; this recommendation has been heeded by past and current program leaders. Three points in the process, including development of assessment plans and systematic-review protocols, have been identified at which public comments will be sought (slide 24, Appendix C). Although the present committee was not shown the approach for acknowledging public comments and incorporating them into the process, the handbook should describe how this will be done. The committee was impressed by other NCEA program activities that engage stakeholders, including dissemination of tools that it has developed, such as the benchmark-dose (BMD) modeling software, and provision of training. EPA also commented that it was moving away from a one-size-fits-all approach to what it termed a portfolio approach, as described in Box 2-1. The move toward a portfolio approach appears to add needbased and context-based flexibility to the IRIS program. EPA used chloroform as an example; it is developing a reference concentration for inhalation exposures and assessing whether the reference concentration protects against carcinogenic effects adequately. The decision to limit the assessment was based on consultation with EPA regulatory programs. Overall, the portfolio approach is expected to conserve agency resources, and it is consistent with the recommendations of the National Academies report, Science and Decisions: Advancing Risk Assessment (NRC 2009). SYSTEMATIC REVIEW: PROBLEM FORMULATION, PROTOCOL DEVELOPMENT, AND EVIDENCE IDENTIFICATION AND EVALUATION The 2014 report offered many recommendations related to systematic review, including problem formulation, protocol development, evidence identification, and evidence evaluation (Chapters 3‒5, NRC 2014). The committee found that the IRIS program has made substantial progress in incorporating systematic-review methods into its process and assessments. Development and implementation of systematic-review methods have been facilitated by the recruitment of the current IRIS program director, who has extensive experience in the development of the methods and their application to chemical risk assessment. The IRIS program has also expanded internal training programs that are designed to improve staff understanding of the methods. BOX 2-1 Environmental Protection Agency Description of Its Portfolio Approach To ensure…support is timely and responsive, NCEA is developing a portfolio of chemical evaluation products employing the principles and state-of-the-art practices of systematic review. The portfolio approach will increase public health protection by:  moving away from a “one-size-fits-all” approach to chemical risk assessment towards a spectrum of assessment products to meet specific decision contexts;
  facilitating the incorporation of new science into risk assessment and decision-making;  tailoring assessments to meet the many needs of decision makers; and,  increasing the number of chemicals that can be evaluated for their effects on human health by utilizing constrained resources in the most efficient manner. Source: EPA (2018a). 6 Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Responses to National Academies Recommendations Furthermore, the IRIS program has developed a number of formal and informal collaborations with groups that are active in systematic review, including the National Toxicology Program Office of Health Assessment and Translation, the World Health Organization (WHO), the European Food Safety Authority, the International Collaboration for Automation of Systematic Reviews, and the Collaborative Approach to Meta-Analysis and Review of Animal Data from Experimental Studies (CAMARADES). Some of those collaborations help to position the IRIS program as a leader in advancing systematic-review methods, such as the development or modification of risk-of-bias tools for animal toxicity studies. The committee was impressed by the efforts of IRIS program management to develop within the IRIS program the scientific expertise needed to conduct systematic reviews. Some notable changes have included the establishment of a systematic-review working group that should lead to increased efficiency and consistency among assessments. Other workgroups that are focused, for example, on epidemiology, physiologically based pharmacokinetic (PBPK) models, and neurotoxicology have been created; these teams of appropriate subject-matter experts are expected to support the IRIS process further through improved rigor of scoping and problem formulation and through improvements in other steps of the systematic-review process. The 2014 report offered numerous recommendations related to systematic-review processes that are accepted as standards of practice in the scientific community. The present committee found multiple examples of the IRIS program’s consideration and implementation of those recommendations, such as the development of systematic-review protocols, inclusion of an information specialist who is trained in systematic-review methods in the work groups, and the use of two-person teams for data extraction and risk-ofbias assessments. The IRIS program is also appropriately using a variety of software tools to assist with literature management (HERO), scoping (SWIFT), screening (Distiller), and data extraction (HAWC). The use of those and other software tools with input from appropriate subject-matter experts should improve efficiency, transparency, and rigor and directly address recommendations in the 2014 report. Many of the operational approaches used by the IRIS program are described in the assessment plans or the systematicreview protocols, and sufficient details are given to provide assurances that standardized systematic-review methods are being developed and applied by the IRIS program. The committee expects future systematicreview protocols to be streamlined and to become less generic when the handbook is completed. The 2014 report also offered several recommendations about evaluating individual studies. Those recommendations encouraged EPA to use or develop tools for assessing risk of bias in different types of studies (human, animal, and mechanistic) and to add quality-assessment items relevant to particular systematic-review questions. EPA has implemented a process for evaluating risk of bias, and several documents that were provided to the committee (for example, EPA 2018b; Orem-Zavaleta 2018) demonstrate implementation of EPA’s risk-of-bias tools and how EPA has augmented them with additional questionspecific elements to assess study validity. The IRIS program, however, should provide information on the choice and use of tools, including its rationale for the choice of particular risk-of-bias domains. Including that documentation in the IRIS handbook will improve transparency. The committee notes that evaluation of risk of bias, although important, is not the only way to evaluate study quality. Accordingly, the IRIS program should show how other important methodologic characteristics of a particular study will be evaluated, and EPA should continue to seek and evaluate additional tools that can help to assess study quality. As part of revisions of the IRIS process, EPA is producing assessment plans and systematic-review protocols. The committee found overlap between those documents; for example, PECO statements are found in both types of documents.3 Indeed, the added value of a two-step process (assessment plan and protocol) was unclear to the committee. It was not immediately clear whether the assessment plan also serves as a “data call” for additional studies that are outside the scope specified by the systematic review but could inform the overall chemical-assessment process. Some additional clarification of terminology and clearer descriptions of how the documents will be used could help the public to understand how chemical assessments move through the IRIS process. 3 A PECO statement is a structured framework that defines a question by specifying population, exposure, comparator, and outcome to be considered in a systematic review. 7 Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Progress Toward Transforming the Integrated Risk Information System Program: A 2018 Evaluation The committee identified several ways in which the IRIS program could benefit from refinements. For example, the link between scoping and problem formulation outlined in the assessment plan and development of the PECO statement was not well described. Improving the description of how scoping and problem formulation are used to focus the goals of a systematic review will lead to greater specificity in descriptions of outcomes, inclusion and exclusion criteria, and other elements found in the systematicreview protocol and will further improve the transparency and scientific rigor of the process. The committee found that the IRIS program included the dates and results of its literature searching and screening (for example, as appendixes) in draft systematic-review protocols that are undergoing public comment. Completing the literature search as part of protocol development is inconsistent with current best practices for systematic review, and the IRIS program is encouraged to complete the public-comment process and finalize the protocol before initiating the systematic review. Doing so will improve transparency in the IRIS process. The committee identified several recommendations in the 2014 report that reflect broad scientific efforts that extend beyond the IRIS program. For example, several recommendations were related to the evaluation and use of mechanistic data in a systematic review. EPA’s systematic-review process indicates that mechanistic data can be considered at various steps; for example, the draft protocol for the IRIS assessment of chloroform (EPA 2018b) describes how mechanistic data will be considered. Although appropriate tools, such as those to evaluate risk of bias in mechanistic studies, are in early stages of development in the broader scientific community, the IRIS program has developed approaches for the evaluation of PBPK models that will be used in assessments (Orme-Zavaleta 2018). The committee expects similar evaluation methods for other types of mechanistic evidence to emerge on a case-by-case basis and to include methods for determining at what stage and how mechanistic data could be used in an IRIS assessment. For example, mechanistic data were used by a National Academies committee to inform development of PECO statements for reproductive outcomes associated with o-phthalate compounds (NASEM 2017a). The committee notes that the use of mechanistic data by the IRIS program is consistent with other EPA programs, such as the Office of Pesticide Programs; for example, in the recent hazard identification conducted for benzo[a]pyrene (EPA 2017b), the IRIS program used mechanistic data extensively. Nonetheless, establishment of a framework for when and how mechanistic data would be identified, evaluated, and used remains challenging. The challenge is not unique to the IRIS program and has been identified for future work in the National Toxicology Program (NTP) handbook for conducting systematic reviews and evidence integration (NTP 2015a, p. 73‒74). Finally, the committee considered best practices for systematic reviews in other medical disciplines. Current best practices recommended by the Institute of Medicine (IOM 2011) suggest that the IRIS teams involved in the systematic-review process should be independent of those involved in regulatory decisionmaking who use the products of the systematic-review teams. The committee notes that the current organizational structure of the IRIS program in the EPA Office of Research and Development is consistent with those best practices. EVIDENCE INTEGRATION The 2011 report recommended standardizing an approach for synthesizing evidence within data streams (human, animal, and mechanistic) and integrating evidence across data streams (NRC 2011, p. 165).4 From 2011 to 2013, the IRIS program moved solidly in that direction, as evidenced by its draft handbook (EPA 2013) and its example applications of the approach in two draft IRIS reports—the Toxicological Review of Methanol (Noncancer) and the Toxicological Review of Benzo[a]pyrene (see NRC 2014, pp. 93‒96). Although the 2014 committee recognized that substantial progress had occurred during 2011‒ 2013, it made several additional recommendations to guide the IRIS program toward a more systematic 4 IRIS uses the phrase evidence synthesis to refer to the task of combining evidence from a given evidence stream, such as human or animal, and the phrase evidence integration to refer to the task of combining evidence from different evidence streams. 8 Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Responses to National Academies Recommendations process for evidence synthesis and integration that would maximize transparency, efficiency, and scientific validity. The major recommendation in Chapter 6 of the 2014 report guided IRIS to choose between making its current guided expert process more transparent and adopting a more structured, GRADE-like,5 process along the lines of the NTP (NRC 2014, p. 105). The IRIS program has explicitly chosen the first option, using structured categories with criteria to guide expert judgment, and EPA has made substantial strides toward more systematic and transparent evidence synthesis (see slides 65‒84, Appendix C; posters D-4 and D-5, Appendix D). Specifically, the IRIS program has created processes and guidelines for synthesizing human evidence and animal evidence that support choosing one category for characterizing the strength of evidence (see slides 82‒84, Appendix C). The guidelines focus on human and animal evidence streams and use mechanistic evidence to inform evidence synthesis and to provide scientific guidance for evidence integration in the steps that follow. In using Bradford Hill criteria to move beyond association to causation and to build on the systematic evaluations of individual study quality conducted in the step before evidence synthesis,6 the IRIS program has created a process for evidence synthesis that is scientifically consistent with the state of the art and that effectively leverages approaches of other programs, such as NTP, that face similar challenges. Increased transparency is evident in the examples and the workshop presentations, but further transparency would be obtained with completion of a handbook that provides more details about processes, reasoning behind decisions, and approaches for presenting results. In the interim, while EPA is completing its handbook, it is releasing protocols for each assessment that include a description of how evidence within each data stream will be synthesized and how evidence from multiple data streams will be integrated. The draft protocol for the IRIS assessment of chloroform (EPA 2018b) was provided as an example. The committee supports EPA’s approach. Integration of evidence across data streams was described by EPA in its presentations (see slides 79‒87, Appendix C; posters D-4 and D-5, Appendix D) and in the draft chloroform protocol (EPA 2018b, pp. 43‒53). Again, the process and framework within which evidence integration takes place (slides 82‒ 84; Appendix C) are consistent with state-of-the-art approaches taken by other scientific institutions or agencies, such as NTP, that face similar challenges. Some questions have been raised about the use of mechanistic data in evidence integration. When animal or human data are extensive, mechanistic data can be used to evaluate the evidence within or across the animal or human data streams rather than as a third stream of evidence to be analyzed separately and then combined with human and animal evidence. When extensive mechanistic data are available and human and animal data on apical end points are sparse, mechanistic data might be used reliably as a third data stream to identify hazards, as has been done for the dioxin-like polychlorinated biphenyls (IARC 2016). Mechanistic data are important in identifying potential adverse outcomes, including ones that are not evaluated in guideline-driven animal testing; in informing dose–response assessment; and in determining the relevance of animal data for human health risk estimation. For example, in the case of phthalates (poster D-7, Appendix D), mechanistic data were used to determine that not all effects on male reproductive development in rodents were relevant for humans, and the data provided a basis for selecting the studies that were most relevant as a starting point in establishing a reference dose. However, EPA acknowledged that understanding of mechanisms relevant to effects of phthalates on development is incomplete, and that uncertainty makes it difficult to estimate risk primarily on the basis of mechanistic information. Although organizing the body of evidence according to a mechanistic framework might at first seem desirable because of biologic relevance, mechanistic frameworks today could probably be completed for only a few chemicals. As noted in the 2014 report, solid conclusions about causality can be drawn without mechanistic information,7 for example, when there is strong and consistent evidence from animal or epidemiology studies. 5 GRADE is defined as grading of recommendations, assessment, development and evaluation. For example, see slide 69 in Appendix C, in which EPA advises using only medium-quality and high-quality studies and incorporating considerations of bias and sensitivity. 7 “The history of science is replete with solid causal conclusions in advance of solid mechanistic understanding” (NRC 2014, p.90). 6 9 Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Progress Toward Transforming the Integrated Risk Information System Program: A 2018 Evaluation Another recommendation from Chapter 6 of the 2014 report concerns expanding EPA’s capacity to perform quantitative evidence integration for hazard identification, for example, by using meta-regression or Bayesian analysis. To avoid compromising efficiency and timeliness in producing assessments, the 2014 report recommended developing such analytic capacities in parallel with other work in the IRIS program. EPA has taken the recommendation seriously and has explored meta-analytic approaches to combining animal data within species to determine whether the evidence indicates a chemical hazard, for example, whether trimethylbenzene poses a neurotoxic hazard (poster D-2, Appendix D). The IRIS program also initiated work on a Bayesian approach to combining data from different animal species (poster D-10, Appendix D).8 The Bayesian work is promising, but application to IRIS assessments has not yet occurred. It is clear that the IRIS program has made progress here; the agency should continue with its efforts in this field. Another recommendation from both the 2011 and the 2014 reports urged the use of more standardized, structured evidence tables to support the evidence-integration narrative9 and emphasized the utility of a somewhat standard template for the narrative. The 2017 Toxicological Profile for Benzo[a]pyrene (EPA 2017b) provides an example of structured evidence tables that directly support the evidence-integration narrative, first for synthesis of individual data streams and then in an integrated summary form that connects evidential categorization with the supporting studies (Table 1-20, page 1-108). The final table lays out the evidence that the chemical is a human carcinogen by first introducing the human evidence on cancer from benzo[a]pyrene or precursors from complex mixtures and the human mechanistic studies and then discussing the findings of in vivo animal studies on tumors associated with multiple routes of exposure, adding the studies of precursor events, and finally presenting the evidence that precursor events are likely to occur in humans. The format is clear, well structured, and straightforward to follow. Although a well-reasoned discussion on noncancer effects is available in the same document, structured-narrative justifications of the evidence-integration process and the conclusion were not as well developed as those on cancer. In the workshop, EPA stated that standardized descriptors for noncancer effects are still needed and are being discussed within the agency. EPA illustrated current thinking regarding the template for evidence integration in the workshop (slide 85, Appendix C) and in the chloroform draft protocol (EPA 2018b). The template has many characteristics of the GRADE approach to evaluating evidence, with similar labels for conclusions about the strength of the evidence within and across data streams. The approach appears to conform with the state of the art and bears considerable similarity to the system used by NTP (NTP 2015a,b). Although the chloroform protocol provides some illustration of EPA’s approach, more detailed guidance and completed examples are needed to judge EPA’s application of the template for evidence integration. In summary, the IRIS program has made substantial strides in meeting the recommendations of the 2011 and 2014 reports regarding synthesis and integration of evidence. The IRIS process that was presented to the committee is dramatically more systematic, transparent, and scientifically defensible than the one presented in the 2010 IRIS Toxicological Review of Formaldehyde (EPA 2010). DERIVATION OF TOXICITY VALUES Recommendations regarding derivation of toxicity values were provided in Chapter 7 of the 2014 report. An important recommendation in that chapter was to “develop criteria for determining when evidence is sufficient to derive toxicity values.” In the workshop, EPA described the overall process and criteria that the agency intends to use to implement that recommendation and indicated that it would develop toxicity values when the evidence-integration conclusion is the “strongest” or a “moderately strong conclusion for a human health effect.” As noted, EPA clarified that the agency intends to systematize processes 8 The Bayesian approach is based on the seminal work of Dumouchel and Harris (1983) and recent work of Jones et. al. (2009). 9 An evidence-integration narrative is a description of the available evidence and the argument for or against a particular hazard. 10 Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Responses to National Academies Recommendations to maintain transparency in reaching the hazard conclusion (slides 132‒133, Appendix C), although standard descriptors for noncancer effects are being reviewed within the agency and are not yet final. EPA’s approach is consistent with the 2014 recommendation that formal dose‒response assessments should be restricted to outcomes on which evidence integration has led to the strongest or a moderately strong conclusion on the given health effect, such as known or likely to be carcinogenic to humans (slide 131, Appendix C). EPA indicated that when there is less strong evidence on a human health effect, such as suggestive evidence of cancer, the decision to develop a toxicity value will be determined by the situation (for example, when there is a well-conducted study and a value would be useful for a decision). However, EPA did not present criteria to be used in such cases. The one example in which criteria have been applied to support the derivation of toxicity values was the chloroprene reassessment (Orme-Zavaleta 2018). In that document, EPA focused its systematic review on publications since the 2010 assessment. EPA concluded that the new studies did not change the conclusions in the 2010 assessment and did not justify a reassessment of human health effects (that is, derivation of new toxicity values). Although commenting on the conclusions in that assessment is beyond the scope of the present committee’s task, the committee acknowledges that EPA’s reassessment described its criteria for evaluating risk of bias and study sensitivity needed to detect a true effect and that it presented criteria for evaluating PK/PBPK studies. Furthermore, EPA explained why each study considered in the final assessment did not change the conclusions reached in the 2010 IRIS assessment and did not justify a reassessment of human health effects. Thus, it is clear that EPA is making progress toward improving transparency in its use of systematic review and expert judgment to inform the derivation of toxicity values directly. Another important recommendation in the 2014 report was that EPA “continue its shift toward the use of multiple studies rather than single studies for dose‒response assessment” (NRC 2014). The present committee noted that progress has been made in the use of multiple studies for dose‒response assessments as exemplified in the recent assessments of ethylene oxide and benzo[a]pyrene (slides 134‒135, Appendix C) and builds on efforts to compare candidate reference doses or concentrations in previous assessments, such as in the 2012 IRIS Toxicological Review of Tetrachloroethylene (EPA 2012). EPA is further developing new tools for visualizing comparisons to communicate the outcome of assessments more effectively, as was demonstrated in the workshop by using HAWC. EPA acknowledged, and the committee agrees, that the development of systematic assessments for many types of mechanistic studies that could contribute to the assessment remains challenging, not only to EPA but to the scientific community generally. However, the process that EPA previously developed to review PK/PBPK models and to describe how they could be used in dose‒response and toxicity-value assessments (EPA 2006) is a good example of best practices. As other forms of mechanistic data become more readily available, partly driven by previous National Academies reports (NASEM 2017b; NRC 2007), the IRIS program should develop new approaches for using such studies to inform dose‒response and toxicity-value assessments (slides 142‒147, Appendix C). Such guidance will improve transparency and encourage new science, whether it is used to support evidence of potential toxicity or, just as important, to provide perspectives on the potential exposure conditions that could reasonably be expected to cause toxicity. The 2014 report also recommended that EPA use formal methods for combining multiple studies and further develop and expand its use of Bayesian and other formal quantitative methods for dose–response assessment and derivation of toxicity values (NRC 2014). EPA has begun to develop and apply tools for meta-regression analysis and Bayesian approaches and has demonstrated their application in case studies (slides 135, 136, 139, and 140, Appendix C; posters D-2 and D-10, Appendix D). Implementation of the recommendation will continue and will require sustained resources and continued capacity-building to develop a process that is ultimately transparent, is replicable, and represents best practices for the future. And it will require close collaborations between domain experts in the biologic and mathematical or statistical disciplines within EPA and with other agencies and stakeholders to establish clear criteria and guidance, including articulation of underlying assumptions, strengths, and weaknesses of each approach. The committee notes that care must be taken when combining results within or between studies in developing dose‒ response relationships inasmuch as multiple mechanisms, each with its own potential dose‒response relationship, might be involved. In such cases, clearly articulated expert judgment, criteria for expert selection, 11 Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Progress Toward Transforming the Integrated Risk Information System Program: A 2018 Evaluation and multidisciplinary collaborations need to be supported and used in the development and application of new mathematical approaches. The 2014 report recommended that EPA develop IRIS-specific guidelines to frame analysis and communication of uncertainty (NRC 2014). EPA has made substantial progress in developing and adopting tools to address uncertainty analysis and communication (slides 136‒138, Appendix C; poster D-6, Appendix D). It demonstrated its work during the workshop and focused on model uncertainty (slide 136, Appendix C) and the probabilistic distribution of toxicity values (slides 137‒138, Appendix C). It further indicated that the IRIS program intends to adopt the WHO/International Programme on Chemical Safety guidance (slide 137, Appendix C) and to provide various calculations when reporting toxicity values, including ranges of risk-specific toxicity values (slide 138, Appendix C) to demonstrate uncertainty. The committee recognizes that the steps taken are in the right direction for an evolving process and encourages EPA to continue to develop and test new tools in collaboration with other agencies and stakeholders. Equally important, the committee encourages EPA to continue its effort to frame uncertainty analysis and communications to address multiple sources of uncertainty surrounding toxicity values. CONCLUDING REMARKS Overall, the committee is encouraged by the steps that EPA has taken, which have accelerated during the last year under new leadership. During the workshop, the committee was impressed by the overall enthusiasm displayed by EPA staff and the substantive progress toward full implementation of systematic review and transparency in IRIS assessments. The committee fully appreciates that changing the process and implementing systematic-review procedures while producing final assessments is a huge challenge for any organization, especially in such a short period (12 months) and with a shrinking staff. Because new tools and approaches will ultimately be needed to implement past National Academies recommendations, especially for incorporating mechanistic information and for integrating evidence across studies, the committee is encouraged by IRIS program efforts to collaborate with other EPA staff, other government agencies, and academe to have the right mix of expertise to develop new approaches and best practices for conducting assessments. REFERENCES DuMouchel, W.H., and J.E. Harris. 1983. Bayes methods for combining the results of cancer studies in humans and other species: Rejoinder. J. Am. Stat. Assoc. 78(382):313-315. EPA (US Environmental Protection Agency). 2006. Approaches for the Application of Physiologically Based Pharmacokinetic (PBPK) Models and Supporting Data in Risk Assessment. EPA/600/R-05/043F. National Center for Environmental Assessment, Office of Research and Development, U.S. Environmental Protection Agency, Washington, DC. August 2006. Available: https://cfpub.epa.gov/ncea/risk/recordisplay.cfm?deid=157668 [accessed March 6, 2018]. EPA. 2010. IRIS Toxicological Review of Formaldehyde (Interagency Science Consultation Draft). EPA/635/R10/002C. U.S. Environmental Protection Agency, Washington, DC [online]. Available: https://cfpub.epa.gov/si/ si_public_record_report.cfm?direntryid=223603 [accessed February 27, 2018]. EPA. 2012. Toxicological Review of Tetrachloroethylene (Perchloroethylene) [CAS No. 127-18-4]. EPA/635/R80/011F. U.S. Environmental Protection Agency, Washington, DC. February 2012 [online]. Available: https://cfpub.epa.gov/ncea/iris/iris_documents/documents/toxreviews/0106tr.pdf [accessed February 27, 2018]. EPA. 2013. Part I: Status of Implementation of Recommendations. Materials Submitted to the National Research Council. Integrated Risk Information System Program, U.S. Environmental Protection Agency. January 30, 2013 [online]. Available: https://www.epa.gov/sites/production/files/2014-06/documents/iris_program_materials_to_nrc_part_1.pdf [accessed February 23, 2018]. EPA. 2017a. IRIS Assessment Plan for Chloroform [CASRN 67-66-3]. EPA/635/R-17/330. Integrated Risk Information System, National Center for Environmental Assessment, Office of Research and Development, U.S. Environmental Protection Agency, Washington, DC. 12 Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Responses to National Academies Recommendations EPA. 2017b. Toxicological Review of Benzo[a]pyrene. EPA/635/R-17/003Fa. Integrated Risk Information System, National Center for Environmental Assessment, Office of Research and Development, U.S. Environmental Protection Agency, Washington, DC [online]. Available: https://cfpub.epa.gov/ncea/iris/iris_documents/documents/toxreviews/0136tr.pdf [accessed February 14, 2018]. EPA. 2018a. A Message from the IRIS Program, January 2018. IRIS Recent Additions, January 25, 2018 [online]. Available: https://www.epa.gov/iris/iris-recent-additions [accessed February 14, 2018]. EPA. 2018b. Systematic Review Protocol for the IRIS Chloroform Assessment (Inhalation) [CASRN 67-66-3]. EPA/635/R-17/486. Integrated Risk Information System, National Center for Environmental Assessment, Office of Research and Development, U.S. Environmental Protection Agency, Washington, DC. EPA. 2018c. IRIS Assessment Plan for Uranium (Oral Reference Dose) (Scoping and Problem Formulation Materials) [CASRN 7440-61-1]. EPA/635/R-17/787. Integrated Risk Information System, National Center for Environmental Assessment, Office of Research and Development. IARC (International Agency for Research on Cancer). 2015. Polychlorinated Biphenyls and Polybrominated Biphenyls. IAEC Monographs on the Evaluation of Carcinogenic Risks to Humans Vol. 107 [online]. Available: http://monographs.iarc.fr/ENG/Monographs/vol107/mono107.pdf [accessed March 22, 2018]. IOM (Institute of Medicine). 2011. Finding What Works in Health Care: Standards for Systematic Review. Washington, DC: The National Academies Press. Jones, D.R., J. Peters., J.L. Rushton, A.J. Sutton, and K.R. Abrams. 2009. Interspecies extrapolation in environmental exposure standard setting: A Bayesian synthesis approach. Regul. Toxicol. Pharmacol. 53(3):217-225. National Academies of Sciences, Engineering, and Medicine. 2017a. Application of Systematic Review Methods in an Overall Strategy for Evaluating Low-Dose Toxicity from Endocrine Active Chemicals. Washington, DC: National Academies Press. National Academies of Sciences, Engineering, and Medicine. 2017b. Using 21st Century Science to Improve RiskRelated Evaluations. Washington, DC: National Academies Press. NRC (National Research Council). 2007. Toxicity Testing in the 21st Century: A Vision and a Strategy. Washington, DC: The National Academies Press. NRC. 2009. Science and Decisions: Advancing Risk Assessment. Washington, DC: The National Academies Press. NRC. 2011. Review of the Environmental Protection Agency’s Draft IRIS Assessment of Formaldehyde. Washington, DC: The National Academies Press. NRC. 2014. Review of EPA's Integrated Risk Information System (IRIS) Process. Washington, DC: The National Academies Press. NTP (National Toxicology Program). 2015a. Handbook for Conducting a Literature-Based Health Assessment Using OHAT Approach for Systematic Review and Evidence Integration. Office of Health Assessment and Translation (OHAT), Division of the National Toxicology Program, National Institute of Environmental Health Sciences, U.S. Department of Health and Human Services, January 9. Available: https://ntp.niehs.nih.gov/ntp/ohat/pubs/ handbookjan2015_508.pdf [accessed March 6, 2018]. NTP. 2015b. Handbook for Preparing Report on Carcinogens Monographs. Office of Health Assessment and Translation (OHAT), Division of the National Toxicology Program, National Institute of Environmental Health Sciences, U.S. Department of Health and Human Services, July 20, 2015. Available: https://ntp.niehs.nih.gov/ntp/ roc/handbook/roc_handbook_508.pdf [accessed March 6, 2018]. Orme-Zavaleta, J. 2018. Response to the Request for Correction (RFC). Letter to Robert Holden, Liskow & Lewis, New Orleans, LA, from Jennifer Orme-Zavaleta, Principal Deputy Assistant Administrator for Science, Office of Research and Development, Washington, DC, January 25, 2018; Attachment 1. EPA Response to the Denka Performance Elastomers (DPE) Request for Correction of the Toxicological Review of Chloroprene (CAS No. 12699-8) In Support of Summary Information on the Integrated Risk Information System (IRIS); Attachment 2. Systematic Review of Chloroprene [CASRN 126-99-80] Studies Published Since 2010 IRIS Assessment to Support Consideration of the Denka Request for Correction (RFC). January 2018 [online]. Available: https://www.epa.gov/ sites/production/files/2018-01/documents/epa_repsonse_to_mr._holdren_jan_25_2018_complete.pdf [accessed February 9, 2018]. 13 Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Appendix A Biographic Information on the Committee to Review Advances Made to the IRIS Process Jonathan M. Samet (Chair) is a pulmonary physician and epidemiologist. He is the dean of the Colorado School of Public Health and previously served as a professor and Flora L. Thornton Chair of the Department of Preventive Medicine of the Keck School of Medicine of the University of Southern California (USC) and director of the USC Institute for Global Health. His research has focused on the health risks posed by inhaled pollutants. He has served on numerous committees concerned with public health: the US Environmental Protection Agency Science Advisory Board; committees of the National Academies, including chairing the Biological Effects of Ionizing Radiation VI Committee, the Committee on Research Priorities for Airborne Particulate Matter, the Committee to Review EPA’s Draft IRIS Assessment of Formaldehyde, the Committee to Review the IRIS Process, and the Board on Environmental Studies and Toxicology; and the National Cancer Advisory Board. He is a member of the National Academy of Medicine. Dr. Samet received his MD from the University of Rochester School of Medicine and Dentistry. Richard A. Corley (retired) was a laboratory fellow at the Pacific Northwest National Laboratory operated by Battelle for the US Department of Energy. Dr. Corley specializes in the development of physiologically based pharmacokinetic models, real-time breath analysis, dermal and inhalation bioavailability, and the development of three-dimensional computational fluid-dynamic models of the respiratory system. He has published numerous peer-reviewed papers on oral, dermal, and inhalation toxicology; on modes of action of a variety of industrial and consumer chemicals; and on pharmacokinetic modeling and its applications in human health risk assessment. Dr. Corley has served on several National Academies committees, including the Committee to Assess the Health Implications of Perchlorate Ingestion, the Standing Committee on Risk Analysis Issues and Reviews, the Committee to Review EPA’s Draft IRIS Assessment of Formaldehyde , and the Committee to Review EPA's Draft State of the Science Paper on Nonmonotonic Dose Response. He received a PhD in environmental toxicology from the University of Illinois at Urbana-Champaign. George Daston is the Victor Mills Society Research Fellow at the Procter & Gamble Company. He has published over 100 articles and book chapters and edited five books in toxicology and risk assessment. His current research efforts are in toxicogenomics and mechanistic toxicology, particularly addressing how findings in these fields can improve risk assessment of chemicals and the development of nonanimal alternatives. Dr. Daston has served as president of the Teratology Society, as councilor and treasurer-elect of the Society of Toxicology, and on the US Environmental Protection Agency Science Advisory Board, the Board of Scientific Counselors of the National Toxicology Program, the National Academies Board on Environmental Studies and Toxicology, and the National Children’s Study Advisory Committee. He is editor-in-chief of Birth Defects Research: Developmental and Reproductive Toxicology. With scientists at the US Humane Society, Dr. Daston manages the AltTox Web site, which is devoted to the exchange of scientific information leading to the development of in vitro replacements for toxicity assessments. Dr. Daston has been awarded the Josef Warkany Lectureship and the Distinguished Service Award by the Teratology Society, the George H. Scott Award by the Toxicology Forum, and the Society of Toxicology’s Best Paper of the Year Award, and he is an elected fellow of the American Association for the Ad- 14 Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Appendix A vancement of Science. Dr. Daston is an adjunct professor of pediatrics at the University of Cincinnati. He earned his PhD in developmental biology from the University of Miami. David Dorman is a professor of toxicology in the Department of Molecular Biomedical Sciences at North Carolina State University. His research interests include neurotoxicology, nasal toxicology, pharmacokinetics, and cognition and olfaction in animals. He has served on advisory boards for the US Navy, the National Aeronautics and Space Administration, the US Department of Agriculture, and the National Toxicology Program. He has chaired several National Academies committees, including the Committee on Endocrine-Related Low Dose Toxicity, the Committee on Predictive-Toxicology Approaches for Military Assessments of Acute Exposures, and the Committee on Design and Evaluation of Safer Chemical Substitutions. He was also a member of the Committee to Review EPA’s Draft IRIS Assessment of Formaldehyde and the Committee to Review the IRIS Process. Dr. Dorman is an elected fellow of the Academy of Toxicological Sciences, is a fellow of the American Association for the Advancement of Science, and is a national associate of the National Academies of Sciences, Engineering, and Medicine. He received a DVM from Colorado State University and completed a combined PhD and veterinary toxicology residency program at the University of Illinois at Urbana-Champaign. Dr. Dorman is a diplomate of the American Board of Veterinary Toxicology and the American Board of Toxicology. Russ Hauser is the chair of the Department of Environmental Health, Frederick Lee Hisaw Professor of Reproductive Physiology, and professor of environmental and occupational epidemiology at the Harvard T.H. Chan School of Public Health. He also holds an appointment at the Harvard Medical School, where he is professor of obstetrics, gynecology, and reproductive biology. Dr. Hauser’s research focuses on the effects of environmental chemicals on reproductive health, pregnancy, and children’s health. He has served on several National Academies committees, including the Committee to Review EPA’s State of the Science Paper on Nonmonotonic Dose Response, the Committee on the Health Risks of Phthalates, and the Committee on Endocrine-Related Low-Dose Toxicity. Dr. Hauser was a member of two US Environmental Protection Agency Science Advisory Boards, served on the US Consumer Product Safety Commission’s Chronic Hazard Advisory Panel that examined the effects of phthalates on children’s health, and is an associate editor of Environmental Health Perspectives. He received his MD from the Albert Einstein College of Medicine and his MPH and ScD from the Harvard T.H. Chan School of Public Health, where he also completed a residency in occupational medicine. He is board-certified in occupational medicine. Karen A. Robinson is an associate professor at the Johns Hopkins University School of Medicine. She also serves as director of the Johns Hopkins University Evidence-Based Practice Center and is a member of the core faculty in the Center for Clinical Trials and Evidence Synthesis at the university’s Bloomberg School of Public Health. Her research focuses on evidence-based health care and evidence-based research. She conducts systematic reviews that are used to develop clinical practice guidelines and to inform other health decisions. She served on the National Academies Committee on Endocrine-Related Low-Dose Toxicity and Committee on Gulf War and Health: Treatment of Chronic Multisymptom Illness. Dr. Robinson received her MSc in health sciences from the University of Waterloo, Ontario, and her PhD in epidemiology from the Bloomberg School of Public Health. Richard P. Scheines is a professor of philosophy and dean of the Dietrich College of Humanities and Social Sciences of Carnegie Mellon University. His research focuses on causal discovery, specifically the problem of learning about causation from statistical evidence. Dr. Scheines also works in building and researching the effectiveness of educational software, including intelligent proof tutors and virtual causality laboratories, and a full-semester course on causal and statistical reasoning. Because of that work, he has a courtesy appointment in the Human-Computer Interaction Institute of Carnegie Mellon. He served on several National Academies committees, including the Committee to Review the IRIS Process. Dr. Scheines received a PhD in the history and philosophy of science from the University of Pittsburgh. 15 Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Progress Toward Transforming the Integrated Risk Information System Program: A 2018 Evaluation Lauren Zeise is director of the California Environmental Protection Agency’s Office of Environmental Health Hazard Assessment. She oversees the department’s activities, which include the development of risk assessments, hazard evaluations, toxicity reviews, cumulative impact analyses, frameworks and methods for assessing toxicity and cumulative effects of vulnerability and environmental exposures on communities, and the department’s activities in the California Environmental Contaminant Biomonitoring Program. Dr. Zeise was the 2008 recipient of the Society for Risk Analysis Outstanding Practitioners Award. She has served on advisory boards and committees of the US Environmental Protection Agency, the Office of Technology Assessment, the World Health Organization, and the National Institute of Environmental Health Sciences. Dr. Zeise has served on numerous National Academies committees, including the Committee on Toxicity Testing and Assessment of Environmental Agents and the Committee on Improving Risk Analysis Approaches Used by the U.S. Environmental Protection Agency. Dr. Zeise received a PhD from Harvard University. Yiliang Zhu is a professor in the Division of Epidemiology, Biostatistics, and Preventive Medicine of the University of New Mexico (UNM) School of Medicine. He directs the biostatistics, epidemiology, and research design cores for the UNM Clinical and Translational Research Center and for the Mountain West Clinical and Translational Research Infrastructure Network, a consortium of 13 universities in seven states. His research focuses on quantitative methods in health risk assessment, including integrative modeling of biologic systems, dose‒response modeling, benchmark-dose methods, and uncertainty quantification. He also conducts research in biostatistics methods, clinical- and health-outcome evaluation, and impact assessment of health-care systems and policies in northwestern rural China. Before joining UNM, Dr. Zhu was a professor at the University of South Florida College of Public Health where he directed the Biostatistics PhD program and the Center for Collaborative Research. Dr. Zhu has served on several National Academies committees, including the Committee on EPA’s Exposure and Human Health Assessment of Dioxin and Related Compounds, the Committee on Tetrachloroethylene, the Committee to Review EPA’s Draft IRIS Assessment of Formaldehyde, and the Committee to Review the IRIS Process. He received a PhD in statistics from the University of Toronto. 16 Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Appendix B Open Session Workshop Agenda COMMITTEE TO REVIEW ADVANCES MADE TO THE IRIS PROCESS SECOND MEETING Open Session: February 1-2, 2018 National Academies of Sciences, Lecture Room 2101 Constitution Ave, NW Washington, DC 20418 OPEN SESSION AGENDA 9:30 Purpose of Open Session and Introduction of Committee Members Jonathan Samet Chair, Committee to Review Advances Made to the IRIS Process Dean, Colorado School of Public Health 9:45 Introduction and Overview of Improvements to the IRIS Program Tina Bahadori Director, National Center for Environmental Assessment U.S. Environmental Protection Agency Kristina Thayer Director, Integrated Risk Information System (IRIS) Division U.S. Environmental Protection Agency 10:45 Discussion with National Academies Committee 11:30 Opportunity for Public Comments to National Academies Committee 12:00 Lunch Break 1:00 Session 1: Systematic Review in the IRIS Program – Evidence Identification EPA Panel Presentations and Discussion with the National Academies Committee on the Following Topics: Scoping, Problem Formulation, and Protocols Literature Searching, Screening, and Inventories 17 Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Progress Toward Transforming the Integrated Risk Information System Program: A 2018 Evaluation 2:00 Opportunity for Public Comments to National Academies Committee 2:15 Session 2: Systematic Review in the IRIS Program – Evidence Evaluation EPA Panel Presentations and Discussion with the National Academies Committee on the Following Topics: Evaluating Individual Studies: Reporting Quality, Risk of Bias, and Sensitivity Evaluating Confidence in a Body of Evidence: Evidence Synthesis and Integration to Reach Hazard Conclusions 3:15 Opportunity for Public Comments to National Academies Committee 3:30 Break 3:45 Session 3: Development and Application of Specialized Tools for Systematic Review EPA Panel Presentations and Discussion with the National Academies Committee 4:30 Opportunity for Public Comments to National Academies Committee 5:00 Break 5:307:00 Poster Session and Demonstrations, West Court FRIDAY, FEBRUARY 2, 2018 8:30 Welcome and Recap from First Day Jonathan Samet Chair, Committee to Review Advances Made to the IRIS Process Dean, Colorado School of Public Health 8:45 Session 4: Study Selection for Developing Toxicity Values, and Advancing Research on Quantitative Analyses for Evidence Integration and Dose-Response Analyses EPA Panel Presentations and Discussion with the National Academies Committee 10:00 Opportunity for Public Com ments to National Academies Committee 10:15 Break 10:30 Collaborations, Training, and Final Thoughts Tina Bahadori Director, National Center for Environmental Assessment U.S. Environmental Protection Agency Kristina Thayer Director, Integrated Risk Information System (IRIS) Division U.S. Environmental Protection Agency 18 Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Appendix B 11:00 Discussion with National Academies Committee 11:45 Opportunity for Public Comments to National Academies Committee 12:30 Adjourn 19 Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Appendix C Presentations by US Environmental Protection Agency 20 Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Appendix C NATIONAL ACADEMY OF SCIENCES COMMITTEE TO REVIEW ADVANCES MADE TO THE IRIS PROCESS February 1-2, 2018 Office of Research and Development NCEA, IRIS INTRODUCTION AND OVERVIEW OF IMPROVEMENTS TO THE IRIS PROGRAM Tina Bahadori* and Kris Thayer [*Speaking] Office of Research and Development NCEA, IRIS 21 Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Progress Toward Transforming the Integrated Risk Information System Program: A 2018 Evaluation  • Created in 1985 to foster consistency in the evaluation of chemical toxicity across the Agency. • IRIS assessments contribute to decisions across EPA and other health agencies. • Toxicity values – Noncancer: Reference Doses (RfDs) and Reference Concentrations (RfCs). – Cancer: Oral Slope Factors (OSFs) and Inhalation Unit Risks (IURs). • IRIS assessments have no direct regulatory impact until they are combined with – Extent of exposure to people, cost of cleanup, available technology, etc. – Regulatory options. – Both of these are the purview of EPA’s program offices. 2 IRIS Provides Scientific Foundation for Agency Decision Making  Clean Air Act (CAA)  Safe Drinking Water Act (SDWA)  Food Quality Protection Act (FQPA) IRIS  Comprehensive Environmental Response, Compensation, and Liability Act (CERCLA)  Resource Conservation and Recovery Act (RCRA)  Toxic Substances Control Act (TSCA) Broad Input to Support • Agency Strategic Goals • Children’s Health • Environmental Justice 3 22 Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Appendix C New Leadership Structure in NCEA • In January 2017, EPA appointed new leadership to the National Center for Environmental Assessment and to its IRIS Program. – NCEA Director: significant experience in the chemical and energy industries, and formerly the Director of ORD’s Chemical Safety for Sustainability National Research Program, Tina Bahadori brings knowledge of TSCA, innovative applications of computational toxicology, and exposure science. – IRIS Program Director: As a recognized leader in systematic review, automation, and chemical evaluations, Kris Thayer brings experience in early partner and stakeholder engagement and input, and demonstrated actions to increase capacity and transparency in assessments. • Improved responsiveness and accountability through Senior Leadership Team. • Integrating across the spectrum of human and ecological RA practices. 4 Drivers for this Study https://www.gao.gov/highrisk/transforming_epa_and_toxic_chemicals/why_did_study Fiscal Year 2017 Appropriations https://www.congress.gov/114/crpt/srpt281/CRPT-114srpt281.pdf 5 23 Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Progress Toward Transforming the Integrated Risk Information System Program: A 2018 Evaluation  NAS (2014) Overarching Statements 2014 em Opvreorcaelsl,sthhaevceobmeemnitm teaedfein,dasndthiattissucblesatrantthiaatl iEmPpArohvaesm enbtrsaicnetdhaenIdRIiSs acting o n t h e r e c o m m e nd a t io ns in the NRC formaldehyde report. The NRC process have been made formaldehyde committee recognized that its suggested changes would take several years and an extensive effort by EPA staff to implement. Substantial Tphreog[2re0s1s1,]hcoowmem zeaddethiantaitsshsourgtgteismteed...”ch[apn.9g]es would take veitrt,eheasrebceoegnnim several years and an extensive effort “EPA has not only responded to the recommendations made in the NRC formaldehyde report but is well on the way to meeting the general EsPyAstheamsantioct-roenvliyew restpaonnddaerdsto fotrhiedernetciofym inm g eannddaatsiosensssm inagdeeviidnetnhcee.N ” R[pC. 51] y to m ee t i n g t h e g e n e r al f o r m ald e h y d e r e p o rt , b u t i s w el l o n t h e w a “... the IRIS program has moved forward steadily in planning for and syim stpelm ateicntrienvgiecw em hasntgaensdianrdesach element of the assessment process. The committee is confident that there is an institutional commitment to pdlefotirnw g atrhde srrte eevvaiidsiolynisnopflathneninpgrofo cersasn...d” i[[m p.p1l3 3e5m]enting changes in each mcovmep ntmomfitthteeeacssoemssmmeenndtspErPoA cefsosr its substantive new approaches, “eTlhem e ceo continuing commitment to improving the process, and successes to Tdhaetec.oO mvmeirtatlelethceom nditsteEePA veillnceowmapplperteoaitcshepsla..n.tnheed com mem exfpoerctitsstshuabtsE EtaPnAtiw w rerevvisiisoionnsswinillatrtaim nsefloyrw matyhaaenIdRItShaPtro w thgerarm evisions will transform the IRIS Program.” [p.135] 6 Previous Phased Improvements to the IRIS Program • Revising the structure of assessments to enhance the clarity and transparency of presentation: - - • Detailing the methods underlying each step of draft development (e.g., literature search strategy). Restructuring the document into separate hazard identification and dose-response chapters. Replacing lengthy study summaries with synthesis text, supported by standardized tables and graphs. Implementing “IRIS Enhancements” – An updated process for developing and reviewing assessments that increases public input and peer consultation at earlier stages of assessment development, and clarifies processes for considering new evidence and scientific issues. • Establishing the SAB Chemical Assessment Advisory Committee (CAAC) • – 5 IRIS assessments completed CAAC review since 2014. Restructuring the IRIS Program to create expertise-specific workgroups and improved assessment oversight. 24 Copyright National Academy of Sciences. All rights reserved. 7 Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Appendix C Quality Management • Assessment Development and Review – Quality management inherent to systematic review methodology (e.g., independent screening of studies) – Rigorous review process includes internal, public, and external peer review • Scientific Support Teams – Systematic review methods (Systematic Review Workgroup) – Systematic review support to chemical assessment teams (e.g., screening, study evaluation, data extraction, use of specialized software, etc. – train the trainer model) – Discipline-specific workgroups (e.g., epidemiology, PBPK, neurotoxicology, etc.) – Executive oversight • • Roles and Responsibilities – Assessment plans, protocols, and draft assessments indicate contributors and roles – Given current budget there is very limited use of contract support to conduct assessments Training – regular training via skill-building seminars, focused discussions, and retreats 8 Improved Practices for Timeliness and Resource Management Current Program and Project Management in IRIS: - Centralized communication processes for providing staff with updates on near-term priorities, template materials, and other process-oriented decisions. - Development and maintenance of templates and checklists for key steps of assessment development using Microsoft SharePoint and Project as collaborative, web-based tools for assessment teams and project managers (document management and storage; scheduling support). - Dedicated IRIS Program staff and on-site programmatic contractor support to facilitate continued implementation of program and project management principles. Program and Project Management Communication Tools and Templates Assessment Team Support 9 25 Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Progress Toward Transforming the Integrated Risk Information System Program: A 2018 Evaluation  GAO 2017 Report • Acknowledged the actions ORD has taken to enable the IRIS Program to produce timely, transparent, and credible assessments in support of EPA’s mission. • Discussions with GAO during and after the release of the 2017 High Risk Report have focused on approaches to demonstrate how management and integrity initiatives within IRIS are supporting the transformation of the program Summary of 2015 and 20 17 GAO High Risk Criteria Ratings of the IRIS Program GAO High Risk Criteria Leadership Commitment Monitoring Action Plan Demonstrated Progress Capacity 2015 Rating Met Partially Met Partially Met Not Met Not Met 2017 Rating Met Met Partially Met Partially Met Partially Met • IRIS is engaged in continual ongoing discussion with GAO regarding recommendations from the 2008, 2012, and 2013 reports. • Of the seventeen recommendations issued in these three reports, as of June 2017, we have successfully closed ten recommendations and are rapidly moving to address the remaining seven. 10 IRIS Multi-Year Agenda • Released to the public December 2015 – Result of a survey EPA program and regional offices for their assessment needs balanced with resource availability. – Other chemicals were also carried over from earlier prioritizations – Reflects global priorities • In FY 2018, reaffirm priorities; identify new or more urgent needs. • Engage states. Group  Chemicals  Manganese  Mercury/methylmercury  1  Nitrate/nitrite  Perfluoroalkyl compounds  Vanadium and compounds  Acetaldehyde  2  Ammonia (oral)  Cadmium and compounds  Uranium  Di‐(2‐ethylhexyl) phthalate  Dichlorobenzene isomers  3  Methyl t‐butyl ether (MTBE)  Nickel and compounds  Styrene  26 Copyright National Academy of Sciences. All rights reserved. 11 Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Appendix C A Portfolio Approach • Moving away from a ‘one-size-fits-all’ approach to risk assessment towards a spectrum of assessment products to meet specific decision contexts; • Facilitating the incorporation of new science into risk assessment and decision-making; • Enabling assessments to be better tailored to meet needs of decision makers; • Increasing the number of chemicals that can be evaluated for their effects on human health by utilizing constrained resources in the most efficient manner. 12 Leading Edge of Science – Systematic Review NAS 2017: Reflections and Lessons Learned from the Systematic Review “….one disadvantage in conducting a systematic review is that it can be time and resource intensive, particularly for individuals that have not previously conducted a systematic review.” [p.157] “The committee discussed at length whether it could provide EPA with advice about when a systematic review should be performed but decided it could not be more specific because that decision will depend on the availability of data and resources, the anticipated actions, the time frame for decision making, and other factors.” [p.157] “The committee also recognized that it might be advantageous for EPA to build on existing systematic reviews that are published in the peer-reviewed literature.” [p.157] “The committee recognizes that the methods and role of systematic review and meta-analysis in toxicology are evolving rapidly and EPA will need to stay abreast of these developments, strive for transparency, and use appropriate methods to address its questions.” [p.157] 13 27 Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Progress Toward Transforming the Integrated Risk Information System Program: A 2018 Evaluation  Leading Edge of Science – New Data Streams Next Generation IRIS • IRIS in the 21st Century – implement recommendations of the NAS 2017 report, Using 21st Century Science to Improve Risk-Related Evaluations; • New Approach Methods – see poster session • Collaborate with Tox21 – build expert-judgment case studies that inform assessment development and fill gaps in assessments, especially for data poor chemicals; – inform where resources should be strategically invested to generate additional data. • Create efficiencies – engage other agencies to share common practices, data, and tools, and more efficiently leverage resources across the federal government. • Refresh science – MOU’s with academia and other federal agencies; strategic staffing; deeper engagement with health agencies in states. 14 How is IRIS Evolving? • Increase transparency and full implementation of systematic review – implement using approaches that foster consistency across the IRIS Program; many active and all new starts address systematic review-related recommendations of 2014 NAS report • Modernize the IRIS Program – through automation and machine learning to expedite systematic review, incorporation of emerging data types • Modularize product lines – implement a portfolio of chemical evaluation products that optimize the application of the best available science and technology. These products will allow IRIS to remain flexible and responsive to clients within the EPA as well the diverse collection of stakeholders beyond EPA, including states, tribal nations, and other federal agencies. • Enhance accessibility – provide outreach and training to make systematic review practices ubiquitous and more accessible; enhance data sharing through publicly available software platforms for assessments developed by EPA, other federal and state agencies, industry, academia and other thirdparties. 15 28 Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Appendix C IRIS has Addressed the Major NAS 2014 Recommendations NAS 2014 Topics IRIS Process Improvements General Process • Quality management pipeline implemented Issues • Program and project management processes implemented (Chapter 2) • Frequent opportunities for stakeholder engagement Future Directions (Chapter 8 “Lessons Learned” and “Looking Forward”) • Processes being implemented include flexibility to incorporate evolving methods in systematic review and risk assessment • Increased collaboration with federal partners and international experts prevents duplication of effort and maintains cutting edge approaches • Current research efforts and training serve to ensure that methods and staff are able to adapt to changing scientific contexts and sources of evidence, including new and emerging data types 16 SESSION 1: SYSTEMATIC REVIEW IN THE IRIS PROGRAM - EVIDENCE IDENTIFICATION Kris Thayer*, Andrew Kraft*, April Luke, Beth Radke, Michele Taylor [*Speaking] Office of Research and Development NCEA, IRIS 29 Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Progress Toward Transforming the Integrated Risk Information System Program: A 2018 Evaluation  Systematic Review A structured and documented process for transparent literature review1 “As defined by IOM [Institute of Medicine], systematic review ‘is a scientific investigation that focuses on a specific question and uses explicit, pre-specified scientific methods to identify, select, assess, and summarize the findings of similar but separate studies.’” [p. 4] (NRC, 2014) 1 Institute of Medicine. Finding What works in Health Care: Standards for Systematic Reviews. p.13-34.The National Academies Press.Washington, D.C. 2011 18 Systematic Review Elements (NAS 2014) "In the context of IRIS, the committee has defined systematic review as including protocol development, evidence identification, evidence evaluation, and an analytic summary of the evidence” Systematic Reviews NAS 2014, Figure S-1 IRIS also considers these phases as part of its systematic review process 19 30 Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Appendix C Scoping, Problem Formulation, and Protocol Development Office of Research and Development NCEA, IRIS IRIS Systematic Review Documents Address several NAS 2014 High Priority (Box 8-1) Recommendations • “EPA needs to...complete documents, such as the draft handbook, that provide detailed guidance for developing IRIS assessments.” (Chapter 2, General Process) • “EPA should include protocols for all systematic reviews conducted for a specific IRIS assessment as appendixes to the assessment.” (Chapter 3, Problem Formulation and Protocol Development) IRIS Handbook: Standard operating procedures and considerations Scoping Systematic Review Protocol Literature Inventory Study Evaluation Data Extraction Evidence Integration Derive Toxicity Values Assessment Developed Assessment Initiated Initial Problem Formulation Assessment Plans: What the assessment will cover Literature Search, Screen Refined Evaluation Plan Organize Hazard Review Evidence Analysis and Synthesis Select and Model Studies Protocols: How the assessment will be conducted 21 31 Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Progress Toward Transforming the Integrated Risk Information System Program: A 2018 Evaluation  IRIS Handbook IRIS Handbook: Approaches and considerations for applying principles of systematic review to IRIS assessments, general frameworks, and examples. Scoping Systematic Review Protocol Literature Inventory Study Evaluation Data Extraction Evidence Integration Derive Toxicity Values Assessment Developed Assessment Initiated Initial Problem Formulation Literature Search, Screen Refined Evaluation Plan Organize Hazard Review Evidence Analysis and Synthesis Select and Model Studies • IRIS Handbook level of detail aimed for EPA staff and contractors, e.g., use of HERO, timelines for internal review steps, etc. • Currently being updated to reflect Agency input, evolving IRIS practices as systematic review approaches are tested through implementation, and public comment received on chemical-specific protocols (e.g., chloroform) • • Evergreen to reflect future advances Anticipate public release in 2018 22 IRIS Assessment Plans and Protocols Scoping Systematic Review Protocol Literature Inventory Study Evaluation Data Extraction Evidence Integration Derive Toxicity Values Assessment Developed Assessment Initiated Initial Problem Formulation Assessment Plans: What the assessment will cover • • Literature Search, Screen Refined Evaluation Plan Organize Hazard Review Evidence Analysis and Synthesis Select and Model Studies Protocols: How the assessment will be conducted (specific procedures and approaches for each assessment component, with rationale where needed) Chemical-specific documents IRIS Assessment Plans (IAPs) are problem formulation and scoping documents that include more elements of systematic review • • Protocols outline methods, including updates to the IAPs • Templates created to promote consistency across the IRIS Program, which is implemented across NCEA divisions and geographical locations IAPs and protocols include proposed “modularity,” targeted focus and use of existing assessments 32 Copyright National Academy of Sciences. All rights reserved. 23 Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Appendix C IRIS Assessment Plans, Protocols, and 7-Step IRIS Process Early Step 1: IRIS Assessment Plans • What the assessment covers • 30-day public comment period + public science meeting Mid-Step 1: Protocols • How the assessment will be conducted • 30-day public Opportunities for Public Comment comment 24 https://www.epa.gov/iris/basic-information-about-integrated-risk-information-system#process IRIS Assessment Plan (IAP) Scoping Systematic Review Protocol Literature Inventory Study Evaluation Data Extraction Evidence Integration Derive Toxicity Values Assessment Developed Assessment Initiated Initial Problem Formulation Assessment Plans: What the assessment will cover • Literature Search, Screen Refined Evaluation Plan Organize Hazard Review Evidence Analysis and Synthesis Select and Model Studies Scoping and initial problem formulation determinations – Background and Agency need, exposure context, objectives and specific aims, key areas of scientific complexity – Includes draft PECO (Populations, Exposures, Comparators, and Outcomes) criteria which outlines evidence considered most pertinent – Internal review of IAP fosters early and focused Agency engagement • Released for a 30-day public comment period + public science discussion (beginning of IRIS Step 1) – Examples: chloroform, ethylbenzene, nitrate/nitrite (Sept 2017), uranium (Jan 2018) 33 Copyright National Academy of Sciences. All rights reserved. 25 Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Progress Toward Transforming the Integrated Risk Information System Program: A 2018 Evaluation  26 IRIS Assessment Plan (IAP) Content From draft uranium IAP (2018) 34 Copyright National Academy of Sciences. All rights reserved. 27 Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Appendix C IAP Can Include Literature Surveys 3  7  5  1  10  4  6  Chronic  1  2  6  1  Developmental  14  Endocrine(thyroid)  6  1  25  3  1  Hematological  Surveys may be developed based on other assessments, manual review of studies, or through use of specialized software applications 3  13 Dermal and ocular  Gastrointestinal  4  Acute  1  Gestational  1  Multi‐generational  60  Cardiovascular  Short‐term  Cancer  Case reports and  case series reports  Outcomes  Surveys inform decisions on targeted focus, e.g., evidence streams to consider core-PECO (versus supplemental), health outcomes likely covered in assessment Animal Stud es  Subchronic  • Human Studies  Controlled exposure  • Nitrate/Nitrite (survey based on IARC 2010 and ATSDR 2017 assessments) Broad surveys to assess extent and nature of evidence, level of effort, type of expertise required Occupational  epidemiology  General population  epidemiology  • Hepatic  3  3  1  2  Immunological  Metabolic disease  8  Musculoskeletal  Neurological and sensory  1  6  1  Renal  1  1  1  Reproductive  3  2  2  9  2  1  Respiratory  Other  1  1  The numbers represent the numbers of studies that investigated a particular  health effect, and not the number of studies that identified a positive  association with exposure. 28 Protocols Scoping Systematic Review Protocol Literature Inventory Study Evaluation Data Extraction Evidence Integration Derive Toxicity Values Assessment Developed Assessment Initiated Initial Problem Formulation Literature Search, Screen Refined Evaluation Plan Organize Hazard Review Evidence Analysis and Synthesis Select and Model Studies Protocols: How the assessment will be conducted (specific) • Assessment specific stand-alone method documents that do not rely on the IRIS Handbook to convey methodology • Comments received on IAP are considered when preparing the protocol (updated IAP text is included in the protocol) • • Released for 30-day public comment period (during Step 1 of IRIS Process) • Protocol is iterative - Knowledge gained during implementation may result in revisions to the protocol to focus on the best available evidence. Major revisions are documented via updates, e.g., changes to specific aims or PECO List of included, excluded, and studies tagged as supplemental will be disseminated through protocols (either during initial release or as an update) 29 35 Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Progress Toward Transforming the Integrated Risk Information System Program: A 2018 Evaluation  Protocol Content Updated IAP text and PECO based on public comments From draft chloroform protocol (2018) 30 Protocol Content From draft chloroform protocol (2018) 36 Copyright National Academy of Sciences. All rights reserved. 31 Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Appendix C Publicly Available Examples Assessment Plans September 27-28, 2017 • Chloroform • Nitrate/nitrites • Ethylbenzene January 26, 2018 • Uranium Protocol • Targeted focus: chloroform, uranium, chloroprene • Modularity: ethylbenzene • Use of existing assessments conducted by others: nitrate/nitrate, uranium (ATSDR assessments) • IAPs and/or protocols will be released for most inprogress assessments • Which document is released depends on extent of refinement in scope compared to previous public sharing and maturity of the draft assessment January 26, 2018 • Chloroform (includes list of included studies) Rapid systematic review • EPA response to the Chloroprene Request for Correction (posted January 29, 2018) 32 Literature Searching, Screening, and Inventories* Office of Research and Development NCEA, IRIS * includes basic methodological details 37 Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Progress Toward Transforming the Integrated Risk Information System Program: A 2018 Evaluation  NAS 2014: High Priority (Box 8-1) Recommendations “...include a section on evidence identification that is written in collaboration with information specialists trained in systematic reviews and that includes a search strategy for each systematicreview question being addressed in the assessment. Specifically, the protocols should provide a line-by-line description of the search strategy, the date of the search, publication dates searched, and explicitly state the inclusion and exclusion criteria...” Scoping Systematic Review Protocol Literature Inventory Study Evaluation Data Extraction Evidence Integration Derive Toxicity Values Assessment Initiated Assessment Developed Initial Problem Formulation Literature Search, Screen Refined Evaluation Plan Organize Hazard Review Evidence Analysis and Synthesis Select and Model Studies • Protocols outline the specifics of the literature search and screening approaches, including inclusion and exclusion criteria in PECO tables • Dedicated information technologists help formulate searches, and screening decisions are tracked in HERO (tagging) • Manual and semi-automated approaches are being used to identify relevant studies • Inventories of basic study methods organize evidence for refinement and evaluation 34 • Changes and updates are documented in the protocol Routine Evidence Identification Processes Inventories Database Searches • Identify peer-reviewed and “gray” (unpublished) literature • PubMed,ToxLine, and Web of Science are standard (others can be included as needed) • Conduct regular search updates • Details of search strategy, dates, and retrieved records are presented in protocols and assessments Screening 1.Title/abstract 2. Full text • Use manual and automated approaches • ≥ 2 screeners • Tag studies as excluded, meeting PECO criteria, or supplemental information • Screening decisions available in HERO • Typically do not apply language-restrictions • Review reference list of included studies and relevant reviews to identify studies missed from database searches • Share list of included studies with public to further ensure all relevant studies included Health Outcome & PBPK Studies • Tag studies by line of evidence and outcome • Distribute to disciplinary experts for review Supplemental Studies • Includes in vitro and other mechanistic evidence (e.g., non-PECO exposure route; non-PECO animal model; toxicokinetic data) • Inventories contain basic study methods for evaluation and prioritization decisions 35 38 Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Appendix C Use of Specialized Software Tools for Literature Search and Screening HERO Literature searching, storage and documentation (tagging) Scoping Systematic Review Protocol Literature Inventory Study Evaluation Data Extraction Evidence Integration Derive Toxicity Values Assessment Developed Assessment Initiated Initial Problem Formulation Literature Search, Screen Refined Evaluation Plan Organize Hazard Review Evidence Analysis and Synthesis Select and Model Studies SWIFT Review Problem formulation • Software tools will be discussed in SWIFT Review Screening prioritization • Tools are being developed and Session 3 and during demo session Distiller (manual) SWIFT Active (SWIFT is a semi-automated/ machine learning tool) Multiple reviewer reference screening and tracking (HERO-tagging) applied through testing • Evergreen - new tools compatible with HERO will be added as performance is characterized 36 Evidence Identification in Protocols special topics would include any specialized software tools 37 39 Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Progress Toward Transforming the Integrated Risk Information System Program: A 2018 Evaluation  PECO Criteria to Identify Studies Populationsa Example from the draft chloroform protocol 38 Example Literature Screening Form *Forms Independently Entered by 2 Reviewers* Draft example based on chloroform using Distiller 40 Copyright National Academy of Sciences. All rights reserved. 39 Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Appendix C Tracking: Literature Flow Diagrams • Track rationale for fulltext exclusions • Use HERO to share repositories of included, excluded, and supplemental studies Example modeled on the draft chloroform protocol 40 Literature Inventories Example Details Routinely Extracted (female reproductive toxicity in animals): • Outcome category (e.g., fertility) and/or Specific endpoint (e.g., number of litters) • Species (e.g., rat; alternative [nonmammalian] animal) • Exposure duration (e.g., chronic; multi-generational; gestational) • Exposure route (e.g., oral [gavage]; in vitro) Assessment-Specific Extraction Details (generic examples): • Exposure levels tested • Test article details, such as purity or isomeric composition Results are Typically Not Included in Inventories Developing Extraction Forms (all 3 lines of evidence) to be interoperable with HAWC 41 41 Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Progress Toward Transforming the Integrated Risk Information System Program: A 2018 Evaluation  Refined Evaluation Plan (optional) Discipline-specific experts consider whether and how to further refine or prioritize studies/outcomes for evaluation (based on study design features) • Health effect studies meeting PECO criteria (e.g., organized by outcome): - Considers ADME and other key science issues (supplemental studies reviewed) - Opportunity to discuss outcome grouping (e.g., based on known biology/MOA) and handling of key science issues during outcome-specific study evaluations - Studies with certain design features or specific outcomes may be selected or prioritized for evaluation and synthesis (e.g., based on exposure duration, administration, or levels tested; or endpoint specificity) • Supplemental mechanistic studies (e.g., organized by test system, mechanistic event, or key characteristic [of carcinogens]) are considered iteratively: - Identifies other studies on specific aim mechanistic questions (e.g., mutagenicity) - Organizes the available evidence to allow for pragmatic evaluations of key issues that arise during review of PECO-specific human and animal studies (Session 2) Refinements are tracked and updated in the assessment protocol 42 IRIS has Addressed the Major NAS 2014 Recommendations NAS 2014 Topics IRIS Process Improvements • Draft IRIS Handbook of program SOPs is being reviewed within EPA General Process Issues (Chapter 2); • IAPs allow early comment on problem formulation Problem Formulation • More frequent Agency engagement facilitates scope refinement and Protocol • Assessment protocols describe methods and allow for iteration Development • Re-occurring staff training and template IAPs and protocols promote (Chapter 3) Evidence Identification (Chapter 4) consistency and quality control • Consultation with information technologists and subject experts • Adopts current systematic review best practices, including use of specialized tools • Transparent documentation (e.g., literature flow diagrams) See Demonstrations: • Sciome Workbench for Interactive computer-Facilitated Text mining (SWIFT Review and SWIFT Active) • Health Assessment Workspace Collaborative (HAWC) • Heath Effects Research Online (HERO) 42 Copyright National Academy of Sciences. All rights reserved. 43 Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Appendix C SESSION 2: SYSTEMATIC REVIEW IN THE IRIS PROGRAM- EVIDENCE EVALUATION Xabier Arzuaga*, Catherine Gibbons*, Barbara Glenn*, Andrew Kraft*, Beth Radke*, Kris Thayer [*Speaking] Office of Research and Development NCEA, IRIS Evaluating Individual Studies: Reporting Quality, Risk of Bias, and Sensitivity Office of Research and Development NCEA, IRIS 43 Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Progress Toward Transforming the Integrated Risk Information System Program: A 2018 Evaluation  NAS 2014 High Priority (Box 8-1) Recommendations on Evidence Evaluation “When considering any method for evaluating individual studies, EPA should select a method that is transparent, reproducible, and scientifically defensible.Whenever possible, there should be empirical evidence that the methodologic characteristics that are being assessed in the IRIS protocol have systematic effects on the direction or magnitude of the outcome.” “EPA should specify the empirically based criteria it will use to assess risk of bias for each type of study design in each type of data stream.” “To maintain transparency, EPA should publish its risk-of-bias assessments as part of its IRIS assessments.” 46 Study Evaluation – Developing an Approach • • • Considered and drew from existing tools for study evaluation. Developed approaches for both epidemiology and toxicity studies that: – Addresses study sensitivity and identifies potential sources of bias. – Transparently presents the criteria/considerations used to consistently evaluate and judge each study/outcome. – Provides access to the rationale for discipline-specific decisions made during the evaluation process. Objective of the approach: Identify the most informative and reliable studies for evidence synthesis and integration. 47 44 Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Appendix C PBPK Model Evaluation Criteria Prior to use, relevant PBPK models will: Example information Biological basis for the model is accurate  e.g., Predicts dose metrics expected to be relevant Scientific • Be thoroughly evaluated based Consideration of model fidelity to the biological system strengthens the scientific basis relative to standard extrapolation (default) approaches  e.g., Can the model describe critical behavior, such as nonlinear kinetics in a relevant dose range, better than the default (i.e., BW3/4 scaling)? on scientific and technical criteria (examples to the left). • Undergo QA/QC on model equations, parameters (including primary/secondary sources), and model code. Principle of parsimony (i.e., model complexity or biological scale should be commensurate with data available to identify parameters) Model describes existing PK data reasonably well, both in “shape” (e.g., matches curvature) and quantitatively (e.g., within a factor of 2−3) For details, please see: Model equations are consistent with biochemical and biological understanding • Poster: Well-documented model code is readily available to EPA and public Systematic evaluations of PBPK models for human health risk assessment Set of published parameters clearly identified, including origin/derivation Parameters do not vary unpredictably with dose • e.g., Any dose dependence in absorption constants is predictable across the Initial dose ranges relevant for animal and human modeling technical • EPA website: Sensitivity and uncertainty analysis has been conducted for relevant exposure levels (local sensitivity analysis is sufficient, though global preferred)  e.g., A sound explanation should be provided when sensitivity of the dose metric to model parameters differs from what is reasonably expected EPA Response to the Request for Correction of the IRIS Toxicological Review of Chloroprene (2018) 48 Evolving Approaches NTPOHAT NTPORoC EPAIRIS SciRAP ROBINS-I ToxRTool NavGuide EFSA 45 Copyright National Academy of Sciences. All rights reserved. 49 Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Progress Toward Transforming the Integrated Risk Information System Program: A 2018 Evaluation  Evaluation of Individual Health Effect Studies Scoping Systematic Review Protocol Literature Inventory Study Evaluation Data Extraction Evidence Integration Derive Toxicity Values Assessment Developed Assessment Initiated Initial Problem Formulation • • Literature Search, Screen Refined Evaluation Plan Organize Hazard Review Evidence Analysis and Synthesis Select and Model Studies General approach same for human and animal studies Evaluation process focused on: – Internal validity/bias – Sensitivity – Reporting quality 50 Development of Evaluation Strategies • • • Questions in IRIS Protocol Template highlight general study attributes or elements to consider Subject-matter knowledge is used to formulate a list of issues to consider in the evaluation Develop a set of considerations based on exposure and outcomespecific knowledge 51 46 Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Appendix C Study Evaluation Overview of Epidemiological and Animal Toxicity studies Individual study level domains Animal Epidemiological Reporting Quality Exposure measurement Selection or Performance Bias Outcome ascertainment Confounding/Variable Control Population Selection Reporting or Attrition Bias Confounding Exposure Methods Sensitivity Analysis Outcome Measures and Results Display Sensitivity   Selective reporting Overall Study Rating Domain Judgment Good High + Adequate Medium - Poor Low -- Critically Deficient Uninformative 52 Individual Domain Ratings for Epidemiological and Animal Toxicity Studies IRIS Judgment Good + - -- Adequate Poor Critically Deficient How to interpret Appropriate study conduct relating to the domain & minor deficiencies not expected to influence results. A study that may have some limitations, but not likely to be severe or to have a notable impact on results. Identified biases or deficiencies interpreted as likely to have had a notable impact on the results or prevent reliable interpretation of study findings. A judgment that the study conduct relating to the domain introduced a serious flaw that is interpreted to be the primary driver of any observed effect or makes the study uninterpretable. Study is not used without exceptional justification. 53 47 Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Progress Toward Transforming the Integrated Risk Information System Program: A 2018 Evaluation  Overall Study Confidence Ratings for Epidemiological and Animal Toxicity Studies Rating High Medium Low Description No notable deficiencies or concerns identified; potential for bias unlikely or minimal and sensitive methodology. Possible deficiencies or concerns noted, but resulting bias or lack of sensitivity would be unlikely to be of a notable degree. Deficiencies or concerns were noted, and the potential for substantive bias or inadequate sensitivity could have a significant impact on the study results or their interpretation. Uninformative Serious flaw(s) makes study results unusable 54 General Considerations to Evaluate Outcomes from Animal Toxicology Studies   Domain Metric Reporting Quality Reporting of information necessary for study evaluation Selection or Performance Bias Allocation of animals to experimental groups Confounding/Variable Control Blinding of investigators, particularly during outcome assessment Control for variables across experimental groups Reporting or Attrition Bias Lack of selective data reporting and unaccounted for loss of animals Exposure Methods Sensitivity Characterization of the exposure to the compound of interest Utility of the exposure design for the endpoint of interest Outcome Measures and Sensitivity and specificity of the endpoint evaluations Results Display Usability and transparency of the presented data 48 Copyright National Academy of Sciences. All rights reserved. 55 Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Appendix C Epidemiology Study Evaluation • Approach based on the Cochrane Risk of Bias in Nonrandomized Studies of Interventions (ROBINS-I)1, modified for environmental and occupational exposures • Start by considering an “ideal” study for each domain, identifying “critical deficiencies”, then developing criteria to define other levels of confidence • Emphasis is on discerning bias that would produce a substantive change in the estimated effect estimate. Sterne, Hernan, et al. ROBINS-I: a tool for assessing risk of bias in non-randomized studies of interventions. BMJ 2016; 355:i4919. 1 56 Epidemiology Evaluation Domains Domain Core Question Exposure measurement Does the exposure measure reliably distinguish between levels of exposure in an appropriate time window? Outcome ascertainment Does the outcome measure reliably distinguish the presence or absence (or degree of severity) of the outcome? Population selection Is there evidence that selection into or out of the study (or analysis sample) was jointly related to exposure and outcome? Confounding Is confounding of the effect of the exposure likely? Analysis Does the analysis strategy and presentation convey the necessary familiarity with the data and assumptions? Are there concerns for study sensitivity? Sensitivity 57 49 Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Progress Toward Transforming the Integrated Risk Information System Program: A 2018 Evaluation  Example of Considerations by Domains Domain Core Question Exposure measurement Does the exposure measure reliably distinguish between levels of exposure in an appropriate time window? Examples of Prompting Questions: • Does the exposure measure capture the variability in exposure among the participants, considering intensity, frequency, and duration of exposure? • Does the exposure measure reflect a relevant time window? • Was exposure measurement likely to be affected by knowledge of outcome or by presence of the outcome (i.e., reverse causality)? Examples of Follow-up Questions: • Is the degree of exposure misclassification likely to vary by exposure level? • If there is a concern about the potential for bias, what is the predicted direction of the bias on the effect estimate? 58 59 50 Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Appendix C Study Evaluation: Final Review in HAWC Questions, instruction text, and drop down rating options are customizable by user 60 Individual Studies in HAWC Medium confidence Uninformative 51 Copyright National Academy of Sciences. All rights reserved. 61 Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Progress Toward Transforming the Integrated Risk Information System Program: A 2018 Evaluation  Study Evaluation Summary in HAWC 62 Publicly available examples • Initial and iterative improvements to study evaluation – – – – Ammonia, Inhalation (final 2016) RDX (peer review draft 2016) TBA (peer review draft 2017) ETBE (peer review draft 2017) • Current methods for study evaluation – Chloroform protocol (2018) – EPA Response to Chloroprene Request for Correction (2018) 63 52 Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Appendix C Evaluating Confidence in a Body of Evidence: Evidence Synthesis and Integration to Reach Hazard Conclusions Office of Research and Development NCEA, IRIS Synthesis and Integration of Evidence Linking Exposure and Health Effects: Purpose Scoping Assessment Initiated Systematic Review Protocol Initial Problem Formulation Literature Inventory Literature Search, Screen Study Evaluation Refined Evaluation Plan Data Extraction Organize Hazard Review Evidence Integration Evidence Analysis and Synthesis Derive Toxicity Values Select and Model Studies Assessment Developed Synthesis: To describe the types of information within each line of evidence (human, animal and mechanistic), and to analyze and present study results regarding a given health effect to facilitate integration judgments. • Decisions about the organization of the synthesis made prior to data extraction Narratives, but not study by study summaries • • Highlight information that informs the hazard evaluation Integration: To develop judgments regarding strength of evidence for a health effect across lines of evidence • A two-step process involving transparent and structured approaches for drawing summary conclusions across lines of evidence 53 Copyright National Academy of Sciences. All rights reserved. 65 Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Progress Toward Transforming the Integrated Risk Information System Program: A 2018 Evaluation  NAS 2014: Relevant Comments and Recommendations The NAS 2014 report discusses the complexities with organizing analyses around mechanism, noting that, “The history of science is replete with solid causal conclusions in advance of solid mechanistic understanding.” (NRC, 2014, p. 90). • The current approach focuses first on the available human and animal studies on health effects, incorporating mechanistic information at various stages of assessment development to clarify identified gaps in understanding (e.g., human relevance of animal-model data). “The risk-of-bias assessment of individual studies should be carried forward and incorporated into the evaluation of evidence among data streams.” (NAS 2014 Recommendation, Box 8-1) • The results of the evaluation of individual studies is a critical component of the current evidence synthesis processes and integration frameworks. 66 NAS 2014: Relevant High Priority (Box 8-1) Recommendations “EPA should continue to improve its evidence-integration process incrementally and enhance the transparency of its process. It should either maintain its current guided-expert-judgment process but make its application more transparent or adopt a structured (or GRADE-like) process...the committee does not offer a preference but suggests that EPA consider which approach best fits...” “EPA should expand its ability to perform quantitative modeling of evidence integration.” • The current approach continues to use a guided expert judgment process, but structured sets of categorical criteria for decision-making within that process are more explicitly defined. • The current frameworks, and documentation of decisions within these frameworks, enhance transparency, reproducibility, and comparability across health effects and assessments; these approaches are evolving within NCEA and across the field. • Current research activities include quantitative methods to integrate evidence across streams (e.g., Bayesian approaches; see Session 4) 67 54 Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Appendix C Synthesizing Evidence on Health Effects – Organization and Structure Some questions about the evidence • What outcomes are relevant to each health hazard domain and at what level (e.g., health effect or subgroupings) should synthesis occur? • What populations were studied (e.g., general population, occupations, life stages, species, etc.) and do responses vary? • Can study results be described across varying exposure patterns, levels, duration or intensity? • Are there differences in the confidence in study results for different outcomes, populations, or exposure? • Does toxicokinetic information explain differences in responses across route of exposure, other aspects of exposure, species, or life stages? • How might dose response relationships be presented (specific study results or across study results)? 68 Scientific Judgment in Analysis and Synthesis of Evidence Scoping Systematic Review Protocol Literature Inventory Study Evaluation Data Extraction Evidence Integration Derive Toxicity Values Assessment Developed Assessment Initiated Initial Problem Formulation • • • • Literature Search, Screen Refined Evaluation Plan Organize Hazard Review Evidence Analysis and Synthesis Select and Model Studies Synthesis of evidence is more than counting the number of “positive” and “negative” studies Must systematically consider the influence of bias and sensitivity when describing study results and synthesizing evidence Synthesis should primarily be based on studies of medium and high confidence (when available) Analysis should try to draw conclusions about the strength of evidence from findings across collections of studies 69 55 Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Progress Toward Transforming the Integrated Risk Information System Program: A 2018 Evaluation  Synthesis Considerations for Determining Strength of Evidence Epidemiology evidence Animal toxicology evidence Study evaluation conclusions (risk of bias, sensitivity) are incorporated into analyses of each of the following considerations (adapted Hill considerations): Consistency Effect magnitude/ precision Biological gradient/ dose-response Coherence Natural experiments pattern of response across Analyze across categories of: Rare,effect but important to highlight arge magnitudes can mitigate • • LExpected within and across studies • Related exposure can mitigate concerns endpoints • about Confidence insome studies’ results concerns bias; smaller effect size ab out ological understanding of organ bias and confounding bi • Given • Study sensitivity is not discounted outright • Adequate Results presented studies mayetc. als o or •disease Exposureacross levels, duration, •system precision can help rule out effect health effect is Informative human and animal h ealth Timing ofevidence exposureabout relativea to development clarify temporal patterns with exposure • Expected • explanation Populations/ species/levels lifestage relationships chance as analyzed and synthesized separa tely.of outcomes•is assessed during study Other explanatory factors Shape of dose-response depend on • • Results presented across curves studies, or evaluation phase Mechanistic evidence is synthesiz ed thatcombined informs in the conclusions regarding outcomes; monotonic increasing not alwa ys meta-analysis may mitigate the human and animal health effect evidence. concerns 70 expectedabout chance Temporality • • Synthesis Examples: Epidemiology TCE and kidney cancer: stratification by utility Highest exposure level graphed for each study RoC Monograph on Trichloroethylene. January 2015. https://ntp.niehs.nih.gov/go/797306 56 Copyright National Academy of Sciences. All rights reserved. 71 Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Appendix C Synthesis Examples: Epidemiology TCE and Kidney Cancer: stratification by exposure level EPA. 2011. Toxicological Review of Trichloroethylene 72 Synthesis Examples:Animal Toxicology Hormone Level Pathology Incidence Behavioral Function All Rodents All Rats All Rats Gestational By Species Medium/ High Confidence Low Confidence Juvenile/Adult 73 57 Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Progress Toward Transforming the Integrated Risk Information System Program: A 2018 Evaluation  Mechanistic Evidence “Mechanistic data represent a wide variety of studies not intended to identify an adverse outcome.” (NRC, 2014) – When evaluating mechanistic evidence, the scope is larger than “in vitro” data – Mechanistic inventories collected at earlier stages may include: • In vivo (cellular, biochemical, molecular) • In vitro or ex vivo (human or animal tissues or cells) • Non-animal or non-mammalian alternative animal models • Big data (‘omics or high-throughput assays) • “Intervention” studies (pharmacologic, environmental, genetic) “…there might be hundreds of in vitro and other mechanistic studies of a given chemical…” (NRC, 2014) “For a given chemical, multiple mechanisms might be involved in a given end point, and it might not be evident how different mechanisms interact in different species to cause the adverse outcome.” (NRC, 2014) 74 Systematic review of mechanistic information requires a different approach “When human data are nonexistent, are mixed, or consistently show no association and an animal study finds a positive association, the importance of mechanistic data is increased...” (NRC, 2014) To narrow the scope of the analyses of mechanistic information, IRIS applies an iterative approach to identifying key mechanistic questions at various stages of the systematic review • Problem formulation identifies predefined analyses (e.g., when a mutagenic MOA is indicated) • Literature inventory allows identification of studies on an organ system that human and animal studies meeting the PECO criteria have not examined • Human and animal evidence syntheses may flag impactful qualitative and quantitative analyses 75 58 Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Appendix C Human and animal evidence syntheses may flag impactful mechanistic analyses – Identify precursor events for apical toxicity endpoints – Inform susceptibility (species, strain, or sex differences; at-risk populations or lifestages) – Inform human relevance of animal data (note: the level of analysis will vary depending on the impact of the animal evidence) – Provide biological plausibility (i.e., to human or animal health effect data when evidence is weak or critical uncertainties are identified) – Establish mechanistic relationships (or lack thereof) across sets of potentially related endpoints/outcomes to inform the consideration of coherence during evidence integration – Aid extrapolation (high-to-low dose; short-to-long duration; route-to-route) – Improve dose-response modeling and quantification of uncertainties 76 Mechanistic Analysis Focused on Specific Questions Examples of when these analyses have been triggered in recent IRIS Assessments: • Benzo[a]pyrene (2017): The descriptor “carcinogenic to humans” was supported by strong mechanistic evidence that established the biological plausibility of the animal findings occurring in humans, despite lack of human exposure data – Key precursors (BPDE-DNA adducts) were identified in humans exposed to PAH mixtures that are specific to B[a]P, form mutational spectra unique to B[a]P, and are associated with cancer in humans • Dichloromethane (2011): The cancer risk estimate was specifically derived for a susceptible subpopulation (GSTT1+/+) identified by the mechanistic evaluation – Differing results in vivo were explainable by species and tissue differences in the availability of GST – PBPK modeling addressed the variability in this population • Documentation and transparency is key for future mechanistic analyses 77 59 Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Progress Toward Transforming the Integrated Risk Information System Program: A 2018 Evaluation  Focused mechanistic evaluations “Several criteria should be considered in assessing in vitro toxicology studies for risk of bias and toxicologic relevance. Relevance should be determined in several domains, including cell systems used, exposure concentrations, metabolic capacity, and the relationship between a measured in vitro response and a clinically relevant outcome measure. Few tools are available for assessing risk of bias in in vitro studies. Because of the nascent status of this field, the committee can provide only provisional recommendations for EPA to consider...EPA should carry out, support, or encourage research on the development and evaluation of empirically based instruments for assessing bias in…mechanistic studies.” (NRC, 2014) • Prioritize studies of relevant endpoints and associated assays by toxicologic relevance (e.g., model systems; dose range; sensitivity and specificity of assay) • Conduct individual study evaluations on the most impactful studies • EPA is exploring the use of existing tools, including adaptations of IRIS study evaluation tools • Organizational frameworks (e.g., EPA’s MOA framework using modified Hill considerations; visual AOP-like constructs) are useful for organizing and documenting these analyses transparently to convey conclusions for evidence integration 78 Moving from Synthesis to Integration Scoping Systematic Review Protocol Literature Inventory Study Evaluation Data Extraction Evidence Integration Derive Toxicity Values Assessment Developed Assessment Initiated Initial Problem Formulation Literature Search, Screen Refined Evaluation Plan Organize Hazard Review Evidence Analysis and Synthesis Select and Model Studies Outputs of Evidence Synthesis Results of Human Health Effect Study Synthesis Evidence Integration Transparent and Structured Processes for Drawing Summary Conclusions Across Lines of Evidence Results of Animal Health Effect Study Synthesis Results of Synthesis of Mechanistic Evidence Informing the Human and Animal Syntheses 79 60 Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Appendix C Evidence Integration Involves a Sequential,Two-Step Process •           Evidence synthesis interpretations for each consideration relating to causality are combined across lines of evidence using transparent, structured frameworks Step Step 1:“Within-Stream” 1:“Within-Stream” Integration Integration Step Step 2:“Across-Stream” 2:“Across-Stream”Integration Integration Characterize Strength Evidence from Judge thethe Strength of of thethe Evidence from the:Human • Human Evidence Stream Studies (Human Evidence Stream Judgment) • Animal Evidence Stream Draw Overall Evidence Integration Conclusions based on: Draw Overall Conclusions for a Human Health Effect • Combined Human and Animal Evidence Streams Human health effect study synthesis conclusions for each consideration are integrated in light of mechanistic evidence in exposed humans or human cells (or other human models) The judgments regarding the strength of the human and animal evidence streams are integrated in light of evidence on the human relevance of the findings in animals, susceptibility, and the coherence of the findings across evidence streams. Characterize the Strength of the Evidence for an Effect in Animals (Animal Evidence Stream Judgment) Animal health effect study synthesis conclusions for each consideration are integrated in light of mechanistic evidence in exposed animals or animal cells (or other relevant models)   80 Within-Stream (Human; Animal Stream) Evidence Judgment Considerations Human Evidence Stream   Animal Evidence Stream Individual Studies • High or medium confidence studies provide stronger evidence within evaluations of each Hill consideration • Interpreting results considers biological as well as statistical significance, and findings across studies Consistency • Different studies or populations increase strength Doseresponse Magnitude, Precision • Different studies, species, or labs increase strength • Simple or complex (nonlinear) relationships provide stronger evidence • Dose-dependence that is expected, but missing, can weaken evidence (after considering the findings in the context of other available studies and biological understanding) • Large or severe effects can increase strength; further consider imprecise findings (e.g., across studies) • Small changes don’t necessarily reduce evidence strength (consider variability, historical data, and bias) • Biologically related findings within an organ system, within or across studies, or across populations (e.g., sex) increases evidence strength (considering the temporal- and dose-dependence of the relationship) Coherence • An observed lack of expected changes reduces evidence strength • Informed by mechanistic evidence on the biological development of the health effect or toxicokinetic/ dynamic knowledge of the chemical or related chemicals Mechanistic • Mechanistic evidence in humans or animals of precursors or biomarkers of health effects, or of changes in Evidence on established biological pathways or a theoretical mode-of-action, can strengthen evidence Biological • Lack of mechanistic understanding does not weaken evidence outright, but it can if well-conducted Plausibility experiments exist and demonstrate that effects are unlikely Light blue rows highlight mechanistic inferences;“temporality” and “natural experiments” not shown 61 Copyright National Academy of Sciences. All rights reserved. 81 Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Progress Toward Transforming the Integrated Risk Information System Program: A 2018 Evaluation  Step 1: Framework for Within-Stream Evidence Judgments The Hill-based considerations are applied to judge the strength of the evidence from human studies and, separately, the evidence for an effect in animals STRONGER EVIDENCE Strength of the Evidence for the Human (i.e. in Human Studies) or Animal Stream (i.e. an Effect in Animals)  Strength of the Evidence for the Human (i.e., in Human Studies) or Animal Stream (i.e. an Effect in Animals)  A sesett ofof co con nsistent high high oror memedium dium co con nfidence, independ independen entt exp expeerimentntss rreasonablyasonably  rulingruling outout altalteernarnattivivee exp expllanatatiionsns;; an anyy cconflictinglicting sesett ofof sstudiesudies isis  weweaaker.Strongest  Addition Additional al critcriteria eria musmustt also lso bebe memet:t:  Evidence Supporting anaEffect Human Human evievid den ence ce ststre reaam: Ob Obserservved ed acracro ossss populapopulattion ionss, wiwitth clecleaar dosdosee‐resp spon onsese  evidencevidencee Anim Animaal ststre reaam: Ob Obserservved ed acracro ossss labslabs oror sp species,ecies, wiwitth multiplemultiple lineslines ofof  A smaller set of high or medium confidence, independent experiments, which may include inconsistent evidence or alternative  explanations (the inconsistent evidence does not discount the positive findings). Includes at least 1 high or medium confidence  study and supporting evidence (e.g., cross‐study coherence)  Mechanistic evidence providing clear support for the exposure‐induced effects (e.g., informing dose‐response, coherence, or MOA  with reasonable confidence) can strengthen weaker sets of evidence to this level  Scenarios where only low confidence experiments are available, or scenarios where medium confidence studies exist, but   conflicting evidence reduces confidence in the effects. Typically, the MOA is not understood. Mechanistic studies alone might prove  Weakest Evidence Supporting an Effect sufficient for this level. Evidence at this level is useful for identifying research needs.  No studies are available, or there is a set of low confidence experiments that are not reasonably consistent.  Inadequate Evidence to Draw a Within-Stream Judgment Consistent evidence demonstrating no effects of exposure (at any tested level) from numerous high confidence experiments,  reasonably ruling out alternative explanations. The data are compelling in that they have examined an extensive range of exposure  Strong Evidence for Lack of an Effect concentrations, exposure durations, and all populations of potential concern (e.g., lifestages; species; sex; etc.) using optimal  methods for both exposure and the evaluation of the health effect(s) of interest  82 Step 2: Framework for Overall Evidence Integration Conclusions STRONGER CONCLUSION Judgments regarding the strength of the human and animal evidence streams are combined to draw a conclusion for a given human health effect h levleveel ofof cecertaintrtaintyy thathatt exp expo osurere cacau uses thethe health health effeffeectct in in humanshumans,, e.e.g.,:g.,:  A veverry highigh mentt fo forr thethe human human evidencevidencee ststrereaam  • TheThe ststrrongegestst evidencevidencee judgjudgmen Strongest Conclusion for a Human Health Effect moderoderaatelyely ststrrong human human evidencevidencee judgjudgmen mentt and and thethe ststrrongegestst animalanimal  • A m mentt alongalongssideide ststrrong mechanismechanistictic evidencevidencee thathatt MO MOAsAs and and  evidencevidencee judgjudgmen i i i h h kk d id fifi dd i l i l d Reasonable certainty that exposure causes the health effect in humans, although some outstanding questions may remain, e.g.,:  The strongest evidence judgment for the animal evidence stream, but not meeting the criterion above  A moderately strong human or animal evidence stream judgment, or the weaker judgments when evidence from the opposite  stream (e.g., mechanistic evidence of precursors supporting coherence) that increases certainty  Conveys some concern that exposure may cause a particular health outcome in humans, but either there were very few studies  that contributed to the evaluation, the evidence was weak or conflicting, and/ or the methodological conduct of the studies was  poor. Given the substantial degree of uncertainty, additional research is encouraged. Scenarios include:  The weakest human or animal evidence stream judgment, or a moderately strong judgment with evidence from the opposite  Weakest Conclusion for a Human Health Effect stream (e.g., null results in well‐conducted mechanistic studies of precursors) that decreases certainty   Exceptionally, strong mechanistic evidence in the absence of conventional human or animal studies  This conveys either a lack of information or an inability to interpret the available evidence, e.g.,:  Inadequate evidence to judge the strength of both the human and animal evidence streams.  Inadequate Evidence to Draw a Conclusion The strongest animal evidence stream judgment with inadequate evidence to judge the strength of the human evidence, and  with strong mechanistic information indicating that the animal evidence is unlikely to be relevant to humans.  A substantial degree of certainty that there is negligible concern for exposure to cause the health effect in humans, e.g.,:  Meeting the criteria for drawing a judgment of ‘strong support for no effect’ for the human evidence stream  Strong Support for No Human Health Effect Meeting the criteria for drawing a judgment of ‘strong support for no effect’ for the animal evidence stream along with  inadequate evidence to judge the strength of the human evidence and strong mechanistic support that the animal models are 83 able to identify an association  83 62 Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Appendix C 84 Evidence Profile Table: Supports the Evidence Integration Narrative “the weight of evidence descriptions need to indicate the various determinants of weight... to be able to understand what elements (such as consistency) were emphasized” [NRC, 2011]; “No matter what method is used to integrate the different kinds of evidence available for an IRIS assessment, using a template for the evidence-integration narrative could help to make IRIS assessments more transparent.” [NRC, 2014] Studies and interpretation Factors that increase Factors that strength decrease strength Summary of findings Within stream evidence judgments Inference across evidence streams Overall conclusion [Health Effect or Outcome Grouping] Evidence from Human Studies (Route) Human relevance of findings in animals  Cross-stream coherence (i.e. for both health effect-specific and mechanistic data)  Other inferences: o Information on susceptibility o MOA analysis inferences: precursors, cross-species inferences of toxicokinetics, or quantitative implications o Relevant information from other sources (e.g., read across; other, potentially related health hazards) Describe conclusion(s) and primary basis for the integration of all available evidence (across human, animal, and mechanistic): “the weight of evidence descriptions need to indicate the various determinants of weight... to be able to understand w hat elements (such as + + +  + + +  “No m atter what method is consistency) were emphasized” [NRC, 2011]; + +  + +  +  +  used to integrate the different kinds of evidence available for an IRIS assessment, using a template for the evidence-integr ation narrative could help to make IRIS assessments more transpar ent.” [NRC, 2014]  References  Study confidence (based on evaluation of risk of bias and sensitivity) and explanation  Study design description  Consistency  Dose-response gradient  Coherence of observed effects (apical studies)  Effect size (magnitude, severity)  Biological plausibility  Low risk of bias/ high quality  Insensitivity of null/ negative studies  Natural experiments  Temporality  Unexplained inconsistency  Imprecision  Indirectness/ applicability  Poor study quality/ high risk of bias  Other (e.g., Single/Few Studies; small sample size)  Evidence demonstrating implausibility  Results information (general endpoints affected/ unaffected) across studies  Human mechanistic evidence informing biological plausibility: discuss how data influenced the within stream judgment (e.g., evidence of precursors in exposed humans). Describe strength of the evidence from human studies, and primary basis: Strongest evidence  ◯ ◯◯Weakest evidence  Could be multiple rows (e.g., grouped by ◯◯◯ – ◯◯ Inadequate  study confidence or population) if this ─  ─ ◯ informs results heterogeneity – ─  ─  Strong evidence for  no effect  Evidence for an Effect in Animals (Route)  References  Study confidence (based on evaluation of risk of bias and sensitivity) and explanation  Study design description  Consistency and Replication  Dose-response gradient  Coherence of observed effects (apical studies)  Effect size (magnitude, severity)  Biological plausibility  Low risk of bias/ high quality  Insensitivity of null/ negative studies  Unexplained inconsistency  Imprecision  Indirectness/ applicability  Poor study quality/ high risk of bias  Other (e.g., Single/Few Studies; small sample size)  Evidence demonstrating implausibility  Results information (general endpoints affected/ unaffected) across studies  Animal mechanistic evidence informing biological plausibility for effects in animals: discuss how mechanistic data influenced the within stream judgment (e.g., evidence of coherent molecular changes in animal studies) Could be multiple rows (e.g., by study confidence, species, or exposure duration) if this informs results heterogeneity Describe strength of the evidence for an effect in animals, and primary basis: Strongest conclusion  ◯ ◯◯Weakest conclusion  ◯◯◯ – ◯◯ Inadequate  ─  ─ ◯ – ─  ─  Strong support for no  human health effect  Summarize the models and range of dose levels upon which the conclusions were primarily reliant + + +  Strongest evidence  + + ◯ + ◯◯ Weakest evidence  ◯◯◯ – ◯◯ Inadequate  ─  ─ ◯ – ─  ─  Strong evidence  for no effect  63 Copyright National Academy of Sciences. All rights reserved. 85 Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Progress Toward Transforming the Integrated Risk Information System Program: A 2018 Evaluation  Evidence Integration Conclusions • For Cancer, conclusions on the integrated evidence for each cancer type (or grouping) are evaluated in the context of MOA information to develop an evidence integration narrative that includes a descriptor for carcinogenicity: - • carcinogenic to humans; likely to be carcinogenic to humans; suggestive evidence of carcinogenic potential; inadequate information to assess carcinogenic potential; or not likely to be carcinogenic to humans For Noncancer Effects, frameworks for evaluating the integrated evidence have been developed to add structure and transparency to the evidence integration narrative(s), which include(s) the relevant exposure context. - IRIS has not yet incorporated standardized descriptors for noncancer effects - The NAS recommended incremental improvements in this area, including recommendations to “Develop uniform language to describe strength of evidence on noncancer effects” [p. 92, 2014] - The specific way in which these conclusions are summarized is currently being tested and discussed within EPA 86 IRIS has Addressed the Major NAS 2014 Recommendations NAS 2014 Topics IRIS Process Improvements Evidence Evaluation • Individual studies are evaluated for reporting quality, risk of bias, and sensitivity (Chapter 5) • Decisions and supporting rationale are clearly documented • Study evaluations impact subsequent assessment decisions Evidence Integration • Structured frameworks provide transparency in expert judgments for Hazard across human, animal, and mechanistic studies (based on Hill) Identification • Standardized templates documenting key evidence integration (Chapter 6) decisions have been developed (evidence profile tables) See Posters and Demonstrations: • Male reproductive toxicity in studies of phthalates (4 posters on a case study for each of the 3 lines of evidence and the overall evidence integration) • Combining data within species (poster on meta-analytical approaches) • PBPK model evaluation for human health assessments (poster) • Health Assessment Workspace Collaborative (demonstration) 64 Copyright National Academy of Sciences. All rights reserved. 87 Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Appendix C SESSION 3: DEVELOPMENT AND APPLICATION OF SPECIALIZED TOOLS FOR SYSTEMATIC REVIEW Kris Thayer*, Michele Taylor*, Amina Wilkins, Xabier Arzuaga [*Speaking] Office of Research and Development NCEA, IRIS NAS 2014: Chapter 8 “Looking Forward” “[EPA] need to consider developing a strategic plan for continuous updating of the IRIS methodology... For example, such a strategic plan should address: – Applying advances in data retrieval and text-mining “The committee also found that the proposed format for the assessments should enhance “user friendliness” and transparency. The evidence tables and data displays in the new documents are moving to the standard practice for systematic reviews.” [p. 136] 89 65 Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Progress Toward Transforming the Integrated Risk Information System Program: A 2018 Evaluation  Current Application of Systematic Review Software • Specialized software tools make the process more efficient • – Time and cost savings, improved data management, increased transparency NOT all systematic review software tools are intended to automate/semiautomate the process, e.g., HAWC helps manage information content • – Currently, automation tools are most advanced for evidence identification Prefer free tools when possible to help address needs of a potentially large community of users in environmental and biomedical sciences • • Incorporate tools after confirming acceptable performance and interoperability with HERO – A toolbox approach, not a “one and only” tool model Organized multiple IRIS staff training sessions in 2017 and created a support team (“train the trainers” model) 90 Research Activities • • • • Developing tools to help automate beyond evidence identification is a long-term research commitment – Major hurdle is lack of training/test sets for model development – Better performance expected for more structured content (e.g., animal bioassay compared to epidemiological studies) Any progress on semi-automation could result in large time and cost savings In 2017, NCEA created an interagency agreement with NTP to leverage resources – Current activities focus on creating test/training sets and model development for basic content of animal studies (e.g., test chemical, species, dose levels, randomization, etc.). – Other parts of EPA can also utilize interagency agreement Innovation challenges may be required to identify solutions for capturing complex content, i.e., table content, information spread across multiple sentences and 91 paragraphs 66 Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Appendix C Suite of Systematic Review Software Tools – Upcoming Demonstrations ARE  TOOLS  INTEROPERABLE  92 SWIFT Review: Scoping and Problem Formulation e  https://www.sciome.com/swift-review/ 93 67 Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Progress Toward Transforming the Integrated Risk Information System Program: A 2018 Evaluation  Increased Efficiency During Scoping and Problem Formulation Can be used to screen studies according to the PECO statement Built-in and user-defined search queries allow targeted surveys of the literature corpus Machine Machine learning learning prioritizes prioritizes relevant relevant literature, literature, reducing the the screening screening burden burden by by at at least least 50% 50% reducing 94 “Tags” Facilitate Searching During Problem Formulation MeSH Terms and Bibliographic Data: Documents originating from PubMed bring along their associated Medline tags, including MeSH Terms, Publication Type, Pharmacological Actions, etc. SWIFT-Review tags are labels assigned to bibliographic documents that are organized into tag categories. For example, the tagIt category “Health Outcomes” includes Queries and Filters: incorporates several pre-defined search the following “Cardiovascular,” and scientists filters.These filterstags:“Cancer,” have been prepared by information When used with the Tag Browser or “Neurological.” or topics of relevance to environmental health scientists. f various Search functionalities, tags facilitate(click increased Details on these filters are documented Helpefficiency > Search during scoping and problem formulation by quickly finding Strategies) the documents you’re interested in. Tags can be assigned both manually and automatically using a variety of mechanisms: Chemical Names:Automatically tags documents that mention chemicals occurring in several chemical lists of relevance to environmental health researchers, such as the nearly 10,000 Tox21 chemicals 95 68 Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Appendix C 96 Built-in and User-Defined Search Strategies 97 69 Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Progress Toward Transforming the Integrated Risk Information System Program: A 2018 Evaluation  Tag Browser Search by Health Outcome 98 Tag Browser Search by Health Outcome 99 70 Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Appendix C Searching Additional Tag Categories Tox21 Chemicals Keyword Text Highlighting 100 Interactive Displays Reveal Patterns of Available Evidence 101 71 Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Progress Toward Transforming the Integrated Risk Information System Program: A 2018 Evaluation  Publication Year by Health Outcome 102 Priority Ranking Reduces Screening Burden 103 72 Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Appendix C Automated Priority Ranking Reduces Screening Burden Topic modeling is a statistical methodology (Latent Dirichlet Allocation or LDA) that automatically computes then categorizes documents according to pre-defined topics. Users can also customize their own topic model by choosing Tools > Build Topic Model Incorporate human curated training sets or manually annotate “included” and “excluded” training “seeds” to automatically priority rank the remaining documents. 104 Seed the model to priority rank 105 73 Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Progress Toward Transforming the Integrated Risk Information System Program: A 2018 Evaluation  Priority Ranking Improves Literature Screening Efficiency Increase screening efficiency Reduce screening burden on average 50% 106 Automated Priority Ranking 107 74 Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Appendix C SWIFT Active Screener Capabilities Improved Ranking Model • • • • Web-based, real-time, collaborative, systematic review software application • • • Option to “seed” studies if relevant on/off topic literature has been identified State-of-the-art statistical models prioritize articles as they are being reviewed Experience suggests screening burden is reduced by at least 50% (likely more) Algorithm improves from screener-input without training “seeds” further increasing efficiency (more efficient than implementing a “seed studies” only model) Incorporates a graphical user interface to provide project status updates User-defined screening levels – Level 1:Title and Abstract – Level 2: Full text screening – Level 3: Conflict Resolution 108 Customize Inclusion/Exclusion Criteria According to the PECO Statement 109 75 Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Progress Toward Transforming the Integrated Risk Information System Program: A 2018 Evaluation  User Input Improves the Algorithm to Priority Rank While Screening 110 “Seed” studies when Relevant On/Off Topic Literature is Identified 111 76 Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Appendix C Manage References with Conflict Resolution – Track and Archive Changes 112 113 77 Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Progress Toward Transforming the Integrated Risk Information System Program: A 2018 Evaluation  SWIFT Active: Data Integration • Active Screener integrates with systematic review tools already in use: – Accepts imports from bibliographic databases and reference curation platforms including SWIFT Review, EndNote, Mendeley, Zotaro, and PubMed – Results from screening in Active Screener can be exported in standard data formats compatible with applications including HAWC and Excel, EndNote, Mendeley, and Zotaro Current Users 114 HAWC: Study Evaluation, Extraction, Visualization and Data Sharing e  https://hawcproject.org/ 115 78 Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Appendix C HAWC Capabilities • • • • Free and open source • • • • Interactive “click to see more” graphics Developed at UNC by Andy Shapiro* with Ivan Rusyn Literature search and initial screening Animal bioassay, epidemiological, and in vitro structured study methods/data extraction and visualization Risk of bias and sensitivity evaluation Modular to work with other tools and maximize flexibility for users Works best in Google Chrome (preferred), Mozilla Firefox, and Safari *current affiliation is National Institute of Environmental Health Sciences/National Toxicology Program (NIEHS/NTP) 116 HAWC: Summarizing Animal Bioassays 117 79 Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Progress Toward Transforming the Integrated Risk Information System Program: A 2018 Evaluation  Epidemiology: Click to See More Display Example from Chloroform 118 Visualizing Epidemiology Evidence Example from Chloroform 119 80 Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Appendix C Visualizing Animal Evidence Chloroform Fetal Survival 120 Visualizing Animal Evidence Chloroform Fetal Survival 121 81 Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Progress Toward Transforming the Integrated Risk Information System Program: A 2018 Evaluation  Multiple Formats to Present Results Chloroform Fetal Survival Animal data can be expressed as effect size, e.g., percent control 122 HAWC: Dose-Response Displays 123 82 Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Appendix C HAWC: Dose-Response Displays 124 HAWC: Download Reports • Entire database for an assessment can be downloaded in Microsoft Excel exports 125 83 Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Progress Toward Transforming the Integrated Risk Information System Program: A 2018 Evaluation  HAWC Benchmark Dose Modeling 126 Advantages • • • • • • Structured extraction to promote consistency and completeness • Creates possibilities for web-based, interactive reports Free, open source and customizable Enhance opportunities for database interpretability Integration with automated data-extraction tools Web-based to promote team collaboration Ability to export data files promotes further analysis of findings and quantification (in assessments or for methods development) 127 84 Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Appendix C IRIS has Addressed the Major NAS 2014 Recommendations NAS 2014 Topics Looking Forward IRIS Process Improvements   • Specialized software tools for efficiency and more user friendly   and transparent formats for evidence display have been adopted • Strategic planning on use of text and data-mining tools and automation • Specialized tools facilitate transparent documentation, consistency across assessments, and database interoperability See Demonstrations: • SWIFT Review and SWIFT Active • Health Assessment Workspace Collaborative • Heath Effects Research Online 128 SESSION 4: STUDY SELECTION FOR DEVELOPING TOXICITY VALUES, AND ADVANCING RESEARCH ON QUANTITATIVE ANALYSES FOR EVIDENCE INTEGRATION AND DOSE-RESPONSE ANALYSES David Bussard*, Jason Lambert*, Ted Berner, Allen Davis, Jeff Gift, Karen Hogan, Leonid Kopylev, Ravi Subramaniam [*Speaking] Office of Research and Development NCEA, IRIS 85 Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Progress Toward Transforming the Integrated Risk Information System Program: A 2018 Evaluation  NAS 2014:Three High Priority (Box 8-1) Recommendations on Quantification • TOXICITY VALUES: “EPA should develop criteria for determining when evidence is sufficient to derive toxicity values.” – Overall hazard conclusions inform decision whether to develop toxicity values. – Better documenting considerations on which studies are carried forward to dose-response. • POINTS OF DEPARTURE (PODs): “EPA should clearly present two dose-response estimates: a central estimate (such as a maximum likelihood estimate or a posterior mean) and a lowerbound estimate for a POD from which a toxicity value is derived.” – Central estimates (MLEs) of BMDs provided in IRIS assessments along with BMDLs. – Will start to use WHO/IPCS approach to characterize distributions in final values. – Model averaging to characterize model uncertainty. • QUANTITATIVE CAPABILITIES: “EPA should expand its ability to perform quantitative modeling of evidence integration; in particular, it should develop the capacity to do Bayesian modeling of chemical hazards. ...The Committee emphasizes that... IRIS assessments should not be delayed while this capacity is being developed.” – Meta-analysis of human and animal studies increasing: hazard decisions and dose-response. – Bayesian methods are being explored to help characterize uncertainty. – New approach methods and assays are increasingly being evaluated quantitatively. 130 Evidence Integration Conclusions Inform when to Develop Toxicity Values Evidence integration conclusion Quantitative toxicity value provided? Strongest conclusion for a human health effect Yes. (for cancer, a descriptor of Known) Moderately strong conclusion for a human health effect (for cancer, a descriptor of Likely) Yes. Weakest conclusion for a human health effect (for cancer, a descriptor of Suggestive) Determined by situation (e.g., may provide values when useful for decision purpose and the evidence includes a well-conducted study) Inadequate information No, although bounding estimate from a study that does not show positive results can be derived where useful for decision purpose. Strong support for no human health effect No. 131 86 Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Appendix C Decision-Making for Advancing Studies to Develop Toxicity Values IRIS has further clarified the considerations that inform the selection of studies to estimate human dose-response relationships (next slide). • IRIS continues to find that this decision process is not reducible to a formula. • Expert judgment is essential for judging the relative merits of individual studies and which studies support more integrative quantitative analyses (e.g., meta-analysis). • IRIS must often utilize studies with a range of attributes and levels of reporting. For example, the available studies on many mission-critical chemicals do not provide data on an individual subject basis. • For full transparency, IRIS continues to emphasize documentation of the factors it weighed in emphasizing certain studies, or combinations of studies, over others. 132 More Explicitly Defining the Attributes IRIS Uses to Evaluate Studies for Derivation of Toxicity Values In addition to qualitative study evaluation judgments (i.e., medium or high confidence studies are preferred), studies are assessed across several study attributes Example Primary Considerations for Selection of Studies for Derivation of Toxicity Values Study attribute Test species Human relevance of the exposure Human studies Human data are generally preferred to eliminate interspecies extrapolation uncertainties (e.g., in toxicodynamics and Animal studies Animals that respond most like humans are preferred. Outcomes associated with species known to show differences in sensitivity can specific health outcomes). provide support with suitable qualification. Exposure Studies involving typical human environmental exposure routes are preferred (e.g., oral, route inhalation). A validated toxicokinetic model can be used to extrapolate across exposure routes. Exposure For chronic toxicity values, chronic or subchronic studies are preferred. Exceptions exist duration (e.g., when a population or lifestage is more sensitive during a particular time window) paradigm Exposure Exposures near the range of typical environmental human exposures are preferred. Studies with a broad exposure range and multiple exposure levels are preferred to levels the extent that they can provide information about the shape of the exposure-response Susceptibility relationship* and facilitate extrapolation to more relevant (generally lower) exposures. Studies that yield risk estimates in the most susceptible groups are preferred. Inclusion of design features in the analysis (e.g., matching procedures, blocking; covariates or other procedures for statistical adjustment) that adequately address the relevant sources of potential critical confounding for a given outcome are preferred. *U.S. EPA Benchmark Dose Technical Guidance (2012) 87 Copyright National Academy of Sciences. All rights reserved. 133 Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Progress Toward Transforming the Integrated Risk Information System Program: A 2018 Evaluation  IRIS Assessments Are Providing Central MLE Estimates of BMDs Along with BMDLs Recent animal study example to the left: Benzo[a]pyrene (EPA, 2017) p. 2-8 Toxicological Review of Benzo[a]pyrene https://cfpub.epa.gov/ncea/iris/iris_documents/docum ents/toxreviews/0136tr.pdf Recent epidemiology example: Ethylene oxide (EPA, 2016) p. 4-109 Toxicological Review of the Inhalation Carcinogenicity of Ethylene Oxide. (EPA, 2016) https://cfpub.epa.gov/ncea/iris/iris_documents/docum ents/toxreviews/1025tr.pdf 134 IRIS is also Presenting Arrays of Candidate Toxicity Values Benzo[a]pyrene (EPA, 2017) 88 Copyright National Academy of Sciences. All rights reserved. 135 Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Appendix C Improvements in Characterizing Uncertainty 1) Model Averaging: characterizing model uncertainty • Currently evaluating several methods • Approach for dichotomous data expected to undergo peer review in 2018 136 Improvements in Characterizing Uncertainty 2) Distributions and Central Estimates: characterizing uncertainty in the human toxicity value • WHO/IPCS guidance (IPCS, 2014) • Risk-specific doses in terms of ranges, for explicitly described: - Effect magnitudes - Confidence levels - Human population incidence rates. • A probabilistic approach to adjustments from animal to human; a framework for refining toxicity values. 137 89 Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Progress Toward Transforming the Integrated Risk Information System Program: A 2018 Evaluation  Improvements in Characterizing Uncertainty WHO/IPCS Approach: IRIS intends to provide such calculations along with traditional Reference Values: • • • • Confidence intervals on risk-specific doses Central estimates Estimates of incidence as a function of dose Use of appropriate probability math for uncertainty adjustments (instead of UFs) to allow for a more probabilistic and scientific value for use in risk assessment By characterizing ranges of risk-specific doses, this provides more than a “conservative” estimate (it provides useful context by estimating the full distribution) 138 Use of Quantitative Modeling to Inform Evidence Integration Meta-Analysis: Increasingly Being Used to Interpret Sets of Results across Similar Populations • Formal tools continue to be used to combine similar human epidemiology studies to improve decisions about hazard and about slope of dose-response. • These approaches have also been used to better understand animal data that differ between studies of similar species and endpoints. • As software tools and best practices become more common and easier to apply to environmental health studies, IRIS intends to consider their use more routinely. Other examples: Libby Amphibole Asbestos (2014) and Trimethylbenzene analysis (Davis and Kraft, 2017) – see poster session; Arsenic assessment (in process) 139 90 Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Appendix C Use of Quantitative Modeling to Inform Evidence Integration Bayesian Approaches: More Frequent Use Across Different Applications, and Research is Ongoing • Characterizing Uncertainty – Bayesian approaches were used to characterize uncertainty in PBPK modeling and evaluate inter-related model inputs (Perchlorate peer review, 2018). – Bayesian Analysis is compatible with the WHO/IPCS Approach for characterizing uncertainty • Model Averaging – Bayesian approaches are being applied to individual BMD models, and then model averaging is used to characterize uncertainty • Meta-Analysis – Bayesian meta-analysis is currently being used to evaluate arsenic epidemiology studies • Bayesian Networks (exploratory research is currently underway) – Possess the potential to integrate across evidence streams and bridge data gaps, borrowing strength from diverse data. – Software and mathematics are currently available. 140 Future work to better meet Agency needs for “benefits analysis” Economics benefits analysis would ideally estimate incidence resulting from different decision options. • We have provided human dose response functions from some analyses based on epidemiology data. (Evaluation of the Inhalation Carcinogenicity of Ethylene Oxide, EPA, 2016). IRIS is also evaluating analogous predictions from animal data that could inform benefits analysis, including modifications of the IPCS approach. 141 91 Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Progress Toward Transforming the Integrated Risk Information System Program: A 2018 Evaluation  Advancing Application of New Approach Methods (NAM) and Data in HHRA • Over the past decade, several reports, books, resource documents, etc. have been published regarding the use of New Approach Methods (NAM) across the human health risk assessment paradigm (i.e., shifting the paradigm) • Numerous labs, centers, workgroups, and initiatives across federal, private, and academic institutions have been formed to advance NAM • EPA/ORD/NCEA, in conjunction with partners (e.g., NCCT, NTP) has been actively engaged in the conceptualization and evaluation of NAM across a broad landscape of HHRA applications 142 NAM Toolbox to Date • Data-mining: ToxRefDB-comprehensive collection and collation of extant hazard and exposure data –(Martin et al. 2009. Env Health Perspect 117: 392-399) • Chemoinformatics: structure-activity/read-across; QSAR –(Wang et al. 2012. Regul Toxicol Pharmacol 63: 10-19; Craig et al. 2014. J Appl Toxicol 34: 787-794) • High-Throughput (HT) Exposure modeling: ExpoCast –(Egeghy et al. 2016. Env Health Perspect. 124(6):697-702) • HT Toxicokinetics: in vitro to in vivo (IVIVE) modeled dosimetry –(Wambaugh et al. 2015. Tox Sci 147: 55-67) • Bioactivity: short-term animal; cell-free and/or cell-based HT assay data – (Judson et al. 2011. Chem Res Toxicol 24: 451-462; Dean et al. 2017. Tox Sci 157(1):85-99) • Adverse Outcome Pathway (AOP): expert-driven identification of signal transduction pathways along the exposure to outcome continuum. –(Edwards et al. 2016. J Pharmacol Exp Ther. 356(1):170-181) 143 92 Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Appendix C Chemoinformatics Expert-driven Read-Across Chemistry Data-poor chemicals • Inferred/interpolated hazard • Surrogate based POD and subsequent derivation of RfVs ADME BP and MP LogP 1° pKa 2° 3° T1/2 topography MW Candidate analog(s) L(N)OAEL, EDx, LD50 Toxicity IRIS-type chemicals • Data-gap filling • Augment WOE • Potential for reducing uncertainties Category approach (A) Target Chemical R1 ADME ADME R-R-R2 (B) R1 (C) R-R-R2 R1 ADME R1 Category ADME R1 R2 ADME R3 (D) R-R-R2 R-R2 ADME = Absorption, Distribution, Metabolism, Elimination • • • • Data-poor chemicals • Data-gap filling • Extrapolated hazard • Less applicable for quantitative assessment currently IRIS-type chemicals • Data-gap filling • Augment WOE • Foundational member of category (i.e., anchor chem) Similarity in structure and physicochemical properties between a chem of concern and a population of analogs Robustness of approach dependent on density of analogs populating a category Highly reliant on WOE supporting toxicity endpoints across category Presumes common Adverse Outcome Pathway or Mode of Action across category members 144 Bioactivity (e.g., transcriptomics) • Close relationship between genotype/phenotype across two different routes of exposure, rodent species, and multiple target tissues • In vitro?? Will need to optimize metabolism protocols; integrate IVIVE Data-poor chemicals • Evidence base for hazard • Empirical dose-response based on pathway perturbations • Reduce need for longer-term animal studies IRIS-type chemicals • Augment WOE (e.g., MOA/AOP) • Opportunity to alert off-target effects • Potential for reducing uncertainties 145 93 Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Progress Toward Transforming the Integrated Risk Information System Program: A 2018 Evaluation  Integrated Application to Risk Assessment RapidTox Dashboard • Associated narrative can be modular based on fit-for-purpose • Systematic WOE always, but can be graded based on decision context • Characterization of qualitative and quantitative uncertainties 146 IRIS has Addressed the Major NAS 2014 Recommendations NAS 2014 Topics Evidence Integration for Hazard Identification (Chapter 6) and Derivation of Toxicity Values (Chapter 7) IRIS Process Improvements • Developing and applying quantitative tools in support of evidence synthesis and integration, including meta-analytical approaches • Expanded development and use of more advanced quantitative methods in software tools, such as BMDS • Developed more explicit criteria for deriving toxicity values, including the intent to derive quantitative toxicity values when IRIS reaches one of the stronger evidence integration conclusions, as well more specific criteria for the evaluation of individual studies • Providing MLE estimates of BMDs, along with BMDLs • Applying and exploring quantitative approaches to better characterize uncertainty, including probabilistic and Bayesian approaches • Quantitative assessment methods will be updated in a continuing, Future Directions strategic fashion, including capacity building (e.g., training; evolving best (Chapter 8 “Lessons practices) for current approaches including meta-analysis, probabilistic Learned” and analyses, and Bayesian methods “Looking Forward”) 147 94 Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Appendix C COLLABORATION, TRAINING, AND FINAL THOUGHTS Tina Bahadori* and Kris Thayer [*Speaking] Office of Research and Development NCEA, IRIS Training and Collaboration • Held multiple training sessions for IRIS Program staff in 2017, ranging from demos, seminars, to retreats. More to come in 2018… • Developed support teams to provide teaching and assistance for systematic review tasks and use of new software (“train the trainer” model) • • Active engagement in the EPA Systematic Review Communities of Practice • Engagement with external stakeholders, other Agency offices, state and other Agencies on systematic review methods and software training – e.g., MOUs with NTP, NIOSH,ATSDR,WHO – Interagency funding agreement with NIEHS/NTP for text-mining and software tool development and evaluation Establishing several academic MOUs to promote hands on training on use of systematic review in chemical assessments 149 95 Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Progress Toward Transforming the Integrated Risk Information System Program: A 2018 Evaluation  International Collaborations Health Health Canada Canada Nav. Nav. Guide Guide CAMARAD CAMARAD ES ES U.S. U.S. Health Health (NTP, (NTP, EPA, EPA, ATSDR,NIOS) NIOS) ATSDR, Europe Health Europe Health (RIVM,SYRCLE, (RIVM,SYRCLE, EFSA,SciRAP) SciRAP) EFSA, WHO/ WHO/ IARC IARC EBTC EBTC EBT EBT Cochrane Cochrane Collaboration/ Collaboration/ GRADE GRADE Aus. Aus. Health Health 150 IRIS has Addressed the Major NAS 2014 Recommendations NAS 2014 Topics IRIS Process Improvements Quality management pipeline implemented Program and project management processes implemented Frequent opportunities for stakeholder engagement Draft IRIS Handbook of program SOPs is being reviewed within EPA Re-occurring staff training and template IAPs and protocols promote consistency and quality control General Process Issues (Chapter 2) • • • • • Problem Formulation and Protocol Development (Chapter 3) • IAPs allow early comment on problem formulation • More frequent Agency engagement facilitates scope refinement • Assessment protocols describe methods and allow for iteration Evidence Identification (Chapter 4) • Consultation with information technologists and subject experts • Adopts current systematic review best practices, including use of specialized tools • Transparent documentation (e.g., literature flow diagrams) 96 Copyright National Academy of Sciences. All rights reserved. 51 1511 Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Appendix C IRIS has Addressed the Major NAS 2014 Recommendations NAS 2014 Topics Evidence Evaluation (Chapter 5) Evidence Integration for Hazard Identification (Chapter 6) IRIS Process Improvements • Individual studies are evaluated for reporting quality, risk of bias, and sensitivity • Decisions and supporting rationale are clearly documented • Study evaluations impact subsequent assessment decisions • Structured frameworks provide transparency in expert judgments across human, animal, and mechanistic studies (based on Hill) • Standardized templates documenting key evidence integration decisions have been developed (evidence profile tables) • Developing and applying quantitative tools in support of evidence synthesis and integration, including meta-analytical approaches • Expanded development and use of more advanced quantitative methods in software tools, such as BMDS 152 IRIS has Addressed the Major NAS 2014 Recommendations NAS 2014 Topics Derivation of Toxicity Values (Chapter 7) IRIS Process Improvements • Developed more explicit criteria for deriving toxicity values, including the intent to derive quantitative toxicity values when IRIS reaches one of the stronger evidence integration conclusions, as well more specific criteria for the evaluation of individual studies • Providing MLE estimates of BMDs, along with BMDLs • Applying and exploring quantitative approaches to better characterize uncertainty, including probabilistic and Bayesian approaches 153 97 Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Progress Toward Transforming the Integrated Risk Information System Program: A 2018 Evaluation  IRIS has Addressed the Major NAS 2014 Recommendations IRIS Process Improvements   NAS 2014 Topics   Future Directions • Processes being implemented include flexibility to incorporate evolving methods in systematic review and risk assessment (Chapter 8 • Increased collaboration with federal partners and international experts “Lessons Learned” prevents duplication of effort and maintains cutting edge approaches and “Looking • Current research efforts and training serve to ensure that methods and Forward”) • • • • staff are able to adapt to changing scientific contexts and sources of evidence, including new and emerging data types Specialized software tools for efficiency and more user friendly and transparent formats for evidence display have been adopted Strategic planning on use of text and data-mining tools and automation Specialized tools facilitate transparent documentation, consistency across assessments, and database interoperability Quantitative assessment methods will be updated in a continuing, strategic fashion, including capacity building (e.g., training; evolving best practices) for current approaches including meta-analysis, probabilistic analyses, and Bayesian methods 154   98 Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Appendix D Posters by US Environmental Protection Agency • D-1: New Approach Methods in Human Health Risk Assessment • D-2: Combining Data within Species: Meta-analysis in IRIS • D-3: Systematic Evaluations of Physiologically-Based Pharmacokinetic Models for Human Health Risk Assessment • D-4: Male Reproductive Toxicity in Animal Studies of Diisobutyl Phthalate (DIBO): A Case Study Application of Systematic Review Approaches • D-5: Male Reproductive Toxicity in Epidemiology Studies of Phthalates: A Case Study Application of Systematic Review Approaches • D-6: Quantitative Evaluation of Uncertainty: APROBA and Beyond • D-7: Mode of Action and Human Relevance Evaluation of Dibutyl Phthalate (DBP)-Induced Male Reproductive System Toxicity • D-8: EPA Dose-Response & Related Software – New & Future Developments • D-9: Evidence Profile Table for DIBP and Male Reproductive Toxicity • D-10: A New Bayesian Approach to Combining Different Species Data 99 Copyright National Academy of Sciences. All rights reserved. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation 100 Copyright National Academy of Sciences. All rights reserved. D-1 Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Copyright National Academy of Sciences. All rights reserved. 101 D-2 Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation 102 Copyright National Academy of Sciences. All rights reserved. D-3 Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Copyright National Academy of Sciences. All rights reserved. 103 D-4 Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation 104 Copyright National Academy of Sciences. All rights reserved. D-5 Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Copyright National Academy of Sciences. All rights reserved. 105 D-6 Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation 106 Copyright National Academy of Sciences. All rights reserved. D-7 Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Copyright National Academy of Sciences. All rights reserved. 107 D-8 Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation 108 Copyright National Academy of Sciences. All rights reserved. D-9 Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Copyright National Academy of Sciences. All rights reserved. 109 D-10 Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Appendix E Committee Findings Regarding 2014 Recommendations 110 Copyright National Academy of Sciences. All rights reserved. Copyright National Academy of Sciences. All rights reserved. 1B 2 2 2 3 2 4A 2 4B 2 [EPA needs to] specifically complete documents, such as the draft handbook, that provide detailed guidance for developing IRIS assessments. When those changes and the detailed guidance, such as the draft handbook, have been completed, there should be an independent and comprehensive review that evaluates how well EPA has implemented all the new guidance. The present committee is completing its report while those revisions are still in progress. Finding The 2014 report reviewed and encapsulated recommendations from the 2011 report, so the present committee focused its review on assessing progress made in implementing recommendations made by the 2014 report. The revised handbook was not provided to the committee. EPA staff indicated that the handbook is under internal agency review and that its public release is expected in 2018. The agency further indicated that standard operating procedures might evolve as the IRIS program gains additional experience in performing systematic review and using emerging methods. The committee expects handbook revisions to be a continuing process, and EPA similarly characterizes the IRIS handbook as “evergreen.” The committee observed that guidance for conducting newly planned IRIS assessments is contained in protocols, and EPA stated that some material currently in protocols might reside in the handbook. The amount of and need for overlap in the protocols and handbook could not be judged without seeing the handbook. EPA should provide a quality-management plan that includes IRIS management has taken multiple steps to ensure high-quality clear methods for continuing assessments of the quality of the management, including the creation of expertise-specific work groups, process. The roles of the various internal entities involved in the systematic-review work groups, and other intermediate structures to process, such as the CASTs, should be described. The improve the quality of the IRIS assessments. EPA has also used the assessments should be used to improve the overall process and SAB Chemical Assessment Advisory Committee to review IRIS the performance of EPA staff and contractors. assessments. Funding for contractors has decreased. When extracting data for evidentiary tables, EPA should use at EPA uses two people to extract data and, when needed, involves a least two reviewers to assess each study independently for risk of third person to resolve conflicts. EPA also uses two people to complete the risk-of-bias evaluation. bias. The reliability of the independent coding should be calculated; if there is good agreement, multiple reviewers might not be necessary. EPA should continue its efforts to develop clear and transparent EPA has adopted the process of soliciting public comment early processes that allow external stakeholder input early in the IRIS through the release of assessment plans and protocols for public process. comment. [EPA] should develop communication and outreach tools that are Although this recommendation was not discussed specifically with tailored to meet the needs of the various stakeholder groups. For EPA, the agency has worked in the past with the National Academies example, EPA might enhance its engagement with the scientific to identify experts that could provide input at IRIS workshops. The community through interactions at professional-society meetings, IRIS Web site provides features for sharing information via socialadvertised workshops, and seminars. In contrast, greater use of media tweets and Facebook. The calendar feature clearly indicates the social media might help to improve communications with schedule for public engagement events on IRIS assessments. EPA environmental advocacy groups and the public. staff also discussed data- and tool-sharing with stakeholders to increase understanding and accessibility of systematic-review practices used to develop IRIS assessments. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Item Chapter Recommendations from 2014 NRC Reporta 1A 2 EPA needs to complete the changes in the IRIS process that are in response to the recommendations in the [2011] NRC formaldehyde report. Evidence Workshop presentations, posters, and discussion Recent IRIS documents (such as plans, protocols, and assessments) and tools. Slides 21‒22 Systematic Review Protocol for the IRIS Chloroform Assessment (Inhalation)b Slides 7‒10, 151 The GAO audit of the IRIS program indicates that improvements in program management have occurred (Slide 10) Slide 39 Systematic Review Protocol for the IRIS Chloroform Assessment (Inhalation) (p. 17, line 1; p. 30, lines 18‒20)b IRIS Web site Slides 24‒25, 29 IRIS Web site Slide 15 (Continued) 111 Finding Evidence Copyright National Academy of Sciences. All rights reserved. 5 2 Similar to other EPA technical-assistance programs, EPA should consider ways to provide technical assistance to underresourced stakeholders to help them to develop and provide input to the IRIS program. This recommendation was not discussed specifically with EPA. 6 2 The stopping rules should be explicit and transparent, should describe when and why the window for evidence inclusion should be expanded, and should be sufficiently flexible to accommodate truly pivotal studies. Such rules could be included in the preamble. The issue of stopping rules was not specifically discussed, but the Systematic Review of Chloroprene IRIS program has completed a rapid review of chloroprene, and this Studies Published Since 2010 IRIS is consistent with this recommendation. Assessmentc 7 2 Regarding promotion of efficiencies, EPA should continue to expand its efforts to develop computer systems that facilitate storage and annotation of information relevant to the IRIS mission and to develop automated literature and screening procedures, sometimes referred to as text-mining. EPA has made considerable progress in developing and upgrading the Health and Environmental Research Online (HERO) database and the Health Assessment Workspace Collaborative (HAWC) computer system to facilitate storage and annotation of data. Those systems are not subject to third party control. EPA is also using other software systems, including the Sciome Workbench for Interactive computer-Facilitated Text-mining (SWIFT) and related products for text-mining. Workshop Demonstrations Slides 36, 92‒116 8 2 More details need to be provided on the recognition and applications of expert judgment throughout the assessmentdevelopment process, especially in the later stages of the process. The points at which expert judgment is applied should be identified, those applying the judgment should be listed, and consideration should be given to harmonizing the use of expert judgment at various points in the process. EPA has developed guided expert judgment to synthesize evidence on the basis of modified Bradford Hill criteria and for integrating evidence across data streams. The agency has developed working groups with expertise (such as PBPK) that can be applied to the assessment process. The draft chloroform protocol identified some situations when expert judgment will be used, including evaluation of studies to identify characteristics that indicate how informative the results are (p. 16, line 21) to perform outcome-specific study evaluations (p. 16, line 24). Systematic Review Protocol for the IRIS Chloroform Assessment (Inhalation)b Slides 8, 48, 69‒86 9 3 EPA should establish a transparent process for initially identifying all putative adverse outcomes through a broad search of the literature. The agency should then develop a process that uses guided expert judgment to identify the specific adverse outcomes to be investigated, each of which would then be subjected to systematic review of human, animal, and in vitro or mechanistic data. EPA has developed assessment plans that provide information about IRIS Assessment Plan for Chloroform the scoping and problem formulation process. The plans are (Scoping and Problem Formulation developed by using expert judgment and input from EPA regional Materials)d offices and other stakeholders. Each assessment plan identifies the specific aims of the systematic review and the PECO statement. 10 3 For all literature searches, EPA should consult with an information specialist who is trained in conducting systematic reviews. EPA staff indicated that they use an information specialist. EPA protocol provides the name of the HERO librarian (see chloroform protocol, page vii);b that person has an MS in library and information science 11 3 EPA should include protocols for all systematic reviews conducted for a specific IRIS assessment as appendixes to the assessment. The IRIS program has developed draft systematic-review protocols that are undergoing public comment before being made final. The protocols contain many of the elements identified by the 2014 Systematic Review Protocol for the IRIS Chloroform Assessment (Inhalation), Appendix Ab 112 Item Chapter Recommendations from 2014 NRC Reporta Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Continued Systematic Review of Chloroprene Studies Published Since 2010 IRIS Assessmentc 12 4 The trajectory of change needs to be maintained. The IRIS program has been responsive to the recommendations made in the 2014 report and is continuing the trajectory of change. The changes appear to have accelerated with the recruitment of new NCEA and IRIS leadership. Workshop presentations, posters, and discussion Recent IRIS documents (such as plans, protocols, and assessments) and tools 13 4 The current process can be enhanced with more explicit documentation of methods. Protocols for IRIS assessments should include a section on evidence identification that is written in collaboration with information specialists trained in systematic reviews and that includes a search strategy for each systematic-review question being addressed in the assessment. Specifically, the protocols should provide a line-by-line description of the search strategy, the date of the search, and publication dates searched and, as noted in Chapter 3, explicitly state the inclusion and exclusion criteria for studies. EPA systematic-review protocols contain descriptions of how evidence will be identified, including relevant search terms and databases to be queried. The protocols also include descriptions of inclusion and exclusion criteria. Systematic Review Protocol for the IRIS Chloroform Assessment (Inhalation), Table 2 (p. 9), p. 12, Appendix Ab Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Copyright National Academy of Sciences. All rights reserved. report as meeting best practices defined by IOM. Furthermore, the chloroprene reassessment included as appendixes the literaturesearch strategy and approaches for evaluating risk of bias in epidemiology and other human studies. The study objective, PECO statement, and methods used to search and screen the literature and evaluate studies were included in the main body of the report. That approach is consistent with this recommendation. The committee expects that some items found in the protocol can be addressed in the handbook. Including the analysis plan in the systematic-review protocols might lead to additional amendments to the protocol that could be minimized if they used a separate analysis plan. Systematic Review of Chloroprene Studies Published Since 2010 IRIS Assessmentc 14 4 Evidence identification should involve a predetermined search EPA systematic-review protocols contain descriptions of how of key sources, follow a search strategy based on empirical evidence will be identified, including relevant search terms and research, and be reported in a standardized way that allows databases to be queried. replication by others. The search strategies and sources should be modified as needed on the basis of new evidence on best practices. Contractors who perform the evidence identification for the systematic review should adhere to the same standards and provide evidence of experience and expertise in the field. 15 4 EPA should consider developing specific resources, such as registries, that could be used to identify and retrieve information about toxicology studies reported outside the literature accessible by electronic searching. In the medical field, clinical-trial registries and US legislation that has required studies to register in ClinicalTrials.gov have been an important step in ensuring that the total number of studies that are undertaken is known. Systematic Review Protocol for the IRIS Chloroform Assessment (Inhalation), Appendix Ab This recommendation goes beyond the scope of the IRIS program Systematic Review Protocol for the and therefore was not addressed by the committee during its review. IRIS Chloroform Assessment Systematic-review protocols indicate that IRIS assessments include (Inhalation) (p. 11, line 14)b only publicly accessible, peer-reviewed information, which should be available through the databases identified by the IRIS program. 113 (Continued) Finding Evidence 16 4 EPA is encouraged to use at least two reviewers who work independently to screen and select studies, pending an evaluation of validity and reliability that might indicate that multiple reviewers are not warranted. It is important that the reviewers use standardized procedures and forms. EPA uses two persons to screen studies. Screeners use a structured form based on the PECO in DistillerSR. Systematic Review Protocol for the IRIS Chloroform Assessment (Inhalation) (p. 12, lines 11‒16)b Slide 39 17 4 EPA should engage information specialists trained in systematic reviews in the process of evidence identification, for example, by having an information specialist peer review the proposed evidence-identification strategy in the protocol for the systematic review. Systematic Review Protocol for the The IRIS assessment team includes an information specialist. The specific tasks completed by that person are not clear. It is hoped that IRIS Chloroform Assessment the handbook will clearly define the roles that the person has in the (Inhalation)b IRIS process. 18 4 EPA should encourage and support research on reporting biases and other methodologic topics relevant to the systematic-review process in toxicology. EPA is supporting and encouraging research through its collaborative efforts described at the workshop. The committee expects EPA research in this field to emerge as the IRIS program continues to develop expertise in systematic-review method development. Slides 79, 91, 149, 145, 150 19 4 EPA should continue to document and standardize its evidence-identification process by adopting (or adapting, where appropriate) the relevant IOM standards described in Table 4-1. It is anticipated that its efforts will further strengthen the overall consistency, reliability, and transparency of the evidence-identification process. Appropriate tools and methods for evidence identification were described and are being used. Systematic Review Protocol for the IRIS Chloroform Assessment (Inhalation)b Workshop presentations To advance the development of tools for assessing risk of bias in different types of studies (human, animal, and mechanistic) used in IRIS assessments, EPA should explicitly identify factors, in addition to those discussed in this chapter, that can lead to bias in animal studies—such as control for litter effects, dosing, and methods for exposure assessment—so that these factors are consistently evaluated for experimental studies. The draft chloroform protocol describes the domains to be considered in the evaluation of epidemiology studies and animal toxicity studies. Domain ratings and their descriptions have also been provided. EPA also presented heat maps of risk-of-bias analyses for studies performed by the IRIS program. Systematic Review Protocol for the IRIS Chloroform Assessment (Inhalation) (Tables 5‒6)b 20A 5 Slides 53, 55 20B 5 Likewise, EPA should consider a tool for assessing risk of bias The 2014 report noted that few tools were available for assessing in in vitro studies. risk of bias in in vitro studies. Fully developed tools that meet the needs of the IRIS program are not available. EPA is exploring adaptations of existing tools for its purpose. Slide 78 21A 5 When considering any method for evaluating individual studies, EPA should select a method that is transparent, reproducible, and scientifically defensible. Systematic Review Protocol for the IRIS Chloroform Assessment (Inhalation)b Systematic Review of Chloroprene Studies Published Since 2010 IRIS Assessmentc Slides 50‒63 EPA has adopted systematic-review methods that are transparent and scientifically defensible. 114 Copyright National Academy of Sciences. All rights reserved. Item Chapter Recommendations from 2014 NRC Reporta Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Continued Whenever possible, there should be empirical evidence that the EPA is using and adapting risk-of-bias tools appropriately. methodologic characteristics that are being assessed in the IRIS protocol have systematic effects on the direction or magnitude of the outcome. The methodologic characteristics that are known to be associated with a risk of bias should be included in the assessment tool. Additional quality-assessment items relevant to a particular systematic-review question could also be included in the EPA assessment tool. Systematic Review Protocol for the IRIS Chloroform Assessment (Inhalation)b Systematic Review of Chloroprene Studies Published Since 2010 IRIS Assessmentc Slides 52‒63 Posters D-4, D-5, D-9 22 5 EPA should carry out, support, or encourage research on the EPA is supporting and encouraging research through its development and evaluation of empirically based instruments collaborative efforts described in the workshop. for assessing bias in human, animal, and mechanistic studies relevant to chemical-hazard identification. Specifically, there is a need to test existing animal-research assessment tools on other animal models of chemical exposures to ensure their relevance and generalizability to chemical-hazard identification. Furthermore, EPA might consider pooling data collected for IRIS assessment to determine whether, among various contexts, candidate risk-of-bias items are associated with overestimates or underestimates of effect. Slides 145, 149 23 5 Although additional methodologic work might be needed to establish empirically supported criteria for animal or mechanistic studies, an IRIS assessment needs to include a transparent evaluation of the risk of bias of studies used by EPA as a primary source of data for the hazard assessment. EPA should specify the empirically based criteria it will use to assess risk of bias for each type of study design in each type of data stream. EPA has adapted existing risk-of-bias tools for its use. Draft protocols describe the domains to be considered in the evaluation of epidemiology studies and animal toxicity studies. Domain ratings and their descriptions have also been provided. EPA also presented heat maps of risk-of-bias analyses for studies performed by the IRIS program. Tools have not been developed for mechanistic studies. Systematic Review Protocol for the IRIS Chloroform Assessment (Inhalation) (Tables 5 and 6)b Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Copyright National Academy of Sciences. All rights reserved. 21B 5 Slides 53, 78 24 5 To maintain transparency, EPA should publish its risk-of-bias assessments as part of its IRIS assessments. It could add tables that describe the assessment of each risk-of-bias criterion for each study and provide a summary of the extent of the risk of bias in the descriptions of each study in the evidence tables. EPA presented example heat maps of risk-of-bias analyses for studies performed by the IRIS program. The heat maps have been included in a recent assessment. Systematic Review of Chloroprene Studies Published Since 2010 IRIS Assessment (Figure 2)c 25 5 EPA should develop terminology for potential sources of bias with definitions that can be applied during systematic reviews. EPA has adapted existing risk-of-bias tools for its use. The draft chloroform protocol describes the domains to be considered in the evaluation of epidemiology studies and animal toxicity studies. Reporting bias was not included as a domain for epidemiology studies, and its omission is not consistent with standard systematicreview methods for assessing risk of bias. Systematic Review Protocol for the IRIS Chloroform Assessment (Inhalation) (Tables 5‒6)b Slides 55, 57 (Continued) 115 Chapter Recommendations from 2014 NRC Reporta Finding Evidence 26 5 Funding sources should be considered in the risk-of-bias assessment conducted for systematic reviews that are part of an IRIS assessments EPA documents funding source, but it is unclear how the data are used. Workshop discussion 27A 5 EPA should contact investigators to obtain missing information that is needed for the evaluation of risk of bias and other quality characteristics of included studies. Investigators are contacted on a case-by-case basis that depends partly on the expected effect of the missing data. IRIS systematicreview protocols also indicate that decisions are made on an assessment-specific basis. If the information is not reported, it is generally not useful to reach out to the study authors. However, if missing study details could change confidence in study conclusions, efforts should be made to contact the study authors. Outreach to study authors is documented and considered unsuccessful if researchers do not respond to multiple e-mail or phone requests within a reasonable period. Systematic Review Protocol for the IRIS Chloroform Assessment (Inhalation) (Table 6 (p. 25); p. 18, line 41)b 27B 5 The committee expects that, as happened in the clinical literature This recommendation goes beyond the scope of the IRIS program in which additional reporting standards for journals were and therefore was not addressed during this review. implemented (Turner et al. 2012), the reporting of toxicologic research will eventually improve as risk-of-bias assessments are incorporated into the IRIS program. However, a coordinated approach by government agencies, researchers, publishers, and professional societies will be needed to improve the completeness and accuracy of reporting toxicology studies in the near future. 28 5 The risk-of-bias assessment of individual studies should be carried forward and incorporated into the evaluation of evidence among data streams. 29 6 EPA should continue to improve its evidence-integration process The IRIS process continues to use a guided expert judgment incrementally and enhance the transparency of its process. It process, but structured sets of categorical criteria for decisionshould either maintain its current guided-expert-judgment making within that process are more explicitly defined. process but make its application more transparent or adopt a structured (or GRADE-like) process for evaluating evidence and rating recommendations along the lines that NTP has taken. If EPA does move to a structured evidence-integration process, it should combine resources with NTP to leverage the intellectual resources and scientific experience in both organizations. The committee does not offer a preference but suggests that EPA consider which approach best fits its plans for the IRIS process. The results of the evaluation of individual studies are a critical component of the current evidence synthesis processes and integration frameworks. Risk of bias is one factor that EPA uses to determine an overall study confidence rating for epidemiology and animal toxicity studies. High- or medium-confidence studies are favored for quantitative dose‒response analysis. Slides 66, 54, 71‒73, 81 Slides 67, 79‒86 116 Copyright National Academy of Sciences. All rights reserved. Item Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Continued 6 EPA should expand its ability to perform quantitative modeling of evidence integration; in particular, it should develop the capacity to do Bayesian modeling of chemical hazards. That technique could be helpful in modeling assumptions about the relevance of a variety of animal models to each other and to humans, in incorporating mechanistic knowledge to model the relevance of animal models to humans and the relevance of human data for similar but distinct chemicals, and in providing a general framework within which to update scientific knowledge rationally as new data become available. The committee emphasizes that the capacity for quantitative modeling should be developed in parallel with improvements in existing IRIS evidence-integration procedures and that IRIS assessments should not be delayed while this capacity is being developed. EPA illustrated its use of meta-analysis of human and animal studies for evidence integration. Bayesian methods are being explored to help to characterize uncertainty and to combine evidence to identify hazard. New methods and assays are increasingly being evaluated quantitatively. Slide 130 Posters provided examples that show how EPA uses new approach methods as part of a chemical assessment process 31 6 EPA should develop templates for structured narrative justifications of the evidence-integration process and conclusion. The premises and structure of the argument for or against a chemical’s posing a hazard should be made as explicit as possible, should be connected explicitly to evidence tables produced in previous stages of the IRIS process, and should consider all lines of evidence (human, animal, and mechanistic) used to reach major conclusions. The 2017 Toxicological Profile for Benzo[a]pyrene shows welldeveloped evidence tables that support the structured narrative and conclusion regarding carcinogenicity. For other effects, the evidence is described as ranging from “strongest evidence for human hazards” to “less robust evidence.” Workshop discussion and the chloroform protocol show progress in template development. EPA staff stated that the approach to standardization of hazard descriptors for noncancer effects is being tested and discussed in the agency. Slides 80‒86 2017 IRIS Toxicological Profile for Benzo[a]pyrenee Systematic Review Protocol for the IRIS Chloroform Assessment (Inhalation)b 32 6 Guidelines for evidence integration for cancer and noncancer end Although EPA has not developed these guidelines, the issue goes points should be more uniform. beyond the IRIS program with respect to agency procedures. However, the IRIS program has developed frameworks for evidence integration and is testing and discussing how conclusions should be summarized. 33 7 EPA should develop criteria for determining when evidence is sufficient to derive toxicity values. One approach would be to restrict formal dose-response assessments to when a standard descriptor characterizes the level of confidence as medium or high (as in the case of noncancer end points) or as “carcinogenic to humans” or “likely to be carcinogenic to humans” for carcinogenic compounds. Another approach, if EPA adopts probabilistic hazard classification, is to conduct formal doseresponse assessments only when the posterior probability that a human hazard exists exceeds a predetermined threshold, such as 50% (more likely than not likely that the hazard exists). Progress has been made. Quantitative toxicity values are restricted to studies with strongest conclusions for a human health effect (for cancer, a descriptor of Known) or a moderately strong conclusion for a human health effect (for cancer, a descriptor of Likely). Criteria are not provided for inclusion of studies that are considered on a case-by-case basis when a weaker conclusion regarding a human health effect (for cancer, a descriptor of Suggestive) is reached. IRIS has not produced final descriptors for noncancer effects and mechanistic studies other than review and application of PK/PBPK models. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Copyright National Academy of Sciences. All rights reserved. 30 Systematic Review of Chloroprene Studies Published Since 2010 IRIS Assessmentc 2017 IRIS Toxicological Profile for Benzo[a]pyrenee Slides 131‒133 (Continued) 117 Chapter Recommendations from 2014 NRC Reporta Finding Evidence 34 7 EPA should continue its shift toward the use of multiple studies rather than single studies for dose-response assessment but with increased attention to risk of bias, study quality and relevance in assessing human dose-response relationships. For that purpose, EPA will need to develop a clear set of criteria for judging the relative merits of individual mechanistic, animal, and epidemiologic studies for estimating human dose-response relationships. Progress has been made toward using multiple studies or end points and comparing multiple candidate toxicity values. IRIS assessments provide one or more candidate toxicity values for use by risk managers. The IRIS program considers the quality of studies when deciding which studies will be advanced for quantitative dose‒response modeling; studies rated as having medium or high confidence will be advanced for dose‒response considerations. Other study attributes—such as relevance of a species to humans, relevance of an exposure route, and susceptibility—might also be considered. EPA is developing new tools for making and visualizing comparisons. Slides 62, 130‒135, 142‒146 2012 IRIS Toxicological Review of Tetrachloroethylened Workshop demonstrations of HAWC and SWIFT EPA recognizes that there is no one-size-fits-all sets of criteria for inclusion of mechanistic studies, but the criteria for evaluating PK/PBPK models and how they are applied in dose‒ response and toxicity-value determinations are a good start. 35 7 EPA should use formal methods for combining multiple studies and the derivation of IRIS toxicity values with an emphasis on a transparent and replicable process. IRIS has begun to develop and apply tools in response to this recommendation. EPA presented two demonstrations for metaregression and Bayesian approaches that showcase the agency efforts. EPA has not presented criteria for when and how new tools should be used. Tool development and application will be a continuing process that requires sustained resources and continued capacity-building. Slide 140 Case studies provided for alternative dose estimates (posters D-2, D-10) 36 7 EPA should clearly present two dose-response estimates: a central estimate (such as a maximum likelihood estimate or a posterior mean) and a lower-bound estimate for a POD from which a toxicity value is derived. The lower bound becomes an upper bound for a cancer slope factor but remains a lower bound for a reference value. EPA indicated that this approach is now standard procedure. Several examples were presented that show comparisons between BMDs and BMDLs and demonstrate how key studies compare with other supporting studies Slides 134, 135; posters 37 7 As the IRIS program evolves, EPA should develop and expand its use of Bayesian or other formal quantitative methods in data integration for dose-response assessment and derivation of toxicity values. Demos show the beginning stage of IRIS efforts on applications of Bayesian methods. Case studies (Poster D-10) EPA has not yet developed criteria for when and how new tools should be used. Slides 136, 139, 140, 143‒146 New research is under way to address New Approach Methods, such as data-mining, cheminformatics, high-throughput exposure modeling and toxicokinetics, and visualization tools. 118 Copyright National Academy of Sciences. All rights reserved. Item Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation Continued 7 Uncertainty analysis should be conducted systematically and coherently in IRIS assessments. To that end, EPA should develop IRIS-specific guidelines to frame uncertainty analysis and uncertainty communication. Moreover, uncertainty analysis should become an integral component of the IRIS process. Efforts are beginning with development of model-averaging approaches and adoption of WHO/IPCS guidance for reporting toxicity values and their uncertainty. IRIS specific guidance has yet to be developed because tools and approaches remain under development. Progress Toward Transforming the Integrated Risk Information System (IRIS) Program: A 2018 Evaluation 38 Cooper et al. (2016)g Slides 137, 138 Case studies (Poster D-9) aNRC. 2014. Review of EPA’s Integrated Risk Information System (IRIS) Process. Washington, DC: The National Academies Press. 2018. Systematic Review Protocol for the IRIS Chloroform Assessment (Inhalation) [CASRN 67-66-3]. EPA/635/R-17/486. Integrated Risk Information System, National Center for Environmental Assessment, Office of Research and Development, U.S. Environmental Protection Agency, Washington, DC. cOrme-Zavaleta, J. 2018. Response to the Request for Correction (RFC). Letter to Robert Holden, Liskow & Lewis, New Orleans, LA, from Jennifer Orme-Zavaleta, Principal Deputy Assistant Administrator for Science, Office of Research and Development, Washington, DC, January 25, 2018; Attachment 1. EPA Response to the Denka Performance Elastomers (DPE) Request for Correction of the Toxicological Review of Chloroprene (CAS No. 126-99-8) In Support of Summary Information on the Integrated Risk Information System (IRIS); Attachment 2. Systematic Review of Chloroprene [CASRN 126-99-80] Studies Published Since 2010 IRIS Assessment to Support Consideration of the Denka Request for Correction (RFC). January 2018 [online]. Available: https://www.epa.gov/sites/production/files/2018-01/documents/epa_repsonse_to_mr._holdren_jan_25_ 2018_complete.pdf [accessed February 9, 2018]. dEPA. 2017a. IRIS Assessment Plan for Chloroform [CASRN 67-66-3]. EPA/635/R-17/330. Integrated Risk Information System, National Center for Environmental Assessment, Office of Research and Development, U.S. Environmental Protection Agency, Washington, DC. eEPA. 2017b. Toxicological Review of Benzo[a]pyrene. EPA/635/R-17/003Fa. Integrated Risk Information System, National Center for Environmental Assessment, Office of Research and Development, U.S. Environmental Protection Agency, Washington, DC [online]. Available: https://cfpub.epa.gov/ncea/iris/iris_documents/documents/toxreviews/ 0136tr.pdf [accessed February 14, 2018]. fEPA. 2012. Toxicological Review of Tetrachloroethylene (Perchloroethylene) [CAS No. 127-18-4]. EPA/635/R-80/011F. U.S. Environmental Protection Agency, Washington, DC. February 2012 [online]. Available: https://cfpub.epa.gov/ncea/iris/iris_documents/documents/toxreviews/0106tr.pdf [accessed February 27, 2018]. gCooper, G.S., R.M. Lunn, M. Ågerstrand, B.S. Glenn, A.D. Kraft, A.M. Luke, and J.M. Ratcliffe. 2016. Study sensitivity: Evaluating the ability to detect effects in systematic reviews of chemical exposures. Environ. Int. 92-93:605-610. bEPA. Copyright National Academy of Sciences. All rights reserved. 119