SCIENTIFIC EXCELLENCE IN THE FORENSIC
SCIENCE COMMUNITY
Alice R. Isenberg* & Cary T. Oien**
INTRODUCTION
The practice of forensic science has existed for centuries. Each year,
hundreds of thousands of cases are closed, suspects cleared, and offenders
convicted through routine, accurate, and reliable forensic testing. Forensic
testing includes chemical analysis to determine the nature of seized drugs;
examinations performed on physical materials such as fibers, glass, and
spent bullet casings; and examination of biological materials such as DNA.1
Tests performed for each of these examinations, regardless of the materials
examined, are strictly prescribed by laboratory policies, supported by peerreviewed research, and lead to accurate and reliable results.2
A casual reader of recent media reports might be led to believe that
forensic science lacks any scientific credibility.3 However, this narrative is
completely inaccurate and at odds with the scientific excellence that exists
throughout the forensic science community. Forensic disciplines are
grounded in diverse sciences such as chemistry, biology, and physics, and
every forensic discipline practiced in an accredited forensic laboratory must
demonstrate that it is reliable, accurate, and fit for its intended use.

* Deputy Assistant Director, Laboratory Division, Federal Bureau of Investigation.
** Senior Forensic Scientist, Laboratory Division, Federal Bureau of Investigation. The
authors would like to thank the Federal Bureau of Investigation Office of General Counsel
and the U.S. Department of Justice for their thoughtful reviews of this paper. This is
publication 18-08 of the Federal Bureau of Investigation Laboratory. Names of commercial
products are provided for identification purposes only, and inclusion does not imply
endorsement by the Federal Bureau of Investigation. The views expressed are those of the
authors and do not necessarily reflect the official policy or position of the Federal Bureau of
Investigation or the U.S. Government.
This Article was prepared as a companion to the Fordham Law Review Reed Symposium
on Forensic Expert Testimony, Daubert, and Rule 702, held on October 27, 2017, at Boston
College School of Law. The Symposium took place under the sponsorship of the Judicial
Conference Advisory Committee on Evidence Rules. For an overview of the Symposium,
see Daniel J. Capra, Foreword: Symposium on Forensic Testimony, Daubert, and Rule 702,
86 FORDHAM L. REV. 1459 (2018).
1. See
Laboratory
Services,
FBI,
https://www.fbi.gov/services/laboratory
[https://perma.cc/6NQP-7YQ7] (last visited Feb. 26, 2018).
2. Id.
3. See Eric S. Lander, Fix the Flaws in Forensic Science, N.Y. TIMES (Apr. 21, 2015),
https://www.nytimes.com/2015/04/21/opinion/fix-the-flaws-in-forensic-science.html
[https://perma.cc/5FV8-BE4M].

39

 40

SCIENTIFIC EXCELLENCE

[Vol. 86

I. ACCREDITATION AND ITS REQUIREMENTS
Accreditation and quality assurance systems assure the public that
accredited organizations are competent and their results can be relied upon.4
Many groups—such as the National Commission on Forensic Science,5 the
National Academy of Sciences,6 the President’s Council of Advisors on
Science and Technology (PCAST),7 and the Department of Justice (DOJ)—
recognize that accreditation is critically important. In fact, in December
2015, the Attorney General directed that all DOJ forensic laboratories must
obtain or maintain accreditation.8
Accreditation is an external assessment of a laboratory’s technical
Accreditation
competence to perform specific types of testing.9
demonstrates that a laboratory is performing its work correctly and
consistent with appropriate standards.10 To maintain this recognition, a
laboratory is periodically reevaluated to ensure its ongoing compliance with
accreditation requirements.11 Laboratory accreditation is internationally
regarded as a reliable indicator of technical competence, and it provides
credibility and public confidence in laboratory operations.12 An accredited
laboratory’s quality assurance system must include written standard
operating procedures, proficiency testing, training programs, processes for
technical review of reports, testimony monitoring, and many other
requirements.13

4. See Forensic Accreditation, ANSI-ASQ NAT’L ACCREDITATION BD.,
https://www.anab.org/forensic-accreditation [https://perma.cc/E4WG-DHYC] (last visited
Feb. 26, 2018).
5. See NAT’L COMM’N ON FORENSIC SCI., RECOMMENDATION ON UNIVERSAL
ACCREDITATION
(2015),
https://www.justice.gov/archives/ncfs/file/477851/download
[https://perma.cc/24BX-93RG].
6. See NAT’L RESEARCH COUNCIL, NAT’L ACADS., STRENGTHENING FORENSIC SCIENCE
IN THE UNITED STATES: A PATH FORWARD 6 (2009), https://www.ncjrs.gov/pdffiles1/nij/
grants/228091.pdf [https://perma.cc/9Q84-RC6S].
7. See President’s Council of Advisors on Sci. & Tech., An Addendum to the PCAST
Report on Forensic Science in Criminal Courts, WHITE HOUSE 4 (Jan. 6, 2017),
https://obamawhitehouse.archives.gov/sites/default/files/microsites/ostp/PCAST/pcast_foren
sics_addendum_finalv2.pdf [https://perma.cc/8A4D-57HX] (“Forensic scientists cite the role
of professional organizations, certification, accreditation, best practices manuals, and
training within their disciplines. PCAST recognizes that such practices play a critical role in
any professional discipline.”).
8. Press Release, U.S Dep’t of Justice, Justice Department Announces New
Accreditation Policies to Advance Forensic Science (Dec. 7, 2015), https://www.justice.gov/
opa/pr/justice-department-announces-new-accreditation-policies-advance-forensic-science
[https://perma.cc/YFK7-NEYL].
9. Forensic Accreditation, supra note 4.
10. See id.
11. See Laboratory Services, supra note 1.
12. The Advantages of Being Accredited, INT’L LAB. ACCREDITATION COOPERATION
(2015), http://ilac.org/?ddownload=898 [https://perma.cc/X9X8-ZLPZ].
13. See Laboratory Services, supra note 1.

 2018]

FORDHAM LAW REVIEW ONLINE

41

According to the Bureau of Justice Statistics, 88 percent of the 409
publicly-funded forensic crime labs in the United States are accredited.14
Unaccredited labs are often very small—less than ten people—and offer
services in a limited number of disciplines. In addition to forensic
laboratories, laboratories performing other types of tests are accredited
This includes
according to the same international standard.15
environmental labs checking for levels of lead in groundwater, chemistry
labs preparing chemicals for consumer use, or food labs ensuring the safety
of our food supply.16
The validation of test methods is also an accreditation requirement.
Validation is the “confirmation, through the provision of objective evidence
(3.8.3), that the requirements (3.6.4) for a specific intended use or
application have been fulfilled.”17 Validation experiments are designed to
determine whether a method yields correct results when the right answer—
the ground truth—is known. These are empirical tests that are conducted
prior to laboratory implementation of a method.
Validation experiments are fundamentally different than equipment
checks, which simply ensure that a particular piece of equipment is
operating within defined parameters.18 Validation may test the limitations
of a method by analyzing a wide range of factors that are relevant and
appropriate to a given application.19 When validation provides insight
regarding the limitations of a method, these limitations should be shared in
legal proceedings.20 The focus of validation is to confirm, through
objective data, that the requirements for a specific intended use are
fulfilled.21 In contrast, method verification is the confirmation that the
laboratory can properly use the method prior to its implementation.22
However, neither method validation nor verification can produce an error
rate for an entire discipline or an individual examiner.

14. ANDREW M. BURCH ET AL., U.S. DEP’T OF JUSTICE, PUBLICLY FUNDED FORENSIC
CRIME LABORATORIES: QUALITY ASSURANCE PRACTICES, 2014, at 1 (2016),
https://www.bjs.gov/ content/pub/pdf/pffclqap14.pdf [https://perma.cc/MYS4-L6JS].
15. See ISO/IEC 17025:2017, ISO, https://www.iso.org/obp/ui/#iso:std:iso-iec:
17025:ed-3:v1:en [https://perma.cc/27J4-Y43U] (last visited Feb. 26, 2018).
The
International Organization for Standardization (ISO) is an independent, non-governmental
international organization with a membership of 161 national standards bodies. All About
ISO, ISO, https://www.iso.org/about-us.html [https://perma.cc/CP5P-BE7M] (last visited
Feb. 26, 2018). The members develop voluntary, consensus-based market-relevant
standards. Id. The ISO/IEC 17025:2017 standard focuses specifically on requirements for
the competence, impartiality, and consistent operation of testing and calibration laboratories.
See ISO/IEC 17025:2017, supra.
16. Forensic Accreditation, supra note 4.
17. ISO 9000:2015, ISO, https://www.iso.org/obp/ui/#iso:std:iso:9000:ed-4:v1:en
[https://perma.cc/ZLW3-RG7L] (last visited Feb. 26, 2018).
18. See Validation Information for Public DNA Laboratories, NAT’L INST. JUST.,
https://www.nij.gov/topics/forensics/lab-operations/Pages/validation.aspx
[https://perma.cc/74J5-Q97C] (last visited Feb. 26, 2018).
19. See id.
20. See id.
21. See id.
22. ISO/IEC 17025:2017, supra note 15.

 42

SCIENTIFIC EXCELLENCE

[Vol. 86

Forensic examiners must complete extensive training to be qualified to
perform casework in accredited laboratories.23 Training programs can be
one to two years, or longer, and require examiners to perform analyses on
samples with a known correct answer.24 The examiner must also
demonstrate a thorough understanding of the science behind the method
employed and an understanding of lab policies, procedures, legal rules,
evidence handling, etc.25 The examiner must undergo oral examinations,
mock trials, and competency tests for which the correct answer is known.26
In addition, all examiners must demonstrate competency to apply the
processes accurately and reliably before they are assigned actual cases.27
Once qualified to conduct casework in an accredited laboratory, every
examiner undergoes continual competency monitoring through proficiency
tests administered at least once per year.28
Testimony monitoring is also a requirement for accredited laboratories.29
The Federal Bureau of Investigation (FBI) Laboratory requires that
examiners request a transcript for each testimony provided.30 FBI
examiners also must follow approved standards for scientific testimony and
reports,31 which document the acceptable range of conclusions expressed in
both laboratory reports and testimony.32 The DOJ is developing similar
documents called Uniform Language for Testimony and Reports,33 as well
as a testimony-monitoring framework, which will apply to all DOJ
laboratories.34 The purpose of these testimony-monitoring activities is to
prevent examiner testimony from exceeding scientific limitations.35
Forensic science research plays a critical role in the culture of continuous
improvement that is part of a rigorous quality assurance program. Such
research seeks to develop new capabilities while providing enhancement
and support for existing capabilities. For example, forensic science
23. See generally AM. SOC’Y CRIME LAB. DIRS./LAB. ACCREDITATION BD.,
SUPPLEMENTAL REQUIREMENTS FOR THE ACCREDITATION OF FORENSIC SCIENCE TESTING
LABORATORIES
(2011),
http://des.wa.gov/sites/default/files/public/documents/About/1063/RFP/Add7Item4
ASCLD.pdf [https://perma.cc/45LX-4NZU].
24. Id. at 21.
25. See id. at 11–13.
26. Id.
27. Id.
28. Id. at 18.
29. U.S. GOV’T ACCOUNTABILITY OFFICE, CHEMISTRY AND TRACE EVIDENCE UNITS
GENERALLY ADHERE TO QUALITY STANDARDS, BUT COULD REVIEW MORE EXAMINER
TESTIMONIES (2017), https://www.gao.gov/assets/690/685507.pdf [https://perma.cc/RS9VXHKV].
30. Id.
31. See Forensic Science, U.S. DEP’T JUST., https://www.justice.gov/archives/dag/
forensic-science#prop [https://perma.cc/APD8-AZLW] (last visited Feb. 26, 2018) (denoting
laboratory standards and reporting requirements).
32. Id.
33. See id.
34. Press Release, U.S Dep’t of Justice, Justice Department Announces Plans to
Advance Forensic Science (Aug. 7, 2017), https://www.justice.gov/opa/pr/justicedepartment-announces-plans-advance-forensic-science [https://perma.cc/56VJ-F39E].
35. See id.

 2018]

FORDHAM LAW REVIEW ONLINE

43

research and development in the 1980s provided the groundwork for
monumental progress in the development and advancement of DNA
testing.36 In addition, the FBI’s studies on latent print examinations have
provided tremendous insight into the reliability of latent fingerprint
examination and the decision-making process of latent print examiners.37
The FBI recognizes the importance of these studies and has begun similar
studies in three pattern-comparison disciplines.
Accreditation, validation, research, training, and testimony monitoring
are important activities for demonstrating the reliable practice of forensic
science. Together, they help ensure the overarching goals of finding the
right answer, correctly communicating that answer, and continuously
improving our ability to deliver quality results.
II. VALIDATION STUDIES
Validation is the process used to determine whether or not a method or
technique is fit for a given application. The PCAST Report (“Report”)
asserts that a forensic discipline must demonstrate “foundational validity”
and “validity as applied” for a discipline to be scientifically valid and
reliable.38 However, the Report conflates two disparate topics in its
discussion of “validity.”
The authors claim that foundational validity requires the performance of
multiple “black box” studies.39 However, black-box studies are decisionanalysis experiments performed across a broad range of practitioners, and
do not validate a specific methodology.40 The authors do not offer any
scientific basis to support their assertions. At the same time, the authors
encourage federal judges to “take into account the appropriate scientific
criteria for assessing scientific validity” in their “gatekeeper” role.41 The
authors use their unique criteria in their effort to discredit numerous
validation studies.42 They argue that, because the research reviewed did not
36. See Peter Gill et al., Forensic Application of DNA ‘Fingerprints,’ 318 NATURE 577
(1985).
37. See generally R. Austin Hicklin et al., Assessing the Clarity of Friction Ridge
Impressions, 226 FORENSIC SCI. INT’L 106 (2013); R. Austin Hicklin et al., Latent
Fingerprint Quality: A Survey of Examiners, 61 J. FORENSIC IDENTIFICATION 385, 385–419
(2011); Bradford T. Ulery et al., Changes in Latent Fingerprint Examiners’ Markup Between
Analysis and Comparison, 247 FORENSIC SCI. INT’L 54 (2015); Bradford T. Ulery et al.,
Measuring What Latent Fingerprint Examiners Consider Sufficient Information for
Individualization Determinations, 9 PLOS ONE 1 (2014); Bradford T. Ulery et al.,
Repeatability and Reproducibility of Decisions by Latent Fingerprint Examiners, 7 PLOS
ONE 1 (2012); Bradford T. Ulery et al., Understanding the Sufficiency of Information for
Latent Fingerprint Value Determinations, 230 FORENSIC SCI. INT’L 99 (2013).
38. PRESIDENT’S COUNCIL OF ADVISORS ON SCI. & TECH., EXEC. OFFICE OF THE
PRESIDENT, FORENSIC SCIENCE IN CRIMINAL COURTS: ENSURING SCIENTIFIC VALIDITY OF
FEATURE-COMPARISON METHODS 44–56 (2016) [hereinafter PCAST REPORT],
https://obamawhitehouse.archives.gov/sites/default/files/microsites/ostp/PCAST/pcast_foren
sic_science_report_final.pdf [https://perma.cc/VJB4-5JVQ].
39. Id.
40. See id. at 65–66.
41. Id. at 41, 142.
42. See generally id.

 44

SCIENTIFIC EXCELLENCE

[Vol. 86

fit the authors’ validation paradigm, these scientific disciplines lack
empirical evidence to support PCAST’s approval as valid science.43 This
position ignores much peer-reviewed research, overlooks critical aspects of
many studies, and fails to acknowledge the empirical value of these studies.
For example, in the firearms discipline, PCAST ignores a large number
of studies based on their criticism of the test designs. Two studies were
discarded because for using a “within-set comparison” design in which the
samples were examined in a pair-wise approach.44 The Report asserted that
it was “impossible to estimate the false positive rate among conclusive
examinations, which is the key measure for consideration” as the reason for
rejecting these studies.45 However, these same studies showed that of 1037
different-source comparisons performed, no false identifications or false
eliminations were reported.46 PCAST dismissed four additional studies
based on the use of a “closed-set” experimental design because the source
gun was always present.47 PCAST opined that “the closed-set design is
problematic in principle,” and was therefore “not appropriate for assessing
scientific validity.”48 In these studies, the researchers utilized particularly
challenging samples that employed consecutively-manufactured firearms.49
This represents the worst-case scenario for toolmarks that carry over from
one machined part to the next. Despite these challenging samples, all of
these studies showed that firearms examiners reliably and accurately
associated the questioned toolmarks with the correct source.50 In a final
example, PCAST ignored another study due to the partly open-set design, in
which some of the questioned samples did not have a matching known
standard.51 Each of these studies provide important insights into the
43. See id.
44. Id. at 107 & nn.319–20; see also Charles S. DeFrance & Michael D. Van Arsdale,
Validation Study of Electrochemical Rifling, 35 ASS’N FIREARM & TOOLMARK EXAMINERS J.
1 (2003); Erich D. Smith, Cartridge Case and Bullet Comparison Validation Study with
Firearms Submitted in Casework, 36 ASS’N FIREARM & TOOLMARK EXAMINERS J. 130
(2005).
45. PCAST REPORT, supra note 38, at 107.
46. Id. at 107 n.320.
47. Id. at 108–09 & nn.324–28; see also David J. Brundage, The Identification of
Consecutively Rifled Gun Barrels, 30 ASS’N FIREARM & TOOLMARK EXAMINERS J. 438, 438–
44 (1998); Thomas G. Fadul Jr. et al., An Empirical Study to Improve the Scientific
Foundation of Forensic Firearm and Tool Mark Identification Utilizing Ten Consecutively
Manufactured Slides, 45 ASS’N FIREARM & TOOLMARK EXAMINERS J. 376, 376–93 (2013);
James E. Hamby et al., The Identification of Bullets Fired from Ten Consecutively Rifled
9mm Ruger Pistol Barrels: A Research Project Involving 507 Participants from Twenty
Countries, 41 ASS’N FIREARM & TOOLMARK EXAMINERS J. 99, 99–100 (2009); Angela
Stroman, Empirically Determined Frequency of Error in Cartridge Case Examinations
Using a Declared Double-Blind Format, 46 ASS’N FIREARM & TOOLMARK EXAMINERS J.
157, 157–75 (2014).
48. PCAST REPORT, supra note 38, at 109.
49. See supra note 47 and accompanying text.
50. See supra notes 44–49 and accompanying text.
51. PCAST REPORT, supra note 38, at 108–09 & n.326; see also THOMAS G. FADUL JR.
ET AL., MIAMI-DADE POLICE DEP’T CRIME LAB., AN EMPIRICAL STUDY TO IMPROVE THE
SCIENTIFIC FOUNDATION OF FORENSIC FIREARM AND TOOL MARK IDENTIFICATION UTILIZING
CONSECUTIVELY MANUFACTURED GLOCK EBIS BARRELS WITH THE SAME EBIS PATTERN
(2013), www.ncjrs.gov/pdffiles1/nij/grants/244232.pdf [https://perma.cc/6UKW-6FYF].

 2018]

FORDHAM LAW REVIEW ONLINE

45

science of firearms analysis and additional empirical support for the validity
of the discipline.
III. ERROR RATE STUDIES
SHOULD NOT BE MISTAKEN FOR VALIDATION
The Report claimed that “the foundational validity of subjective methods
can be established only through empirical studies of examiner’s
performance . . . such studies are referred to as ‘black-box’ studies.”52
Black-box studies are a way to analyze the decisions made by a range of
examiners under defined conditions.53 However, black-box studies should
not be mistaken as a way to establish the validity of a specific method or the
error rate for an entire discipline. The entire body of research and testing
relative to a particular forensic method provides support for its scientific
validity—not simply the number of black-box studies performed.
Some forensic commentators conflate distinct issues when criticizing the
reliability of forensic disciplines.54 These issues concern whether: (1) a
forensic discipline is scientifically valid and based upon sound scientific
principles; (2) individual practitioners can identify the right answer when
that answer is known (personal error rate); (3) different practitioners obtain
the same answer when reviewing the same materials and data; and (4) there
is a universal error rate for a specific discipline.55 The Report focuses on
the fourth issue,56 which has nothing to do with the method validity, but
rather the decisions made by examiners under a defined range of conditions.
Furthermore, it is problematic when scientific validity is confused with the
legal standard for admissibility. Unfortunately, the Report only exacerbates
the confusion.
Regarding microscopic hair analysis, PCAST discussed a 2002 paper by
Max M. Houck and Bruce Budowle.57 The study found that in 11 percent
of cases in which hairs were microscopically associated, DNA analysis
revealed that the samples originated from different individuals.58
Unfortunately, many misinterpret the results of this study to mean that
microscopic hair comparison has an 11 percent error rate. PCAST correctly
noted that these associations may not have been incorrect but, instead, were
simply characteristics that were shared by chance.59 Because microscopic
hair comparison involves class-level associations, hair cannot be used as a
52. PCAST REPORT, supra note 38, at 49.
53. Id.
54. See generally KELLY PYREK, FORENSIC SCIENCE UNDER SIEGE: THE CHALLENGES OF
FORENSIC LABORATORIES AND THE MEDICO-LEGAL INVESTIGATION SYSTEM (2007).
55. See id. at 241–43 (discussing the need to disclose errors, error rates, and sources of
errors in forensic science experiments to maintain confidence in the scientific integrity of the
results).
56. See generally PCAST REPORT, supra note 38.
57. Id. at 28 & n.33; see also Max M. Houck & Bruce Budowle, Correlation of
Microscopic and Mitochondrial DNA Hair Comparisons, 47 J. FORENSIC SCIS. 1, 1–4
(2002).
58. PCAST REPORT, supra note 38, at 28.
59. Id.

 46

SCIENTIFIC EXCELLENCE

[Vol. 86

unique identifier.60 One key point of the Houck and Budowle study that
PCAST did not discuss was the combined power of discrimination by the
joint use of microscopic and mitochondrial DNA analysis.61 Instead, the
Report characterized the false positive rate for microscopic hair comparison
in the study as applicable to the technique in general.62
CONCLUSION
Science continuously evolves and is built upon observation, study, and
experience that spans hundreds of years. The justice system would not be
well served by the exclusion of reliable forensic methods and techniques
that provide valuable information to a wide range of stakeholders. Critical
reviews of past and current practices assist in the continual process of
evaluation and improvement. However, they do not invalidate the entire
body of past scientific research and achievement.63

60. Class-level evidence encompasses a group of objects or persons with characteristics
that are shared by the group. The characteristics are not unique to a particular object or
person but serve to place the evidence into a smaller group of objects.
61. Houck & Budowle, supra note 57, at 2–4. See generally PCAST REPORT, supra
note 38.
62. See generally PCAST REPORT, supra note 38.
63. See ERNST MAYR, THE GROWTH OF BIOLOGICAL THOUGHT: DIVERSITY, EVOLUTION,
AND INHERITANCE 831 (1982) (“All interpretations made by a scientist are hypotheses, and
all hypotheses are tentative. They must forever be tested and they must be revised if found
to be unsatisfactory. Hence, a change of mind in a scientist, and particularly in a great
scientist, is not only not a sign of weakness but rather evidence for continuing attention to
the respective problem and an ability to test the hypothesis again and again.”).