Case Document 153-1 Filed 10/17/17 Page 1 of 51

 

EXHIBIT 

 

Case 1:15-cr-00565-VEC Document 153-1 Filed 10/17/17 Page 2 of 51

Declaration of Nathaniel Adams
I, Nathaniel Adams, declare pursuant to 28 U.S.C. § 1746, that I have personal knowledge of
the following, and if called upon to do so, could and would testify competently to the matters
contained herein.

1 Qualifications
I have a Bachelor of Science degree with a major in Computer Science from Wright State
University (Dayton, Ohio). I am enrolled in the Graduate School at Wright State University,
pursuing a Master of Science degree in Computer Science. I am employed as a Systems
Engineer at Forensic Bioinformatic Services, Inc. in Fairborn, Ohio. My duties include analyzing
electronic data generated during the course of forensic DNA testing; reviewing case materials;
reviewing laboratory protocols; and performing calculations of statistical weights, including
custom simulations for casework and research projects. I actively use, develop, and maintain a
number of software programs to assist with these efforts. I have been involved in several
reviews of probabilistic genotyping analyses in criminal cases, including FST. In 2014 I attended
a week-long workshop on interpreting forensic DNA mixtures using emerging software solutions.
In January 2016, I was retained in a criminal case and inspected the source code of the
commercially-available probabilistic genotyping program STRmix™. Due to a non-disclosure
agreement that I signed, I am not allowed to discuss the findings of my review of STRmix™
outside of that particular case.

2 Overview
I have been asked by Sylvie Levine and Christopher Flood to conduct a code review of the
Forensic Statistical Tool (FST). I have previously provided a statement dated May 27, 2016
regarding the utility of code reviews when evaluating probabilistic genotyping systems such as
FST.

Page 1 of 22

 Case 1:15-cr-00565-VEC Document 153-1 Filed 10/17/17 Page 3 of 51

3 Setup
3.1 Materials provided
For this review I received a set of materials on a flash drive. These materials included a
Microsoft Visual Studio solution 1 named “FST_Production” which includes source code and
application configurations. Two database backup files were included in the solution.
I have also received the New York City Office of the Chief Medical Examiner (NYC OCME)
“Forensic Statistical Tool Validation” documents in PDF format as well as casework materials
describing DNA analyses conducted by OCME in the case US v. Kevin Johnson.

3.2 Review environment
Microsoft Visual Studio 2015 Community Edition was used for all code inspection, compilation,
debugging, and execution on a PC running Microsoft Windows 7 Professional Service Pack 1
(64-bit). Microsoft SQL Server 2016 Express on a PC running Microsoft Windows 8.1
Professional (64-bit) was used for restoration of the database backups and all database
functionality of the FST system. Portions of inspection and testing utilized Microsoft Internet
Explorer version 11.0.

3.3 Build
In Microsoft Visual Studio, the term “build” is equivalent to the concept of compile-and-go 2,
which automates multiple configurable steps of translating the source code to an executable
program, including the compilation 3 step. That equivalency is maintained in this document.
Upon inspection, the majority of the source code appears to be written as a mix of HTML,
Javascript, and the C# (pronounced “C sharp”) languages for Microsoft’s ASP.NET (pronounced
“A-S-P dot net”) framework. ASP.NET provides a client-server architecture widely used for

1

Solution: “Group of related projects with which a user works.” In this context, a “user” is a user
of Visual Studio, e.g. a software developer, rather than a user of the FST software. (Microsoft
Developer Network, “Visual Studio 2015 SDK Glossary.” Available:
https://msdn.microsoft.com/en-us/library/bb166253.aspx)
2
Compile-and-go: “an operating technique in which there are no stops between the compiling,
linking, loading, and execution of a computer program” (ISO/IEC/IEEE, “Systems and software
engineering -- Vocabulary,” ISO/IEC/IEEE 24765:2010(E), vol. 2010.)
3
Compile: “to translate a computer program expressed in a high-order language into its
machine language equivalent” (ISO/IEC/IEEE, “Systems and software engineering -Vocabulary,” ISO/IEC/IEEE 24765:2010(E), vol. 2010.)
Page 2 of 22

 Case 1:15-cr-00565-VEC Document 153-1 Filed 10/17/17 Page 4 of 51

distributed, multi-user systems such as websites and applications. FST features a user interface
accessible via a web browser running on a local computer. FST also utilizes a user
management system for security, a server-side computational engine, and a database for
information storage and retrieval, though these systems appear to be hidden from a normal
user’s interface.
I had to modify portions of code and application configurations in order to get a copy of FST
running on my local systems. The steps taken to build a working copy of FST are described in
[Appendix A – Refactoring]. Following these steps, I was able to access the FST user interface
as described in [Appendix B – FST Interface].

4 Assumptions
4.1 Code version
I assumed that the Visual Studio solution and code contained therein provided to me is identical
to the solution used to build the version of FST used in OCME’s DNA analysis conducted in the
case US v. Johnson, i.e. the “code base” of FST v2.5.

4.2 Documentation
I assumed that materials describing the design and development of FST would be provided to
me.

4.3 Build and operating environments
Expected differences between the executables built on my PC and built by OCME are limited to
those introduced by different versions of Visual Studio, Microsoft SQL Server, and operating
systems used by me and by OCME. I assume these differences are minimal. To explore
potential differences, I attempted to replicate the calculations performed by OCME’s version of
FST used in the case US v. Johnson. Upon comparison, results from my build and OCME’s
build of FST appear to be indistinguishable for the samples OCME analyzed in the case US v.
Johnson. See OCME materials with Bates stamp KJ_00173 and KJ_00174 and [Appendix C –
US v. Johnson reproductions].

4.4 Validation materials
I assumed that all materials included in the FST validation documents apply to the version of
FST provided to me unless indicated otherwise.

Page 3 of 22

 Case 1:15-cr-00565-VEC Document 153-1 Filed 10/17/17 Page 5 of 51

5 Review
Primary goals of this review included:
Aim 1 - To compare the instructions (source code) and observed behaviors of FST to the
requirements, specifications, and design documents (describing intended behaviors)
generated during the development and maintenance of FST. A conceptual description of
these materials is available in my May 27, 2016 statement.
Aim 2 - To compare the coding style and standards with which FST was developed with
practices common to the field of software engineering.
Aim 3 - To perform a limited set of tests on the system to identify any behaviors that
detectably depart from intended operations.
Screenshots included in the appendices accompanying this statement are documentation of
actions taken during the course of my review.

5.1 Documentation
5.1.1 Software engineering materials
As described in an article by Coble, et al. in “DNA Commission of the International Society for
Forensic Genetics: Recommendations on the validation of software programs performing
biostatistical calculations for forensic genetics applications” 4, “International industry standards
apply to software validation, verification and test documentation.” Standards can also be
codified by professional organizations, government agencies, or internal to an organization such
as OCME.
In the materials I have been provided, I have not seen any reference to standards governing the
development, testing, or validation of FST. Materials contained in FST Validation Vol. 1,
“Methods,” could be considered germane to descriptions of program requirements, specification,
design, and testing. However, these materials do not cover the entire span of the program such
that it could be precisely replicated from these materials or sufficiently tested against during the
validation & verification stage of the software development process.

4

M. D. Coble, J. Buckleton, J. M. Butler, T. Egeland, R. Fimmers, P. Gill, L. Gusmão, B.
Guttman, M. Krawczak, N. Morling, and others, “DNA Commission of the International Society
for Forensic Genetics: Recommendations on the validation of software programs performing
biostatistical calculations for forensic genetics applications,” Forensic Sci. Int. Genet., vol. 25,
pp. 191–197, 2016.
Page 4 of 22

 Case 1:15-cr-00565-VEC Document 153-1 Filed 10/17/17 Page 6 of 51

Enumerated requirements can increase transparency of the development materials by allowing
for references made in the source code to the collection of requirements. For example, a
comment in the code can that states something along the lines of, “satisfies requirement 5.2.2,”
could allow a non-author of the source code to connect a particular functional code segment to
a particular intended behavior of the software. Ideally, all source code used to build a software
program could be attributed to specific requirements defining that program’s intended behavior.

5.1.2 Manuals
A manual for operating FST is included in the FST Validation materials. The provided manual
includes instructions and figures that indicate differences between the version of FST described
in the validation materials and the version provided to me. See FST Validation Vol. 1,
“Methods,” Sec. VI, “Manual,” Figures 1D-E and [Appendix B – FST Interface, Figure B.2 and
Figure B.3] accompanying this statement. Notable differences include the “Version 2.5” logo,
replacement of a “Degraded Type” dropdown box with a “Lab Kit” dropdown box, and the
differences in comparison types available.
From the materials provided to me, it is unclear which portions of the FST software underwent
additional development between the time of producing the FST Validation materials and the
provision of the FST solution in the case US v. Johnson. In order for the FST Validation
materials to provide relevant context to this review (and to casework in general), it is important
to fully understand if the differences between versions of FST evaluated during validation and
during this review are entirely cosmetic or if any functional behaviors differ.

5.2 Testing
Software testing 5 can be performed in many ways and from many perspectives. The extent of
testing expected to be performed during the development of a program is proportional to the
importance of its correctness. That is, it is qualitatively more important to conduct extensive
testing on a software system regulating a pacemaker than in a video game. While software
defects extant in a video game’s code can be potentially overlooked by users or rectified by
software updates, injury or loss of life cannot be so simply addressed.

5

Testing: “an activity in which a system or component is executed under specified conditions,
the results are observed or recorded, and an evaluation is made of some aspect of the system
or component” (ISO/IEC/IEEE, “Systems and software engineering -- Vocabulary,”
ISO/IEC/IEEE 24765:2010(E), vol. 2010.)
Page 5 of 22

 Case 1:15-cr-00565-VEC Document 153-1 Filed 10/17/17 Page 7 of 51

Similarly, the extent of software testing expected of a software developer should be increased
when the operation of the software is difficult to test comprehensively prior to its use in a realworld system. Software defects affecting a medical device and a space probe were addressed
in my May 27, 2016 statement.

5.2.1 Unit tests
One common perspective of testing is the unit test 6, whereby individual subcomponents of a
larger program are tested for correctness, given pairs of known inputs mapped to known
outputs. Allowable inputs and expected behaviors should be defined by the requirements and
specifications enumerated for the program.
Subcomponents tested by unit testing can be combined together for integration or component
testing. Eventually, the program can be tested in its entirety. As described in my May 27, 2016
statement, there are noted difficulties (or impossibilities) for testing whole probabilistic
genotyping systems due to the lack of known ground truths for likelihood ratio calculations
performed on noisy casework data.
Some portions of the FST Validation could be considered unit tests, e.g. the tests and results
described in FST Validation Vol. 1, “Methods,” Sec. IV, “Program Evaluation.” However, table
1F in this section describes testing across only 11 of the 15 autosomal loci typed by the
Identifiler genotyping kit used in this study, and these 24 total evaluations involved changing
multiple variables between tests (numerator and denominator hypotheses as well as template
amount).

5.2.2 Extent of testing
Test coverage 7 can be quantified in a number of ways. Branch 8 coverage can be quantified by
comparing the number of branches tested to the number of total branches. Path 9 coverage can
6

Unit test: “a test of individual programs or modules in order to ensure that there are no analysis
or programming errors.” (ISO/IEC/IEEE, “Systems and software engineering -- Vocabulary,”
ISO/IEC/IEEE 24765:2010(E), vol. 2010.)
7
Test coverage: “the degree to which a given test or set of tests addresses all specified
requirements for a given system or component” (ISO/IEC/IEEE, “Systems and software
engineering -- Vocabulary,” ISO/IEC/IEEE 24765:2010(E), vol. 2010.)
8
Branch: “a computer program construct in which one of two or more alternative sets of
program statements is selected for execution” (ISO/IEC/IEEE, “Systems and software
engineering -- Vocabulary,” ISO/IEC/IEEE 24765:2010(E), vol. 2010.)
9
Path: “in software engineering, a sequence of instructions that may be performed in the
execution of a computer program” (ISO/IEC/IEEE, “Systems and software engineering -Vocabulary,” ISO/IEC/IEEE 24765:2010(E), vol. 2010.)
Page 6 of 22

 Case 1:15-cr-00565-VEC Document 153-1 Filed 10/17/17 Page 8 of 51

be similarly quantified. Branches or paths of particular importance can be selected for testing or
emphasized over less critical parts for more rigorous testing performed during development.
Exploration into the amount of testing needed can even help developers reduce a program’s
complexity before any testing is conducted, proactively reducing the complexity and quantity of
tests needed.
It is impossible to objectively test a program’s actual behavior against its intended behavior
unless its intended behaviors are objectively defined (i.e. in requirements and specifications).
Any ambiguities in the specified requirements are necessarily addressed subjectively by the
developers. It is difficult to assess the appropriate extent of software testing even when
specifications are precisely defined. It is substantially more difficult to determine the appropriate
extent of software testing when the object of the testing has no objective standard for correct
behavior.
The use of software programs as a component of the system of forensic DNA analysis prompts
the questions “when” and “how” probabilistic genotyping software systems should be tested.
Coble, et al. suggest4:
The DNA Commission differentiated developmental from internal (laboratory) validation
and emphasizes that the software used is an integral part of the evidential process and
should not be treated as a separate and isolated component. [emphasis added]
I strongly disagree with this guidance. It is my experience (to the extent that I am free to
comment on it) that the development of probabilistic genotyping software is substantially less
mature than the field of software engineering itself, especially in the domain of software testing,
validation, and verification. When a probabilistic genotyping system has been developed, it
should be tested on its own, in accordance with strictly defined test criteria developed from its
own requirements and specifications documents. If it sufficiently passes all tests in isolation,
then and only then should it be considered for comprehensive testing within the greater forensic
DNA analysis process.
If the software is tested in isolation, aberrant behaviors, including outright defects, do not risk
being “explained away” by the same noisy profile data that inhibits our ability to assert ground
truths for testable likelihood ratios. The field of software engineering has had its own standards
for validating software for decades, and it is very reasonable to draw on these general
standards directly.

Page 7 of 22

 Case 1:15-cr-00565-VEC Document 153-1 Filed 10/17/17 Page 9 of 51

For reference, the Scientific Working Group on Forensic DNA Analysis Methods published a 12page document 10 in 2015 on validating probabilistic genotyping systems, only a portion which
applied to the development of software, and none of which addressed software engineering
standards. The International Society for Forensic Genetics (ISFG) published a 15-page
document4 in 2016 on validating probabilistic genotyping systems. ISFG makes reference to two
long-extant, 100+ page documents 11,12 published by the Institute of Electrical and Electronics
Engineers (both of which are already marked as superseded) and a 14-year old guidance
document 13 from the Food and Drug Administration. Multiple frameworks for validating generalpurpose software have long been codified and continue to be revised by organizations such as
IEEE. A framework for validating biological software with complex and safety-risk-prone
components has been considered and codified by the federal government. While the field of
forensic DNA has no public and widely used software standards specific to the field, we can
draw on standards describing good practices that generally apply to all software development
processes.
I have seen no quantitative description of test coverage in the FST Validation materials. The
time that it would take to quantitatively evaluate the extent of testing FST has undergone is
outside the scope of this review.

5.2.3 Test code and automated software testing
“Test” code can be written parallel to a software program’s code and is one of the most common
practices throughout software engineering. The purpose of test code is not to provide any
functional behavior or fulfill any requirement of the software, but rather to ensure that the
program’s code fulfills its requirements by testing its actual behavior against its expected
behavior. Unit tests are commonly written as code segments within the software’s solution.
Multiple test frameworks exist for assisting developers with automated testing, some even
writing the code automatically. Modern development environments, including Visual Studio,

10

Scientific Working Group on DNA Analysis Methods. Guidelines for the Validation of
Probabilistic Genotyping Systems. Available at - http://www.swgdam.org/publications
11
IEEE Standard for Software and System Test Documentation. (2008). IEEE Std 829-2008.
12
IEEE Standard for System and Software Verification and Validation . (2012). IEEE Std 10122012 (Revision of IEEE Std 1012-2004).
13
Food and Drug Administration. (2002). General principles of software validation; final
guidance for industry and FDA staff. Available at http://www.fda.gov/MedicalDevices/DeviceRegulationandGuidance/GuidanceDocuments/ucm08
5281.htm
Page 8 of 22

 Case 1:15-cr-00565-VEC Document 153-1 Filed 10/17/17 Page 10 of 51

include features for executing automated software tests each time the program is built, ensuring
that no new defects are detected after modifications have been made to the program’s code.
A notable advantage of writing test code is that, even in the absence of precise requirements
and specifications definitions, test code can serve as a de facto specification since all tests
should have explicit pass/fail criteria. These tests’ pass/fail criteria can implicitly describe a
program’s expected behaviors. An entire approach to software development, called test-driven
development, utilizes this approach to software development rather than the traditional
hierarchical process described in the appendix of my May 27, 2016 statement.
Similar to unit tests, higher-level component and integration testing, regression testing, and
random testing can be automatically executed prior to building a solution or as a step of the
compile-and-go build process. A practice called “nightly builds” involves taking the snapshot of
the current code in a version control system and attempting to build the solution, triggering any
automated testing processes specified therein. In this way, developers are more prone to notice
software defects caused by their recent modifications, allowing them to make any needed
changes before the defect becomes more tightly integrated into the system.
I have seen no code indicating that any test code has been written for, or automated software
testing has been performed on, FST.

5.3 Validation and verification (v&v)
A definition of validation is provided by IEEE as, “The confirmation, through the provision of
objective evidence, that the requirements for a specific intended use or application are fulfilled.”
It is necessarily difficult to validate a program for which explicit requirements and specifications
have not been established.
It is worth considering, due to the incomplete and at-times ambiguous descriptions of FST’s
intended behaviors, whether FST has been validated, partly validated, or not yet validated from
a software engineering perspective.
Using IEEE’s definition of verification, “The process of evaluating a system or component to
determine whether the products of a given development phase satisfy the conditions imposed at
the start of that phase,” it is difficult to assess what verification processes have been undertaken
during FST’s validation and continued use.

Page 9 of 22

 Case 1:15-cr-00565-VEC Document 153-1 Filed 10/17/17 Page 11 of 51

I have seen no document resembling a software development plan describing stages of
development, against which comparisons for verification of FST could be made.
It is reasonable to expect that unanticipated requirements changes were made following the
failure to validate FST for four-person mixtures, which presumably gave rise to the note on the
login screen seen in [Appendix B– FST Interface, Figure B.1]. Since this change necessarily
followed at least a portion of the validation study conducted by OCME, and along with the
changes and undocumented behavior noted in [5.1.2 Manuals and 5.6 Undocumented
behavior], further investigation is indicated into the software validation and verification
processes conducted on FST.

5.3.1 Perception of v&v
Overlapping terms of art between software engineering and forensic biology should be
distinguished. A “validation study” in the field of forensic DNA is ultimately intended to determine
the limits of certainty with which a laboratory, utilizing certain processes, can confidently arrive
at a given conclusion. Ideally, the confidence with which the conclusion is stated and the
significance of the conclusion are quantified. The manifestation of forensic DNA’s “statistical
weights” in the form of random match probabilities, probabilities of inclusion, and likelihood
ratios are direct responses to this ideal.
However, validation of a software system is a comparison of a program’s intended and actual
behaviors. The limits of confidence with which the software can report a particular conclusion is
part of a greater process of scientific inquiry and is not part of the validation efforts from a
software engineering perspective. Conflation between these two concepts is detrimental to
understanding the relationship of both. A validation study of a forensic DNA analysis process
involving software should incorporate the uncertainties inherent to the software system(s) and
should only follow the conclusion that the software itself has been validated.

5.3.2 Independence
Regarding the independence of software validation, the FDA states13:
Validation activities should be conducted using the basic quality assurance precept of
"independence of review." Self-validation is extremely difficult. When possible, an
independent evaluation is always better, especially for higher risk applications.
IEEE defines22 “independent” as, “performed by an organization free from control by the
supplier, developer, operator, or maintainer.”
Page 10 of 22

 Case 1:15-cr-00565-VEC Document 153-1 Filed 10/17/17 Page 12 of 51

I am unaware of any independent review of FST’s development process prior to this review.

5.4 Code maintenance
5.4.1 Code ownership
The concept of code ownership is that the developer who originally wrote or currently maintains
the code is an authority on its behavior. Should a future developer uncover issues in the existing
code or create issues when modifying the code, the author can be used as a human reference.
Over 100 source code files are included in the FST solution, each presumably having been
authored or at least modified by a human developer. I conducted a keyword search of the entire
solution for terms indicating authorship of original code or modifications made to existing code. I
located three files that have comments 14 explicitly describing authorship. I located only five files
that had any comments attributing a contribution or modification to a particular developer.
Developers noted in these comments are Vivien Song, Dhrubajyoti “Dhruba” Chattopadhyay,
and “win.” I found no other references to whomever “win” is.
Overall, it is unclear from my review who authored the majority of the code constituting the FST
solution. It is possible that inspection of the version control system could provide further
information. See [5.4.4 Version control] for further discussion.

5.4.2 Code attribution
Numerous references to the popular coding website Stack Overflow 15 are observed throughout
the code, including the sole comment describing the routine “ByteArrayHelper” in the file
“IndividualReportPrinter.cs”. Without any elaboration as to the underlying function of the routine,
the comment simply states, “/// This helper class is used to find patterns in byte arrays. Found
on stack overflow. Thank you, Internet!”
Taking code from unknown sources can lead to using code that is poorly understood and
possibly dangerous. No URL’s to the original Stack Overflow web page are provided for
reference by future developers. The original author is not acknowledged, nor is contact
information for him/her provided. Further implications of using unattributed, unlicensed,
“borrowed” code is outside of the scope of this review.

14

Comment: “information embedded within a computer program, job control statements, or a set
of data that provides clarification to human readers but does not affect machine interpretation”
(ISO/IEC/IEEE, “Systems and software engineering -- Vocabulary,” ISO/IEC/IEEE
24765:2010(E), vol. 2010.)
15
http://stackoverflow.com/
Page 11 of 22

 Case 1:15-cr-00565-VEC Document 153-1 Filed 10/17/17 Page 13 of 51

5.4.3 Coding style
5.4.3.1 Coding conventions
The Microsoft Developer Network describes coding style guidelines 16:
The C# Language Specification does not define a coding standard. However, the
guidelines in this topic are used by Microsoft to develop samples and documentation.
Coding conventions serve the following purposes:
•

They create a consistent look to the code, so that readers can focus on content,
not layout.

•

They enable readers to understand the code more quickly by making
assumptions based on previous experience.

•

They facilitate copying, changing, and maintaining the code.

•

They demonstrate C# best practices.

Coding style guidelines are generally stylistic, rather than functional, conventions governing the
development of software. A Google search for the phrase “c# guide style OR conventions”
returned 1.83 million search results on October 26, 2016. Top results include coding style
guides or conventions published by Microsoft, academic institutions, private individuals, and
companies. There is no one “correct” style guide, but consistent stylistic practices throughout an
organization, or at least a specific solution, increase code comprehension and consequently
decrease the rate and significance of software defects 17.
I found no references to coding conventions or style guides in the FST solution or FST
Validation materials. Upon inspection of the FST source code, disparate coding styles are
indicated. While portions of some code feature inline comments for nearly every line of code,
long sections of code are uncommented in other files, with no annotations including even
summary descriptions of purpose. Without explicit descriptions of intended behaviors, reviewers
and developers are left to infer the purpose of the code from its own function. In the absence of

16

“C# Coding Conventions (C# Programming Guide)” (Microsoft Developer Network. Available
at - https://msdn.microsoft.com/en-us/library/ff926074.aspx)
17
Defect: “a problem which, if not corrected, could cause an application to either fail or to
produce incorrect results” (ISO/IEC/IEEE, “Systems and software engineering -- Vocabulary,”
ISO/IEC/IEEE 24765:2010(E), vol. 2010.)
Page 12 of 22

 Case 1:15-cr-00565-VEC Document 153-1 Filed 10/17/17 Page 14 of 51

external definitions of intended functionality, a developer runs the risk of justifying the behavior
of existing code by its sheer existence. See also [5.2 Testing].

5.4.3.2 Code smells
Martin Fowler describes code smells as, “A code smell is a surface indication that usually
corresponds to a deeper problem in the system.” 18 In this sense, a smell is not a defect in itself
but is a deviation from good coding practices, which can indicate underlying software defects.
Coding conventions and style guides purposely prevent code smells. Coding practices such as
writing long routines 19 or complex classes 20 can obfuscate underlying issues due to the difficulty
of comprehending large segments of code.
The FST source code presents a number of basic smells such as routine length and complex
classes. For example, the “DoCompare” routine involved in computing likelihood ratios is
approximately 200 lines long, including comments and whitespace. The “configureComparison”
method differentiates between 38 different hypothesis pairs and constitutes approximately 400
lines of the “ComparisonData.cs” file that is, itself, over 800 lines long. Routines such as
Database.SaveCase require 20 or more ordered input parameters. An uncertain developer
invoking this routine could potentially transpose two of these input parameters, resulting in
aberrant behavior that might go undetected.
Some source code files consisting of only a single class’ implementation are exceptionally long.
The “Database.cs” class file is approximately 1,900 lines long and defines only one class –
“Database”.
Magic numbers 21 are used throughout the FST solution. In the event of a needed change to
these values during development, all hard-coded values must be located and modified since

18

Fowler, M. “CodeSmell”. Available at - http://martinfowler.com/bliki/CodeSmell.html
Routine: “a subprogram that is called by other programs and subprograms” (ISO/IEC/IEEE,
“Systems and software engineering -- Vocabulary,” ISO/IEC/IEEE 24765:2010(E), vol. 2010.)
20
Class: “1. an abstraction of the knowledge and behavior of a set of similar things. 2. a static
programming entity in an object-oriented program that contains a combination of functionality
and data. Syn: type NOTE Classes are used to represent the notion of ‘things whose knowledge
or actions are relevant’.” (ISO/IEC/IEEE, “Systems and software engineering -- Vocabulary,”
ISO/IEC/IEEE 24765:2010(E), vol. 2010.)
21
Magic number: “literal value that is used by a program directly rather than being embedded in
a named constant or variable” (ISO/IEC/IEEE, “Systems and software engineering -Vocabulary,” ISO/IEC/IEEE 24765:2010(E), vol. 2010.)
19

Page 13 of 22

 Case 1:15-cr-00565-VEC Document 153-1 Filed 10/17/17 Page 15 of 51

they are dispersed throughout the code rather than being centrally, or at least consistently,
located.
More subjective code smells are prevalent in FST, such as excessive commenting of code [see
5.4.3.3 Comments] and long variable names such as the 131-character-long
“compareMethodIDSerializationBackupDoNotMessWithThisVariableBecauseItsAnUglyHackAro
undACrappyLimitiationOfTheFCLsCrappyXMLSerializer” seen in the file “ComparisonData.cs”.

5.4.3.3 Comments
Comments provide annotations for developers to reference but do not affect the functional
behavior of the source code. Comments can be generated manually by developers or
automatically by a development environment (e.g. Visual Studio). In the C# language, lines
preceded by a double-slash (“//”) are not translated to machine code and can therefore serve as
non-functional annotations of the code. Similarly, lines of old or deprecated code can be
prepended with a “//” to remove from the functional body of code – this is often observed during
the development stage of a software program.
Many comments in the source code I have reviewed appear to be manually generated by
developers, e.g. line 353 of “Comparison.cs”, which states “// here we iterate to find the LR for
each of the races (they are different because the frequency values are different per-race)”. Such
comments serve to increase comprehension of the code and are ultimately beneficial to the
software development efforts. Elsewhere, many of the comments appear to be auto-generated
by Visual Studio without being sufficiently filled-in with English-language descriptions of program
behavior. These differences result in an inconsistent commenting convention.
However, many portions of the FST code are without comments for dozens or hundreds of lines,
including some entire routines and class definitions. As stated above, the same mechanism that
indicates to the compiler that a line is not to be considered code (i.e. a literal “comment”) can be
used to remove lines of actual code from the functional behavior of a program. In the file
“ManageUsers.aspx.cs”, there are approximately 150 lines of uncommented code with four lines
containing code that are prepended by “//”.
Other large sections of code are entirely “commented-out,” indicating a change in requirements
or implementation of that section of code. Reasons for maintaining “commented-out” code
rather than deleting it are future reference, future re-implementation, use of the code as a test
during development, or simply untidy maintenance practices.

Page 14 of 22

 Case 1:15-cr-00565-VEC Document 153-1 Filed 10/17/17 Page 16 of 51

Excessive commenting can detract from developer comprehension of the code. In the same file
described above, “ComparisonData.cs”, a sequence of otherwise intuitively-named variables is
interspersed with comments, needlessly breaking up the flow:

Figure 1 – ComparisonData.cs lines 66-77.
Excessive comments detract from the readability of the code, in this case, doubling the length of
the code segment. Excessive comments increase maintenance time costs and complexity, as
code modified during the normal development cycle or as a bug fix must be accompanied with
any needed changes to the comments describing the code’s intended (modified) behavior.

5.4.4 Version control
IEEE defines “version control” as, “establishment and maintenance of baselines and the
identification and control of changes to baselines that make it possible to return to the previous
baseline” 22. Normally, when a file is saved to a hard drive, the previous version of the file is
overwritten by the new version, permanently discarding any previous information contained
therein. Version control systems allow for iterative changes to be made to all files in the system,
which in turn emphasizes code ownership and allows for modifications to be tracked.
Common technologies freely providing version control functionality include Apache’s Subversion
(SVN) and Git. I found references to SVN in only the FST.Web submodule’s
“FST_Production_Service” folder, indicating that the use of SVN version control might have only
applied to a portion of the FST system, specifically its report-generating feature. I observed no
references to Git version control systems or any other version control system.

22

ISO/IEC/IEEE, “Systems and software engineering -- Vocabulary,” ISO/IEC/IEEE
24765:2010(E), vol. 2010.
Page 15 of 22

 Case 1:15-cr-00565-VEC Document 153-1 Filed 10/17/17 Page 17 of 51

From the materials provided to me, it is unclear if the FST code base is maintained in a version
control system.

5.4.5 Dead code
As described by Debray, et al 23, “dead code refers to computations whose results are never
used.” Visual Studio provides basic detection of dead code upon attempting to build a solution.
Dead code identified in the FST solution by Visual Studio include a number of import “directives”
(i.e. imported packages 24) specified in many files that are never used in those files. Possible
causes for unused imports are modified specifications or designs; updates to the ASP.NET
framework between versions of Visual Studio; and import code copied and pasted between files
with different package requirements. From the materials provided to me, I cannot be certain as
to why these unused imports are included in the FST solution.

5.5 FST Database
5.5.1 Empty tables
The “FST_DB” database backup I was provided includes tables named “Cases,” “Known,” and
“Tests.” These tables are empty, unlike other tables in this database, which contain information
on dropout rates, allele frequencies, etc. I do not know if entries in these tables exist in OCME’s
version of the database but were removed before the database backup was made for provision
in this case; if I was provided a “test” or “developmental” database backup; or if these tables are
also empty in the FST build used by OCME for casework.

5.5.2 “CaseTypes” table and “ComparisonData.cs”
The same “FST_DB” database backup I was provided includes a table name “CaseTypes.” It
contains six entries:

23

Debray, S. K., Evans, W., Muth, R., & De Sutter, B. (2000). Compiler techniques for code
compaction. ACM Transactions on Programming Languages and Systems (TOPLAS), 22(2),
378–415.
24
Package: “a separately compilable software component consisting of related data types, data
objects, and subprograms“ (ISO/IEC/IEEE, “Systems and software engineering -- Vocabulary,”
ISO/IEC/IEEE 24765:2010(E), vol. 2010.)
Page 16 of 22

 Case 1:15-cr-00565-VEC Document 153-1 Filed 10/17/17 Page 18 of 51

Figure 2 - "CaseType" database table contents
The contents of the table “CaseTypes” appear to be pairs of hypotheses representing
numerators and denominators for likelihood ratio calculations. The source code file named
“ComparisonData.cs,” within routine named “configureComparison,” describes 38 similarlooking hypotheses:

Page 17 of 22

 Case 1:15-cr-00565-VEC Document 153-1 Filed 10/17/17 Page 19 of 51

Figure 3 - Cases 1 and 2 from "ComparisonData.cs", routine “configureComparison”
From my inspection of the behavior of the FST system, the user interface shown in [Appendix
B– FST Interface, Figure B.3] displays six possible comparisons which correspond to the
“CaseTypes” table entries. It is my understanding that the comparisons performed by OCME in
the case US v. Johnson are of types #3, “Comparison + Unknown / 2 Unknowns,” and #4,
“Comparison + 2 Unknowns / 3 Unknowns.” See OCME materials with Bates stamp KJ_00173
and KJ_00174.
Attempts to select the third and fourth items from the comparison dropdown list resulted in the
error page seen in [Appendix B – FST Interface, Figure B.4]. It seemed that while items #3 and
#4 in this dropdown list describe the comparisons made by the OCME in the case US v.
Johnson, the FST version I built was incapable of performing these comparisons as described.
The dropdown list appears to pull its contents from the “CaseTypes” table in the database rather
than from the source code of the program, itself.
However, upon further inspection, it appears that the comparisons dropdown list is merely a
placeholder for parallel functionality in the code, e.g. the first dropdown list item maps to the
calculation described under “case 1” in [Figure 3] above. I inspected the code to find:
Page 18 of 22

 Case 1:15-cr-00565-VEC Document 153-1 Filed 10/17/17 Page 20 of 51

Figure 4 - Cases 3-6 from "ComparisonData.cs", routine “configureComparison”
The reason selecting #3 or #4 from the list resulted in an error page is because “case 3” and
“case 4” trigger exceptions in the code. When I selected options #5 or #6 from the dropdown list,
the comparisons performed appear to not be the comparisons listed in the dropdown box
(corresponding to the “CaseTypes” database table), but rather the comparisons described in
“case 5” and “case 6,” respectively, in the “configureComparison” routine as seen in [Figure 4]
above. That is, I had to select the comparison listed as “Comparison + Known + Unknown /
Known + 2 Unknowns” in order to perform the actual comparison “Comparison + Unknown / 2
Unknowns.” Additionally, I had to select the comparison listed as “Comparison + 2 Knowns / 2
Knowns + Unknown” in order to perform the actual comparison “Comparison + 2 Unknowns / 3
Unknowns.”
The comparisons described in [Appendix C – US v. Johnson reproductions] appear to support
this inference, including an artifact of the database table “CaseTypes” descriptions of the
comparisons appearing in the “Debug out” files shown in [Appendix C– US v. Johnson
reproductions, Figure C.5 and Figure C.10].
It is unclear to me why this aberrant behavior occurs or if this behavior of FST is also observed
by FST users at OCME during casework analysis.

Page 19 of 22

 Case 1:15-cr-00565-VEC Document 153-1 Filed 10/17/17 Page 21 of 51

5.6 Undocumented behavior
An instance of undocumented behavior of the FST program is noted in the calculation of
likelihood ratios performed in “Comparison.cs”. A routine included in the source code file
(“Comparison” class), named “CheckFrequencyForRemoval” appears to perform the following
behavior:
1. Check all replicate (evidentiary) genotypes for any locus that contains alleles whose
frequency sums to ≥ 0.97 in any of the four subpopulations (“Asian,” “Black,”
“Caucasian,” and “Hispanic).
2. Remove these loci from the likelihood ratio calculations.
During this review, I encountered no notice, either intended or actual, provided to the user of
FST that any loci were removed from the likelihood ratio calculation. I found no indication that
this behavior is intended during my examination of FST-related publications and the FST
Validation materials. Testing I performed [see Appendix D – Single-locus Testing] indicates that
this behavior is real. Equivalent locus likelihood ratios of “1” were reported for single-locus tests,
regardless of whether the “comparison” (reference) profile shared alleles with the evidentiary
item or not, as long as the sum of allele frequencies was ≥ 0.97 at that locus.
It is unclear from this review how often this is encountered during casework; whether an FST
user is aware that a locus is ignored when it ignored by the FST software; or how this has
affected or can affect casework mixture analysis and reporting. I am not aware of any studies
conducted on this feature’s impact on casework samples and the calculation of their statistical
weights reported by OCME.
Such departures from the published descriptions of FST’s behavior during its actual operation
raise the question if additional undocumented behaviors, either intended or actual, affect
casework samples during the operation of the FST software or the reporting of its likelihood ratio
results.

6 Conclusion
This review should not be considered complete, comprehensive, or exhaustive. I am available to
continue reviewing the materials I have been provided or to review additional materials
previously not provided.

Page 20 of 22

 Case 1:15-cr-00565-VEC Document 153-1 Filed 10/17/17 Page 22 of 51

I am ultimately unsure if the materials provided to me are the same materials that were used
while conducting the study documented in the FST Validation materials or casework analyses in
the case US v. Johnson.
Reviewing the materials I was provided, I did not leave with the impression that FST was
developed by an experienced software development team, especially in regards to adherence to
coding conventions, use of general software development standards, or even basic good
practices such as using consistent coding styles; attributing authorship to code segments; or
writing automated software tests.
The presence of undocumented behaviors and differences in appearance between the version I
reviewed and the version shown in the FST Validation materials calls into question when
development on FST ceased and how this affects or might affect the significance of any
scientific conclusions predicated on FST Validation materials.
Given the lack of a set of detailed and explicit software requirements and specifications, it is
difficult to evaluate to what extent FST's testing reflects rigorous and appropriate software
testing. Consequently, the idea that FST is a "validated" software program should be revisited
and a more thorough review conducted.
Given the materials I was provided, the time I was able to review them, and the issues I
identified, I recommend that:
1. Explicit requirements and specifications of the intended (not necessarily extant)
behaviors be written such that no undocumented or unexpected behaviors exist and
against which software testing of FST can be conducted.
2. A formal software test plan be written and described in terms of coverage.
3. Software tests needed for validation be written as automated tests that can be executed
and reported on by an automated software testing framework.
4. The FST Validation materials and publications regarding FST be compared to the
validated version of FST, with any discrepancies noted.
Until these steps are followed, the correctness of the behavior of the FST software should be
seriously questioned.

Page 21 of 22

 Case 1:15-cr-00565-VEC Document 153-1 Filed 10/17/17 Page 23 of 51

October 27, 2016

Page 22 of 22

 Case 1:15-cr-00565-VEC Document 153-1 Filed 10/17/17 Page 24 of 51

Appendix A – Refactoring
I made modifications to the database, application configurations, and code in order to assemble
a working FST client-server system on my local PC’s.

A.1 Databases
Aside from the modifications described below, I made no further changes to the FST databases
I was provided.

A.1.1 Restoration of backups
I received backup copies of two databases, named “FST_DB.bak” and “FST_Membership.bak.”
I restored these databases into Microsoft SQL Server.

A.1.2 Database security
The restored databases contained user profiles and logins with security identifiers (SIDs) that I
was not provided. I was able to insert a new user into each database with full privileges to
bypass the need for transferring or replicating existing SIDs.

A.2 Application configurations
Aside from the modifications described below, I made no further changes to FST’s application
configuration.

A.2.1 .NET packages
Multiple packages freely provided by Microsoft are required for building FST:
•

Microsoft.ReportViewer.Common

•

Microsoft.ReportViewer.WebForms

•

Microsoft.ReportViewer.WinForms

I received these packages on the flash drive containing the FST solution but I had to reestablish
references from the solution to the packages prior to building the solution.

A.2.2 Database connections
The FST solution has multiple references to a database server. These references are serverspecific, so they had to be regenerated by my server and specified in multiple configuration files
to point to my local server.
Section 2 - 1 of 2

Appendix page 1 of 24

 Case 1:15-cr-00565-VEC Document 153-1 Filed 10/17/17 Page 25 of 51

A.2.3 Project properties
I changed the “Startup Project” property to attempt to start all four major components of the FST
solution: FST.Common, FST.Web, FST_WindowsService, and FSTServiceConsoleHost.
FST_WindowsService had to be manually installed and started to operate.

A.2.4 FST_WindowsService
Using the Visual Studio developer’s command prompt, I used the Microsoft-provided
‘installutil.exe’ program to install the FST_WindowsService “debug”-built executable as a
background Windows service. I then started the service manually.
The service is named “FST_Charles”.

A.3 Code modifications
Aside from the modifications described below, I made no changes to the FST solution’s source
code.

A.3.1 Secure login
When a user first connects to the FST server (via a web browser), the user is presented with a
login page that requires a username and password. The login attempt is negotiated by the
project FST.Web’s “Login.aspx.cs” file, specifically the routines “btnLogin_Click” and
“IsAuthenticated.” I commented-out portions of this code and set the IsAuthenticated method to
return “true” regardless of the login credentials supplied.
By this method, I am able to “log in” using the “admin” login without providing a legitimate
password since no password check is performed.

Section 2 - 2 of 2

Appendix page 2 of 24

 Case 1:15-cr-00565-VEC Document 153-1 Filed 10/17/17 Page 26 of 51

Appendix B – FST Interface

Figure B.1 – Login screen for FST.

Section 3 - 3 of 22 pages long

Appendix – Page 3 of 24

 Case 1:15-cr-00565-VEC Document 153-1 Filed 10/17/17 Page 27 of 51

Figure B.2 – Homepage for new analysis.

Section 3 - 4 of 22 pages long

Appendix – Page 4 of 24

 Case 1:15-cr-00565-VEC Document 153-1 Filed 10/17/17 Page 28 of 51

Figure B.3 – Available “case types” dropdown menu on homepage for new analysis.

Section 3 - 5 of 22 pages long

Appendix – Page 5 of 24

 Case 1:15-cr-00565-VEC Document 153-1 Filed 10/17/17 Page 29 of 51

Figure B.4 - FST error page.

Section 3 - 6 of 22 pages long

Appendix – Page 6 of 24

 Case 1:15-cr-00565-VEC Document 153-1 Filed 10/17/17 Page 30 of 51

Appendix C – US v. Johnson reproductions

Figure C.1 - Replicate 1 for "back strap and grips # 2"

Section 3 - 7 of 22 pages long

Appendix – Page 7 of 24

 Case 1:15-cr-00565-VEC Document 153-1 Filed 10/17/17 Page 31 of 51

Figure C.2 - Replicate 2 for "back strap and grips # 2"

Section 3 - 8 of 22 pages long

Appendix – Page 8 of 24

 Case 1:15-cr-00565-VEC Document 153-1 Filed 10/17/17 Page 32 of 51

Figure C.3 - Comparison profile for "back strap and grips # 2"

Section 3 - 9 of 22 pages long

Appendix – Page 9 of 24

 Case 1:15-cr-00565-VEC Document 153-1 Filed 10/17/17 Page 33 of 51

Figure C.4 - FST report reproducing results for “back strap and grips # 2”

Section 3 - 10 of 22 pages long

Appendix – Page 10 of 24

 Case 1:15-cr-00565-VEC Document 153-1 Filed 10/17/17 Page 34 of 51

Figure C.5 - "Debug out" file for Caucasian subpopulation for "back strap and grips # 2" FST analysis

Section 3 - 11 of 22 pages long

Appendix – Page 11 of 24

 Case 1:15-cr-00565-VEC Document 153-1 Filed 10/17/17 Page 35 of 51

Figure C.6 - Replicate 1 for "slide grip grooves and mag release"

Section 3 - 12 of 22 pages long

Appendix – Page 12 of 24

 Case 1:15-cr-00565-VEC Document 153-1 Filed 10/17/17 Page 36 of 51

Figure C.7 - Replicate 2 for "slide grip grooves and mag release"

Section 3 - 13 of 22 pages long

Appendix – Page 13 of 24

 Case 1:15-cr-00565-VEC Document 153-1 Filed 10/17/17 Page 37 of 51

Figure C.8 - Comparison profile for "slide grip grooves and mag release"

Section 3 - 14 of 22 pages long

Appendix – Page 14 of 24

 Case 1:15-cr-00565-VEC Document 153-1 Filed 10/17/17 Page 38 of 51

Figure C.9 - FST reproducing results for "slide grip grooves and mag release"

Section 3 - 15 of 22 pages long

Appendix – Page 15 of 24

 Case 1:15-cr-00565-VEC Document 153-1 Filed 10/17/17 Page 39 of 51

Figure C.10 - "Debug out" for Caucasian subpopulation for "slide grip grooves and mag release" FST analysis

Section 3 - 16 of 22 pages long

Appendix – Page 16 of 24

 Case 1:15-cr-00565-VEC Document 153-1 Filed 10/17/17 Page 40 of 51

Appendix D – Single-locus Testing

Figure D.1 - Replicate 1 for CSF testing where comparison alleles are observed in the replicate

Section 3 - 17 of 22 pages long

Appendix – Page 17 of 24

 Case 1:15-cr-00565-VEC Document 153-1 Filed 10/17/17 Page 41 of 51

Figure D.2 - Comparison for CSF testing where comparison alleles are observed in the replicate

Section 3 - 18 of 22 pages long

Appendix – Page 18 of 24

 Case 1:15-cr-00565-VEC Document 153-1 Filed 10/17/17 Page 42 of 51

Figure D.3 - FST results for CSF testing where comparison alleles are observed in the replicate

Section 3 - 19 of 22 pages long

Appendix – Page 19 of 24

 Case 1:15-cr-00565-VEC Document 153-1 Filed 10/17/17 Page 43 of 51

Figure D.4 - "Debug out" data for Asian subpopulation for CSF testing where comparison alleles are observed in the replicate *Note
no other “Debug out” files were produced for this analysis

Section 3 - 20 of 22 pages long

Appendix – Page 20 of 24

 Case 1:15-cr-00565-VEC Document 153-1 Filed 10/17/17 Page 44 of 51

Figure D.5 - Replicate 1 for CSF testing where comparison alleles are not observed in the replicate

Section 3 - 21 of 22 pages long

Appendix – Page 21 of 24

 Case 1:15-cr-00565-VEC Document 153-1 Filed 10/17/17 Page 45 of 51

Figure D.6 - Comparison for CSF testing where comparison alleles are not observed in the replicate

Section 3 - 22 of 22 pages long

Appendix – Page 22 of 24

 Case 1:15-cr-00565-VEC Document 153-1 Filed 10/17/17 Page 46 of 51

Figure D.7 - FST results for CSF testing where comparison alleles are not observed in the replicate

Section 3 - 23 of 22 pages long

Appendix – Page 23 of 24

 Case 1:15-cr-00565-VEC Document 153-1 Filed 10/17/17 Page 47 of 51

Figure D.8 - "Debug out" data for Asian subpopulation for CSF testing where comparison alleles are not observed in the replicate
*Note no other “Debug out” files were produced for this analysis

Section 3 - 24 of 22 pages long

Appendix – Page 24 of 24

 Case 1:15-cr-00565-VEC Document 153-1 Filed 10/17/17 Page 48 of 51

Curriculum Vitae

Name:

Nathaniel D. Adams

Current employer:

Forensic Bioinformatic Services, Inc.

Address:

2850 Presidential Drive, Suite 160
Fairborn, OH 45324 USA

Phone:

+1 (937) 426-9270 (office)

Email:

Adams.201@Wright.edu
Adams@Bioforensics.com

Present position:

Systems Engineer

Current duties:

Operating custom and commercial software for independent review of
forensic STR DNA analysis; reviewing materials generated during the
course of STR DNA testing; performing data analysis for research and
case work; and reviewing current scientific and industry literature.

Educational background:

B.S. ​Computer Science
Wright State University
Fairborn, OH
M.S. ​Computer Science, in progress, 2014-present
Wright State University
Fairborn, OH

Professional experience:

Systems Engineer
Forensic Bioinformatic Services, Inc.
Fairborn, OH
2012 – present

Presentations:

N. Adams. The Science and the State of Probabilistic Genotyping.
Questioning Forensics: Inside the Black Box. The Legal Aid Society.
October 28, 2016.
N. Adams. Weka in Java: Classification and Clustering. Data Science
and Security Cluster, Wright State University. September 2, 2016.
Fairborn, OH.

1

 Case 1:15-cr-00565-VEC Document 153-1 Filed 10/17/17 Page 49 of 51

N. Adams, D. Krane. Black Boxes and Due Process: Transparency in
Expert Software Systems. 68th Annual Meeting of the American
Academy of Forensic Sciences. February 26, 2016. Las Vegas, Nevada.
D. Krane, C. Rowland, N. Adams. Disputed DNA Stats for a Low-level
Sample: A Case Study. 68th Annual Meeting of the American Academy
of Forensic Sciences. February 26, 2016. Las Vegas, Nevada.
N. Adams, R. Chakraborty, C. Rowland, D. Krane. Complex Mixtures
and the Minimum Number of Contributors: A Case Study (poster). 68th
Annual Meeting of the American Academy of Forensic Sciences.
February 26, 2016. Las Vegas, Nevada.
R. Koppl, D. Krane, N. Adams. Minimizing and Leveraging Bias in
Forensic Science. National Institute of Standards and Technology
International Symposium on Forensic Science Error Management. ​July
24, 2015. ​Washington, DC.
N. Adams. Is the Tail Wagging the Dog: Reviewing Probabilistic
Genotyping Software. National Forensic College of National Academy
of Criminal Defense Lawyers and Cardozo Law School. June 10, 2015.
New York City, NY.
N. Adams. Introduction to Pattern Mining. Bioinformatics Research
Group, Wright State University. March 24, 2015. Fairborn, OH.
S. Al-Awadi, M. Sabbaha, N. Adams, A. Marshall, C. Rowland, D.
Krane. Pairwise Comparisons as a Means of Validating Iraqi Muslim
and Christian Allele Frequency Databases (poster). 67th Annual
Meeting of the American Academy of Forensic Sciences. February 19,
2015. Orlando, FL.
N. Adams, A. Marshall. The Science (and Pseudoscience) of DNA
Profiling. The Kettering-Moraine Public Library. June 21, 2014.
Kettering, OH.
N. Adams. Allele Frequencies: How the Same Statistic Varies and Why
It Should. Forensic DNA for Trial Attorneys, Cook County Office of
the Public Defender. May 30, 2014. Chicago, IL.
D. Krane, N. Adams. The Science (and Pseudoscience) of DNA
Profiling. Wright State University Foundation Board of Trustees
Meeting, Wright State University. October 24, 2013. Fairborn, OH.
N. Adams, A. Marshall. Forensic DNA Analysis. Bioinformatics
Research Group, Wright State University. April 3, 2013. Fairborn, OH.

2

 Case 1:15-cr-00565-VEC Document 153-1 Filed 10/17/17 Page 50 of 51

Continuing education:

Questioning Forensics: Inside the Black Box. The Legal Aid Society.
October, 2016. New York, NY.
DNA and More: Developments in Forensic Science, Cook County
Office of the Public Defender. June, 2016. Chicago, IL.
68th Annual Meeting of the American Academy of Forensic Sciences.
February, 2016. Las Vegas, NV.
National Institute of Standards and Technology ​International
Symposium on Forensic Science Error Management. July, 2015.
Washington, DC.
National Forensic College of the National Academy of Criminal
Defense Lawyers and Cardozo Law School. June, 2015. New York
City, NY.
Forensic Bias and Error: Causes and Correction, Cook County Office of
the Public Defender. May, 2015. Chicago, IL.
DNA Mixture Interpretation Software Workshop, Midwestern
Association of Forensic Scientists. June, 2014. St. Louis, MO.
67th Annual Meeting of the American Academy of Forensic Sciences.
February, 2015. Orlando, FL.
Science in the Courtroom for the 21​st​ Century: Current Issues in DNA
Litigation. Cook County Office of the Public Defender. May, 2013.
Chicago, IL.

Professional organizations:

Institute of Electrical and Electronics Engineers. Student member.
Association for Computing Machinery. Student member.

Testimony:

State v. Emanuel Fair. King County Superior Court, Seattle, WA.
Pretrial hearing on probabilistic genotyping.

Student organizations and
recognition:

Data Science and Security Cluster, 2015-present. Wright State
University, Fairborn, OH.
Bioinformatics Research Group, 2012-present. Wright State University,
Fairborn, OH.
Dean’s Leadership Institute, 2013-2014. College of Engineering and
Computer Science. Wright State University, Fairborn, OH.
College Award for Senior Design, College of Engineering and
Computer Science, May, 2014. Wright State University, Fairborn, OH.
Choose Ohio First Scholarship
3

 Case 1:15-cr-00565-VEC Document 153-1 Filed 10/17/17 Page 51 of 51

Recognition for Excellence in Service-Learning, 2013. Wright State
University, Fairborn, OH.
Lockheed Martin Scholarship

4