Tiffany Roy, JD D-ABC
ForensicAid, LLC
Tiffany.Roy@gmail.com
Dear Commission Members,
My name is Tiffany Roy and I am a forensic DNA expert and lawyer. I am writing to express my
grave concern about a series of events that have transpired in the state of New York with regard to
forensic DNA evidence being analyzed there. My concerns involve the February 2018 audit of the New
York State Police Crime Laboratory by ANAB, the subsequent remedial actions taken by the New York
State Police Lab and accepted both by ANAB and the DNA Subcommittee, and my review of documents
related to NYSP Lab Case: 17HL-03487.
The ANAB Audit
I have concerns about the placement of the findings from the February 2018 ANAB audit of the
NYSP Lab. There were several findings of concern regarding the DNA Unit in this audit. The audit team
from ANAB consisted of Rebecca Dian, David Jackson, Kenneth Jones, Britton Morin, Catherine Nigra
and Kacie Waiters. All of these individuals have been identified by ANAB as experts qualified to perform
audits in the area of forensic DNA analysis. These auditors work in public crime laboratories, some of
them in technical leadership positions and quality management.
Several serious findings were reported in section 8, the validation section of the audit document.
These describe some interpretations being performed which were not validated by the lab. The Y-Filer
and Globalfiler validations were reviewed and compared to the procedures at the lab. There were several
analyses being performed which were not verified through validation. Specifically, the Y-Filer validation
was only performed for mixtures up to two contributors, but procedures allowed analysts to interpret
higher order mixtures (3+ contributors). This means any Y-Filer cases processed at NYSP which included
mixtures of more than two individuals were analyzed using methods that were not verified to be valid or
reliable.
Similar findings were reported for the Globalfiler Kit, which is now the standard STR kit used by
the laboratory. The audit team assessed the 4:1 ratio used by the laboratory for identifying major
contributors and found that the validation did not support the use of this method. Additionally, like YFiler, analyses on Globalfiler samples were also being performed which were not fully validated by the
lab. Auditors noted there were procedures for interpreting four person mixtures even though the
validation only tested mixtures up to three people. Any casework analysis performed on Globalfiler four
person mixtures was not validated by the lab. Further, auditors noted no validation or procedure guidance
on how to identify a major mixture, which would be required for the use of the restricted CPI. The
auditors noted the analysts did not appear to understand the application of the restricted CPI, and they
reviewed and cited three cases as objective evidence of nonconformance.
The finding involving the restricted CPI is the most concerning to me because the auditors
identified cases which had been affected by this nonconformance. I contrast this audit with the audit
performed on the DC Department of Forensic Sciences in April 2015 1 which resulted in the cessation of
DNA casework at that lab until the technical nonconformances were addressed. The allegations here are
similar to those found at the DC Lab. It is unclear to me why the restricted CPI concerns were not
included in section 9 under analytical procedures, specifically 9.6.4. In this except taken from the actual
February 2018 audit, the standard reads;

1

ANAB Audit of DC Department of Forensic Sciences Surveillance and Remote Surveillance Audit Report, April 24,
2015. https://dfs.dc.gov/page/anab-report-april-2015

1

 The auditors expressed to the New York State Police that the procedures were insufficient and
analysts were confused in interviews. They provided the following feedback to NYSP about the CPI
issue, which auditors curiously listed under requirement 4.2.1:

This clearly describes that the laboratory has not documented interpretation procedures to the
extent necessary to provide clear guidance on the proper application of the restricted CPI. I am concerned
this was not addressed as a technical or analytical concern in any way. It may also be a management
concern, but it is certainly technical. The issues here closely resemble the issues addressed in the audit of
the DC lab, including several findings of validation deficiencies and improper case handling. The auditors

2

 further provided NYSP with three cases they identified as being processed incorrectly using restricted CPI
as objective evidence.

I am deeply concerned that though these cases were identified to the laboratory through some
formal or informal communication, they were not identified in any of ANAB’s formal audit
documentation. I am also concerned the CPI issue was listed under standard 9 in the DC Audit, but not in
the NYSP audit. The DC audit had comparable technical and analytical nonconformances, but the
outcomes were very different. The following is an excerpt from the DC audit report on actions required
by ANAB;

The issues described above by Pat Bencivenga in the DC audit could have been written about the
NYSP audit. Verbatim. There were technical issues with incomplete validations, as well as a similar issue
regarding application of the CPI statistic which affected case work. There was a failure of the quality
management system to address these issues before casework was affected, and a failure of not stopping
casework when auditors identified these issues to management, including the DNA technical leader.
In my experience dealing with CPI issues are often not discovered by auditors. In Washington DC
as well as in Broward County, audit teams were not able to discover the complex and specific issues
identified by external independent experts. I contrast those situations with the NYSP audit. In this case,
ANAB need only rely on the expertise of their own experts. To their credit, this audit team did identify
this complicated issue. However, there appears to breakdown in the system response.
At no point in the formal written corrective actions do NYSP state they will review cases
identified by the auditors, or any other cases. They propose clarified wording in their mixture macro tool
and additional training for their analysts. It is unclear who will provide that training and whether or not
technical leadership and lab management will also undergo additional training. Lead assessor, Nita Bolz,
notifies the laboratory that she consulted with the audit team and the proposed remedial measures were
deemed acceptable to rectify the nonconformity. It’s unclear what discussions took place, but I am
3

 concerned these measures were not afforded due scrutiny by the DNA audit team. The cases identified by
the audit team were active cases. The auditors recognized that the restricted CPI was not applied properly
in those cases. As such, any corrective action should have formally included a review of those cases to
determine the extent to which casework was affected. This is particularly concerning to me given that the
cases identified by the audit team had not yet been adjudicated.
I am concerned that because the merger between ASCLD LAB and ANAB resulted in a revised
audit document, the auditors were not as familiar with the new document, and as such, misplaced some of
these nonconformances. I am concerned the new document created confusion on where to include certain
findings. Whatever the reason, I urge this Commission to investigate why these major nonconformances,
similar in many respects to the findings in the DC audit, resulted in starkly different responses. I urge this
Commission to communicate directly with the DNA auditors themselves to understand the importance of
these nonconformances rather than relying on the interpretation of the NYSP Lab. This Commission has
the authority to interview each auditor, as well as members of the quality assurance team at NYSP.
I also urge ANAB, Pamela Sale and Laurel Farrell, to investigate why nonconformities that were
clearly described to the laboratory as analytical procedure nonconformities were listed in the management
documentation section rather than the FBI QAS audit document. I echo the sentiments of the Texas
Commission on Forensic Science in their report on US v. Torney 2. Accrediting bodies play an important
role in ensuring the quality of the work done in the Nation’s Crime Labs. While it’s important to evaluate
the verification of procedures and validation of technologies with in the labs, when nonconformances are
discovered, it’s also vitally important to evaluate the impact of those nonconformances on case work if
we are to maintain public trust in the profession. This is the reason why auditors review case files during
an audit.
The NYSP Response
I have serious concerns regarding the NYSP response to the nonconformities identified by the
audit. In general, there are several steps involved in investigating a nonconformity that involve root cause
analysis and corrective action that do not appear to have been addressed in this investigation. The
recommended steps of a proper root cause analysis are;
•
•
•
•
•
•
•
•
•
•

Occurrence of adverse event
Safety/Quality personnel notified
Safety/Quality manager assigns a Safety Assessment Score to the event
Safety/Quality manager charters an interdisciplinary team to investigate
Gather and Analyze Info related to the adverse event
Determine root cause and contributing factors
Determine actions to address root cause
Determine how to measure outcomes
Present analysis and proposed actions to leadership
Implement actions and examine outcomes for effectiveness3

I am concerned with the failure to properly assess outcomes of the nonconformances and how the
remedial measures will prevent these nonconformances from occurring in the future. What was the effect
of these adverse events on the casework? In order to know this, a review of the case work affected by
2

Texas Commission on Forensic Science Final Report on National Medical Services, Inc (NMS) DNA Analysis in
Case of U.S. v. Torney retrieved at http://www.txcourts.gov/media/1441526/final-report-national-medical-servicesinc-nms-in-case-of-us-vs-torney-04202018.pdf
3
Guidelines for the Use of Root Cause Analysis (RCA) to Reduce Error and Improve Quality in Forensic Science
Laboratories retrieved from
https://www.nist.gov/sites/default/files/documents/2016/11/22/guidelines_for_the_use_of_root_cause_analysis_to_r
educe_error_and_improve_quality_in_forensic_science_labs.hollway.labmgmt.pdf

4

 these issues should have been performed. Reviews of affected Y-Filer, Globalfiler and restricted CPI case
work should have part of the formal corrective action plan. A laboratory system truly interested in quality
would want to know how casework was impacted, even if that includes a comprehensive retroactive
review.
At the March 21, 2018 Commission meeting, the following exchange took place between Marvin
Schechter and Dr. Russell Gettig;
(Unofficially transcribed from video of 3-21-18 Commission meeting)
Starting 0:29:22
MS: What I’m more zeroing in on, Mr. Wickenheiser, is the issue of, did this finding of a lack of
understanding or being more specific prompt you to do a review of any past reports of DNA
testing and reporting to see if the results were correct in view of these new findings?
RW: Ok, perhaps if I could refer that to Dr. Gettig.
RG: Yeah, one thing I wanted to mention is that--it’s the analysts who are being interviewed--and
in any of our analysis, the report goes through an extensive review process. It goes through a
peer review process, ok, by a peer, another analyst. It goes through what we call a secondary
review, usually by a supervisor or somebody who, an analyst who is well versed in the technology
and the protocols. And finally, there’s an administrative review done on it. So, to my knowledge,
nothing improper has gone out in any of our reports because anything that would be improper
would be caught on a review and corrected. A lot of times it wasn’t necessarily that something
was wrong, it’s a matter of the actual interpretation, to make sure the interpretation that is being
followed by the analyst follows, you know, follows the protocol.
MS: So, so if the interpretation was incorrect based upon, not on the substantive part or the
repeatability, but the interpretation was wrong, and the finding of ANAB was that people didn’t
understand exactly how to interpret these, did the fact of that lack of interpretation-notwithstanding that you had a process in place for catching mistakes beforehand--after this
finding, did it cause you to go back and take a look and see if there had been any
misinterpretations based upon the lack of understanding of what ANAB was now finding?
RG: Well they had flagged three cases in their review and we went and looked at those cases and
there was nothing improper, or incorrect, that went out in the report in those cases. Currently,
you know, I’m reviewing all of this particular kind of interpretation. Analysts will come and I’ll
review it and I haven’t found any problems at this point. But I have not looked at any past cases.
MS: You have not?
RG: I have not looked specifically now at any past cases, no. Because I don’t have any reason to
believe that they are not correct.
Ending at 0:32:14 4
I struggle to understand Dr. Gettig’s line of reasoning given that the auditors identified a list of
cases, presumably for a reason. Why else would the auditors “flag” cases for the lab to review? The fact
4

Unofficial Transcript of excerpts from March 21, 2018 meeting Video Clips at: https://www.youtube.com/watch?v=S6dPG3s1Bo&t=0s&list=PLcfRwlrt2Ir0sG_G5XQP9WJ3rPBeIDdxW&index=32

5

 that Dr. Gettig reviewed the three cases and found no issue makes me question his understanding of the
use of the restricted CPI. Like Dr. Gettig, I have also reviewed data from one of the identified cases. I can
conclusively say there were serious errors in the identification of a major mixed profile and the
application of the restricted CPI calculation. I agree with the ANAB auditors and I disagree with Dr.
Gettig. I have concerns that these errors were not identified at the laboratory because many of the
analysts, including the technical leadership, have incomplete understanding of when restricted CPI should
apply. The problems are the hardest to address when bad information flows from the top down. In that
event, errors would not be detected in any technical or administrative review. And further, there appears
to be an unresolved discrepancy between Dr. Gettig’s assessment of the identified case files and the
auditors’ assessment. There should have been a detailed discussion between Dr. Gettig and the DNA
auditors to determine exactly what the issues were that the auditors saw and why those were problematic.
There is no documentation that this exchange ever occurred.
Additionally, I agree with the concerns of the auditors: The method employed for determining a
major contributor at NYSP Lab DNA Unit is unreliable and unsupported by good science. See the excerpt
from Appendix A authored by the auditors from the February 2018 QAS audit;

Not only do I believe the 4:1 ratio method being employed by NYSP is improper in their
application, I also believe it is unverified by the data described by the NYSP in their validation
summaries.
I reviewed the revised NYSP protocols provided to the DNA Subcommittee in the last meeting on
May 11, 2018. Even the new protocols demonstrate a suspect-driven approach to analysis and use of the
CPI. The very definition of suspect-driven CPI is that the interpretation varies depending on who is being
compared to the data. In version 5 of the Interpretation of STR DNA Profiles (Globalfiler) drafted on
4/6/2018, I was alarmed to see the statement;

The protocol, which can be accessed publicly using the link in the footnotes, follows with several
pages of suspect-centric interpretation 5. Evidence DNA profiles are interpreted before comparison to
known standards. It’s driven by the data alone, determinations as to what loci are suitable for comparison
and statistical calculations must be made before the known samples are introduced to the process. The
data is set, and the interpretation is set, regardless of what “reference samples are available for
comparison.” 6 The NYSP indicated to the DNA Subcommittee members that the procedures they
reviewed are draft versions, not the final copies. The DNA Subcommittee should request the final
working versions of these procedures in their next meeting. It also does not appear that the ANAB
auditors reviewed these documents, and these should be provided to ANAB also.
5

http://www.criminaljustice.ny.gov/pio/open-meetings/5-11-2018-dna/18.05.11%20DNA%20Draft%20Binder_Legal_RED-1of5.pdf
6
Evaluation of forensic DNA mixture evidence: protocol for evaluation, interpretation, and statistical calculations
using the combined probability of inclusion Bieber et al. BMC Genetics (2016) 17:125 DOI 10.1186/s12863-0160429-7

6

 Other references are made to the suspect-centric analysis. In the Urfan Mukhtar memo dated
February 28, 2018, guidance is provided on changes to the interpreted loci following a comparison. See;

It is unclear what the communication with Joel Sutton included, but the instruction here is vague
and supports a practice that is not acceptable. It indicates that it is acceptable to drop that locus and proceed
with an inclusion and statistical calculation without the “single locus.” While the instance described may
not necessitate an exclusion, it certainly does not allow for an inclusion/CPI calculation given the
determination to drop the locus would be made post comparison. The suspect-centric approach to analysis
is widely rejected by the DNA Community because it can result in false inclusions and overstated statistical
evidence weights with match or inclusion associations. This practice is expressly prohibited by the
SWGDAM interpretation guidelines and is further described as inappropriate in Evaluation of forensic DNA
mixture evidence: protocol for evaluation, interpretation, and statistical calculations using the combined
probability of inclusion by Bieber et al. 7
I urge the Commissioners to investigate the proposed remedial actions more closely. The
Commission, through the DNA Subcommittee, must examine the actual data from the Y-Filer and
Globalfiler validation studies instead of relying on the summary prepared by NYSP. A quorum of the
DNA Subcommittee has reviewed and approved the NYSP proposed remedial measures, but it is unclear
what their review entailed. Without the actual data, I am concerned the DNA Subcommittee was painted
an incomplete picture of the validations. Additionally, Dr. Fred Bieber was not present during the DNA
Subcommittee meeting where these matters were discussed. He is a foremost expert in the area of mixed
DNA profile analysis and the Combined Probability of Inclusion. I respectfully request that he be
provided an opportunity to review these documents, in light of my concerns, to determine their validity. It
may also be necessary to involve experts outside the Commission who have expertise in addressing these
types of systemic lab issues, such as those involved with the review of the Austin, Texas and Washington,
DC labs. No matter who performs the function, a review of mixed DNA casework, especially casework
involving the application of major and minor profile resolution, mixed major profile resolution and the
application of the restricted CPI should be undertaken.
Lab Case: 17HL-03487
I was contacted in late May regarding a consulting review on case 17HL-03487. I was asked to
review a sample of significance in the case, Item 015J-K, which was identified as swabs from the magazine
of a 9mm pistol. The sample was a complex mixture with low level data. My interpretation of the profile
varied significantly from what was reported by NYSP. Using a binary analysis, I found that Omarrio
Morrison was excluded from the major profile. The analyst from the NYSP, Cheryl Strevell, found that
Omarrio Morrison could not be excluded, and the probability of selecting an unrelated individual with a
profile that would be included in the mixture is 1 in 14.33 million.
The analysis of 015J-K by Ms. Strevell and the NYSP constitutes a false inclusion. I could find no
documentation in the case file to support her use of the restricted CPI. Ms. Strevell reported a single major
profile matching an unidentified male, not a mixed major profile as would be necessary to employ a
7

Evaluation of forensic DNA mixture evidence: protocol for evaluation, interpretation, and statistical calculations
using the combined probability of inclusion Bieber et al. BMC Genetics (2016) 17:125 DOI 10.1186/s12863-0160429-7

7

 restricted CPI. In the case file, I did find a draft report retained in the case file where Ms. Strevell also
initially excluded Mr. Morrison from the interpretable portion of the profile. Sometime between the initial
analysis on November 17th 2017 and the final draft on December 8, 2017, Ms. Strevell’s own opinion
changed from exclusionary to inclusionary on 015J-K based on some interaction with one of her reviewers.
This change was made after comparison to the known standards associated with the case and necessitated
removal of 4 loci she had previously deemed suitable for comparison. I found no documentation in the case
file to explain why 4 exculpatory loci had been removed post comparison, despite recommendations by the
National Commission on Forensic Science 8 and legal requirements under Brady v. Maryland 9 to maintain
this documentation in the case file and provide it to defense counsel for review. Removal of loci post
comparison indicates a biased, suspect-centric approach. Based on the Mukhtar memo, this practice would
not be allowed at their laboratory today.
The report for case 17HL-03487 was issued on December 8, 2017 and the case file was “flagged”
by the DNA auditors from ANAB in February 2018. In May 2018, three months after it had been identified
as problematic by the ANAB auditors, the DNA evidence from Lab Case: 17HL-03487 was admitted over
objection into evidence at the trial of Omarrio Morrison.
I respectfully request to have my concerns added to the agenda for the June 15, 2018 Commission
meeting. It lies well within the authority of the Commission to investigate fully all of the claims contained
in this letter. I implore this Commission, ANAB and any other body with the authority to do so, to formally
investigate the concerns that have been described in this letter using the full breadth of their resources. The
systems in place to identify problems with forensic science in the state of New York must act to prevent
further introduction of bad science into it’s courtrooms and to preserve public trust in its justice system.
Certainly false inclusions, suspect-centric interpretation, overstated evidence weights and incorrect case
evidence analysis is not something the Commission condones.
Sincerely,

Tiffany A. Roy, JD D-ABC

8

National Commission on Forensic Science Views of the Commission Documentation, Case Record and Report
Contents retrieved from https://www.justice.gov/archives/ncfs/file/818191/download
9
Brady v. Maryland 373 u.s. 83

8