Questioning the Reliability of DNA Testing

The LA Times runs what it wants to think of as a serious investigative piece on the reliability of current DNA testing practices. And while there's plenty of investigation, it's also plenty alarming:

State crime lab analyst Kathryn Troyer was running tests on Arizona's DNA database when she stumbled across two felons with remarkably similar genetic profiles.

The men matched at nine of the 13 locations on chromosomes, or loci, commonly used to distinguish people.

The FBI estimated the odds of unrelated people sharing those genetic markers to be as remote as 1 in 113 billion. But the mug shots of the two felons suggested that they were not related: One was black, the other white.

In the years after her 2001 discovery, Troyer found dozens of similar matches -- each seeming to defy impossible odds.

As word spread, these findings by a little-known lab worker raised questions about the accuracy of the FBI's DNA statistics and ignited a legal fight over whether the nation's genetic databases ought to be opened to wider scrutiny.

But here's the thing: A nine-loci "match" (because forensic scientists don't use those terms in court and are cautioned against ever using those terms) is not and can never be as statistically significant as a 13-loci match. And the way Troyer ran her searches created results that don't tell the whole story:

Bureau officials say critics have exaggerated or misunderstood the implications of Troyer's discoveries.

Indeed, experts generally agree that most -- but not all -- of the Arizona matches were to be expected statistically because of the unusual way Troyer searched for them.

In a typical criminal case, investigators look for matches to a specific profile. But the Arizona search looked for any matches among all the thousands of profiles in the database, greatly increasing the odds of finding them.

As a result, Thomas Callaghan, head of the FBI's CODIS unit, has dismissed Troyer's findings as "misleading" and "meaningless."

In other words, Troyer wasn't looking for one needle in a haystack, but searched all haystacks for all possible needles and then was "shocked" to discover the presence of more than one needle in a haystack. Which isn't to say that DNA testing standards should remain at the status quo, especially as techniques improve and detection levels grow ever smaller, and the FBI (and other state and local crime labs) figure out how to improve interpretation and minimize what are often catastrophic errors with forensic evidence. But there is an easy, albeit time-consuming from a validation standpoint, way to get around this: bump up the number of loci required for statistical significance.

Comments

[Ali]

Just back from Harrogate -

Sarah and I were at the inaugurual Harrogate 6 years ago, and I can tell you that 2008's crime-writing festival was the best - superbly organised, huge attendance - many events sold out, and the bar stayed open very late. Then again Simon Kernick was the festival chair and last night we left our drinks when dawn broke -

Will be reporting at The Rap Sheet in due course, but first some sleep is required

Ali

[Mo]

Just because one guys black and the other is white doesn't mean they aren't related? Isn't that a not so rarely occurring trope in hardboiled fiction?

What is coming up here is that if the judicial system is going to bring science into the courtroom, they are also going to have to accept that "scientific truth" is a moving target. Do defendants have the right to know how many DNA samples match their own in the state's database? That's the underlying question here, and I don't see how the answer is going to stay "no" forever. However, do we retry everyone every time technology advances? The answer to that one isn't yes.

[Bill Peschel]

It sounds like saying that, because you find two fingerprints that match in 9 out of 13 places, that fingerprinting doesn't work.

[Jim Winter]

9 out of 13 matches might, in reality, mean nothing in the final analysis, but you can bet there are some giddy defense attorneys planning to trot this out to what they hope is a gullible jury.

Then again, remember, there was a bloody trail all the way back to Brentwood, and the best they could do to OJ was find him "responsible."

[C.E. Petit]

Any potential problems are not with the DNA test or data comparisons per se; they are with the administrative aspects of the system, which are not scientific -- they are human. For example, Troyer's "9/13 match" ASSUMES that the record showing one man was black, one man was white are correct (and, as a previous commenter pointed out, it wouldn't be the first time that someone was not entirely aware of his "true" ancestry and ethnic origin!).

This is the same problem as is pervasive in drug testing. I know enough about the sample-handling protocols that I would never accept a positive test with inquiring every step of the way into handling and administrative procedures... and I'd expect to find a nontrivial error at least 20% of the time, even with in-laboratory sample provision, and more than that in the field.