« New assorted links | Main | Economicwoman.com »
CSI on Trial
...to judge by the most comprehensive study on the reliability of forensic evidence to date, the error rate is more than 10% in five categories of analysis, including fiber, paint and body fluids. ...DNA and fingerprints are more reliable but still not foolproof....a 2005 study in the Journal of Criminal Law & Criminology suggests a fingerprint false-positive rate a bit below 1%, a widely read 2006 experiment shows an alarming 4% false-positive rate.
How can we preserve the usefulness of forensic evidence while protecting the public when it breaks down? The core problem with the forensic system is monopoly. Once evidence goes to one lab, it is rarely examined by any other. That needs to change. Each jurisdiction should include several competing labs. ...This procedure may seem like a waste. But such checks would save taxpayer money. Extra tests are inexpensive compared to the cost of error, including the cost of incarcerating the wrongfully convicted....
Other reforms should include making labs independent of law enforcement and a requirement for blind testing. When crime labs are part of the police department, some forensic experts make mistakes out of an unconscious desire to help their "clients," the police and prosecution. Independence and blind testing prevent that.
That's forensics expert Roger Koppl writing in Forbes. If anything I think Koppl is being kind to CSI. Take bullet lead analysis a procedure used by the FBI for decades that turns out to have no scientific validity whatsoever.
Full Disclosure: Koppl's op-ed is based on a paper in a book called Law Without Romance edited by Ed Lopez to be published by Independent Institute where I am director of research.
Posted by Alex Tabarrok on May 15, 2008 at 09:45 AM in Law | Permalink
Comments
Competing labs would help, but that may solve only a small part of the problem. Any test, even if correctly done, will always have a non-zero false positive rate. This is unavoidable. The other source of error is, well, error itself. Competition can help solve the unforced error problem, but not the underlying false positive rate.
Posted by: Craig at May 15, 2008 9:59:48 AM
It's amazing that such no-brainers sound like novel concepts when applied to government.
Blind tests? Audits? Skepticism? Objectivity? Statistics!?!
How 'bout you present the jury with all the facts. "This study shows fingerprints have a 1-4% error rate. Paint chips have an 10% error rate." If you don't want that presented in court, then don't use it for evidence, just for investigation."
Of course, the obvious solutions aren't used BECAUSE they don't want the problem solved, because they don't pay for the errors. Fancy sounding tests are used and experts present the results for the very purpose of slanting the impressionable jury.
If we don't propagate errors in engineering we get points off. Or, in the real world bridges fall over and space satellites miss their targets. In government, you get a conviction and a promotion. The problem is there is no natural cost of error, it must be imposed on them.
Of course, I won't even bring up how this relates to letting courts decide issues like evolution and teaching science.
Posted by: Andrew at May 15, 2008 10:51:10 AM
It makes me sad to say this, but I really don't believe that Law enforcement, including prosecutors, really gives a darn about convicting innocent people. Stories like the FBI's bullet lead "evidence" and the "bite marks" expert below are so disappointing.
http://www.reason.com/news/show/125151.html
Posted by: Mercutio.Mont at May 15, 2008 11:54:59 AM
My experience from frequent jury duty in DC leads me to believe that police "know" who the bad guys are, and prosecutors are willing to bring a case on evidence that is less than perfect because they are sure they are guilty of something if not the crime they are on trial for. This additude no doubt contributes to the way forensic evidence is handled.
Posted by: joan at May 15, 2008 11:58:09 AM
@ andrew
that is how it works. the prosecution needs to introduce evidence about the procedure, but the defense can always question the prosecution's witness and nail him on the procedure's unreliability or even bring its own witness in. the problem is that 99% reliability, though not perfect, is pretty damn close in a jury's eyes. if i told you the democrats have a 99% chance of winning the election this fall, do you have a reasonable doubt that it will happen?? the prosecutors ought to be required to confirm the tests on their own before bringing the case at all. and no, defendants can't afford to run these tests themselves.
Posted by: andres at May 15, 2008 2:16:40 PM
Interesting that the article is in Forbes. Here's another Forbes article about a crime lab belonging to a surprising company.
http://www.forbes.com/forbes/2008/0421/102.html
Posted by: mikesdak at May 15, 2008 2:22:44 PM
"if i told you the democrats have a 99% chance of winning the election this fall, do you have a reasonable doubt that it will happen?"
Yes, because I wouldn't believe you. I'd also be allowed to do my own research. Ask for other expert opinions, and not just take your word as you presented it, regardless of how good my defense attorney is.
Posted by: Andrew at May 15, 2008 3:19:29 PM
I loved the article, but I think Koppl doesn't push it far enough when he says the problem is monopoly, and therefore we need the government to require multiple tests, etc. That's like saying the problem with oil prices is OPEC, and that's why we need to ask Saudi Arabia to pump more.
Case in point: Koppl discusses a guy who was wrongly convicted of rape and held for four years. His compensation? $118,000. If those are the penalties the government faces for mistakes, no wonder they are so sloppy. In a voluntary system where people could patronize different legal frameworks (and yes we can argue about how/whether that would work), I think the fines might be such that the agencies that survived the competition fixed the leaky roofs over their crime labs.
Posted by: Bob Murphy at May 15, 2008 4:32:37 PM
@andrew
you missed my point. the article says, "a 2005 study in the Journal of Criminal Law & Criminology suggests a fingerprint false-positive rate a bit below 1%." my question is whether a 99% chance of something being true leaves you with a reasonable doubt that it is, in fact, true. we can go down the list to the other methods and other percentages, and somewhere along the line, it'll break down--there will be enough of a likelihood that the test gave a false-positive to sustain a reasonable doubt. i'm thinking a false positive rate of over 10% is definitely there. under 5% is probably not. in between, i'm not sure.
my point is only that jurors are frequently presented with these error rates, which you think is some sort of novel idea--"How 'bout you present the jury with all the facts." the problem is not that they aren't given the facts, but that the facts themselves support the legal conclusion of guilt, even if the moral implications of a 1% false positive rate are alarming, to say the least.
as i understand it, you don't think you should believe it when you are told that something has a 99% accuracy rate. your skepticism is something of which you are obviously proud and i'm sure it serves you well. but are you seriously suggesting that jurors go off on their own, in the midst of trials, to determine the accuracy of the tests being used by a party?? that's what the adversarial system is for. the party's opponent will challenge it, and if they fail to persuade the jury, they lose. you don't honestly believe for a second that having jurors conduct independent evaluations of evidence or evidentiary methods is a good idea, whatever the merits of skepticism and independent evaluation in our daily lives. don't pretend otherwise.
Posted by: andres at May 15, 2008 5:03:15 PM
Even if libertarian fantasy competing privatized crime labs existed and were 100% correct, why would you think that would make a significant difference?
You would have to presume that conviction is an accurate process: but there's no good measure of it, and lots of reasons why we know it is frequently inaccurate.
But it makes a nice story for libertarians to bash government with, even if it is stupid.
Posted by: Mike Huben at May 15, 2008 7:50:43 PM
With regard to bullet lead analysis (actually trace element analysis), the definitive experiment seem easy to design. Buy a couple of hundred boxes of ammunition. Take some bullets from each and analyze them for whatever trace elements you desire. And then see if a blinded observer can match them up as to box of origin. My guess is that it won't be possible, but it seems that the FBI (or anyone else) has never done the experiment. That said, you have to understand that someone "from the FBI Laboratory" has a tremendous impact on a jury, which will be composed mostly of poorly educated, low IQ, scientifically illiterate individuals, and that such an expert's testimony is likely to be entirely believed without question. For an enjoyable experience as to how a pompous expert witness from a government laboratory spewing bogus testimony can be blown out of the water by a really sharp defense expert, rent a DVD of "My Cousin Vinnie."
Posted by: Ned at May 15, 2008 9:07:13 PM
Even if libertarian fantasy competing privatized crime labs existed and were 100% correct, why would you think that would make a significant difference?
You would have to presume that conviction is an accurate process: but there's no good measure of it, and lots of reasons why we know it is frequently inaccurate.
Mike, I'm not sure I understand your objection. I am saying Koppl is focusing too narrowly on the way the monopoly government handles one particular aspect of the criminal justice system.
In response, you seem to be saying that I am overlooking another way that the government botches things in this realm.
I don't see how this renders my point "stupid." Again, maybe my point is useless because unfeasible, but you conceded its workability upfront.
So what am I missing here?
Posted by: Bob Murphy at May 15, 2008 9:08:57 PM
The jury instructions tell the jury that "all evidence is evidence including what you believe." This evidence includes circumstantial, real, and documentary evidence. The forensics evidence has probative value and should be allowed in court. I agree competition would add to the accuracy of the evidence.
Posted by: Mike Fladlien at May 15, 2008 10:30:21 PM
I don't get it - if you believe "well this test has an error rate of 1% then it's worthless and is inadmissible as evidence" - how is any one going to get a conviction? Are people going to get cold feet and say "since we can't ever be 100% percent sure we can never convict anyone"? How would a private system be any better? The anarcho-Libertarian approach becomes obvious let homeowners decide how they will and won't defend themselves and their right to defend themselves begin and ends at their fenceline.
Posted by: Gil at May 15, 2008 10:47:23 PM
I don't get it - if you believe "well this test has an error rate of 1% then it's worthless and is inadmissible as evidence" - how is any one going to get a conviction?
Well, part of the answer is that you focused on the best stat of all the ones Koppl mentioned. To repeat:
...to judge by the most comprehensive study on the reliability of forensic evidence to date, the error rate is more than 10% in five categories of analysis, including fiber, paint and body fluids. ...DNA and fingerprints are more reliable but still not foolproof....a 2005 study in the Journal of Criminal Law & Criminology suggests a fingerprint false-positive rate a bit below 1%, a widely read 2006 experiment shows an alarming 4% false-positive rate.
And so I think Koppl's point here is that when someone might go to jail for the rest of his life, steps to increase the reliability ought to be taken. If airplanes had a 1% failure rate, that would be pretty serious.
Posted by: Bob Murphy at May 16, 2008 10:08:12 AM
So, it's 1-4% based on research papers. These would be controlled (hopefully) experiments where they are NOT testing the soft-corruption of monopolistic labs trying to please their customer. They are likely only testing sample error, not bias.
The other problem is with the jury, who is over-influenced by authority (not that the system pounds this into them or anything).
From the provided link on the bullet analysis thing;
"I thought it was very important to our client's conviction. It was the single piece of physical evidence corroborating their story. And it came from, you know, it came from the mountaintop."
According to this lawyer, people think that because the FBI said it, it must be gospel.
I don't know much about trials, but I know a good bit about labs. They would have to pay me a lot of money to get up and tell people I was 100% sure of a result of a study anyone I know of could afford to run. Oh, wait. They couldn't pay me enough money to do that.
"you don't think you should believe it when you are told that something has a 99% accuracy rate."
Were the jurors told that the FBI bullet testing had an accuracy rate of...zero!?!
Posted by: Andrew at May 16, 2008 10:25:58 AM
"I don't get it - if you believe "well this test has an error rate of 1% then it's worthless and is inadmissible as evidence" - how is any one going to get a conviction?"
There used to be this little thing called witnesses. Tests can support real evidence.
It's also important how statistics are worded. I'll try to find the example I read one time about DNA evidence, but it goes something like this. The probability that this DNA is someone else's is X, but the probability that this DNA is the defendant's is Y. X and Y are not the same.
And, I don't think anyone here is arguing for throwing stuff away that has value. I basically said that crappy tests should be used for investigation, and less crappy tests should be laid bare for the jury to decide, and I'll add maybe the whole system needs an overhaul because it doesn't seem like our country's problem is NOT convicting enough people. The lawyers seem to disagree, which goes along with one of my tenets in life, never complain to someone for whom the system is working.
We also spend an @$$load on trials. We could probably spend a bit less on trials, improve tests, convict a few less innocent people, and come out way ahead.
Why would competing labs work? Oh, I don't know, maybe it's just a theory based on the idea that competition helps in almost everything else. Start with the theory that decoupling the interests in investigation and analysis and prosecution will improve objectivity. Add some adversarial competition between labs to keep them on their toes. Then you go from there and do some experiments, then do some trials with a few. Send tests to both labs and see how often they agree for a while. Have them cross-check. Or, maybe you are right, there's nothing to this idea.
Posted by: Andrew at May 16, 2008 10:34:59 AM
It should say "The probability that this DNA is NOT someone else's is X"
Btw, "how is any one going to get a conviction?"
I guess convictions didn't exist prior to forensic science, right?
And, now that we have a society that is pretty much approaching a meritocracy, there is a lot more incentive for people not to commit crime. We are also richer, so it may be time to rethink our paradigm about the tradeoffs between ensuring security versus making sure we don't convict innocent people, where those tradeoffs exist.
Posted by: Andrew at May 16, 2008 10:42:01 AM
"I don't get it - if you believe "well this test has an error rate of 1% then it's worthless and is inadmissible as evidence" - how is any one going to get a conviction?"
There used to be this little thing called witnesses.
*epic facepalm*
Witnesses are even *less* reliable than the physical evidence just criticized, genius!
***
I want to look at the criminal justice system as a "guilt-finding device". Based on all the handwringing, it has a "false positive" (wrongful conviction) rate of 1%, or 10%, or something like that, and is characterized as "too high".
Someone forgot to ask: what's the false *negative* rate? What's the fraction of crimes committed (and let's confine it to real crimes, not paper crimes) where the perp isn't convicted. I'm not just talking about not-guilty verdicts, but also, cases where:
-the crime isn't reported or discovered
-the police don't pursue the case, or the at least the right perp
-the prosecutor declines to go through
-the charges are dismissed by a judge
I'd put the false negative rate at over 99% across real crimes, ~10% for homicide.
While I think the alternatives proposed can decrease both the false negative and false positive rates, and improve the welfare of victims, I don't think it's meaningful, within a system, to complain about the false positive rate without also comparing it to the false negative rate. (Remember, "It's better for 1000 guilty to go free than one innocent get convicted" is a slogan, not an actual logical justification.)
Imposing ever-stricter conviction standards and safeguards will decrease false positives, but increase false negatives.
IIRC, the Middle Ages had obscenely high standards for guilt in homicide (one witness to the act or a confession were required), and, stuck between a rock and a hard place (can't convict, can't let murderers roam), law enforcement felt no inhibitions against underhanded tactics to get convictions, like beating confessions out of people.
Another metric more important than false positive rate, would be: how difficult it is for any given person to reduce his own chances of being falsely convicted.
Posted by: Person at May 16, 2008 11:10:47 AM
The prosecution has the ethical duty to bring up exculpatory evidence--otherwise the case should go to appeal.
Posted by: Mike Fladlien at May 16, 2008 12:04:31 PM
Oh, okay, then we throw out witnesses too. You guys are a trip.
Suggest ways to improve a system, and to you I must be saying toss it all out. When it is stated that convictions would be impossible if tests were improved or scuttled (e.g. bullet analysis) I suggest that there actually were, believe it or not, convictions prior to all these new fangled scientific-like, and not-so-scientific forensics, and the experts to schill them at trial. One of these technologies was witnesses. Well, the ridicule spews forth. I suggest we improve one link in the chain, and the criticism is levied that you haven't improved the entire chain, so it is all for nought. Fine guys. We all get the justice system you deserve.
No, erring on the side of non-conviction over false-conviction isn't a slogan. It's a tradeoff. And, it isn't even a constraint. We could in fact improve both sides of the equation if people didn't automatically jump to either fur 'em or agin 'em. But, given only those two choices, I err on the side of non-conviction.
Posted by: Andrew at May 16, 2008 2:40:56 PM
A 5% error rate on a forensic process single piece of evidence might not be very convincing , but what of the likelihood of 5 independent pieces of coincident evidence, each with a 5% likelihood of error? (0.05^5 = 0.00003% probability)
The weight of the evidence can largely - but not entirely - overcome statistical error rates with individual pieces of evidence. imho.
Posted by: Shane M at May 16, 2008 3:14:10 PM
So, expert witnesses aren't...witnesses? They have to testify to the results of the tests. We can also have tests for other witnesses. I'm not opposed to polygraphs. You'd still have to convince the jury they aren't fool proof.
"Imposing ever-stricter conviction standards and safeguards will decrease false positives, but increase false negatives."
Not necessarily true. It just increases the cost of positives. All we have to do is decide to pay the cost. We can reduce false positives and false negatives. Off the top of my head I can think of:
1) wait until you have sufficient evidence by expanding investigation
2) spend more on the process by freeing up resources from hard/expensive to convict victimless crimes to those with victims/witnesses
3) streamline trial process and obtain higher quality jurors, maybe assigning a devil's advocate role
4) redirect resources currently wasted on negative return investments (bullet analysis)
5) give me half a day and I could probably come up with 100 ways the miracle of the web could improve the investigation and trial process. How much of the current process is based on information transaction costs? Send the fingerprint electronically to 100 labs in India.
6) finally, beat confessions out of people as you suggest
If you consider the conviction rate in terms of statistical process control and total quality management, then yes, all other things being equal, tightening the control limits at the end of the process results in more falling outside tolerance. But, all other things aren't equal. You can tighten the variability by improving the process.
People say, "they already do this" and "they already do that" as if the legal system fell out the sky fully formed without need for improvement or rethinking as technology advances. Who owns the overall process? It seems like the falsely accused are the only ones with a solid and direct incentive to improve it. Not a good customer base.
Posted by: Andrew at May 16, 2008 3:36:45 PM
"The weight of the evidence can largely - but not entirely - overcome statistical error rates with individual pieces of evidence. imho."
True, and that's the thinking we need.However...
"There's all together something like 2,500 cases that the FBI analyzed since the 80s. There could be, you know, 30, 40, 50, who knows -- instances where people were wrongfully convicted," Scheck says.
Apparently, a lot of cases hinged on one particular test that turns out to be one particular load of B.S. Not that it proves that other tests aren't B.S. But the fact that they grudgingly admit one is, doesn't prove the other ones aren't either.
I'd say at 50 innocent people, $1 million dollars each per year of false incarceration wouldn't be an unreasonable restitution. You could improve a lot of legal processes for a billion bucks or so.
Posted by: Andrew at May 16, 2008 3:40:40 PM
I'd say at 50 innocent people, $1 million dollars each per year of false incarceration wouldn't be an unreasonable restitution. You could improve a lot of legal processes for a billion bucks or so.
I personally don't mind your estimate here, but I just want to point out that one of the advantages of a market economy is that individuals don't need to decide on rules in this way; the price of things emerges spontaneously.
So that's what I was getting at in my earlier post about a truly market-based system of law enforcement. (Tabarrok has written/edited a lot in this area; why doesn't he jump in here?? :)) We're all arguing here about how much falsely convicted people should be paid, how many independent tests should be conducted, what the right tradeoff is between false positives and false negatives, etc. But to me this is like Cubans arguing about how many loaves of bread a kidney transplant should cost, etc.
Again, it's a bit tricky to even imagine how you could meaningfully have a "free market" in law enforcement, evidentiary procedures, etc., but if you could it would solve all of these problems. The ratio of false positives to false negatives would be a market outcome, just as the number of broken eggs to non-broken eggs in cartons in your grocery store.
Posted by: Bob Murphy at May 16, 2008 4:17:10 PM
