« New assorted links | Main | Economicwoman.com »
CSI on Trial
...to judge by the most comprehensive study on the reliability of forensic evidence to date, the error rate is more than 10% in five categories of analysis, including fiber, paint and body fluids. ...DNA and fingerprints are more reliable but still not foolproof....a 2005 study in the Journal of Criminal Law & Criminology suggests a fingerprint false-positive rate a bit below 1%, a widely read 2006 experiment shows an alarming 4% false-positive rate.
How can we preserve the usefulness of forensic evidence while protecting the public when it breaks down? The core problem with the forensic system is monopoly. Once evidence goes to one lab, it is rarely examined by any other. That needs to change. Each jurisdiction should include several competing labs. ...This procedure may seem like a waste. But such checks would save taxpayer money. Extra tests are inexpensive compared to the cost of error, including the cost of incarcerating the wrongfully convicted....
Other reforms should include making labs independent of law enforcement and a requirement for blind testing. When crime labs are part of the police department, some forensic experts make mistakes out of an unconscious desire to help their "clients," the police and prosecution. Independence and blind testing prevent that.
That's forensics expert Roger Koppl writing in Forbes. If anything I think Koppl is being kind to CSI. Take bullet lead analysis a procedure used by the FBI for decades that turns out to have no scientific validity whatsoever.
Full Disclosure: Koppl's op-ed is based on a paper in a book called Law Without Romance edited by Ed Lopez to be published by Independent Institute where I am director of research.
Posted by Alex Tabarrok on May 15, 2008 at 09:45 AM in Law | Permalink
Comments
Competing labs would help, but that may solve only a small part of the problem. Any test, even if correctly done, will always have a non-zero false positive rate. This is unavoidable. The other source of error is, well, error itself. Competition can help solve the unforced error problem, but not the underlying false positive rate.
Posted by: Craig at May 15, 2008 9:59:48 AM
It's amazing that such no-brainers sound like novel concepts when applied to government.
Blind tests? Audits? Skepticism? Objectivity? Statistics!?!
How 'bout you present the jury with all the facts. "This study shows fingerprints have a 1-4% error rate. Paint chips have an 10% error rate." If you don't want that presented in court, then don't use it for evidence, just for investigation."
Of course, the obvious solutions aren't used BECAUSE they don't want the problem solved, because they don't pay for the errors. Fancy sounding tests are used and experts present the results for the very purpose of slanting the impressionable jury.
If we don't propagate errors in engineering we get points off. Or, in the real world bridges fall over and space satellites miss their targets. In government, you get a conviction and a promotion. The problem is there is no natural cost of error, it must be imposed on them.
Of course, I won't even bring up how this relates to letting courts decide issues like evolution and teaching science.
Posted by: Andrew at May 15, 2008 10:51:10 AM
It makes me sad to say this, but I really don't believe that Law enforcement, including prosecutors, really gives a darn about convicting innocent people. Stories like the FBI's bullet lead "evidence" and the "bite marks" expert below are so disappointing.
http://www.reason.com/news/show/125151.html
Posted by: Mercutio.Mont at May 15, 2008 11:54:59 AM
My experience from frequent jury duty in DC leads me to believe that police "know" who the bad guys are, and prosecutors are willing to bring a case on evidence that is less than perfect because they are sure they are guilty of something if not the crime they are on trial for. This additude no doubt contributes to the way forensic evidence is handled.
Posted by: joan at May 15, 2008 11:58:09 AM
@ andrew
that is how it works. the prosecution needs to introduce evidence about the procedure, but the defense can always question the prosecution's witness and nail him on the procedure's unreliability or even bring its own witness in. the problem is that 99% reliability, though not perfect, is pretty damn close in a jury's eyes. if i told you the democrats have a 99% chance of winning the election this fall, do you have a reasonable doubt that it will happen?? the prosecutors ought to be required to confirm the tests on their own before bringing the case at all. and no, defendants can't afford to run these tests themselves.
Posted by: andres at May 15, 2008 2:16:40 PM
Interesting that the article is in Forbes. Here's another Forbes article about a crime lab belonging to a surprising company.
http://www.forbes.com/forbes/2008/0421/102.html
Posted by: mikesdak at May 15, 2008 2:22:44 PM
"if i told you the democrats have a 99% chance of winning the election this fall, do you have a reasonable doubt that it will happen?"
Yes, because I wouldn't believe you. I'd also be allowed to do my own research. Ask for other expert opinions, and not just take your word as you presented it, regardless of how good my defense attorney is.
Posted by: Andrew at May 15, 2008 3:19:29 PM
I loved the article, but I think Koppl doesn't push it far enough when he says the problem is monopoly, and therefore we need the government to require multiple tests, etc. That's like saying the problem with oil prices is OPEC, and that's why we need to ask Saudi Arabia to pump more.
Case in point: Koppl discusses a guy who was wrongly convicted of rape and held for four years. His compensation? $118,000. If those are the penalties the government faces for mistakes, no wonder they are so sloppy. In a voluntary system where people could patronize different legal frameworks (and yes we can argue about how/whether that would work), I think the fines might be such that the agencies that survived the competition fixed the leaky roofs over their crime labs.
Posted by: Bob Murphy at May 15, 2008 4:32:37 PM
@andrew
you missed my point. the article says, "a 2005 study in the Journal of Criminal Law & Criminology suggests a fingerprint false-positive rate a bit below 1%." my question is whether a 99% chance of something being true leaves you with a reasonable doubt that it is, in fact, true. we can go down the list to the other methods and other percentages, and somewhere along the line, it'll break down--there will be enough of a likelihood that the test gave a false-positive to sustain a reasonable doubt. i'm thinking a false positive rate of over 10% is definitely there. under 5% is probably not. in between, i'm not sure.
my point is only that jurors are frequently presented with these error rates, which you think is some sort of novel idea--"How 'bout you present the jury with all the facts." the problem is not that they aren't given the facts, but that the facts themselves support the legal conclusion of guilt, even if the moral implications of a 1% false positive rate are alarming, to say the least.
as i understand it, you don't think you should believe it when you are told that something has a 99% accuracy rate. your skepticism is something of which you are obviously proud and i'm sure it serves you well. but are you seriously suggesting that jurors go off on their own, in the midst of trials, to determine the accuracy of the tests being used by a party?? that's what the adversarial system is for. the party's opponent will challenge it, and if they fail to persuade the jury, they lose. you don't honestly believe for a second that having jurors conduct independent evaluations of evidence or evidentiary methods is a good idea, whatever the merits of skepticism and independent evaluation in our daily lives. don't pretend otherwise.
Posted by: andres at May 15, 2008 5:03:15 PM
Even if libertarian fantasy competing privatized crime labs existed and were 100% correct, why would you think that would make a significant difference?
You would have to presume that conviction is an accurate process: but there's no good measure of it, and lots of reasons why we know it is frequently inaccurate.
But it makes a nice story for libertarians to bash government with, even if it is stupid.
Posted by: Mike Huben at May 15, 2008 7:50:43 PM
With regard to bullet lead analysis (actually trace element analysis), the definitive experiment seem easy to design. Buy a couple of hundred boxes of ammunition. Take some bullets from each and analyze them for whatever trace elements you desire. And then see if a blinded observer can match them up as to box of origin. My guess is that it won't be possible, but it seems that the FBI (or anyone else) has never done the experiment. That said, you have to understand that someone "from the FBI Laboratory" has a tremendous impact on a jury, which will be composed mostly of poorly educated, low IQ, scientifically illiterate individuals, and that such an expert's testimony is likely to be entirely believed without question. For an enjoyable experience as to how a pompous expert witness from a government laboratory spewing bogus testimony can be blown out of the water by a really sharp defense expert, rent a DVD of "My Cousin Vinnie."
Posted by: Ned at May 15, 2008 9:07:13 PM
Even if libertarian fantasy competing privatized crime labs existed and were 100% correct, why would you think that would make a significant difference?
You would have to presume that conviction is an accurate process: but there's no good measure of it, and lots of reasons why we know it is frequently inaccurate.
Mike, I'm not sure I understand your objection. I am saying Koppl is focusing too narrowly on the way the monopoly government handles one particular aspect of the criminal justice system.
In response, you seem to be saying that I am overlooking another way that the government botches things in this realm.
I don't see how this renders my point "stupid." Again, maybe my point is useless because unfeasible, but you conceded its workability upfront.
So what am I missing here?
Posted by: Bob Murphy at May 15, 2008 9:08:57 PM
The jury instructions tell the jury that "all evidence is evidence including what you believe." This evidence includes circumstantial, real, and documentary evidence. The forensics evidence has probative value and should be allowed in court. I agree competition would add to the accuracy of the evidence.
Posted by: Mike Fladlien at May 15, 2008 10:30:21 PM
I don't get it - if you believe "well this test has an error rate of 1% then it's worthless and is inadmissible as evidence" - how is any one going to get a conviction? Are people going to get cold feet and say "since we can't ever be 100% percent sure we can never convict anyone"? How would a private system be any better? The anarcho-Libertarian approach becomes obvious let homeowners decide how they will and won't defend themselves and their right to defend themselves begin and ends at their fenceline.
Posted by: Gil at May 15, 2008 10:47:23 PM
I don't get it - if you believe "well this test has an error rate of 1% then it's worthless and is inadmissible as evidence" - how is any one going to get a conviction?
Well, part of the answer is that you focused on the best stat of all the ones Koppl mentioned. To repeat:
...to judge by the most comprehensive study on the reliability of forensic evidence to date, the error rate is more than 10% in five categories of analysis, including fiber, paint and body fluids. ...DNA and fingerprints are more reliable but still not foolproof....a 2005 study in the Journal of Criminal Law & Criminology suggests a fingerprint false-positive rate a bit below 1%, a widely read 2006 experiment shows an alarming 4% false-positive rate.
And so I think Koppl's point here is that when someone might go to jail for the rest of his life, steps to increase the reliability ought to be taken. If airplanes had a 1% failure rate, that would be pretty serious.
Posted by: Bob Murphy at May 16, 2008 10:08:12 AM
So, it's 1-4% based on research papers. These would be controlled (hopefully) experiments where they are NOT testing the soft-corruption of monopolistic labs trying to please their customer. They are likely only testing sample error, not bias.
The other problem is with the jury, who is over-influenced by authority (not that the system pounds this into them or anything).
From the provided link on the bullet analysis thing;
"I thought it was very important to our client's conviction. It was the single piece of physical evidence corroborating their story. And it came from, you know, it came from the mountaintop."
According to this lawyer, people think that because the FBI said it, it must be gospel.
I don't know much about trials, but I know a good bit about labs. They would have to pay me a lot of money to get up and tell people I was 100% sure of a result of a study anyone I know of could afford to run. Oh, wait. They couldn't pay me enough money to do that.
"you don't think you should believe it when you are told that something has a 99% accuracy rate."
Were the jurors told that the FBI bullet testing had an accuracy rate of...zero!?!
Posted by: Andrew at May 16, 2008 10:25:58 AM
"I don't get it - if you believe "well this test has an error rate of 1% then it's worthless and is inadmissible as evidence" - how is any one going to get a conviction?"
There used to be this little thing called witnesses. Tests can support real evidence.
It's also important how statistics are worded. I'll try to find the example I read one time about DNA evidence, but it goes something like this. The probability that this DNA is someone else's is X, but the probability that this DNA is the defendant's is Y. X and Y are not the same.
And, I don't think anyone here is arguing for throwing stuff away that has value. I basically said that crappy tests should be used for investigation, and less crappy tests should be laid bare for the jury to decide, and I'll add maybe the whole system needs an overhaul because it doesn't seem like our country's problem is NOT convicting enough people. The lawyers seem to disagree, which goes along with one of my tenets in life, never complain to someone for whom the system is working.
We also spend an @$$load on trials. We could probably spend a bit less on trials, improve tests, convict a few less innocent people, and come out way ahead.
Why would competing labs work? Oh, I don't know, maybe it's just a theory based on the idea that competition helps in almost everything else. Start with the theory that decoupling the interests in investigation and analysis and prosecution will improve objectivity. Add some adversarial competition between labs to keep them on their toes. Then you go from there and do some experiments, then do some trials with a few. Send tests to both labs and see how often they agree for a while. Have them cross-check. Or, maybe you are right, there's nothing to this idea.
Posted by: Andrew at May 16, 2008 10:34:59 AM
It should say "The probability that this DNA is NOT someone else's is X"
Btw, "how is any one going to get a conviction?"
I guess convictions didn't exist prior to forensic science, right?
And, now that we have a society that is pretty much approaching a meritocracy, there is a lot more incentive for people not to commit crime. We are also richer, so it may be time to rethink our paradigm about the tradeoffs between ensuring security versus making sure we don't convict innocent people, where those tradeoffs exist.
Posted by: Andrew at May 16, 2008 10:42:01 AM
"I don't get it - if you believe "well this test has an error rate of 1% then it's worthless and is inadmissible as evidence" - how is any one going to get a conviction?"
There used to be this little thing called witnesses.
*epic facepalm*
Witnesses are even *less* reliable than the physical evidence just criticized, genius!
***
I want to look at the criminal justice system as a "guilt-finding device". Based on all the handwringing, it has a "false positive" (wrongful conviction) rate of 1%, or 10%, or something like that, and is characterized as "too high".
Someone forgot to ask: what's the false *negative* rate? What's the fraction of crimes committed (and let's confine it to real crimes, not paper crimes) where the perp isn't convicted. I'm not just talking about not-guilty verdicts, but also, cases where:
-the crime isn't reported or discovered
-the police don't pursue the case, or the at least the right perp
-the prosecutor declines to go through
-the charges are dismissed by a judge
I'd put the false negative rate at over 99% across real crimes, ~10% for homicide.
While I think the alternatives proposed can decrease both the false negative and false positive rates, and improve the welfare of victims, I don't think it's meaningful, within a system, to complain about the false positive rate without also comparing it to the false negative rate. (Remember, "It's better for 1000 guilty to go free than one innocent get convicted" is a slogan, not an actual logical justification.)
Imposing ever-stricter conviction standards and safeguards will decrease false positives, but increase false negatives.
IIRC, the Middle Ages had obscenely high standards for guilt in homicide (one witness to the act or a confession were required), and, stuck between a rock and a hard place (can't convict, can't let murderers roam), law enforcement felt no inhibitions against underhanded tactics to get convictions, like beating confessions out of people.
Another metric more important than false positive rate, would be: how difficult it is for any given person to reduce his own chances of being falsely convicted.
Posted by: Person at May 16, 2008 11:10:47 AM
The prosecution has the ethical duty to bring up exculpatory evidence--otherwise the case should go to appeal.
Posted by: Mike Fladlien at May 16, 2008 12:04:31 PM
Oh, okay, then we throw out witnesses too. You guys are a trip.
Suggest ways to improve a system, and to you I must be saying toss it all out. When it is stated that convictions would be impossible if tests were improved or scuttled (e.g. bullet analysis) I suggest that there actually were, believe it or not, convictions prior to all these new fangled scientific-like, and not-so-scientific forensics, and the experts to schill them at trial. One of these technologies was witnesses. Well, the ridicule spews forth. I suggest we improve one link in the chain, and the criticism is levied that you haven't improved the entire chain, so it is all for nought. Fine guys. We all get the justice system you deserve.
No, erring on the side of non-conviction over false-conviction isn't a slogan. It's a tradeoff. And, it isn't even a constraint. We could in fact improve both sides of the equation if people didn't automatically jump to either fur 'em or agin 'em. But, given only those two choices, I err on the side of non-conviction.
Posted by: Andrew at May 16, 2008 2:40:56 PM
A 5% error rate on a forensic process single piece of evidence might not be very convincing , but what of the likelihood of 5 independent pieces of coincident evidence, each with a 5% likelihood of error? (0.05^5 = 0.00003% probability)
The weight of the evidence can largely - but not entirely - overcome statistical error rates with individual pieces of evidence. imho.
Posted by: Shane M at May 16, 2008 3:14:10 PM
So, expert witnesses aren't...witnesses? They have to testify to the results of the tests. We can also have tests for other witnesses. I'm not opposed to polygraphs. You'd still have to convince the jury they aren't fool proof.
"Imposing ever-stricter conviction standards and safeguards will decrease false positives, but increase false negatives."
Not necessarily true. It just increases the cost of positives. All we have to do is decide to pay the cost. We can reduce false positives and false negatives. Off the top of my head I can think of:
1) wait until you have sufficient evidence by expanding investigation
2) spend more on the process by freeing up resources from hard/expensive to convict victimless crimes to those with victims/witnesses
3) streamline trial process and obtain higher quality jurors, maybe assigning a devil's advocate role
4) redirect resources currently wasted on negative return investments (bullet analysis)
5) give me half a day and I could probably come up with 100 ways the miracle of the web could improve the investigation and trial process. How much of the current process is based on information transaction costs? Send the fingerprint electronically to 100 labs in India.
6) finally, beat confessions out of people as you suggest
If you consider the conviction rate in terms of statistical process control and total quality management, then yes, all other things being equal, tightening the control limits at the end of the process results in more falling outside tolerance. But, all other things aren't equal. You can tighten the variability by improving the process.
People say, "they already do this" and "they already do that" as if the legal system fell out the sky fully formed without need for improvement or rethinking as technology advances. Who owns the overall process? It seems like the falsely accused are the only ones with a solid and direct incentive to improve it. Not a good customer base.
Posted by: Andrew at May 16, 2008 3:36:45 PM
"The weight of the evidence can largely - but not entirely - overcome statistical error rates with individual pieces of evidence. imho."
True, and that's the thinking we need.However...
"There's all together something like 2,500 cases that the FBI analyzed since the 80s. There could be, you know, 30, 40, 50, who knows -- instances where people were wrongfully convicted," Scheck says.
Apparently, a lot of cases hinged on one particular test that turns out to be one particular load of B.S. Not that it proves that other tests aren't B.S. But the fact that they grudgingly admit one is, doesn't prove the other ones aren't either.
I'd say at 50 innocent people, $1 million dollars each per year of false incarceration wouldn't be an unreasonable restitution. You could improve a lot of legal processes for a billion bucks or so.
Posted by: Andrew at May 16, 2008 3:40:40 PM
I'd say at 50 innocent people, $1 million dollars each per year of false incarceration wouldn't be an unreasonable restitution. You could improve a lot of legal processes for a billion bucks or so.
I personally don't mind your estimate here, but I just want to point out that one of the advantages of a market economy is that individuals don't need to decide on rules in this way; the price of things emerges spontaneously.
So that's what I was getting at in my earlier post about a truly market-based system of law enforcement. (Tabarrok has written/edited a lot in this area; why doesn't he jump in here?? :)) We're all arguing here about how much falsely convicted people should be paid, how many independent tests should be conducted, what the right tradeoff is between false positives and false negatives, etc. But to me this is like Cubans arguing about how many loaves of bread a kidney transplant should cost, etc.
Again, it's a bit tricky to even imagine how you could meaningfully have a "free market" in law enforcement, evidentiary procedures, etc., but if you could it would solve all of these problems. The ratio of false positives to false negatives would be a market outcome, just as the number of broken eggs to non-broken eggs in cartons in your grocery store.
Posted by: Bob Murphy at May 16, 2008 4:17:10 PM
"This procedure may seem like a waste. But such checks would save taxpayer money. Extra tests are inexpensive compared to the cost of error, including the cost of incarcerating the wrongfully convicted...."
What costs are associated with the wrongfully convicted? Most must be people of few means who couldn't afford an effective defense in the first place; they are unlikely to gain retrials. And the loss of these people is probably insignificant to the economy. The expense of their incarceration is a wash, since someone would be in jail for the crime one way or the other.
Posted by: p.a. at May 16, 2008 7:07:27 PM
Bob Murphy:
First, if you want to use the dichotomy monopolistic versus competitive, then you have to say that jury trial is monopolistic the same way that elections are monopolistic or markets are monopolistic. After all, markets are imposed by governments instead of competing with alternative arrangements such as socialism or mixed models.
And of course there is competition in a trial: that is the nature of an adversarial system. Good competitors are rewarded on both sides: prosecution and defense.
Nor is there a history of "better" methods of conviction for crimes in the sense of accuracy, though there has been an enormous amount of experimentation in the innumerable governments, national and local. Largely because there's no good way to measure the accuracy rate: no better "truth" to compare to.
We know that juries can be biased. We know that witnesses have a far higher error rate than forensics labs. We know that defense and prosecution attempt innumerable invalid arguments attempting to sway or mislead juries. We know that there are "hanging judges". We know that laws can be unjust and biased.
But there's no evidence that "let the market sort them out" will work any better than "kill them all and let god sort them out". Perhaps markets could fix this one source of evidence: but of course, there is the same moral hazard of corruption because of the large stakes. The difference is that hiding behind private rights, we're less likely to observe the corruption than in a public process.
Posted by: Mike Huben at May 17, 2008 5:07:07 AM
The only Libertarian solution (for those wanting a Libertarian solution) is the one I suggested - let property owners deal with trespassers and let two people in the street dispute and duke it out or they can find an arbitrator. The police, courts, judges, jails, juries, laws, etc., are all entities of a central government system.
Posted by: Gil at May 18, 2008 1:52:37 AM
No, Gil, Those are not all entities of a central government system. Courts, judges, juries, and laws are all entities of private arbitration systems as well. And police and prisons would be potential entitles of such non-central government legal systems were they not forcibly banned from competing with the government monopoly.
Posted by: Micha Ghertner at May 18, 2008 10:41:50 AM
Mike Huben,
After all, markets are imposed by governments instead of competing with alternative arrangements such as socialism or mixed models.
Nonsense. Markets are simply the sum of voluntary transactions conducted by freely acting individuals. For that matter, socialism can be voluntary too, if people freely choose to aggregate their wealth and income unanimously. Neither of these need to be imposed by governments.
And of course there is competition in a trial: that is the nature of an adversarial system. Good competitors are rewarded on both sides: prosecution and defense.
Yes, some competition is better than none. Even more is better than some.
Nor is there a history of "better" methods of conviction for crimes in the sense of accuracy, though there has been an enormous amount of experimentation in the innumerable governments, national and local. Largely because there's no good way to measure the accuracy rate: no better "truth" to compare to.
Hmm...I wonder why that is? I'm reminded of the reason why anti-death penalty groups have such great difficulty finding a single, legally proven case of wrongful state execution. Could it have something to do with the fact that the government tends to dispose of evidence once the execution has taken place and in numerous other ways makes it difficult to impossible to prove it has murdered an innocent person? Might this have anything to do with the fact that the state is *gasp* a monopoly, answerable to no higher authority but itself and the conservative lackies who prefer its very observable and frequent corruption to the scary thought of change to the totally foreign and unknown?
I mean, it's amazing we ever got democracy, trials, and the adversarial system in the first place; certainly whatever we had before that--however corruption infested--was still better known and therefore safer-feeling, amirite?
Posted by: Micha Ghertner at May 18, 2008 10:58:08 AM
You got to be kidding M Ghertner. If two parties have a dispute they want to non-violently resolve then they find an arbitrator they can both abide by. They would then both state their case and how each is right and the other is wrong. The arbitrator presumedly is able to sort out the problem and show where the rightness and wrongness is and shows how the two parties can come to a truce and resolve the problem. On the other hand how is the rigmarole of the justice system not part of a centralised system? Where would private operators get their right to a law or legal code that they could use physical force to restrain or punish somebody? Why would a known violent offender submit to a private justice system? History has shown protecton from crime in a village has been the realm of the individual within the comman law agreement between other villagers as to what constitutes a crime and what a home owner can use as defence.
Posted by: Gil at May 20, 2008 2:21:06 AM
Gil,
Where would private operators get their right to a law or legal code that they could use physical force to restrain or punish somebody?
Where does government get this right? Depending on your answer to that question, you may or may not have your answer to the original question.
Why would a known violent offender submit to a private justice system?
Generally, because the social incentives of not submitting look even worse. But even in a case where a known violent offender doesn't mind cutting all ties with a society, there is nothing stopping people from hunting him down and trying him anyway.
Posted by: Micha Ghertner at May 23, 2008 5:46:26 PM
