« The most cited economics articles in the last five years | Main | No Immunity for the Telecoms »
Updated numbers on violent deaths in Iraq
I've cited the Lancet numbers myself (in qualified fashion), but maybe the estimate of a million Iraqi deaths is far too high:
A new survey estimates that 151,000 Iraqis died from violence in the three years following the U.S.-led invasion of the country...For the new study, however, surveyors visited 23 times as many places and interviewed five times as many households. Surveyors also got more outside supervision in the recent study; that wasn't possible in the spring of 2006 when the Johns Hopkins survey was conducted..."Overall, this is a very good study," said Paul Spiegel, a medical epidemiologist at the United Nations High Commission on Refugees in Geneva. "What they have done that other studies have not is try to compensate for the inaccuracies and difficulties of these surveys, triangulating to get information from other sources."
Here is more, here is the study itself. This new estimate is probably not the final word, but you will recall that anyone who questioned the older Lancet estimate was pilloried at length; there is a lesson here - Thy Shall Not Use Thy Blog to Squelch Heretics -- and I am curious to see who will offer mea culpa and who will not. "The two estimates aren't as different as they look" is one way of spinning it; "I was wrong" is another.
Posted by Tyler Cowen on January 10, 2008 at 07:51 AM in Data Source | Permalink
Comments
I wonder if these new figures will get much press, other than blogs. I also find it strange that there are no comments. Anybody out there?
Posted by: macquechoux at Jan 10, 2008 8:19:26 AM
The . It also reports the correct estimate for the original study - 600,000, not "one million".
It may be worth noting that this new study apparently only counts "civilian" deaths, while the Lancet study, iirc, counted all deaths, whether civilian, military or insurgent. The Iraqi ministry of health has also revised its pre-war estimates of death rates for this study, which obviously makes a big difference.
The Guardian quotes one of the authors of the original study, who is supportive even though he claims "overwhelming evidence" the study new study's estimates are too low (it doesn't go into what this evidence might be). He also says the studies' findings are actually less different than they appear - a doubling as opposed to a tripling of death rates, accodring to the Guardian.
It should be noted, the Iraqi government was apparently previously relying on IBC figures of around 40,000, so this increases their estimate by over three times. (The British government tends to refer to IBC figures too; I don't know about the US.)
Posted by: Tim at Jan 10, 2008 8:46:59 AM
My prior was that the Lancet studies were shocking and hard to believe but the best estimate available. This new information should lead to revised priors. In search for the discrepancy between the Lancet studies and this questions will include why the pre-war mortality rate is so different, and whether this implies under/over-reporting in the NEJM/Lancet study/ies.
But:
What is the estimate of one million deaths? (Neither of the John Hopkins / Bloomberg studies report this.)
Also, I would be interested to have spelled out the lesson you refer to is? My impression is that the article you link to is a mostly entirely valid criticism of sundry obvious statistical fallacies committed by people refusing to review their priors even when intellectually unable to provide a reason for their stance. It could only have raised the standard of debate. And it does not refer to Megan McArdle (who would not be silly enough to make those mistakes), nor to the arguments she made.
Meanwhile, innumerate criticism of the John Hopkins / Bloomberg study is alive and well; here's the latest National Journal version: http://news.nationaljournal.com/articles/databomb/index.htm. No wonder Deltoid and others have been kept busy.
Posted by: per at Jan 10, 2008 8:47:08 AM
So Fred Kaplan was "pilloried" by Daniel Davies, was he? Anyone got a picture of poor Fred in the stocks?
Posted by: Kevin Donoghue at Jan 10, 2008 9:13:13 AM
There are still two points that aren't much changed by the numbers:
(1) Nobody should be comforted by "only" 150k deaths.
(2) There is responsibility, and then there is blame; the difference is more than just semantics. Responsibility falls to the US for security, but blame falls to the murderers who actually did the killing. If anything, the higher casualty figures were an argument for more troops, not less.
Posted by: Independent George at Jan 10, 2008 9:19:45 AM
"but the best estimate available": it's a dangerous game to accept any old rubbish as an estimate rather than to say "there is no worthwhile estimate available". I'm also struck by the tedious debates about statistical validity - all statistical analyses depend on key assumptions of trust (did they actually do what they claimed?) and representativeness (were the samples, assuming they really existed, representative of the population?). Fiddling about with technical details when you are unable to answer these two questions satisfactorily is pretty feeble stuff.
Posted by: dearieme at Jan 10, 2008 9:31:57 AM
dearime: yes, of course; if one is happy to dismiss as fraud any study whose findings do not conform with one's priors one can learn nothing new. That, too, may be called a dangerous game.
Posted by: per at Jan 10, 2008 10:21:53 AM
Well said dearime! People are all-too prone to accept 'confidence intervals' around an estimate as a measure of how confident we should be that something is true.
But statistics is mostly summary - and summarized garbage is still garbage.
Above quite a small sample size, the representativeness of a sample is vastly more important than the size of a sample - that was the message of the famous 1936 Presidential election Gallup poll story - http://en.wikipedia.org/wiki/George_Gallup
Posted by: Bruce G Charlton at Jan 10, 2008 10:35:59 AM
So only 150,000 Iraqis died. And here I was thinking the war was unjustified. My bad. Obviously it was an even-handed reponse to the deaths of 3000 New Yorkers
Posted by: Joe at Jan 10, 2008 11:23:41 AM
(((snip))) "Two of the study's co-authors told the National Journal that they opposed the war and submitted their findings to Lancet with the insistence that it appear before the election.
Much of the data for the study, which was organized by Johns Hopkins University, was collected by Iraqi researcher Riyadh Lafta, who once worked for Saddam, the Nat. Journal
said." (((snip)))
All you need to know...
Cheers
JR
Posted by: JR at Jan 10, 2008 11:25:06 AM
One obvious potential bias is that victimized groups are far more likely to have fled the country, and therefore were not surveyed. Since millions reputedly fled their homes, this is not a trivial problem. I wonder how they addressed it?
Posted by: steve at Jan 10, 2008 11:26:35 AM
Dear JP,
"All you need to know..." is a poor conclusion.
For one, you are making the weak guilt by association fallacy.
Secondly, please identify which government researchers in pre-US Invasion Iraq -didn't- work for Saddam.
Thirdly, describe the relationship and degree of closeness to Saddam and his cult of personality that Lafta enjoyed.
Fourthly, describe Lafta's degree of political and philosophical devotion to Baathism.
As it is, your conclusion that Lafta's work is unreliable because of a preported relationship to Saddam is very weak. As with any good science, examine the method and try to reproduce the results following that same method in order to validate the results.
Posted by: Not All at Jan 10, 2008 11:50:29 AM
Dearest "Not all,"
Credibility is an absolute when presenting such a study. Those two issues I touched on destroy any possible credibility. I didn't even have to point out that the funding for the Lancet report came from George Soros.
Good grief...you can't possibly be defending the original numbers. Right?
Cheers,
JR
Posted by: JR at Jan 10, 2008 12:05:53 PM
The estimate of 1 to 1.2 million deaths was NOT given by any study or researcher, so far as I remember. It was a very rough extrapolation from the end date of one of the Lancet studies to the present time, and it roughly matched an extrapolation from another and separate determination of Iraqi death rates.
Posted by: Lee A. Arnold at Jan 10, 2008 12:08:14 PM
Tyler, I don't see how this study means defenders of the Lancet study owe anyone an apology. Either the Lancet critics had sound objections about the methodology of the Lancet study, in which case they were always owed an apology, or they were making unsound objections and are still not owed an apology. I don't believe anyone's argument turned on what a future study would show.
Posted by: washerdreyer at Jan 10, 2008 12:38:09 PM
"So only 150,000 Iraqis died. And here I was thinking the war was unjustified. My bad. Obviously it was an even-handed reponse to the deaths of 3000 New Yorkers"
Is there really people, who are stupid enough to believe stuff like this? Do these people believe that the US Marines went from house to house to kill civilians? Do they not understand, that the majority of these victims were killed by the same people who attacked the WTC? Why are so many people so fricking retarded?
Posted by: Sven at Jan 10, 2008 12:58:48 PM
Do they not understand, that the majority of these victims were killed by the same people who attacked the WTC?
This would seem to be a new revelation.
Posted by: shecky at Jan 10, 2008 1:04:53 PM
It is interesting to contrast the reasonable tone of Megan's original post with the vehement response then and some of the agitated posters here who still want to cling to the Lancet study. The Lancet study made a splash because it was a big number. The impact would have been much less had the number been 150,000. Futhermore, the death rate must be much lower now after the success of Petraeus' strategy.
Posted by: Rich Berger at Jan 10, 2008 1:15:00 PM
Obviously, this hasn't gotten much press play- this is the first I have read of it.
Posted by: Yancey Ward at Jan 10, 2008 1:34:29 PM
Thou shalt not use thy...
Thou- you
Thy- your.
Posted by: anotherlmrr at Jan 10, 2008 2:14:17 PM
Thou shalt not use thy...
Thou- you
Thy- your.
Posted by: anotherlmrr at Jan 10, 2008 2:14:40 PM
I do wonder what the impact would have been had this study come out first.
Arguably, the Lancet study has pushed the Iraqi government towards doing this study - which was one of the hopes of the authors.
Can we move past the methodological issues distinguishing the studies to other things? In particularly, how do we interpret this kind of loss of life, how does it make us feel. The British government has been pushing the IBC figure of 40,000 deaths; if they now accept this official Iraqi figure, does it change anything? Is there anything they, or others would do differently if they believe (or are forced to admit) more than three times as many people have died as they previously estimated? Is there any kind of economic calculus going on - say, a relation between how much pro-occupation people are willing to spend in Iraq and the severity of Iraqi casualties?
What does it change if we accept the Lancet figures? Do we have (as Stalin would say) a threshold after which more deaths become effectively meaningless to us? I wonder whether this is the case, whether precise figures only become useful later, when we're comparing the scale of historical events.
I don't imagine this study will make much news, because it's not particularly useful as propaganda for any political position, and for those interested in the truth, that means getting into questions of statistical and epidemiological methodology. Which is not something the media are very interested in (because generally, neither is anyone else).
Posted by: Tim at Jan 10, 2008 2:18:37 PM
Curious why apparently in this crowd the following two assertions are made:
1. The study that says *all* fatalities (including disease, poor medical care, etc.) are high must be wrong.
2. The study that says that all *violence* caused fatalities are lower must be right.
Why aren't the same doubts applied to the 1st, not applied to the second? Since the studies measure two different quantities, it's possible that both are correct.
I don't know what the correct answer to the number of *total* fatalities is, but when I draw the conclusion that the civilian toll in civil war(s) is usually high, I don't derive it from political priors, I derive it from history. The Lancet study is certainly consistent with history. The new study may indicate the actual toll was mercifully less. But the idea that the war is a most excellent adventure because it *only* directly kills 50,000 people per year (for the economists: at a cost of hundreds of billions of dollars per year) is ah, wrong.
Posted by: Russell L. Carter at Jan 10, 2008 2:34:02 PM
As I blogged in 2006, the Lancet study total number of deaths was probably overestimated because Iraq is too dangerous a place to have carried out the study in the random fashion that was claimed:
"Or it could be that the interviewers got in contact ahead of time with neighborhood leaders to see if their presence would be welcome to reduce their chances of being killed. (That's not good random surveying hygiene, but are you going to blame them?) Then, in a neighborhood where the local big shot wanted their presence, he might have passed the word around to aggrieved families to get ready to tell their stories to the interviewers when they showed up. This could cause a bias upward in the number of deaths reported. [It would account for the very large number of interviews per day and the high proportion of interviewees with death certificates of their loved ones.]
"The more I think about the mechanics of carrying out the survey on the street without getting killed, the more I suspect that the Iraqi interviewers didn't actually implement the purely random survey design that the American professors from MIT and Johns Hopkins dreamed up for them. It would be nuts to to let luck determine which streets you'd choose, as the report claims they did. You'd want to only go where you knew you'd be safe. Then you'd tell the Americans you did exactly what they told you to do."
http://isteve.blogspot.com/2008/01/new-study-150000-violent-deaths-in-iraq.html
By the way, I estimate that American troops have been firing 275,000 rounds of ammunition per day in combat.
Posted by: Steve Sailer at Jan 10, 2008 3:11:56 PM
By the way, this estimate of 151,000 violent deaths was only for the first 3.0 years of this 4.8 year long (and counting) war. The first year of the war was pretty peaceful, so the total violent deaths under this methodology should be approaching a quarter of a million by now.
And then there is the big increase in nonviolent deaths reported in this latest study.
But it's all worth it because we've made Iraq safe for Iranian influence!
Posted by: Steve Sailer at Jan 10, 2008 3:31:28 PM
Caveat: I have not yet analyzed this new study for biases the way I carefully reviewed the Lancet study, so I don't know how much it should be trusted.
Posted by: Steve Sailer at Jan 10, 2008 3:42:47 PM
Just to get the numbers straight, the Lancet study estimated that, from the March 2003 invasion through June 2006, there were 601,000 violent Iraqi deaths (95% confidence interval: 426,000-794,000). This new study estimates 151,000 violent Iraqi deaths over the same time period (95% confidence interval: 104,000-223,000). Correct?
Posted by: Dan at Jan 10, 2008 3:59:16 PM
July 2006 is the end date for the Lancet study, not that it makes much difference. Since the two studies are fundamentally based on a comparison of rates though, it's also worth taking not of them:
The Lancet study gives figures of 5.5/1000 people per year before the invasion and 13.3 after
The new study gives 3.17 before and 6.01. (In all cases, these are deaths from all causes.)
In other words, the new has death rates at almost half the lancet study both before and after the invasion. If you assume whatever biases exist are consistent pre-and post invasion, then the two show an increase that is not shockingly dissimilar (124% vs 89%).
But given the differences in the count of pre-invasion mortality, I don't know what to make of these statistics.
Posted by: Tim at Jan 10, 2008 4:39:51 PM
Well, I am someone who threw the Lancet study at some folks when it came out.
However, I always granted that there was lots of uncertainty about the numbers.
I also note that the only other number out there at the time was something on
the order of 30,000, which President Bush was quoting. That was the number I
used the Lancet study to trash, and that number still looks ridiculous.
The number of extra nonviolent deaths is supposedly the same as the number of
violent ones. They did an arbitrary split. So, the total is just over 300,000,
with lots of those due to the lost power, water, sewage, and so forth, thanks to
the war.
Indeed, to JR, and some others, the differences between the studies are not really
all that great. The main source of the difference is the murky estimate of the
prewar death rate. In terms of the death rate since, they are not all that different.
Posted by: Barkley Rosser at Jan 10, 2008 4:48:49 PM
Dan, You're not right. The Lancet number is for "excess deaths," not just violent death. The new NEJM study has a more comparable (violent + nonviolent) number that is higher. It's about 300,000 (since Table 2 shows that 53.5% of post-invasion deaths are due to injuries in armed conflict).
Posted by: J at Jan 10, 2008 5:02:29 PM
Oops. It looks a little more complicated than I calculated above. In this new study, "intentional" + "armed conflict" only equals 14% of all deaths. About 1 million deaths. But then you run into the problems with excess deaths based on pre-invasion rates.
Posted by: J at Jan 10, 2008 5:15:01 PM
comments are more insightful then Tyler's editorializing.
Posted by: Jor at Jan 10, 2008 5:52:24 PM
If only I could squelch people with my blog. But Megan McArdle is stilling writing rubbish about the Lancet study and Tyler Cowen is still linking to her.
Posted by: Tim Lambert at Jan 10, 2008 7:37:26 PM
"One obvious potential bias is that victimized groups are far more likely to have fled the country, and therefore were not surveyed. Since millions reputedly fled their homes, this is not a trivial problem. I wonder how they addressed it?"
Posted by: steve
IIRC the Lancet study just ignored it (they'd go on to another house). This is an undercounting bias; but don't hold your breath until Lancet critics cite it - they seem to not see that particular flaw.
The latest study had a larger undercounting bias, from refusal to go into certain areas where it was feared that 'many enter; few leave'. ~11%? Even more than in the Lancet study. They attempted to impute the death rates there. Considering that the IBC project would be expected to undercount more in the worst areas, I'd be suprised if imputation based on those counts were not biased downwards.
Posted by: Barry at Jan 10, 2008 7:42:31 PM
"It is interesting to contrast the reasonable tone of Megan's original post with the vehement response then and some of the agitated posters here who still want to cling to the Lancet study. "
Posted by: Rich Berger
When somebody does a study, and a propagandist with no background at all claims that it couldn't be true, because her (MBA) 'gut' says so, a vehement response is called for.
Posted by: Barry at Jan 10, 2008 7:46:51 PM
Tyler,
There seem to be some pretty serious and substantive objections to your implication that mea culpas are in order. Will you be gracing us with a response? "You are all just rationalizing your priors" is one way of spinning it; "I didn't really read and understand the new study" is another.
Posted by: Mark at Jan 10, 2008 7:54:12 PM
Sven:
"Is there really people, who are stupid enough to believe stuff like this? Do these people believe that the US Marines went from house to house to kill civilians? Do they not understand, that the majority of these victims were killed by the same people who attacked the WTC? Why are so many people so fricking retarded?"
We're not retarded, we simply remember the recent history - the US blew away the government, killed anybody who tried to (rightly, wrongly or whatever) impose order, and quite deliberately did not police Iraq itself. The stunningly obvious result, predicted beforehand, was that Iraq fell into sh*tland.
Posted by: Barry at Jan 10, 2008 7:55:46 PM
"anyone who questioned the older Lancet estimate was pilloried at length"
Not true: David Kane, "Jane Galt" and a few other functionally innumerate ideologues were pilloried at length--and deservedly so.
Posted by: engels at Jan 10, 2008 8:03:05 PM
Claims by conservatives that Iraq War opponents were too quick to trumpet the Lancet results leaves conservatives open to criticisms that they are trumpeting the IFHS results before understanding them fully. The fact is that the mainstream response to the Lancet study was cautiously pessimistic ("even if it's off by a factor of two it still means the Bush Administration has been lowballing us") and the conservative reaction was shrill and defensive. Now conservatives are claiming that the Johns Hopkins researchers are biased against America, and that the IFHS survey conducted by the Iraqi Ministry of Health is less biased. Their responses don't strke me as intellectually honest, (although they would be quick to point out that doesn't mean they are wrong.)
But the whole controversy between the two sets of numbers is a red herring: Both the Johns Hopkins and the IFHS estimates of civilian deaths are much higher than official estimates that were coming from Iraqi, British or American governments at the time. If the Johns Hopkins study had said 150,000 iraqis were killed, that would have been big news, too.
Posted by: MostlyAPragmatist at Jan 11, 2008 6:43:40 AM
Well Barry, if you are not a specialist in the area, like a demographer or statistician, but do have a feel for numbers, you can make an intelligent assessment of the correctness of a study like the Lancet's. I can think of at least a couple of examples of outrageous claims that were obviously whoppers, but were repeated in the press. For one, I remember that Mitch Snyder claimed that there were 3 MM homeless in the US (more than one out of 100 Americans at the time). Real numbers were in the 250K-600K range. Taking Mitch's number and applying to Manhattan, for example, would probably have meant about 15,000 at the time - clearly not consistent with the numbers seen.
We also heard claims of incredible numbers of deaths due to the sanctions imposed on Sadaam - weren't the numbers like 500K per year? That would have been a death rate of about 2% for the Iragis, unreasonably high. BTW, this new study debunks those claims, too. We also found out that the money from oil sales was getting to Iraq, but that Saddaam was using it for new palaces, among other things.
Tim - too bad your blog can't squelch annoying commenters like Megan.
Posted by: Rich Berger at Jan 11, 2008 8:50:39 AM
Did someone "squelch" Megan McArdle? Last time I heard, she was being published in a Big Media magazine with over a million readers, mostly parroting the US government party line in the cause of defending the world's most powerful state and its military. I don't really see her as a persecuted dissident, but I am not surprised to learn that the "libertarians" who populate this comments section see things differently...
Posted by: engels at Jan 11, 2008 9:56:13 AM
At this point, I think the defenders of the original study need to pony up and tell us why _they_ think the Lancet number was so far off. No claims that you're just correcting "functionally innumerate" critics.
Posted by: Zach at Jan 11, 2008 10:21:08 AM
I'll start
1) Lack of internal controls and reality checks. I really like how the NEJM study also asked females about mortality among siblings. It's a cheap way to get two relatively independent estimates of the same quantity. Another approach might be to assign clusters to 'A' and 'B' groups before the study and compare the A and B estimates.
2) Lack of external controls and reality checks. Differing from Iraq body count and government figures by a factor of 10 is embarassing. Especially since most of the dead in the original study had been issued death certificates. If I were reviewing the original paper, I would ask the authors to account for the disparity between their estimated number of death certificates issued and the official government count.
Here again I like the NEJM approach of comparing the distribution of violence among provinces to Iraq body count.
3) If the statistical methods used for the Lancet study are really valid, I have to wonder if the surveyers implemented the protocol carefully enough. Call it the lesson of the exit polls: slight departures from planned statistical methods can skew results by large amounts.
Posted by: Zach at Jan 11, 2008 11:10:47 AM
Zach,
Your request for defenders of the original study to explain why they think the original study was so far off begs the question, since you seem to be assuming in advance the the Lancet number was too high, and therefore any new result which is lower is therefore closer to the truth. Now, you've made some arguments as to why the new study is better, but there are also some arguments about why it might be under-counting (more people have fled the region at this point, and it seems reasonable to suppose that people who have had family members killed are more likely to flee) which you've not addressed.
It seems like the right conclusion at this point is that the number of excess deaths is somewhere between 300K and 600K, modulo error bars. This study reaffirms the earlier conclusion that IBC and government numbers (such as Bush's 30K claim) are ridiculously off.
One last point: It is true that the factor of 10 difference between the original study published in the Lancet and the government figures was embarrassing, but it was not an embarrassment to the Lancet or the authors of the paper. If you disagree with this, can you point me to any published research (hopefully done before the current war) that shows that body-count style tallies (whether performed by government or media) of wartime deaths are _more_ accurate than cluster surveys? If there were any such research, I'd be astounded that the war boosters haven't already trotted it out.
Posted by: Mark at Jan 11, 2008 1:22:03 PM
Hate to say it but all this recimination back-and-forth about which study is correct is bullshit. Anyone out there think we would be quibbling about how many tens of thousands (set aside hundreds of thousands) of American civilians had been killed in an morally and politically disastrous (set aside illegal) invasion?
Sure the integrity of science as a tool for ascertaining something like truth is important. But we are talking about tens of thousands of civilians dead because Bush and company lied repeatedly and we let them get away with it. Let's worry about truth there.
Posted by: JJ at Jan 11, 2008 1:26:45 PM
Hate to say it but all this recimination back-and-forth about which study is correct is bullshit. Anyone out there think we would be quibbling about how many tens of thousands (set aside hundreds of thousands) of American civilians had been killed in an morally and politically disastrous (set aside illegal) invasion?
Sure the integrity of science as a tool for ascertaining something like truth is important. But we are talking about tens of thousands of civilians dead because Bush and company lied repeatedly and we let them get away with it. Let's worry about truth there.
Posted by: JJ at Jan 11, 2008 1:28:16 PM
As several of us keep pointing out in comments sections, and as daniel has just said at Crooked Timber, the first Lancet paper actually agrees very closely with this NEJM paper if you exclude the Fallujah outlier.
If we just stick to the violent deaths, the Lancet1 paper wasn't explicit about this, but Tim Lambert and others pointed out that its midrange violent death toll (excluding the Fallujah outlier) was about 60,000. Over 17 months that's about 120 deaths per day. Now look at the NJEM paper. How many violent deaths per day does it give? Around 120. The close agreement is coincidental, given the error bars, but remember that critics of the Lancet1 paper weren't just saying that its estimate was outlandish if you included Fallujah (they might have been right about that). They were saying it was outlandish without Fallujah. Iraq Body Count's death count at that time (they corrected it upwards by a few thousand later on) was 15,000, about 4 times less. None of the critics said that it was reasonable to think the true violent death toll might be 3-4 times higher--until the Lancet paper was published the prowar people thought Iraq Body Count's numbers were inflated.
Posted by: Donald Johnson at Jan 11, 2008 1:41:54 PM
DJ-
That would be a sweet explanation except the 60,000 number was not the one that got the press. If the inclusion of one outlier changed the results so much, how much stock can you put in the conclusion?
Posted by: Rich Berger at Jan 11, 2008 1:50:28 PM
The number that got into the press was 100,000--of that 100,000, roughly 60,000 were from violence. The 100,000 excess is the number without the Fallujah outlier. And it appears to be consistent with this latest paper.
Posted by: Donald Johnson at Jan 11, 2008 3:20:05 PM
DJ -
There were two Lancet studies, one in 2004 and one in 2006. The 2006 study , as noted here,indicated that "'655,000 Iraqis killed since invasion'
Sarah Boseley, health editor
Wednesday October 11, 2006
The Guardian
The aftermath of a Baghdad bomb attack - a study published in the Lancet estimates that 655,000 Iraqis have died as a result of the war. Photograph: Getty Images
The death toll among Iraqis as a result of the US-led invasion has now reached an estimated 655,000, a study in the Lancet medical journal reports today
Posted by: Rich Berger at Jan 11, 2008 3:45:08 PM
I know, Rich. I've been following the arguments over this since 2004. I was just pointing out that the earlier study is in agreement with this NJEM paper for the period up until Sept 2004. And that earlier study was trashed by various people, but it appears to be correct that Iraq Body Count is only capturing a fraction of the total death. The main question now is the size of that fraction. Whether the later study (the 2006 paper) is correct or whether this NJEM paper is right I have no idea. Maybe we'll know someday if Iraq is ever at peace.
Posted by: Donald Johnson at Jan 11, 2008 4:31:32 PM
At this point, I think the defenders of the original study need to pony up and tell us why _they_ think the Lancet number was so far off. No claims that you're just correcting "functionally innumerate" critics.
I think you need to explain what you mean by "so far off". Both studies go to excruciating pains to explain the difficulty in getting accurate numbers, and publish very large confidence intervals -- confidence intervals when overlap when comparing apples:apples or oranges:oranges.
This is a problem primarily with the press, which takes one number and acts as if the study has claimed that this exact number happened, even when the researchers quite explicitly report a probability distribution, rather than a number. The number reported by the press for each study is only a best guess over a very wide range.
Most criticisms of the lancet study that go beyond "my gut doesn't like the number they got" would apply equally well (or equally poorly) to this study. Why is this one gold and the old one rotten? just because this one is more recent? What good reason do you have to trust this new study more than the Lancet study other than it being closer to your original priors?
AFAICT, both studies were reasonable, given their tremendously difficult constraints. We had one reasonable data collection, now we have two. My bayesian response to the first report was to pretty much accept it, since there appeared to be no strong opposing data (the body count methodology clearly represents a lower bound, not an estimate). Now there is some real new data, so I'll adjust my beliefs in the matter to figuring the actual number is probably somewhere within the overlap of the confidence intervals of the two studies. Meaning I think it's very likely the central lancet number is high, but also somewhat likely the new central number is low.
What justification do you have for throwing out the Lancet study completely and yet accepting the new one 100%?
Posted by: Michael Sullivan at Jan 11, 2008 5:23:17 PM
What seems to be disputed about the older study? If we assume that the violent death excess totted up by the newer one is correct, and if we assume that table 3 in the report is correct, then I get:
[(6.01-3.17)/(1.09-0.10)]*150,000~400,000
Since this is in agreement with the lower bound on the confidence interval in the Lancet study (~400,000), how can anyone say these findings differ in any significant way? Is there something wrong with my math?
Posted by: ScentOfViolets at Jan 11, 2008 6:06:57 PM
I'm not "defending" the conclusions of any particular study, Zach, just pointing out that whatever conclusion we arrive at on this issue--by reasoned consideration of the best scientific evidence as it becomes available--we will owe no apologies to David Kane or Megan McArdle since they contributed nothing positive to this process.
Posted by: engels at Jan 11, 2008 7:00:32 PM
There are more than two studies involved in this discussion. There are four statistical studies (first and second Lancet, UN study, NEJM study), and at least two actual enumerations of some subclass of fatalities -- Iraq Body Count and the official government estimate. Actual enumeration is by far the gold standard if the data is complete. If the original study were well designed, it should have been comparable to the actual enumerations, which are exact, or nearly so, for the subclass of fatalities measured by the more precise technique. The original Lancet study could not possibly be correct, because 90 odd percent of its fatalities had associated death certificates, and nowhere near 500,000 death certificates were issued, by actual count. That's a reality check, and it's one that the Lancet study failed miserably.
It seems like the right conclusion at this point is that the number of excess deaths is somewhere between 300K and 600K, modulo error bars.
Completely wrong. (Dare I say, innumerate?) The original study was terrible. It doesn't belong in the discussion at all.
A useful lesson for reading statistical studies: statistical error bars only have meaning in the context of a function with a known theoretical distribution. Failed statistical studies are close to worthless. You're far better off taking the very best study and its associated error bars than trying to do a mental meta-study with ill-defined protocols in a poorly conceived attempt to include all the data.
Posted by: Zach at Jan 11, 2008 10:24:24 PM
Zach, lots of errors there (also, your remarks on "mental meta studies" are both patronising and wrong).
[If the original study were well designed, it should have been comparable to the actual enumerations, which are exact, or nearly so, for the subclass of fatalities measured by the more precise technique]
But these subclasses are "violent deaths of civilians reported in at least two English language news sources", which no possible manipulation of the Lancet data could correspond to, and "deaths certified at a hospital which reported its statistics to the Health Ministry", ditto. You can see why this doesn't work, can't you?
[The original Lancet study could not possibly be correct, because 90 odd percent of its fatalities had associated death certificates]
no, this would be the second Lancet study, not the first.
[and nowhere near 500,000 death certificates were issued, by actual count]
no, there is no such actual count. Nobody, certainly not the Iraqi Health Ministry, claims that the Iraqi government have accurate statistics for the number of death certificates issued. This is a common mistake that's been made since 2006 in discussion of the Lancet studies. Even in the USA and UK, mortality statistics are only published with a significant lag, because collating and tabulating locally issued death certificates is a fiddly, time-consuming and low-priority task. In Iraq (particularly given that the survey period contains at least 12 months during which there was no government to send them to), it's obvious that the statistical infrastructure wasn't providing this output.
Think about this, Zach - this is an Iraqi government survey that the NEJM has just published. If they had accurate mortality statistics, why would they be doing these surveys?
Posted by: dsquared at Jan 12, 2008 9:07:10 AM
With a little googling, I was able to find an article referencing an official Iraqi accounting of death certificates, complete with explanations of where and how they are incomplete. This put the figure, before corrections for these errors, near 50,000 at the beginning of 2006.
http://www.commondreams.org/headlines06/0625-03.htm
My remarks on mental meta studies were a little patronising, and I apologize for that, but it is a good technique to use in the face of unreliable studies or particularly difficult measurements. Studies can err due to statistical error -- the designed-in error that comes from not using an infinite study size -- and also from implementation error -- the error that comes from using the wrong model distribution, not implementing your protocols perfectly, systematically missing something, etc. These are more study-specific. My problem with doing mental meta studies is that if you don't understand the different studies well enough, you end up carrying the sins of the father unto the nth generation -- a single rotten study keeps throwing you off. In a regime where the statistical error dominates, you can lump different studies together. In a regime where implementation error dominates, you should find the very best study or studies and treat them as the gold standard.
Posted by: Zach at Jan 12, 2008 1:19:27 PM
Zach, what you wrote was gibberish:
A useful lesson for reading statistical studies: statistical error bars only have meaning in the context of a function with a known theoretical distribution.
This is most definitely not true, and this is explained in very basic statistics courses. Haven't you ever heard of the Central Limit theorem? If not, I don't think you're qualified to make these kinds of statements.
Posted by: ScentOfViolets at Jan 12, 2008 4:33:29 PM
Violets, my wording was unclear. What I meant to say was what I said in my reply to Dsquared. That is, quoted statistical error bars aren't the same thing as real-life error bars. That got mixed up with a second point, that statistical techniques work far better when the object of study has a known theoretical distribution, and the study can verify that they're seeing the correct distribution -- ie, that they know their system well enough that they can correct for implementation errors. If you talk to experimenters, you'll find that it can be quite difficult to get things working so well that statistical error in a single run is actually the dominant error term.
It's important to avoid what you might call the statistics 101 fallacy -- the belief that things are working perfectly, even when the underlying measurement is difficult and might fail in poorly understood ways. That's why my suggested changes to the Lancet study revolve around introducing internal and external cross-checks.
Posted by: Zach at Jan 12, 2008 6:37:02 PM
I'm not following you. What's the difference between 'real life' error bars and 'statistical' error bars? And are you talking about error bars as a synomymn for a confidence interval?
Just to be sure we're on the same page here, we're talking about how closely the estimator, x-bar, approaches the the parameter, mu, right? That is, how good an approximation x-bar is of mu?
Posted by: ScentOfViolets at Jan 12, 2008 8:01:06 PM
Statistical error bars measure the difference between a particular calculated average and the true average. Real life error bars measure the difference between a particular calculated average and the platonic ideal quantity that you were trying to measure in the first place.
Consider a common experimental scenario: on a particular day when everything is working, you can rerun the experiment a million times and measure a particular quantity as having value x with a tiny statistical error bar. But when you come in the next day, random jitter, thermal expansions and contractions, slight changes in where the optics focus, etc, mean that you can repeat the experiment another million times and measure the same quantity as having value y. The error bar is just as small, but x and y may differ by many times the statistical error. That happens because the error due to not controlling everything perfectly dominates the statistical error seen in one run. The run-to-run error bars are what I'm calling the real-life error bars; they measure the error associated with repeating the measurement under putatively the same conditions.
Posted by: Zach at Jan 12, 2008 8:32:39 PM
I see what you're saying, but you have said it in a nonstandard way. Speaking as someone who has a slight bit of familiarity with subject, how do you come to know what you know about statistics? This certainly isn't the way the material is customarily taught, at least in my experience.
Posted by: ScentOfViolets at Jan 12, 2008 11:52:54 PM
Zach, I can't see any basis for deciding that the Lancet study was the "dreadful" one and the NEJM one is the "good" one on methodological grounds alone. It is not as if the NEJM study is flawless after all - not visiting 11% of the clusters and stitching the data together from ad hoc assumptions based on the IBC dataset is at least as much of a problem as any of the abstruse issues people raised with respect to Lancet 2 and much more than Lancet 1. I don't think we should reject any of the studies, but if we're going to chuck one out, the only basis for doing so appears to be an a priori decision that the Lancet 2 number was "too high", and that's not really statistics.
Posted by: dsquared at Jan 13, 2008 3:44:46 PM
According to the British government, the Lancet study's methodology was sound (http://www.iht.com/articles/ap/2007/03/26/europe/EU-GEN-Britain-Iraq-Death-Toll.php):
March 26, 2007
LONDON: British government officials backed the methodology used by scientists who concluded that more than 600,000 Iraqis have been killed since the U.S.-led invasion in 2003, the British Broadcasting Corp. reported Monday.
The government publicly rejected the findings, published in The Lancet medical journal in October. But the BBC said documents obtained under freedom of information legislation showed advisers concluded that the much-criticized study had used sound methods.
The Lancet study, conducted by researchers from Johns Hopkins University in Baltimore and the Al Mustansiriya University in Baghdad, estimated that 655,000 more Iraqis had died since March 2003 than one would expect without the war. The study estimated that 601,027 of those deaths were from violence.
The researchers, reflecting the inherent uncertainties in such extrapolations, said they were 95 percent certain that the real number lay somewhere between 392,979 and 942,636 deaths.
The conclusion, based on interviews of households and not a body count, was disputed by some experts, and rejected by the U.S. and British governments.
U.S. President George W. Bush said he did not consider it "a credible report," and Prime Minister Tony Blair's official spokesman said the study had extrapolated from an unrepresentative sample of the population.
However, the chief scientific adviser to the Defense Ministry, Roy Anderson, described the methods used in the study as "robust" and "close to best practice."
A memo from Anderson's office to senior officials, obtained by the BBC World Service, said the chief scientist "recommends caution in publicly criticizing the study."
In another document, a government official — whose name has been blanked out — said "the survey methodology used here cannot be rubbished, it is a tried and tested way of measuring mortality in conflict zones."
Posted by: Charlie at Jan 14, 2008 1:27:29 PM
Free weight loss resources is to help you make healthy food and lifestyle choices for you and your family. You'll find topics that will help improve your overall health and sense of well-being related to weight loss, relationships, diet, exercises, nutrition, vitamins, prevention, and more. The first goal of dieting is to stop further weight gain. The next goal is to establish realistic weight loss goals. While the ideal weight is a BMI of 20-25, this is difficult to achieve for many people. Thus success is higher when a goal is set to lose 10% to 15% of baseline weight as opposed to 20% to 30% or higher. It is also important to remember that any weight reduction in an obese person would result in health benefits.
Hoodia is currently the most publicized and talked about natural weight loss product in America . It has been featured on CBS 60 Minutes, ABC, NBC Today, BBC, CNN and in other major electronic and print media.
Posted by: herbal at May 11, 2008 10:35:03 AM
