Mortality Studies Again

Kevin Drum, apparently a more careful reader than I, gets an apples-to-apples comparison of today's new study with the earlier Lancet study:

The difference in their estimate of total excess deaths (655,000 vs. 393,000) isn't huge for a study with such inherent difficulties, but the difference in the violent death rate is. The Lancet study calculates that 92% of all post-invasion excess deaths were from violent causes, while WHO figures it at 38%.

Be all that as it may, The New York Times observes that "because of its timing, the study missed the period of what is believed to be the worst sectarian killings, during the latter half of 2006 and the first eight months of 2007." The Lancet study, clearly, having been from even earlier also missed that period. So even if you have a great deal of confidence that one or the other of these studies got things right, the studies' figures are probably badly outdated by now.

One issue this whole controversy raises relates to casualty figures from civil wars that aren't political hot potatoes in the United States. When a study comes out in Iraq, there's intense political pressure from one side or another (or both) to expose real and imagined methodological flaws. But what about things like the millions who've died recently in Congo where the studies get done in a context where nobody's seriously trying to work the refs?

Comments

Kevin actually makes a minor arithmetical error.

Basically, the WHO point estimate of post-invasion violent deaths is 1.09 per thousand person-years (Table 3 of the NEJM paper, which is currently available on the NEJM website). However, the pre-invasion estimated rate is .10 per thousand person-years, so the estimated increased rate for violent deaths is .99 (not 1.09) per thousand person-years.

As Kevin notes, the estimated increase in the rate of total deaths is 2.84 per thousand person-years.

So the estimated increase in total deaths is 151,000*2.84/0.99 or about 433,000.

Posted by Michael I | January 10, 2008 5:51 PM

Historically it's generally been true that most civilian wartime deaths are due to the disruptions of war not to violence. I think this was even true in WWII, maybe Vietnam as well. To be sure, Iraq has seen a level of prolonged anarchy that is unusual for wars (where conquerors usually crack down hard, but at least in doing so restore basic order and security) but it would surprise me if the deaths from violence in Iraq dwarfed the deaths from stavation and disease.

Posted by Jonf | January 10, 2008 5:53 PM

Jonf-

Actually, the Iraq War follows the pattern set by most wars in human history, save only the last couple hundred years(we're talking van Creveld and 1st, 2nd, 3rd, 4th generation warfare here), a pattern already ressurrected in Africa. Which is why the Congo deserves more attention.

Posted by Greg | January 10, 2008 6:02 PM

Ach, I reread your comments, Jonf, and I should rephrase that I mean this period of anarchy lasting so long is a very common situation in war until very recently.

Obviously, disease has killed, particularly in warfare, more people than anything short of nuclear weaponry ever could.

Posted by Greg | January 10, 2008 6:12 PM

The study is confusing. Table 3 does show an increase in deaths per 1000 person-years from violence of 0.99 and from all causes of 2.84. So the death rate from nonviolent causes increased by 1.85. But the authors write "Overall mortality from nonviolent causes was about 60% as high in the post-invasion period as in the pre-invasion period." Did the rate increase but the number of deaths ("overall mortality") decline due to population changes? They also write that they found lower post-invasion mortality from non-violent causes than the Lancet study (372 deaths per day vs. 416 deaths per day.). And yet the Lancet study attributes onlyIt is also unclear which numbers refer to Iraqis in total and which refer only to civilians.

Posted by Mixner | January 10, 2008 6:18 PM

Not only is cluster-sampling methodology widely accepted -- it's part of a Congressional finding of fact in an amendment regarding the DR Congo (S.2125) sponsored by Sens. Obama & Brownback that was signed into law by President Bush.

(7) A mortality study completed in December 2004 by the International Rescue Committee found that 31,000 people were dying monthly and 3,800,000 people had died in the previous six years because of the conflict in the Democratic Republic of the Congo and resulting disintegration of the social service infrastructure, making this one of the deadliest conflicts since World War II. - Signed by President, Dec 22, 2006

Posted by joejoejoe | January 10, 2008 6:41 PM

The Congo is a classic example of what the War Nerd calls "War Without Battles:" various roving armies refuse to meet each other in sustained combat but instead massacre villages. So the civilians flee into the brush.

That raises a difficult methodological issue: What happens to them next? Do they find refuge somewhere else? Die of starvation? Die of infections that would be curable if they could get to a clinic? Get bitten by poisonous snakes? All of the above?

Posted by Steve Sailer | January 10, 2008 6:58 PM

Mixner, I think the authors meant that the the post-invasion non-violent death rate is about 60% higher than the pre-invasion non-violent death rate.

If you look at Table 3, and do the appropriate subtractions, you get a pre-invasion non-violent death rate point estimate of 3.07 per thousand person-years and a post-invasion non-violent death rate point estimate of 4.92 per thousand person-years. Which is about a 60% increase.

Posted by Michael I | January 10, 2008 7:11 PM

Michael I.

Maybe that's what they meant. But it's not what they wrote. It's hard to know. I see several inconsistencies of this kind, and I've only looked at the paper briefly.

Posted by Mixner | January 10, 2008 7:19 PM

Not being a statistician or a mortality expert, I'd prefer to wait until the Lancet people have analyzed the new report and express their opinion of the methodology and results.

Or someone with equal credentials.

It would be nice for the Iraqis if the lower figures are better, but it doesn't change the basic situation of the war.

But that's exactly what will happen if the lower figures are accepted - the right wing nuts will start crowing about how it was all "worth it" since "only X hundred thousand" died whereas "Saddam killed three hundred thousand" (or whatever figure having no basis except mythology.)

Posted by Richard Steven Hack | January 10, 2008 7:48 PM

"The difference in their estimate of total excess deaths (655,000 vs. 393,000)"

The WHO/NEJM didn't publish any "estimate of total excess deaths". 393,000 is someone else's back of napkin estimate imputed to the WHO/NEJM.

Looking at the discussion in the paper about the violent death estimate of 151,000 the WHO did a lot of serious calculating and adjustments of the data to arrive at this number, accounting for a range of issues which affected the final numbers.
I doubt they'd do a "total excess deaths" estimate from this data with just a few clicks of the calculator.

The WHO 151,000 vs. Lancet's 601,000 (both for violence) is the apples to apples comparison. Manufacturing your own "total excess deaths" estimate for WHO and comparing this to one from Lancet is the apples to oranges comparison. Even if the numbers come closer they aren't measuring the same deaths (apples to oranges).

If one of these is right, the other is wrong.

Posted by martine42 | January 10, 2008 9:53 PM

If one of these is right, the other is wrong.

And of course they could both be wrong. Perhaps the true number is closer to 50,000. When two studies using the same basic methodology produce estimates that differ by a factor of four, they should both be treated with great skepticism.

Posted by Mixner | January 10, 2008 10:11 PM

And maybe the true figure is one million.

Why assume it's lower?

Nobody on the right wing nut side believes that Saddam had any trouble killing three hundred thousand, despite no specific evidence for the figure.

Posted by Richard Steven Hack | January 10, 2008 10:53 PM

Um, the Congo war wasn't exactly a civil war. It was started when Uganda and Rwanda invaded the DRC in support of their attempt to overthrow Kabila's government. Almost certainly with Anglo-American greenlighting, if not direct support (almost everybody has looked the other way on that issue).

Posted by MFB | January 11, 2008 3:10 AM

Les Roberts (co-author of the papers in the Lancet) comments here:

http://scienceblogs.com/deltoid/2008/01/ifhs_study_on_violent_deaths_i.php

and here:

http://scienceblogs.com/deltoid/2008/01/les_roberts_replies_to_wsj_edi.php

Posted by buermann | January 11, 2008 3:28 AM

The WHO 151,000 vs. Lancet's 601,000 (both for violence) is the apples to apples comparison.

No, this isn't right; you're implicitly throwing away the information that a) the NEJM study has a big increase in nonviolent deaths b) the potential for misclassification of deaths as violent or nonviolent depending on who's asking has always been controversial here. It's rather like counting a single tray of apples then declaring "this grocer has no fruit!" while ignoring a massive great pile of oranges.

Posted by dsquared | January 11, 2008 3:53 AM

Roberts somments on Al's suggestion that the data collector was a Baathist:

"Your suggestion that our Iraqi colleague Riyadh Lafta was suspect because he recorded child mortality during his career is particularly ironic. He was one of few professors in the country that never joined the Baath Party."

Apparently Al got this crap from a bullshit WSJ editorial - a typical Al source.

His comments in the first link above are interesting. He also says he'll have more to say - presumably he needs to study the report more.

Posted by Richard Steven Hack | January 11, 2008 4:25 AM

In regards to Matt's question about less politically sensitive wars, the accuracy of death toll estimates is most likely worse.

First, there are fewer researchers in the field, so the total amount of dat is simply smaller.
Second, it is impossible to be a close observer of any of these horrific conflicts without becoming emotionally invested in some proposed solution. So the level of bias by the researchers is unlikely to be any less, but there will be less oversight, and fewer opposing voices to check their methodology, so biased results are less likely to be challenged. (Biased in the sense of skewed too high or low due to methodological flaws, not political chicanery)

The WHO was recently forced to reduce its AIDS incidence estimates for most of Africa by 1/4th because of flawed methodology, but it took 10 years before anyone noticed that researchers had been taking their "representative samples" at prenatal clinics and assuming that this could be extrapolated directly to the population at large.

Despite the fact that Matt doesn't like the criticism directed at the Lancet study, such criticism (even the politically motivated sort) is useful for determining the correct answer. A study that cannot stand up to challenge isn't worth very much.

(Full disclosure: I think the Lancet report was seriously flawed, and that its authors were more concerned with its political impact than with its rigor. I also believe that at least 150,000 Iraqis died as a result of the invasion, and that such numbers should encourage a little humility on the part of anyone peddling a grandiose solution to the countries ills, then or now.)

Posted by heedless | January 11, 2008 6:33 AM

dsquared,

No, this isn't right; you're implicitly throwing away the information that a) the NEJM study has a big increase in nonviolent deaths b) the potential for misclassification of deaths as violent or nonviolent depending on who's asking has always been controversial here. It's rather like counting a single tray of apples then declaring "this grocer has no fruit!" while ignoring a massive great pile of oranges.

Yes, it is right. As martine42 noted, the new study does not provide an estimate of total excess deaths. The only apples-to-apples comparison that can be made between the two studies is their estimates of excess deaths from violence, and those numbers differ by a factor of four.

The bogus numbers for total excess deaths attributed to the new study by the likes of Kevin Drum and Tim Lambert do not come from the study but are the results of crude calculations made by those writers using certain other numbers from the study.

Posted by Mixner | January 11, 2008 11:32 AM