Fuzzy Casualty Math

Here's Ilan Goldenberg's chart of the Pentagon's changing story about civilian casualties in Iraq:

Hat tip to Brian Katulis at CAP who clued me on to this issue and Spencer Ackerman has already got a great post on this.

Basically there are more serious questions about the violence numbers that are being reported out of Iraq. The Pentagon is congressionally mandated to produce a quarterly progress report to Congress measuring stability in Iraq. Each of these reports has a graphic measuring sectarian violence. The last four reports were August, 29 2006 (pg 35), November 30, 2006 (pg 24), March 2, 2007 (pg 17) and June 7, 2007 (pg 17).

I graphed the levels of sectarian violence from these various reports and found some confusing trends. The abnormalities have been labeled A, B and C. (There is no difference between the November report and the March report and thus they overlap).

It'd be nice to not need to hyper-scrutinize every random bit of official government data this way, but the idea that the Bush administration has no credibility on Iraq isn't just a cliché -- based on his record, one has no choice but to inquire and to be very suspicious.

Comments

Goldenberg hands off to Yglesias. Tackle by blog readers. Gain of 2. Third down.

Posted by Fred | August 31, 2007 10:08 AM

There we have the congenital liar and hate monger, Fred, thinking how to lie and peddle more hate. Fred, do your wretched best.

Posted by Jennifer | August 31, 2007 10:16 AM

I agree with your conclusion, and Goldenberg's:

Clearly certain types of violence have been taken out and others have been added. What we need is some transparency.

Official information is oftentimes self-serving, so scrutiny and skepticism are, for citizens, synonymous with sophistication, regardless of which party is in power.

That said, I bet the proximate cause for the shift in number reporting is more opportunistic than diabolical. Think about it like this:

In any given situation -- in any control group -- there are going to be "casualties." For instance, in 2004 there were 16,137 murders in the US, which comes to about 1345 a month. Of course, the US has ten times the population of Iraq, so bear that in mind.

Now let's say you were occupying the United States, and you wanted an accurate way of measuring progress in stability operations. Clumping the "ordinary" murders in with the "extraordinary" would give you an inflated number, and, depending on the ratio of ordinary to extraordinary, a distorted picture (this is a basic "signal to noise" problem). Therefore, after some thought, an intelligent person should find some statistically sound way to "cancel out" the noise leaving only the relevant information in the report.

The reason I agree with the conclusion that transparency would be nice is because the change in the Pentagon's reporting tactic has all the signs of a legitimate information filter. What I want to know is how they computed their baseline number. Which murders classify as "relevant to measuring security progress," and which classify as noise? That's where we need to look.

Nevertheless, the fact that the reporting strategy changed is not ipso facto proof of subterfuge.

Posted by JA | August 31, 2007 10:39 AM

MSNBC reports that some of the most supportive republican senators were almost shot out of the air while leaving baghdad after "inspecting all of the security gains" on the ground!http://www.msnbc.msn.com/id/13538730/

I cant handle all of the lies and cheery propaganda by the administration and it supporters anymore!!!

Posted by centrist | August 31, 2007 10:51 AM

Introducing better, more refined sampling filters that separate criminal violence from insurgent deaths is not a bad thing.

And the reports that the Surge is working rely not just on that stat but reports from NGOs like Iraq Body Count, tribal leaders, sectarian leaders, reporters, US commanders - that violence is headed sharply downwards as AQ in Iraq is being routed and slaughtered, the Sunnis have begun laying off the Shiites, and in turn the Shiites are also moving away from their extremist elements and Iranian agent's efforts to kill Sunnis.

What we should have been doing years ago.

The postwar was massively botched, no doubt, but the danger now is with Leftists that are emotionally invested in wanting a major American defeat and humiliation because "it would really hurt Bush!".

Posted by Chris Ford | August 31, 2007 10:57 AM

I tend to think that 1) reporting strategy may very well have underwent a legitimate change 2) estimates may be revised with new data. These strike me as the simplest and most obvious explanations. I don't think we have any reason here to believe that there is something untoward in these estimates changing.

I agree some transparency and addressing the changed estimates would be nice; if my consulting firm issues an updated report and anything changes we're expected to explain why. The government, of course, faces a drastically different set of incentives than my consulting firm. The magnitude of these changes are high enough that some sort of explanation is warranted. I think it's just typical incompetance rather than a deliberate attempt to confound war opponents.

Posted by Wilson | August 31, 2007 11:02 AM

Terrible graph! Even the Economist's charts are easier to decipher.

Posted by Rula Lenska | August 31, 2007 11:29 AM

The strangest number in the figure is the date November-07 -- ooops.

"Nevertheless, the fact that the reporting strategy changed is not ipso facto proof of subterfuge." is true, however the fact that the reporting strategy changed and the DOD refuses to release raw data and the formulae it used is, in itself, an inexcusable violation of basic principles of empirical data analysis.

Personally I feel not doubt that this refusal is due to the fact that if they were transparent their dishonestly would be revealed. As a matter of principle, one should always assume so, even though there have been cases in which opaque data processing is honest stupidity not fraud. The DOD's persistent refusal to explain the processing makes it hard for me to understand how anyone can seriously entertain they hypothesis that the DOD is acting in good faith http://www.mcclatchydc.com/227/story/18927.html
"U.S. officials say the number of civilian casualties in the Iraqi capital is down 50 percent. But U.S. officials declined to provide specific numbers, and statistics gathered by McClatchy Newspapers don't support the claim."

Processed numbers should never be accepted as a substitute for the rawest data available. No reputable academic journal would publish the DOD's report as a contribution to pure science. Still less can it be accepted as an input into life and death decision making.

Posted by Robert Waldmann | August 31, 2007 11:30 AM

If we have to explain how Iraqis died then the terrorists will have won.

Posted by Jeffrey Davis | August 31, 2007 12:12 PM

Processed numbers should never be accepted as a substitute for the rawest data available. No reputable academic journal would publish the DOD's report as a contribution to pure science.

Yes. That is absolutely true and extends to virtually all Professional Journals in addition to academic Journals, and to virtually all disciplines including Chemistry, Physics, Medicine, Psychology, etc. Moreover, the methodology used to obtain the raw data is also a normal requirement. These requirements are simply a normal part of universally accepted standards in the objective search for truth and advancement of knowledge.

Great observation.

Posted by JustMe | August 31, 2007 12:24 PM

..."The postwar was massively botched, no doubt, but the danger now is with Leftists that are emotionally invested in wanting a major American defeat and humiliation because "it would really hurt Bush!"."...

Really? That's the danger? Wow, and here I was thinking that a military success doesn't guarantee a political success. Or that the invasion of Iraq seriously destabilized the Middle East, which then impacts Afghanistan, then Pakistan and then onto India. In the west, of course, Israel, Syria, Lebanon, Turkey (who knows how they'd react if the Kurds declare independence, apparently you do). Then of course there's the Sunni/Shia, tribalism, growing dispossessed Muslim populations in Europe, the radicalization of Muslims in Asia.

No, apparently, none of that matters. What matters is that the body count in Iraq ticks down from absolutely heinous, to just obscene. But I can't see our "future glorious victory" the because of my irrational hatred of "W". I'm not capable of learning the lessons from the turn of the century British experience with the "country" of Iraq, “’cause Bush really gets under my skin”. When I assume that the invasion and occupation of Iraq and all that follows was perhaps the worst strategic military decision (in both terms of sheer stupidity as well as loss of life, treasure, morality, sanity, etc.) in world history, I’m not being rational because I think that Bush “is a mean old pissy-pants”.

Let me tell you, I don’t give a shit about George anymore. The real danger, my friend, is that the morons who convinced turd-blossom’s buddy that this was a good idea are still with us. Still trying to salvage their reputations (and their consciences, if they have them) by pretending that up is down, black is white, failure is success, and if the “defeatocrats” would just stop pointing out the insanity of this horrible decision,…, well, then everything would turn out just fine, and we’d all be sucking on lolly-pops and farting rainbows. And, George, the man with the “real vision thing”, would go down in history as “The Decider” who saved us all from our petty, irrational emotions. Let me know when that happens, won't you, so I'll know when to cheer...

Posted by Rihilism | August 31, 2007 1:55 PM

At one point in my career, I spent several days at Harvard Law School attending a seminar on negotiation. The central point of the seminar was the recommendation for a well thought out and well researched fall back position, referred to as a "BATNA" (Best Alternative To a Negotiated Agreement).

Why me I don't know, but during one seminar lecture a business executive from New Jersey turned to me and said loudly,

"This guy is probably one of the smartest bastards in the world, but when the Union Rep. walks into the room, takes his coat off, and puts his handgun on the table, try telling him about your fucking batna."

"What business are you in?", I asked quietly, attempting to influence him to lower his voice.

"Preformed concrete products," he answered loudly.

**********

In my view both the professor and business executive were right. There are many truths in the world.

Posted by JustMe | August 31, 2007 1:57 PM

oops, wrong thread ... oh well, time for coffee or something..

Posted by JustMe | August 31, 2007 2:08 PM

Anybody who treats ANYTHING coming out of the Pentagon as something remotely resembling the truth is a complete idiot.

When a military person opens his mouth in an official capacity, he is undoubtedly lying.

US casualties are up over last year, civilian casualties are up over last year, infrastructure is down over last year - there is NO "good news" in Iraq.

The US invasion of Iraq is responsible for an estimated one million dead Iraqi civilians, and well over a million displaced persons and there are probably a million or two more wounded. That's ten percent of the Iraqi population devastated by the actions of the United States. And that doesn't even count the devastation of the infrastructure or the economy and the general health problems resulting from that.

Period. End of story.

Oh, wait, not the end - we still have Iran coming up which ought to be good for doubling, tripling or more all of the above.

By the end, George Bush will be closing in on Stalin, Mao and Hitler as the biggest mass murderer in history. He's already WAY past Saddam.

And thirty percent of the US population - like this character Ford here - still support him and the ongoing mass murders in Iraq by US bombings and trigger-happy military goons.

Tell me again how we didn't deserve 9/11. The only problem with 9/11 is that the wrong 3,000 people got killed.

Posted by Richard Steven Hack | August 31, 2007 8:09 PM

Hack,
Yeah but hey..We got Saddam..you know the guy behind 911..er a had wmd..or wait a minute...he killed Iraqis by the thousand as well..

What in the hell are we doing!!!!

Posted by centrist | August 31, 2007 9:45 PM