Bad Incentives

Josh Marshall on Zogby polls:

If there wasn't a record, you might think that Obama was heading for a major upset victory in PA next week. Zogby has Obama just one point behind Clinton at 45%-44%.

But remember, Zogby was out in front this year predicting Obama's big wins in California and Ohio too. So it's hard for me to put too much weight in this sounding.

This illustrates a real problem with the public polling game, namely the lack of incentives to get things right. Presumably there's some level of consistent wrongness at which people stop giving you the links, readership, buzz, and whatever else it is you're looking for but it's really not clear where that is. And, indeed, for your average media poll where the objective is to produce an "interesting" article accompanying the poll, you're probably better off being wrong.

Suppose I somehow screwed up my polling and got the result that 50 percent of African-Americans say they'll vote for John McCain in November if Hillary Clinton wins the nomination. Does that sound plausible? No. Would it be a big story if I had a poll that said it was true? Yes. And if I'm in the business of producing big stories, then that means I run with the poll and come away very happy with a day's job well done.

Comments

I don't pay attention to the issue, but me sense is that Zogby's record is especially bad, going back several election cycles. Does anyone have a handle on his?

Posted by Will Allen | April 17, 2008 2:04 PM

Pollsters should be required to bet ten percent of their assets on the elections every cycle. And their bets should be tied to their polling. That would clear out the bad pollsters pretty quickly.

Posted by fostert | April 17, 2008 2:04 PM

Man, I type worse than Yglesias!

Posted by Will Allen | April 17, 2008 2:06 PM

Presumably there's some level of consistent wrongness at which people stop giving you the links, readership, buzz, and whatever else it is you're looking for but it's really not clear where that is. And, indeed, for your average media poll where the objective is to produce an "interesting" article accompanying the poll, you're probably better off being wrong.

Isn't this precisely the problem with political punditry as well?

Posted by Christmas | April 17, 2008 2:07 PM

I agree. Sadly nearly all main stream media currently revolves around the same idea (I.e. Last nights debate, and pretty much the past month+ of news coverage)

Posted by kevinq | April 17, 2008 2:10 PM

I don't pay attention to the issue, but me sense is that Zogby's record is especially bad, going back several election cycles. Does anyone have a handle on his?

I haven't followed this too closely, but my recollection is that Zogby uses some fairly unorthodox weighting procedures and sampling methods, based on his gut instincts. When his gut instincts are correct he yields accurate results. When they aren't, well...

Posted by Matt Stevens | April 17, 2008 2:13 PM

My gripes with polling:

1. The General election is months away. The political junkies and pundits are well versed in the policies and platforms (generally) of the candidates so we're ready to make an informed decision today on who we're going to vote for. Joe 6-pack, and all those people who watch reality TV, have no clue who they'll vote for.

2. It categorizes people way too easily. Even in Red states such as Texas, plenty of plenty of people voted for Kerry. It presumes that people are supposed to vote based on which category boxes they fit in. If I'm a white guy, who lays drywall, making $45,000, then I'm supposed to be pro-war, skeptical of immigrants, and vote as a reagan democrat. People aren't so static or aught be so easily categorized.

MY, you're a Jew of Cuban/Hispanic ancestry. Is the Cuban in you supposed to be skeptical of Obama b/c of the brown/black divide? Is the Jew in you going to be skeptical that Obama will be insufficiently supportive of Israel. Is the liberal NYer in you supposed to go with the latte sipping choice.

2000 = Soccer Mom
2004 = Nascar Dad
2008 = Subprime deadbeat? TMZ geek? internet porn surfing dad? I drive a dodge stratus?

Posted by nattyb | April 17, 2008 2:16 PM

Actually, I get the sense that pollsters are being called on their past performance in this election cycle. IIRC, every reference I have seen to this new Zogby poll has carried the caveat, "Remember, he did screw up California" or words to that effect.

Posted by alkali | April 17, 2008 2:18 PM

I think you're being a little unfair. Just this morning, I've heard a couple of journalists (one on TV and one on a WaPo webchat) talk about all the bad polls coming out of PA right now. My polls of choice for this election arethe Gallup daily tracking poll and the summaries at Real Clear Politics, which includes Gallup as one of the Polls as one of its elements.

Posted by THS | April 17, 2008 2:21 PM

But isn't the public polling just a loss-making advertisement for their private polling business? In which case, accuracy really does count.

Posted by Boring Commenter | April 17, 2008 2:22 PM

i think this is the unraveling of captitalism.

you don't make money by being the best pollster, but the one who gets the most attention (for outlier polls) or the one who can make everybody *think* they are the best.

the idea that the market picks the best is not the case anymore. the market picks what people think is the best and thanks to advertising and gaming mainstream media coverage, what people think is the best almost never is.

Posted by david in norcal | April 17, 2008 2:29 PM

But isn't the public polling just a loss-making advertisement for their private polling business? In which case, accuracy really does count.

Exactly. In the short run, it might be nice to make it in the news with a spurious finding, but it's really harmful over the long run.

Posted by Phil | April 17, 2008 2:32 PM

Mark Penn, however, proves that there is a market for spectacularly inaccurate private polling fitting a convenient narrative.

Zogby's probably just doubling down on his previous failures and hoping that if PA surprises everyone, then it'll negate his CA/OH/etc. predictions.

Posted by phil | April 17, 2008 2:47 PM

Zogby is junk, and has been for years now. Why do creditable people like Josh Marshall continue to publish and discuss their polling results? Why give them any credit at all? Just ignore Zogby, everybody, please. Their polling is meaningless. Most polling is meaningless, but theirs especially so.

Posted by onceler | April 17, 2008 3:22 PM

ARG seems to have reached a point of badness that nobody talks about it.

Posted by John | April 17, 2008 3:28 PM

"I don't pay attention to the issue, but me sense is that Zogby's record is especially bad, going back several election cycles. Does anyone have a handle on his?

Posted by Will Allen | April 17, 2008 2:04 PM"

Put it this way: I've gotten to the point that in a close election, I don't want Zogby to be saying my choice will win. I haven't done statistical analysis on his history, but he does seem to be a bit of a dim bulb with numbers. Add in the fact that it's fun to say Zogby (it sounds like a mythical villian from Babylon in a Ghostbusters movie) and everyone remembers that when he fucks up, he fucks up embarrassingly and rather publicly.

Posted by Reality Man | April 17, 2008 3:29 PM

Well, in defense of Josh (and I don't usually defend him, but when he's right, he's right), he also stated below the quote Matt used that the last five polls released have shown decreasing Clinton leads in PA.

So he criticized Zogby for being out of step and unreliable, but he also noted that the polls are starting to show the usual Obama pattern of closing the gap the closer he gets to the primary.

And last night's Clinton show could conceivably erode Clinton's lead badly in the last days before the PA primary. Or not.

Posted by Richard Steven Hack | April 17, 2008 5:47 PM

Let's not forget that Zogby is a Superdelegate, and he is for Obama. That might color the questions or the results. Or not.

Posted by Jammu | April 17, 2008 6:07 PM

Gee...I never would've thought that a polling outfit that partnered up with NewsMax would, y'know, SUCK.

C'mon. Seriously. NewsMax???

Posted by Michael | April 17, 2008 7:39 PM