The Myth of Polling

The obvious problem with the polls you see all the time about how the public feels about such and such an issue is that these surveys don't tell you whether the people actually care about the issue or not. Taegan Goddard, meanwhile, glosses The Opinion Makers forthcoming from David W. Moore:

The author — a former senior editor of the Gallup Poll — says that today's opinion polls misfire due to an intrinsic methodological problem: survey results don't differentiate between "those who express deeply held views and those who have hardly, if at all, thought about an issue."

Kevin Drum is puzzled:

This is disturbing. Either Moore managed to find a publisher for a book thesis about as obvious as "college students like to drink," or else Moore's thesis actually isn't as bog obvious as I think it is. I'm not sure which is worse.

Or there's a third option: his thesis really is as obvious as I think it is, but everyone keeps pretending not to know it anyway. Which means it's worth a book. Good luck, David!

I think that option number three is correct. Nobody who thinks about this stuff a lot could possibly fail to have thought of Moore's point, but at the same time politicians and their aides very frequently do act as if they don't understand this. I think the reason is that referring to polling data, even bad data, is a good CYA mechanism when you need to make difficult decisions. A consultant who says "we don't have any valid data on this question, but I think you should do X" is going to get blamed if X doesn't turn out right. But if he can point to some data, and say that he's not making the recommendation, he's just pointing to the numbers then if things go south it isn't really his fault.

This is a pretty common organizational flaw. The natural tendency is to try to maximize whatever it is that you have a good measurement of, even if the measured quantity is only questionably related to what you're trying to do. Politicians know how to get an issue poll in the field, and there aren't great metrics for getting the information you would really want. So campaigns often go to war with the data they have, even while knowing that the data's no good.

Comments

Hasn't this already become conventional wisdom, particularly on gun control?

Posted by WillieStyle | July 31, 2008 9:28 AM

This is silly. While I'm sure there are many polls that don't guage depth of feeling, others do. Some do it directly with multiple choice answers to questions like "How much do you care about this issue? 1-not at all ... 5-Very much". Some do it obliquely with questions like "Did you vote in last election?" etc.

When people actually want to know the information about which they are polling, they make the effort to do it right. The problem is that most polls are not taken in order to obtain information. They are done either for promotional purposes or for entertainment. The latter doesn't require accuracy, and the former usually requires innaccuracy.

Where's the advance from my publisher? I can stretch that out to 300 pages, I'm sure.

Posted by Njorl | July 31, 2008 9:32 AM

Just one caveat: I think politicians and their aides often may act like they don't understand the limits of polling in part because they are putting on such an act for the media and the public--in other words, they are spinning the polls because the media reports on polls and those reports may influence public opinion. But I strongly suspect at least some politicians and their aides are well-aware of the limits of polling.

Posted by DTM | July 31, 2008 9:35 AM

Social scientists don't claim that responses to polls and surveys indicate 'deeply' held views.

They do argue that a person's responses often predict and/or are associated with behavior -- and that is largely an empirical claim, not a psychological one.

Posted by El Cid | July 31, 2008 9:37 AM

Here's the thing: in political campaigns, issues function primarily as indicators of character and values. John McCain is running on the Surge not because he thinks people will agree with him on the issue, but because it shows that he's a big tough guy who can stick to his guns and yadda yadda yadda.

For this purpose, it doesn't matter so much whether an opinion about an issue is deeply held, because you can build a narrative around what your position on th issue says about you regardless.

Posted by joe from Lowell | July 31, 2008 9:50 AM

" But if he can point to some data, and say that he's not making the recommendation, he's just pointing to the numbers then if things go south it isn't really his fault."

Paging Mark Penn.

Posted by John | July 31, 2008 9:54 AM

Isn't responding to a poll at all evidence of a certain level of involvement?

When my phone rings, I make a choice to get involved or to just hang up. It's a small thing, but I think I'm doing my part when I take the two minutes to push a few buttons and listen to a few questions.

Posted by Trevor J | July 31, 2008 9:57 AM

Yes, I recognize that this is sad and not doing nearly enough, but it's more than a lot of people do.

Maybe I'm wrong though and even the apathetic are flattered for the opportunity to voice their opinion.

Posted by Trevor J | July 31, 2008 10:01 AM

While I'm sure there are many polls that don't guage depth of feeling, others do. Some do it directly with multiple choice answers to questions like "How much do you care about this issue? 1-not at all ... 5-Very much".

I think a better methodology would be to present people with a choice of two hypothetical candidates, each of which holds a different position on several issues, like gun control, abortion, etc. Then, ask them who they would vote for. By mixing up the set of candidate's positions, you could get a good idea of the relative importance of the issues in driving voting behavior, divorced from personal details about the candidates.

Posted by Jim W | July 31, 2008 10:02 AM

Isn't responding to a poll at all evidence of a certain level of involvement?

Trevor, you tell me!

Posted by John | July 31, 2008 10:02 AM

Oh my god, that's depressing, John. I guess what I mean is the level of how deeply someone cares about the issue, not necessarily how much they know.

Plus, turn on a camera and suddenly everyone's excited to share. A clipboard is far less enticing.

Posted by Trevor J | July 31, 2008 10:15 AM

Joe from Lowell pretty well nails it. As a whole, the electorate neither understands nor cares about political issues as such. They respond to personalities, to narratives, and most importantly, to their own personal circumstances. Polling an issue can't tell you what people think, because, as a gestalt, they often don't think anything.

Such polling could be useful in formulating a way to market certain approaches, but if you try to use it to determine a direction for leadership it will take you nowhere.

Isn't responding to a poll at all evidence of a certain level of involvement?

When my phone rings, I make a choice to get involved or to just hang up. It's a small thing, but I think I'm doing my part when I take the two minutes to push a few buttons and listen to a few questions.

All this means is that you haven't burned out on responding to surveys yet. I feel very strongly about this upcoming election, but I don't think I'll answer any pollsters' questions unless they are willing to pay me for my time.

Posted by jimBOB | July 31, 2008 10:19 AM

Interesting thesis. The problem is it's not really true. Pollsters commonly do several things that distinguish those with or without "deeply held views" and those who have or have have not "thought about an issue."

1) They ask questions about which issues voters are concerned about. Once they have this, they can crosstab those who are/are not concerned (have deeply held views) with Issue X and what their views are on Issue X.

2) They ask follow up questions that get at intensity. For instance, after asking people's party ID or vote preference they will ask whether the respondent holds that view "strongly" or "not so strongly". This obviously gets right at "deeply" or "strongly" held views.

3) They will ask people to assess their own level of knowledge on Issue X. Again, that can then be crosstabulated with their position on Issue X so you know that those who say they know a lot about Issue X feel one way and those who say they know nothing at all about Issue X feel another way. Granted, its far from a perfect measure but it does allow you to stratify the results and see how things change as knowledge increases or decreases.

Posted by Junius Brutus | July 31, 2008 10:58 AM

Jim W said, 'I think a better methodology would be to present people with a choice of two hypothetical candidates, each of which holds a different position on several issues"

Jim: that methodology is used routinely in political polls.

Posted by Junius Brutus | July 31, 2008 11:06 AM

As I've noted on this site before: its simply misguided to make a blanket attack on pollsters on methodological grounds. They're really not as dumb as Matthew thinks.

The real problem with pollsters is that they have a professional interest in avoiding accountability. This means they have a strong conservative bias. Not conservative in the ideological sense, but conservative in that innovation will be eschewed in favor of the tried and true. That's because a pollster won't be held accountable for a losing campaign that used a conventional wisdom strategy while they will be held accountable for a losing campaign that used an unconventional strategy that they recommended.

The problem is much less a matter mishandling the data the poll produces, and much more a question of what goes into the poll the in the first place. Many powerful lines of argument will be left by the wayside, going untested, and effectively rejected without getting a fair test. Its an agenda setting problem.

Posted by Junius Brutus | July 31, 2008 11:20 AM

An excellent, timely, and insightful post about a very important subject.

Now I'll go back and read the comments.

Posted by robert powell | August 1, 2008 8:23 AM

Excellent comments.

All polls are obviously not alike. Some are sloppy, some are highly scientific and disciplined, and everything in between.

In my view the essential question is whether the ultimate goal is selling a product, or finding the truth. Polls can help and hurt both enterprises.

Posted by robert powell | August 1, 2008 8:37 AM