Data! Run! Hide!

I find this tidbit from Michelle Cottle's latest reporting on Mark Penn fascinating:

What's more, being The Man With The Data gives Penn a formidable edge in any debate over strategy. It is almost impossible to argue Penn down, say colleagues, because he brandishes his polling data like a weapon. And so, his fellow advisers explain, in the eternal debate over whether to keep the message focused on Hillary's strength and readiness or to try and humanize her, Penn would simply whip out data showing that "readiness" was the way to go. When anyone argued against going negative on Obama, Penn would point to more numbers.

Pollsters really are the witchdoctors of modern campaigning. Possession of the secrets of The Numbers lends a mystical heft to their strategic arguments. And yet, if pollsters actually had reliable methods at hand for conducting this kind of work, pollsters wouldn't be brand names. You'd have to hire a pollster, of course, but pollsters would be commodity products where any one of several firms would all give you more-or-less the same methods and produce more-or-less the same results. Instead, though, we know that Mark Penn habitually produces different advice from a Stan Greenberg or a Celinda Lake.

Comments

You'd have to hire a pollster, of course, but pollsters would be commodity products where any one of several firms would all give you more-or-less the same methods and produce more-or-less the same results.

Which is exactly how Obama does it, right? They contract from a handful of polling operations, I believe.

The problem is that you can't have your Chief Strategist also be your Chief Pollster especially when he is secretive about his methodology.

"Hey guys, guess what? Remember that theory of mine you guys didn't believe? Well I just conducted a SUPER SECRET POLL (that you can know nothing about) that says I was 100% right. Isn't that awesome?"

The guy sounds like an absolute charlatan, and it's really an indictment of the Clinton's that they can't see that even when seemingly every one in their campaign does.

Posted by J.W. Hamner | April 22, 2008 12:40 PM

you wrote that a little too quickly, i think.

there's two separate issues being conflated here: the first is the reliability of polls, the second how the polling data is interpreted.

by and large, polling reliability if you do your polls correctly is pretty good.

understanding what the polling data is actually telling you and how to employ it is the "advice" component, and that really is entirely hit-or-miss.

(i like to recall, in this context, that even at the height of prewar support for our splendid adventure in iraq, if you drilled down into the polling data, you discovered that only just over 1/3 supported a war where there would be more than 1,000 US casualties. assuming that mark penn helped convince clinton to vote for the war resolution, one can't imagine that he actually paid any attention to that critical data point.)

Posted by howard | April 22, 2008 12:40 PM

Which is why it seems odd that Mark Penn's numbers give him any leverage at all within the campaign. Certainly the same people that work with him know how unreliable numbers have been, especially in this campaign. You would think they would dismiss him out of hand.

Posted by jj | April 22, 2008 12:42 PM

Trivia question. The lines "If you can't take the heat get out of the kitchen" can be attributed to:

a. Harry Truman
b. Hillary Clinton
c. Eddie Jordan
d. Petey

Mail your answers to myglesias at gmail:) Lucky winners get free copies of HITS and a subscription to Matt's exclusive NBA punditry on a game by game basis.

Posted by KC | April 22, 2008 12:45 PM

if pollsters actually had reliable methods at hand for conducting this kind of work, pollsters wouldn't be brand names.

Yes, and that can never really happen - no poll will ever be definitive in the sense you're talking about. Polling, or any kind of social research, does not in any sense replace judgement, and that goes both for how the polls/research are designed and for how the data are interpreted. Of course one wants data. But there is no getting around judgement, which I think is essential to keep in mind when thinking about this stuff. Judgement is not the same thing as bias. How do you tell the difference? You can't tell the difference if you aren't alert to the fact that there is one.

Penn is exemplary vis a vis my point. He precludes other people's judgement (and biases) altogether because he won't let anyone see his methodology. Some other pollsters may be partial bullshit artists (even unwittingly), but Penn is pure.

Posted by jonnybutter | April 22, 2008 12:52 PM

In retrospect, option b should be read as "Mark Penn's polling data"

Posted by KC | April 22, 2008 12:54 PM

Sadly, all of the phones in Mark Penn's phone bank were crosswired to a hall phone in a dorm on the campus of the University of Delaware. The students would take turns making up humorous responses to the polling questions. "Tell them you want to be a sniper." was often overheard, amid a chorus of chuckles.

Posted by Njorl | April 22, 2008 1:51 PM

The person ultimately responsible for any campaign is always the candidate. It certainly says something about Hillary for trusting this charlatan. And it says something about what her influences would be in office.

Posted by mpowell | April 22, 2008 2:06 PM

Pollsters don't do secret polls with secret methods. Polling is reliable enough -- a lot more than a lot of people think. When a pollster spins its not the numbers that they spin, its the interpretation thereof.

Pollsters tend to have 2 advantages when it comes to setting strategy on a campaign:

1) they know more about the numbers, not because they whip out fake ones when they need to but because they have real expertise and they have spent hours poring through them that no one else on the campaign has spent.

2) A guy like Penn -- who I dislike by the way -- has a shitload of real world, high-level campaign experience. He has probably worked on literally hundreds of federal races of all kinds. A presidential candidate has probably been through only a few, a congressional candidate maybe several but most of them probably being safe incumbent re-election efforts. Campaign managers can handle only 1 or 2 big campaigns per cycle, while a pollster can handle dozens.

The only other person with the experience to stand up to the pollster is the media consultant but they are usually more Mac than PC and don't do numbers, so they're still at a disadvantage.

Posted by Junius Brutus | April 22, 2008 2:16 PM

There is a third key advantage I should have mentioned: agenda setting. If some ideas aren't reflected on the oll in the first place, then its a done deal -- alternative views will have no evidence to be backed up with.

The drafting of a poll questionnaire is a collaborative effort but at every step the pollster has the obvious advantage: they write the first draft. After that, changes are incremental and the pollster brings the first two advantages I listed in my previous post to bear in arguments about changes as well.

This is where a lot could be done to curb the advantage of pollsters.

Posted by Junius Brutus | April 22, 2008 2:27 PM

Which is exactly how Obama does it, right? They contract from a handful of polling operations, I believe.

That kind of misses Matt's point. Matt's point is that polling is witchcraft, whether it's polling that Obama gets or whether it's polling that the Clintons get.

The problem with the Clinton campaign, apparently, is that this kind of witchcraft has been elevated to a role in the campaign that far surpasses any insights it can provide.

Apparently the Obama campaign has been less inclined to take that approach, although who really knows. It may be that since Mark Penn is an asshole and a pollster, we may be finding out more about how idiotic his polling is over, say, the polling done by whoever does it for Obama.

Posted by Joe Strummer | April 22, 2008 2:39 PM

Sorry for the serial posting but one other thing. Matthew says, "if pollsters actually had reliable methods at hand for conducting this kind of work, pollsters wouldn't be brand names."

This is a really weak argument. Think: basketball coaches (I know Matthew understands that one), lawyers, doctors, generals etc. Its not physics but it ain't postmodern philosophizing either.

Posted by Junius Brutus | April 22, 2008 2:50 PM

The basic problem is when people misunderstand what polling can and cannot do. Polling can generally do a good job of telling you what the people in the target population are thinking right now. Polling generally cannot do a good job of predicting the future beyond a relatively short horizon, particularly not hypothetical futures. Indeed, that is partially because people themselves are not particularly good at predicting hypothetical futures. So, for example, generally if you ask people questions of the form, "What would you feel about X if Y happened?", they will guess wrong, as measured by what they end up actually feeling about X in cases where Y happens.

In short, polls aren't magical crystal balls. And if you truly understand that, pollsters are unlikely to exert undue influence upon you.

Posted by DTM | April 22, 2008 3:03 PM

Polling is not witchcraft. Its based on statistics. Its very effective.

All of you who base your opinion of the value of political polling because you've heard about horserace numbers being off from time to time, have no idea how polling is used in campaigns.

The most important use of polling in a campaign is to craft and target message. And all that you need is comparative evaluations that allow you to make decisions about which are your top messages, it doesn't matter if you get the numbers exactly right as long as they are not systematically biased in a way that gets the results in the wrong order.

Posted by Junius Brutus | April 22, 2008 3:05 PM

Comment of the week, Njorl.

Posted by Gabriel | April 22, 2008 3:12 PM

Junius Brutus:

While reputable pollsters don't "do secret polls with secret methods", Mark Penn is accused of that very thing.

Here is an TPM Cafe post from a year ago about it:

http://tpmcafe.talkingpointsmemo.com/2007/05/09/the_real_case_against_mark_pen/

And from all the campaign reporting I've seen, he really does "pull it out of his ass" and doesn't show anyone any data... maybe it's all "proprietary" methodology and he doesn't want anyone to steal his firms secrets, but the lack of transparency doesn't give one confidence IMHO.

Posted by J.W. Hamner | April 22, 2008 3:14 PM

Comment of the week, Njorl.

Posted by Gabriel | April 22, 2008 3:14 PM

It is a very rare event indeed when a pollster decides to spend tens of thousands of dollars of his own money to do a poll related to a campaign when he can usually get 99% of what he wants on a campaign-funded poll anyway. They can't do a secret poll and then use it for a campaign anyway because of campaign finance laws -- unless they make the results public, which is not the case here ex hypothesi.

As far as sharing data, they share the crosstabs so they do share the data to that extent. It is an even rarer thing to have anyone else on a camapaign who would have the slightest idea what to do with the raw data.

Posted by Junius Brutus | April 22, 2008 3:26 PM

It is a very rare event indeed when a pollster decides to spend tens of thousands of dollars of his own money to do a poll related to a campaign when he can usually get 99% of what he wants on a campaign-funded poll anyway. They can't do a secret poll and then use it for a campaign anyway because of campaign finance laws -- unless they make the results public, which is not the case here ex hypothesi.

As far as sharing data, they share the crosstabs so they do share the data to that extent. It is an even rarer thing to have anyone else on a camapaign who would have the slightest idea what to do with the raw data.

Posted by Junius Brutus | April 22, 2008 3:28 PM

Another reason political polling isn't commodified is that its really a niche business. You have to combine mathematical expertise with the political expertise of a serious political junky, policy expertise, and communications abilities. You don't find a lot of people who combine all these traits and they're certainly not as common or as indistinguishable as bushels of wheat or bricks of gold.

Posted by Junius Brutus | April 22, 2008 3:39 PM

Hamner: I went to the link you provided but the guy doesn't know what he's talking about. Here is the meat of his complaint:

"unlike almost any other Democratic pollster, he never shows his work. Indeed, my first criticism of Penn was here in TPM Cafe last July, responding to an op-ed he co-wrote with James Carville making broad assertions such as that "Democratic and even independent women are thrilled with the idea" of Senator Clinton running for president, all without a single piece of data to support them. It is telling to compare the web sites of Penn,Schoen and Berland and that of Greenberg's firm,Greenberg,Quinlan & Rosner: Both firms do plenty of work that is proprietary, and both have corporate clients. But Greenberg's site is full of actual data -- the link above goes to a page with 192 reports on U.S. politics, eleven since the beginning of this year alone! Penn's site hasnothing; a link to
"read samples of our thinking" goes to a page with links to those same data-free op-eds! In short, we have no way of knowing whether Penn's demographic analysis of the electorate is as rigorous as Pew's or Greenberg's or whether he,
if you'll forgive me some technical jargon, pulls it out of his ass."

Even trying to be charitable, all I can say is that this critique is foolish. No one -- and I mean no one, incuding Greenberg or anyone else -- puts the results of campaign polls on the internet. The idea is simply idiotic -- like asking a football coach to publish his playbook.

But pollsters always share their data with the campaign. The first thing they do after a poll comes out of the field is send literally hundreds of pages of solid numbers -- the "crosstabs" -- to the campaign.

The problem isn't the quality of the data or hidden data, the problem is with what they make of that data. Penn's "office-park dad" analysis was a load of crap for the reasons mentioned in the linked post, but that's not because he did a secret poll or jiggered the numbers.

Posted by Junius Brutus | April 22, 2008 4:16 PM

It's not just in politics - same holds sway in the business world. You can make all the arguments you want about why a policy is good or bad for a company, but the guy who has the data - who can "prove" that his idea will result in XX profit more than the other guy's - always wins. This, incidentally, is why consultants are so powerful and despised.

Of course, the guy with the numbers is not always right - even the best number is built out of a lot of estimates and interpretation that leave a lot of room for error, discount the impact of difficult to measure variables, and often aren't seeking the right number to begin with. The problem is that there is no other way to really do it - once you get to the highest levels, everyone is smart and can make their idea sound plausible, so unless you can back them up with a numerical fact base which you can contrast with the other ideas, there is no way of making an objective choice. Still, it's a sick part of our world and one that pervades both campaigns and businesses.

Posted by Vermando | April 22, 2008 4:31 PM

I actually think message testing is a good example of where politicians may be asking too much out of their pollsters. Again, that basically amounts to asking the people you are surveying to predict how they would respond to hypothetical events, and people generally don't do a particularly good job of making such predictions.

Indeed, consider just the simple point that your opponent and other third parties might respond to your message in some unanticipated way. How is a poll supposed to help you predict how people will feel after not just hearing your message, but also hearing the responses? It can't, of course, but ultimately that is what matters to you as a politician.

Posted by DTM | April 22, 2008 5:14 PM

DTM:

Its quite common to present both sides of an argument, so testing responses is not as hard as you may think, although learning how to interpret the results and how they will play in the real world is where the art of polling and strategy comes in.

The bottom line is there certainly are limitations to polling message but its a lot better than wild ass guesses. If you were going to put millions of dollars and your career on the line, would rather go with someone's intuition or inferences from representative samples of the voting population drawn by experts who have been through hundreds of campaigns in the past?

Posted by Junius Brutus | April 22, 2008 5:37 PM

Wait, Penn knows what the numbers mean? I thought we weren't going to find out till at least season 5.

4,8,15,16,23,42

Posted by I Drink Your Milkshake | April 22, 2008 6:08 PM

Junius Brutus,

You wrote: "Its quite common to present both sides of an argument, so testing responses is not as hard as you may think ..."

That is why I originally wrote, "your opponent and other third parties might respond to your message in some unanticipated way," because you can't test a response you didn't anticipate. Generally, that is another fundamental problem with hypothetical polls: the real world tends to quickly get off script.

You also wrote: "If you were going to put millions of dollars and your career on the line, would rather go with someone's intuition or inferences from representative samples of the voting population drawn by experts who have been through hundreds of campaigns in the past?"

I don't view it as an either/or situation. If I was advising candidates who wanted to do message testing--and who could easily afford it--I would tell them to go ahead. But I would also help those candidates understand that the data they will get from such polls is of limited and uncertain value, no matter what their pollsters might suggest to the contrary.

Indeed, the most problematic word in what you wrote is "inferences". Polls give you data, and that is what pollsters should be giving you as well. As soon as your pollster starts giving you "inferences", you may need to start looking for another pollster.

Posted by DTM | April 22, 2008 6:35 PM

Uh, DTM, every time you calculate statistics you are making what is known in mathematics as a statistical "inference".

Also, pollsters don't present polling results as certain at all. The always cover their ass. But they will argue quite forcefully, armed with statistical inferences over other peoples' wild ass guesses.

I don't care how smart you are or how much of a political junkie, you simply can't intuit your way into knowing which 3 out of 30 chosen possible messages pops with the most intensity amongst specific targeted subgroups (that you were able to identify e.g. using before and after horserace questions to identify the subgroups that disproportinately makeup the group of those who shift their vote after hearing the messages) -- and do that with any accuracy in all the different media markets in 15 or 20 swing states.

Posted by Junius Brutus | April 22, 2008 6:54 PM