With the actual polls in South Carolina closing at 7:00 p.m. ET, it might seem a bit late to look at what Mark Blumenthal wrote at Pollster.com about the uncertainty of the pre-primary polls. Plenty of folks already blogged about it in the conventional way, wondering whether we could be in for the sort of surprise we got in New Hampshire. Yet Blumenthal’s analysis is compelling on a different, more general level as well.
Regular visitors here know that one of my favorite themes is that laypeople often interpret public opinion polls far too literally, with an under-appreciation of how much art there is mixed with the science involved in polling. That notion feeds one of my other favorite themes, which is that the media (and by extension, much of the public) focuses too much on public opinion polling in covering political campaigns, rather than seeing polling as a tool campaigns bring to bear in synergy with organization, message, etc. Blumenthal’s analysis of the South Carolina polling provides a case study illuminating these themes.
In discussing the variation in the pre-primary polling, Blumenthal notes the differences in the “undecided” percentage created by the differences of question wording, context, survey mode and interviewer technique for each poll.ÂÂ
A subset of the last factor is whether the pollster conducts the interviews with live interviewers or with an automated methodology. Blumenthal notes that even professional pollsters debate whether automated polling produces different results and whether those so interviewed are more willing to cast a “secret ballot” or whether those willing to participate in automated polls tend to be more opinionated.
Blumenthal also looks at the differing methods pollsters use to select samples, from “likely voter” screens, to random dialing vs. starting from a list of registered voters, to variations in the demographics of each sample.
The larger value of Blumenthal’s analysis is that these differences can be compared within the context of a specific, discrete event. Often, polls on other topics tend to be spread out over months in a way that does not allow for a head-to-head comparison. An election allows for this direct comparison, revealing the significant differences among pollsters and their methodologies. The broader lesson, as I have suggested, is that laypeople — including journalists — tend to attach much more literal meaning to a numerical response to a question than professional pollster would. Polling is a valuable tool, but those who rely too much on them are like the person who owns only a hammer and sees every problem as a nail.
Also: Inasmuch as I tend to stick up for the impact organization has on campaigns, I must flag a piece by Chris Frates at the Politico about how the Clinton campaign sought to maximize its effort in South Carolina by focusing on Congressional districts which favor Sen. Hillary Clinton by age, race and income. Again, paying attention to the rules for delegate allocation and how a candidate can use polling and organization to exploit those rules tells much more of the story than looking at the topline horse race numbers.
Good stuff. I’d like to have a rolling contest for blogs and MSM regarding who makes the worst predictions regarding the primaries.
I have no idea whose going to win the election in ’08…I dont even care all that much.. but I do know I just saw Cloverfield and it was far more satisfying than Blair Witch… A very enjoyable, thrill-ride of a movie that’s well worth the price of admission. Now if I can just figure out how Slusho & The Tagruato Corp figure into it all…
Dan, that would be fun and interesting, though I expect the reult would be more a comparison of who did worse. The MSM often has hard deadlines and (in broadcasting) time/space constraints that systemically helps drive a more superficial analysis. Bloggers have more time to seek out expertise on the net at sites like Pollster.com and perhaps deeper analysis. However, bloggers on a story like this tend to impose deadlines on themselves, itching for the traffic that comes from calling “FIRST!” on the newest poll. Moreover, bloggers are (even if to their credit) often openly biased, and their analyses may skew — consciously or sub-consciously — toward those biases.
Furthermore, one of the points implied in my original post is that even with time and expertise is that analysis of polling can only take one so far in making any sort of prediction about an election.
Ouroboros,
Just stay out of midtown.
Obama Wins
Women, Minorities Affected Most
— NY Times
I don’t understand why teh NPR keeps telling me that Obama is the first African American person ever to be in the position of possibly being the first African American person ever to be president when that’s what they already said about Jesse in the 80’s. NPR can’t just erase Jesse like that, can they? I think maybe they can, but I think we have to fight to keep his memory alive, cause it ain’t right.
happyfeet,
You’re gonna wind up in the re-ed camps if Clinton wins. I know you know what a nonperson is. It’s is doubleplusungood to ever make a reference as you did. Tomorrow’s scheduled high in Barrow is -19.
Ohnoes. That means I’m gonna have to magic marker my name in all my clothes again. I just now got rid of all those finally except for my red Members Only jacket but I only ever wear that ironically.
I think it’s being medically proved that one who’s suffered Clinton fatigue in the past is liable to suffer a faster onset upon relapse.
I guess that the polls and pundits that saw Hill and Silky performing well in SC were not all that right.
Since John Edwards has no discernible soul, he is going to just keep plugging away, relegating himself further and further to the back of the dustbin of history. Only bad part about that, is that he might go back to channelling dead kids in courtrooms. Maybe he does less damages, perpetually running for something that he will never win.
NPRs satellite is crashing.
[…] Rasmussen suggests. I have previously written that events like primaries help demonstrate the uncertainties in polling that usually remain hidden when there are not simultaneous competing polls to compare.  This is […]
[…] PW visitors know that one of my recurring themes is that laypeople often interpret public opinion polls far too literally, with an […]