Centerfield

From my biostastics course I learned how complicated polling data and statistical inference is. From a project I did I learned how many surveys are poorly written. I'm not sure how interested the public (educated or otherwise) is in the subtleties of polling science.

I've always been a bit confused by the notion of "partisan" pollsters. Polling is supposed to be a science, an objective attempt to understand the state of political or cultural opinion. Once you identify yourself as a Democratic or Republican pollster I think your credibility plummets. You are clearly then in the business of telling people what they want to hear. You're more pimpster than pollster. I note that although there are certainly physicists who are Democrats and physicists who are Republicans, the two groups do not reach identifiably different conclusions.

But it's different with polling. Physicists presumably are in the business to discover how the world operates; there really doesn't exist a "Republican" physics or a "Democratic" physics (although the USSR attempted to create a Marxist-type genetics). With polls, one of the functions is to find out objectively how people actually think about a particular issue. To do this, pollsters have to conduct the surveys in a scientifically valid way. Campaigns use these kinds of survey because they really do need to know how they stand. But another function of polls is to persuade voters that others think a particular way about a particular issue. To do this, a pollster may want to conduct the survey in a way that skews the results, ie, loaded questions, etc. This is where partisanhip comes in and why you have Republican and Democratic pollsters. Credibility doesn't really matter because it's sort of like spin; once a story is out there that xx% think such about an issue or candidate, that story takes on a life of its own, even if it's based on a less valid poll.

But there are other ways that pollsters can participate in campaigns without being whores. They provide legitimate data to the parties on demographics and advice on strategy. Of course, this leads to poll-driven campaigns and governing, but that's another story.

Besides what has already been mentioned--specifically the phrasing of the questions, etc., I'd also like to see more disclosures as far as the demographics that composed the sample.

For example, remember how during the Presidential Campaign there would be these widely variating numbers thrown around. Bush would be up 10 points in one and trailing by 4 in another. To the average TV viewer/news browser, that's all they would see, leaving them perhaps very confused as to where the race really stood. (Or, the reliability of polling.)

However, it often appeared that upon digging deeper into those individual polls, the samples would be skewed presumeably to reach the desired outcome. You'd see a heavier weight of Republicans in the Bush-leading poll, and perhaps more Independents in the Bush-trailing poll. This tactic seemed most apparent after the individual conventions when the respective candidates led by wide margins. (Or didn't.)

One more note...as it relates to all of these SS polls...obviously demographics are huge. If they are calling home phones during the day for samples, who are they more likely to get?

Well, sometimes you have partisan pollsters, but sometimes you have pollsters hired by partisans.

Pollsters are prone to pleasing their customers, which is a problem that, say, physicists are less likely to encounter to the same degree. Then there are polls that are bad because they were comissioned by newspapers that just want to print a colorful visual like a table or infographic as a vehicle for a story, but don't want to pay for a well-done in-depth exploration of an issue.

I'm not sure the public is interested in the specifics of polling science either, but that doesn't mean that it would be bad to add value to polls by 1)doing them better, and 2) paying closer attention to polls conducted by nominally unbiased inquisitors.

I might be wrong, but I really think you could get interesting results if you still allowed people to express their opinions via choice but also allowed them the honest option of saying "but I haven't been following it closely" or "I don't really know that much about it."

Hey Mike, thanks for the visit, I've been enjoying your rants over at the mighty middle.

And if you did a SS poll that first checked people's knowledge with a brief MC quiz, the poll might even do a little good beyond measuring attitude. If 1000 americans were polled on SS, and 100 of them, via the quiz, were driven to somehow acknowledge " I really don't know much about this," it might encourage some small additional fraction of the people to dig a little deeper. Even if some small fraction was willing to acknowledge a distinction between an opinion pulled from one's rectal cavity as opposed to one based on a bit of information, it'd be a start. Even if a little knowledge can be a dangerous thing.

I've never really thought the problem was in the polling. Rather, the problem is in the reporting. Newsmedia ought to adopt a standard for reporting various polls, including:

The exact phrasing of the question,
The basis of the sample selected,
The time of day of the poll,
Precise demographics of the sample,
A statement of purpose by the polling entity, and
Anything else that I haven't thought of but will help the reader understand the poll results.

I agree with those commenters above that polling is more science than art. It should be reported with its parameters so that the various polls can be compared with each other.

The numbers are straight math, so you could call that part statistical science. The questions are art (or artifice). I know, I used to write such things. You're trying to quantify opinions. Or, if you're less than upfront, trying to shape them. Nothing scientific about it. Opinions are not objective material.

I'm not the only one who noticed some problems with the structuring and interpretation of the WaPo poll. John Henke of QandO blog also notes the discontinuity.

One way to judge individual polling bias is to see how well they predicted the election.
There are sites and archives where you can judge the accuracy of an organization. (I would post the links but experienced a hard drive melt down since the election.)

I seem to remember that Harris was the best.

Personally I always liked Pew and the polling organizations of Peter Hart (D) and Bill McInturff (R). (NBC News/Wall Street Journal Poll)

Real clear politics keeps a running average of most major polls. I don't think they do their math right but its better than nothing.

http://www.realclearpolitics.com/polls.html

There are so many problems with polls:

1. Wording of polls and how it is said or written. If the pollster is reading the questions to the respondent, even the tone placed on words will place a subjective slant of the response.
2. When and where the poll is done. Daytime vs. evening, phone vs. random person-to-person, weekday vs. weekend.
3. Demographics. This one I really believe is worthless if done over the phone.
4. Paid vs. unpaid participation. Some pollsters will pay $ if you go into a room at some mall and answer some questions.

And most of these methods will not be given with the output of the poll.

...yeah, and sometimes I wish that more poll questions came equipped with a None of the Above response, because I think that might be another popular answer.

That said, though, I can't imagine either entry being mentioned much in reports on a poll, except for the entire poll itself, which always gets much narrower readership, of course. Sometimes, IMHO, unfortunately, one has to read both sides a bit to figure out what's going on (Iraq, and now SS).