What Does A Bad Poll Prove?

Nothing. Yesterday’s Bloomberg News poll is getting a lot of publicity, mainly for its purported finding that respondents approve of President Obama’s immigration decree by a 64-30 margin. But the poll had numerous other findings that cheered Democrats, as indicated by Bloomberg’s own headline: “Obama Leads In Poll As Voters View Romney As Out Of Touch.” And, in fact, in yesterday’s survey supposed likely voters favored Obama over Mitt Romney by a remarkable 13 points, 53-40.

This result is obviously an outlier, but that hasn’t prevented it from being widely repeated and commented upon. The obvious question is, why is it an outlier? Anyone who follows polls knows that the answers you get depend on large part on whom you ask, so the first question always is, what was the composition of the respondent pool?

For purposes of comparison, take a look at Bloomberg’s March poll. Just three months ago, Bloomberg found Obama and Romney locked in a 47-47 tie. Does anyone seriously believe that Obama’s stumbling campaign has produced a 13-point gain since March? Of course not. The difference lies in the composition of the sample.

In March’s survey, Bloomberg did not ask respondents to indicate whether they were Republicans or Democrats. Bloomberg did, however, ask the generic Congressional preference question, which is a fair proxy. The March respondents (including leaners) favored Republican candidates 46-44. The respondents in yesterday’s poll, on the other hand, favored Democrats 48-41. So there was a nine-point swing in favor of Democrat-leaning respondents, which, combined with the margin of error, accounts for the poll’s skewed result. In June, Bloomberg did ask the party affiliation question; Democrats (again, counting leaners) outnumbered Republicans 38-33. (In the Rasmussen survey, by way of contrast, 33.4% of likely voters say that they are Democrats, compared with 35.7% who describe themselves as Republicans.)

Why did Bloomberg over-sample Democrats in yesterday’s poll? It may have been largely random, but the fact that the survey was done on a Friday-Saturday-Sunday-Monday cycle no doubt contributed. If you telephone people on the weekend, you will over-sample Democrats, because Republicans go out more, on the average.

If you run a bad poll and produce an outlying result, you can get lots of publicity, especially if your results favor Democrats. But you can’t prove anything with a bad poll.

Bloomberg’s 64/30 finding on the immigration question is of course infected by the poll’s bad sample. But that particular question suffered from other defects as well. This was the question:

President Obama announced that the U.S. would halt the deportation of some illegal immigrants if they came here before age 16, have been in the country for five years, have no criminal record, are in school or have a high school diploma or have been honorably discharged from the military. Do you agree or disagree with this new policy?

As many others have already noted, this question misrepresents Obama’s announced policy, at least implicitly, in several respects. First, the only age mentioned in the question is 16. Respondents were not told that persons up to age 30 are eligible for the president’s amnesty. Second, Bloomberg’s question gave the impression that no one convicted of a crime would be eligible for the amnesty. But that is not correct. In fact, under the administration’s guidelines, an illegal resident can have two misdemeanors on his record without being disqualified, and more than three if they were on the same day and part of the same act. Moreover, the executive branch has discretion to determine what is a “significant” misdemeanor offense. And, finally, the question makes no reference to the important procedural question, i.e., whether it was proper for Obama to implement a policy of selective non-enforcement of the immigration laws with no Congressional action.

All of that said, I don’t doubt that the administration’s policy, in substance, would command majority support. That is, after all, why Obama cherry-picked this particular partial amnesty, in order to enhance his reelection chances. But it is impossible to determine what the actual public support for Obama’s policy might be on the basis of the flawed Bloomberg poll.

Responses