The Weighting Game

My friends at the CBS News blog Public Eye have responded to my criticism of the New York Times/CBS poll and accompanying New York Times story on attitudes towards the use of warrantless surveillance in the war on terror. I mentioned three criticisms: (1) that the poll sample included a disproportionate number of Democrats compared to Republicans, (2) that the poll, while containing dozens of questions, never asked what respondents thought about the type of surveillance program that the Bush administration says it has implemented, and (3) that the New York Times article about the poll results failed to accurately reflect the extent to which respondents favor warrantless surveillance to combat terrorism. My focus was on the second and third points.
Public Eye doesn’t dispute my second point and seems to agree with my third. However, it provides an informative defense of the poll’s “weighting” techniques. As I understand the argument, CBS wants a sample that accurately reflects the sex, race, age, education level, etc of the American population. It calls the people in its surveys randomly, but then weights its results to make sure that African-Americans or younger Americans, for example, are represented in the sample to the same extent they are represented in the population. But it performs no such adjustment based on political affiliation.
This practice appears to have support among experts in the field, and therefore I should not have characterized it as “the usual MSM trick of over-sampling Democrats.” For what it’s worth, however, the practice strikes me as flawed. If you’re trying to figure out what types of ice cream Americans like most, it makes sense to use a sample that reflects the general demographics of the U.S without worrying about political party affiliation. But if you’re tying to figure out who Americans will vote for, how they feel about the president, or how they feel about an intensely political issue, I think it makes little sense to ignore political affiliation. Nor, as far as I can see, would doing so exclude the possibility of using a sample that also reflects other demographic characteristics of Americans.
The counter-argument, I think, is that one’s political affiliation — as opposed to one’s age and gender for example — is not constant even from election cycle to cycle. But that doesn’t strike me as an argument for excluding it from consideration altogether. Political affiliation is fairly constant, and what it lacks in complete consistency, it probably makes up for in relevance at least in the many cases where affiliation is constant.

Notice: All comments are subject to moderation. Our comments are intended to be a forum for civil discourse bearing on the subject under discussion. Commenters who stray beyond the bounds of civility or employ what we deem gratuitous vulgarity in a comment — including, but not limited to, “s***,” “f***,” “a*******,” or one of their many variants — will be banned without further notice in the sole discretion of the site moderator.

Responses

The Weighting Game

My friends at the CBS News blog Public Eye have responded to my criticism of the New York Times/CBS poll and accompanying New York Times story on attitudes towards the use of warrantless surveillance in the war on terror. I mentioned three criticisms: (1) that the poll sample included a disproportionate number of Democrats compared to Republicans, (2) that the poll, while containing dozens of questions, never asked what respondents thought about the type of surveillance program that the Bush administration says it has implemented, and (3) that the New York Times article about the poll results failed to accurately reflect the extent to which respondents favor warrantless surveillance to combat terrorism. My focus was on the second and third points.
Public Eye doesn’t dispute my second point and seems to agree with my third. However, it provides an informative defense of the poll’s “weighting” techniques. As I understand the argument, CBS wants a sample that accurately reflects the sex, race, age, education level, etc of the American population. It calls the people in its surveys randomly, but then weights its results to make sure that African-Americans or younger Americans, for example, are represented in the sample to the same extent they are represented in the population. But it performs no such adjustment based on political affiliation.
This practice appears to have support among experts in the field, and therefore I should not have characterized it as “the usual MSM trick of over-sampling Democrats.” For what it’s worth, however, the practice strikes me as flawed. If you’re trying to figure out what types of ice cream Americans like most, it makes sense to use a sample that reflects the general demographics of the U.S without worrying about political party affiliation. But if you’re tying to figure out who Americans will vote for, how they feel about the president, or how they feel about an intensely political issue, I think it makes little sense to ignore political affiliation. Nor, as far as I can see, would doing so exclude the possibility of using a sample that also reflects other demographic characteristics of Americans.
The counter-argument, I think, is that one’s political affiliation — as opposed to one’s age and gender for example — is not constant even from election cycle to cycle. But that doesn’t strike me as an argument for excluding it from consideration altogether. Political affiliation is fairly constant, and what it lacks in complete consistency, it probably makes up for in relevance at least in the many cases where affiliation is constant.

Notice: All comments are subject to moderation. Our comments are intended to be a forum for civil discourse bearing on the subject under discussion. Commenters who stray beyond the bounds of civility or employ what we deem gratuitous vulgarity in a comment — including, but not limited to, “s***,” “f***,” “a*******,” or one of their many variants — will be banned without further notice in the sole discretion of the site moderator.

Responses