Chronicle reader rep ignores substantive criticisms

I didn't start blogging to do media criticism. In thirteen months of blogging at Perry vs World, I haven't criticized or cavilled. The only exception I can think of is when I offered an opinion that the state's reporters were not treating Kinky seriously enough, when it was becoming clear that he could be a credible candidate.

Nor did I start DeLay vs the World with any desire or expectation that I would have to criticize the media. However, the Chronicle clearly deserves it. They commissioned a poll that ended up being flawed. They did not recognize those flaws, and instead published the poll on the front page, including the most flawed results prominently in a graphic.

However, mistakes happen. What is truly sad is that that Chronicle will not admit mistake, instead choosing to stand behind the poll in its entirety, despite detailed entreaties from myself and others that they correct or clarifying their misleading reporting.

Chronicle reader rep James Campbell is given space in the Sunday op-ed section to defend the Chronicle's flawed survey (it's a survey, not a poll, but I often use the terms interchangably). Campbell's column delineates exactly how ignorant he is of the substantive criticism of the survey. Campbell clearly is not knowledgable about polling. Why he chose to offer his opinion on something on which he lacks knowledge, I will not speculate.

Let's go over the important substantive criticisms.

1. 28% of district voters voted straight ticket Republican in 2004. Those who study voting know that voting is a habit, and voting habits are very difficult to break. Thus, the Chronicle's reported poll result of 22% for DeLay would indicate that one in four straight ticket Republican voters won't vote straight ticket this election cycle, and that no one else would vote for DeLay. While I suppose that there are some scenarios that this could happen, they all involve DeLay being in jail.

In other words, this part of the poll doesn't reflect real-world realities. Good pollsters think long and hard about these issues before they put a poll into the field, because when a poll does not reflect real-world realities, then the poll is worthless.

2. They identified Steve Stockman as "a former Republican Congressman" in the general election question. This strongly overstates Stockman's general election support and undermines the accuracy of the results. Good pollsters think very carefully about how they label candidates, and a good poll would never label Stockman this way.

3. The reported primary figures are basically worthless when you combine the following two reasons:
a) the sample size makes the margin of error very large. Once you get past 5% margin of error, as this survey does, they are not accurate enough to be worthwhile.
b) 38% of registered voters in the Chronicle's survey claimed they were "certain" or "likely" to vote in the upcoming Republican primary. However, in 2004 only 0.65% actually voted. The huge, huge disparity between the two numbers means that it cannot be confidently said that the Chronicle's survey respondents represent the people who will actually vote in the Republican primary.

It is obvious that the people who the Chronicle claimed would vote in the Republican primary do not represent the people who actually vote in the Republican primary. According to the Chronicle's data, only 58% of the people who will vote in the Republican primary voted for Republican DeLay over Democrat Richard Morrison.

4. The survey undersamples Republicans. When you push include leaners in party totals, the partisan demographics are 36%D, 50%GOP, and 10% strict independent, in a district that is about 62-63% Republican. That's a pretty big discrepancy that is not easily explained.

Campbell utterly fails to address any of these points. Instead he offers this:

We invite readers to review the poll online. The dispassionate will agree that no matter how the numbers are analyzed, the emerging picture will show eroded support for DeLay in the district.

Moreover, the charge that the poll's methodology was flawed is inaccurate. Neither the Chronicle nor the pollsters deliberately distorted poll results. The poll did not use the entire sample, including Democrats, to measure DeLay's support in the GOP primary.

"The only data reported in the paper was from people in Distict 22 who said they were certain and likely to vote in the Republican primary," Murray said. "If you have a sample with 560 people total, that's usually when you report that the margin of error is plus or minus 4 percent. But if you look at a subgroup, like certain or likely Republican primary voters, that's just 213 people. Well, the margin of error is larger, probably about 7 percent at that level. Sometimes people (pollsters) aren't as consistent in pointing out that for subgroups, the error margins are larger."

Murray conceded that critics may have a valid point that the poll should have noted when its margin of error was greater for subgroups.

"What we did — and what the paper reported — was to look at the sample and interpret in terms of the primary, people who told us that they were, a) certain or likely to vote in the Republican primary ... . While everybody was asked, 'If you did vote in the Republican primary how would you vote?' we didn't use the data for the entire sample ... . It's a little bit confusing, but it's a misleading attack on the survey."

I have not heard anyone (including the DeLay campaign) argue that DeLay's poll numbers were not hurt by the news cycles preceeding the poll, as Campbell claims. The DeLay campaign, however, has vociferously complained that the Chronicle poll understates DeLay's Republican support. In this case, the DeLay campaign appears to be right. The Chronicle's polls on the Republican primary are statistically worthless.

Campbell argues that "[n]either the Chronicle nor the pollsters deliberately distorted poll results." I agree. Frankly, it appears that both the Chronicle and the pollsters rushed a poll into the field that they hadn't thought through. As a result, the numbers are flawed, ESPECIALLY the numbers that they've reported for the primary. The poll isn't deliberately distorted, but the poll and its attendant coverage has been very sloppy.

Murray simply refuses to address the fact that he polls 38% of registered voters for an election in which less than 1% vote. Anytime you do that, you ruin the reliability of your data. Murray should know that.

The Chronicle's steadfast refusal to address the substantive criticisms of this poll is a shame. If the Chronicle believes in the truth, they ought issue a correction that a) the reported Republican primary results cannot be supported and b) that, while DeLay's support surely has slipped after many negative news cycles, this poll has methodological errors which render the results imprecise.

Post-writing Memo: this post may be updated as time and candor allows. I'm not yet sure I got the tone right. My goal is that this post should incite the Chronicle to greater accuracy, not antagonize them.