Cohen's summary of PolitiFact data on the Republican field of candidates very prominently featured the following (from a post Cohen wrote back in September of 2011):
PolitiFact only looks at statements that pique its interest. Here’s how Bill Adair, the editor of PolitiFact, described the process: “We select statements that we think readers will be curious about. If someone hears a claim and wonders, ‘Is that true?’ then it’s something that we’ll check.”Perfect.
In other words, if Mitt Romney says the sky is blue, PolitiFact doesn’t bother grading the statement as true. So there is a sampling bias at play here. Accordingly, the following numbers should be interpreted with caution. They aren’t perfect indicators of the honesty of each candidate, and conclusions like “Candidate X lies the most” or “Candidate Y is the most truthful” should probably not be drawn from the data.
PolitiFact, take note.
(1/31/2012) Jeff adds: A commenter to this post brings up an excellent point, and it's one I happen to agree with. Cohen deserves credit for at least recognizing some of the flaws, and including a disclaimer. The fact that it happened in the NYT is also worthy of note. In my view, however, it's unfortunate that Cohen immediately disregards his own reservations about the legitimacy of the report cards and goes on, in detail, to provide an analysis of them anyway. A colored chart with percentages provides an undeserved air of scientific authenticity. I'm not impressed.
The flaws with PolitiFact's report cards go deeper than Cohen implied. We've published a new post that explains these problems in detail. You can read it here.
You shouldn't laud Micah Cohen too much. Despite noting the inherent sampling bias, he continued on with his numerical analysis as if that were the only issue. "They aren’t perfect indicators of the honesty of each candidate" is quite an understatement. The numbers Cohen generates have far more to do with the Politifact staff's regard for the the candidates than it has to do with the candidates' honesty.
ReplyDeleteI'm working on an addendum to the post that addresses just the issues you raise. Hopefully it will be posted by tonight. Thanks for commenting.
ReplyDelete"They aren’t perfect indicators of the honesty of each candidate" is quite an understatement.
ReplyDeleteQuite true.
But Cohen benefits hugely by comparison to PolitiFact's own handling of the data. And I interpreted him charitably by crediting him with understating the degree to which selection bias undermines conclusions about candidates' honesty. A statement like Cohen's should be the minimum requirement for PolitiFact whenever it publishes candidate report cards and the like.
"crediting him with understating"
ReplyDeleteTo clarify, the credit associated with understatement is as a literary device ("Einstein was no dummy").