Cohen's summary of PolitiFact data on the Republican field of candidates very prominently featured the following (from a post Cohen wrote back in September of 2011):
PolitiFact only looks at statements that pique its interest. Here’s how Bill Adair, the editor of PolitiFact, described the process: “We select statements that we think readers will be curious about. If someone hears a claim and wonders, ‘Is that true?’ then it’s something that we’ll check.”Perfect.
In other words, if Mitt Romney says the sky is blue, PolitiFact doesn’t bother grading the statement as true. So there is a sampling bias at play here. Accordingly, the following numbers should be interpreted with caution. They aren’t perfect indicators of the honesty of each candidate, and conclusions like “Candidate X lies the most” or “Candidate Y is the most truthful” should probably not be drawn from the data.
PolitiFact, take note.
(1/31/2012) Jeff adds: A commenter to this post brings up an excellent point, and it's one I happen to agree with. Cohen deserves credit for at least recognizing some of the flaws, and including a disclaimer. The fact that it happened in the NYT is also worthy of note. In my view, however, it's unfortunate that Cohen immediately disregards his own reservations about the legitimacy of the report cards and goes on, in detail, to provide an analysis of them anyway. A colored chart with percentages provides an undeserved air of scientific authenticity. I'm not impressed.
The flaws with PolitiFact's report cards go deeper than Cohen implied. We've published a new post that explains these problems in detail. You can read it here.
