Speaking of Selection Bias...

Alan Colmes has now jumped into the discussion (by repeating a Political Wire post) about the Smart Politics review of PolitiFact's selection bias.

In a truly amazing feat, Colmes (and Political Wire) managed to completely ignore the point of Eric Ostermeier's article, presenting the finding that PolitiFact more frequently rates Republicans negatively as proof that Republicans lie more often.

Check out the Colmes headline-

Republicans’ Statements Untrue Three Times As Often As Those Of Democrats

Colmes made absolutely no mention of the fact that Ostermeier was suggesting that the lopsided figures point to selection bias at PolitiFact. This is how Colmes treated the study-

"An analysis by Smart Politics of more than 500 stories during the past year show that statements made by Republican politicians are false three times as often as those made by Democrats."

Representing Ostermeier's piece as the polar opposite of what it is takes either an act of denial or more likely a shoddy job at research.

Compounding the error, Colmes, et al. ignore the focus of the study. Ostermeier's article dealt specifically with the issue of selection bias, i.e. the notion that PolitiFact's editors are more inclined to select outlandish or questionable statements from the GOP. This still leaves the issue of the quality of the ratings themselves untouched. As Ostermeier noted-

"Assuming for the purposes of this report that the grades assigned by PolitiFact are fair (though some would challenge this assumption), there has nonetheless been a great discrepancy regarding which political parties' officials and officeholders receive the top ratings and those that are accused of not telling the truth."

(bold emphasis added)

Colmes' careless posting of misleading information exemplifies how PolitiFact's opinionated brand of journalism can be misrepresented as objective fact, and repeated throughout the echo chamber.



Feb. 16, 2011  Corrected misspelled version of Eric Ostermeier's last name in one of the middle paragraphs.  Also added some new hyperlinks and a definite article in the first sentence.

Smart Politics on selection bias at PolitiFact

Research associate Eric Ostermeier of "Smart Politics" at the University of Minnesota has published a study of PolitiFact's content and finds a disturbing vein of partisanship in PolitiFact's selection bias:

Assuming for the purposes of this report that the grades assigned by PolitiFact are fair (though some would challenge this assumption), there has nonetheless been a great discrepancy regarding which political parties' officials and officeholders receive the top ratings and those that are accused of not telling the truth.

Ostermeier goes on to note PolitiFact's focus on untrue statements by the party out of power--perhaps unusual considering the press considers itself the watchdog of government.

Ostermeier's approach is exactly right in taking PolitiFact's grade groupings as an indication of PolitiFact's selection bias rather than as a measure of candidate truthfulness.

A key chart from the report:

As Hot Air's Ed Morrissey points out, it remains possible to interpret the data to mean that Republicans simply utter more false and "Pants on Fire" statements than Democrats.

But is that the best explanation in terms of the empirical data?

I would again call attention to the divide between the "False" and "Pants on Fire" ratings.  PolitiFact to date has offered no metric for objectively distinguishing one from the other.  This in turn strongly suggests that subjective judgment serves as the ultimate criterion for grading a statement as "Pants on Fire."

 ***

Importantly, Ostermeier also points out that PolitiFact's presentation encourages readers to use the ratings to draw conclusions about the subjects receiving the ratings:

The question is not whether PolitiFact will ultimately convert skeptics on the right that they do not have ulterior motives in the selection of what statements are rated, but whether the organization can give a convincing argument that either a) Republicans in fact do lie much more than Democrats, or b) if they do not, that it is immaterial that PolitiFact covers political discourse with a frame that suggests this is the case.

Feb 11, 2011: Corrected typo affecting the word "Minnesota." My apologies to that fine state.
June 14, 2018: Clarified the third paragraph (first after the quotation) by removing a "that" and an -ed suffix.