Thursday, February 10, 2011

Smart Politics on selection bias at PolitiFact

Research associate Eric Ostermeier of "Smart Politics" at the University of Minnesota has published a study of PolitiFact's content and finds a disturbing vein of partisanship in PolitiFact's selection bias:
Assuming for the purposes of this report that the grades assigned by PolitiFact are fair (though some would challenge this assumption), there has nonetheless been a great discrepancy regarding which political parties' officials and officeholders receive the top ratings and those that are accused of not telling the truth.
Ostermeier goes on to note PolitiFact's focus on untrue statements by the party out of power--perhaps unusual considering the press considers itself the watchdog of government.

Ostermeier's approach is exactly right in taking PolitiFact's grade groupings as an indication of PolitiFact's selection bias rather than as a measure of candidate truthfulness.

A key chart from the report:

As Hot Air's Ed Morrissey points out, it remains possible to interpret the data to mean that Republicans simply utter more false and "Pants on Fire" statements than Democrats.

But is that the best explanation in terms of the empirical data?

I would again call attention to the divide between the "False" and "Pants on Fire" ratings.  PolitiFact to date has offered no metric for objectively distinguishing one from the other.  This in turn strongly suggests that subjective judgment serves as the ultimate criterion for grading a statement as "Pants on Fire."


Importantly, Ostermeier also points out that PolitiFact's presentation encourages readers to use the ratings to draw conclusions about the subjects receiving the ratings:
The question is not whether PolitiFact will ultimately convert skeptics on the right that they do not have ulterior motives in the selection of what statements are rated, but whether the organization can give a convincing argument that either a) Republicans in fact do lie much more than Democrats, or b) if they do not, that it is immaterial that PolitiFact covers political discourse with a frame that suggests this is the case.

Feb 11, 2011: Corrected typo affecting the word "Minnesota." My apologies to that fine state.
June 14, 2018: Clarified the third paragraph (first after the quotation) by removing a "that" and an -ed suffix.

No comments:

Post a Comment

Thanks to commenters who refuse to honor various requests from the blog administrators, all comments are now moderated. Pseudonymous commenters who do not choose distinctive pseudonyms will not be published, period. No "Anonymous." No "Unknown." Etc.