Thursday, January 2, 2020

PolitiFact's "Pants on Fire" bias in 2019

As our research has documented, PolitiFact has consistently failed to offer any objective means of distinguishing objectively between false political claims and ridiculously false political claims.

On the contrary, PolitiFact's founding editor, Bill Adair, said decisions about the "Truth-O-Meter" ratings are "entirely subjective." And current editor Angie Drobnic Holan in 2014 explained the difference between the "False" and "Pants on Fire" ratings by saying "the line between 'False' and 'Pants on Fire' is just, you know, sometimes we decide one way and sometimes decide the other."

Given the understanding that the difference between "False" and "Pants on Fire" rests on subjective grounds, we have conducted ongoing research on the chances a claim PolitiFact considers false will receive the "Pants on Fire" designation.

Our research suggests at least two things.

First, PolitiFact National is biased against Republicans.

Second, the statement selection process renders "Truth-O-Meter" ratings an entirely unreliable guide to candidate truthfulness even assuming the subjective ratings are objectively accurate(!).

Without further ado, an updated chart for both political parties showing the percentage of false ratings given the "Pants on Fire" rating:

We'll address one potential criticism right off the bat.

We should expect a higher percentage for the party that lies more!

We would agree with that criticism if the PolitiFact data stemmed from objective considerations in the fact checks. We have no evidence to support that and considerable evidence to counter it (see above). All the evidence suggests the "Pants on Fire" rating is a purely subjective judgment.

Subjective judgment is incompatible with neutrality.

A Review of the Findings

False statements from Democrats were rated "Pants on Fire" just 9.09 percent of the time in 2019, tying the record low set in 2009. The Republican percentage stayed very close to its historic baseline, which cumulatively stands at 27.21 percent. The long-term average for Democrats dropped slightly to 17.41 percent. Over PolitiFact National's entire history, Republicans are about 60 percent more likely to receive the subjective "Pants on Fire" rating.

That's bias.

PolitiFact's wildly unscientific selection process

The Trump presidency ought to end permanently any supposition that PolitiFact's story selection process approximates random (scientific) representative selection in any way.

Of the 14 "Pants on Fire" ratings given to Republicans in our 2019 data, 13 went to President Trump. The other one went to Mr. Trump's son-in-law, Jared Kushner.

Of the 39 "False" ratings given to Republicans in our 2019 data, 29 went to Mr. Trump.

Combined, then, 42 of 53 of Republicans' false "Truth-O-Meter" ratings went to Mr. Trump.

For comparison, in 2011 PolitiFact rated 88 Republican claims false with none of them coming from Mr. Trump. From 88 down to 10? Is the Republican Party, aside from Trump, that much more honest with the passage of time? Nonsense. That hypothesis is completely implausible on its face.

The explanation is painfully simple: As PolitiFact admits, its editors choose the claims PolitiFact rates. They use editorial judgment to select stories, not scientific selection. The editors are, in the words of the editors, "not social scientists."

If anything counts as a proof that the supposedly unbiased fact checkers at PolitiFact are deliberately pulling the wool over the eyes of their readers, it is PolitiFact's unrepentant use of its aggregated ratings as voter guides.

They have to know better.

They do it anyway. And the practice was part of PolitiFact's aim from the start, which helps explain why it won't go away.


  1. Um. . . why would they rate any comment from Donald Trump in 2011? A better comparison would have been to an actual Republican leader in 2011. This was a decent post until that nonsense. So politifact is probably as biased as politifact (eye roll here).

  2. Happypessimist wrote:

    **Um. . . why would they rate any comment from Donald Trump in 2011?**

    They probably rated comments from Trump in 2011 (Obama's birth certificate, for example), but they wouldn't count as Republican statements for the sake of our data set. I think perhaps you don't understand the nature of the research.

    **A better comparison would have been to an actual Republican leader in 2011.**

    Our research does not compare any two leaders. It compares the PolitiFact view of the parties as a whole, discounting ratings resulting from a Republican attacking a Republican or a Democrat attacking a Democrat. Our point is that Trump exemplifies the selection bias in PolitiFact's ratings. Likely if Trump never ran for president the numbers wouldn't look much different. They'd find x "False" or "Pants on Fire" statements from Republicans with Trump or without. The question is why they found so few aside from those coming from Trump.

    It's just an illustration of selection bias. I think you tried to make it something other than what it was.


Thanks to commenters who refuse to honor various requests from the blog administrators, all comments are now moderated. Pseudonymous commenters who do not choose distinctive pseudonyms will not be published, period. No "Anonymous." No "Unknown." Etc.