PolitiFact's "Pants on Fire" bias in 2020 [Updated Dec. 31, 2020]

Readers, please do not neglect the Dec. 31, 2020 update near the bottom of the post. Thanks!

 

As we noted in a post one week ago, we changed how we're conducting the "Pants on Fire" bias study to include all of PolitiFact and not just PolitiFact National.

In the past, that might have meant a softening of the liberal bias we find at PolitiFact. But the data appear to show that the state franchises have shifted left to more closely match the leftward lean at PolitiFact National.

The study looks at the percentage of false statements (that's "False" plus "Pants on Fire") that PolitiFact labels "Pants on Fire" for Republicans and Democrats. We count candidates, partisan elected officials or partisan political appointees. Attorney General William Barr, for example, would count as a Republican while holding the AG office under a Republican administration but not as a civilian outside the government.

In 2020, through today, PolitiFact was 4.61 times more likely to rate a claim it regarded as false as "Pants on Fire" if it came from a Republican instead of a Democrat. [Note: 12-31-2020: These numbers may reflect a minor transcription error for the number of "Pants on Fire" claims and probably slightly exaggerate PolitiFact's left-leaning bias. See update below]

Some may think to ask, "Why would that mean PolitiFact is biased? Maybe Republicans just lie more."

It counts as bias because all the evidence shows PolitiFact's "Pants on Fire" rating is a subjective judgment. PolitiFact has never offered a justification for the distinction between the two ratings that runs any deeper than because we felt like it.

PolitiFact defines a "False" rating as "The statement is not accurate."

PolitiFact defines a "Pants on Fire" rating as "The statement is not accurate and makes a ridiculous claim."

We posit that "ridiculous" is not an objective measure unless somebody makes the effort to define it in objective terms. We find an absence of that effort over the whole of PolitiFact's history.

We also note that PolitiFact's founding editor Bill Adair and current Editor-in-Chief Angie Drobnic Holan have made statements that appear to admit the "Pants on Fire" rating is subjective.

Unless an objective basis exists for the "Pants on Fire" rating, the "Republicans lie more" premise does not help explain it except in terms of confirmation bias.

Inside the Numbers

Republican claims PolitiFact regarded as false were rated "Pants on Fire" 28.8 percent of the time. That's only very slightly above the average PolitiFact National established between 2007 and 2019. So why was the bias measurement so much higher this year?

This: PolitiFact was extremely reluctant to give a "Pants on Fire" rating to a Democrat.

PolitiFact only issued three "Pants on Fire" ratings to Democrats in 2020. That's out of 48 claims it regarded as false, resulting in a figure of 6.25 percent. That's far below the PolitiFact National average for Democrats between 2007 and 2019 (about 17 percent).

This Was Predictable

When PolitiFact unveiled and subsequently developed its "Truth-O-Meter" rating system, we saw the day coming when the ratings would clearly reveal bias. We expected to see similar claims receiving different ratings. And once we identified the very likely subjective nature of the "Pants on Fire" rating, we anticipated we would be able to produce a solid evidence of bias using analysis of PolitiFact's ratings.

The problem for PolitiFact stems from the fact that it is impossible to assign ratings objectively and subjectively at the same time and in the same sense.

We have sent our 2020 findings to the International Fact-Checking Network, along with the suggestion that it examine the compatibility between subjective fact checker rating systems and its stipulation (2.1) requiring fact checkers to rate claims by the same standard no matter the source of the claim.

We think either the IFCN must erase that stipulation from its requirements or else fact checkers seeking IFCN verification need to abandon the use of subjective rating systems.

Update Dec. 22, 2020

A new "Pants on Fire" rating for Claudia Tenney (R-NY) brings the PoF percentage for Republicans up to 29.1 percent from the 28.8 percent reported above.

Here's a chart based on the updated data:

Update Dec. 31, 2020

PolitiFact North Carolina came through with a late "Pants on Fire" rating that changes our numbers yet again, and helped bring to our attention a transcription error that probably affected the percentages reported above. We added notes in the above text to highlight that imprecision.

Still, the late "Pants on Fire" rating to the North Carolina Republican Party, which will soon get its own write up here at PFB, boosts the GOP percentage above what we have on our graph to 29.2 percent. That means falsehoods PolitiFact regarded as "Pants on Fire" were 4.58 times more likely to receive a "Pants on Fire" rating coming from a Republican compared to a Democrat.