PolitiFact's "Pants on Fire" bias in 2022

Back in 2012, we started an ongoing study of PolitiFact's bias particular to its decisions on "False" versus "Pants on Fire" ratings, given that the difference counts to all appearances as "entirely subjective."

This post updates that research with observations about 2022.

The "Pants on Fire" Bias

PolitiFact has never described an objective grounds for deciding between a "False" rating and a "Pants on Fire" rating for its "Truth-O-Meter." Our research approach predicts that PolitiFact's bias will drive a preference for one party over the other in making those decisions. That bias we express as the "PoF Bias number," where 1.0 shows perfect balance between the two parties. A figure below 1.0 shows PolitiFact favoring Republicans and a figure over 1.0 shows PolitiFact favoring Democrats.

 

For 2022, PolitiFact scored its third-highest PoF Bias number since it started in 2007. Of note, these figures include all of PolitiFact's state franchises. Tracking PolitiFact National by itself, as we once did, shows a sharp lean to the left during the 2010-2015 period. State franchises provided the balance shown during that time. The more recent spikes in bias likely stem from a combination of new, more left-leaning franchises and fact checker zeal over President Trump.

The cumulative PoF Bias number stands at a relatively modest 1.33 even with recent left-leaning spikes. So. over its history PolitiFact is 33 percent more likely to give a claim PolitiFact deems false a "Pants on Fire" rating for a Republican compared to a Democrat. That figure for PolitiFact National from 2007 through 2019 was 56 percent.

That variation, by the way, supports the hypothesis that different PolitiFact fact checkers display differing trends in their fact-checking. If, for example, PolitiFact National leans more left than state franchises then increased control over those franchises by National should show increased left-leaning bias.

Trends

We observe a fascinating trend at PolitiFact toward lower numbers of fact checks for politicians. Probably thanks to the lure of social media dollars, PolitiFact's timeline over time shows increased checking of social media claims. That's understandable, as social media reimburse fact-checking partners like PolitiFact for their work.  We noted that trend in a separate dataset including PolitiFact's fact checks of all U.S. politicians. We also see it reflected in this study focused only on "False" and "Pants on Fire" ratings of politicians, though with a notable spike during the 2020 election year.

Why was PolitiFact able to find over 100 false claims from Democrats each year from 2010 through 2012 but unable to crack the peak of 68 in the years since? Though we've noticed PolitiFact claiming politicians have started communicating with greater care, we do not find that explanation plausible without specific supporting evidence. Selection bias likely serves as an adequate explanation. And PolitiFact's numbers show an increased reluctance to rate false statements from Democrats as "Pants on Fire."

In 2022 PolitiFact established a new low rate from Democrats by rating only 6.06 percent of Democratic Party false claims as "Pants on Fire." That edged marks of 6.12 percent for 2018 and 6.38 percent in 2020.

PolitiFact also gave Republicans "Pants on Fire" ratings for the lowest annual rate ever, at 21.7 percent. Of course the rate for Republicans receiving that subjective and severe rating was over three times greater than for Democrats.

Monthly Tracking

We experimented with creation of a monthly chart for 2022's numbers. We expected the chart to show reasonable stable trends in the numbers despite the relatively low number of ratings in 2022. Small datasets should show relatively greater variation with randomized sample sizes. PolitiFact, of course, makes no apparent attempt to randomize its dataset. Selection bias may explain the relatively stable numbers we see from the chart.