Changes to the "Pants on Fire Bias" study

Back in 2011 PolitiFact Bias started a study of one measure of PolitiFact's bias. Our study took PolitiFact's "Truth--O-Meter" ratings and looked at the differential between its application of the "False" rating and the "Pants on Fire" rating.

We had noted that the only difference PolitiFact advertised between a "False" rating and a "Pants on Fire" rating is that PolitiFact considers the latter "ridiculous" but not the former. So a "False" statement is a false statement and a "Pants on Fire" statement is a statement that is both false and ridiculous.

Save us your comments like "Well, to me a "Pants on Fire" means PolitiFact believes it was an intentional lie." That's not how PolitiFact has ever defined it.

We kept the study updated for every year from 2007 through 2019. For PolitiFact National, claims viewed as false by PolitiFact (rated either "False" or "Pants on Fire") were over 50 percent more likely to receive a "Pants on Fire" rating than one from a Democrat.

Because "ridiculous" seems like a subjective measure and we could find no unspoken or hidden objective rationale justifying a "Pants on Fire" rating, we judged it was likely a subjective measure. So PolitiFact's preference for doling out the "Pants on Fire" rating to Repubicans instead of the "False" rating, compared to the same measure for Democrats, we argue counts as a legitimate measure of political bias.

PolitiFact's various state operations have varied considerably from PolitiFact National in terms of their "Pants on Fire" bias measure. We say the variations between them support the hypothesis that the ratings are subjective.

Change Time

In 2020, PolitiFact revamped its website. Instead of publishing material from the state franchises to a special domain for each state, PolitiFact changed to a tagging system.

That was not good for our study.

PolitiFact did not, for example, reserve the tag "Wisconsin" for fact checks stemming from the staff at PolitiFact Wisconsin. Any post or article with content dealing with Wisconsin might receive that tag. That means that categorizing the data to match what we did from 2007 through 2019 would mean a giant headache.

To keep things simple, from 2020 onward we'll be lumping all of PolitiFact into one hopper. We will no longer track the state operations separately. That makes things easy. We just have to review that the claimant counts as a Republican or Democrat and log appropriately.

We don't think the trends will change much for all of PolitiFact compared to what we measured for PolitiFact National over the years. The operations that were kinder to Republicans have either shifted left (bubble effect?) or dropped out. We expect Repubicans to be about 50 percent more likely than a Democrat to receive a "Pants on Fire" rating for a statement deemed false. But we'll see.

Future PoF bias charts will not appropriately compare directly to those from the past, like this one:

 

That will be the final series graph from the first run of the study.

Afters:

It's worth noting again, we suppose, that "Republicans lie more!" does not help explain the disparity in the graph if our hypothesis about the subjectivity of the ratings is correct. We have no good reason for thinking it is incorrect.

The graph measures percentages, not raw numbers of ratings. Democrats making one "Pants on Fire" statement to go with nine "False" statements ends up as the same percentage as Republicans making 100 "Pants on Fire" statements to go with 900 "false" statements.