The "Pants on Fire" bias study updated through 2023

 We have updated our "Pants on Fire" bias study with data from 2023.

What is it? We use a spreadsheet to track all "False" and "Pants on Fire" ratings given to partisan Republicans or Democrats whether candidate, officeholder or appointed administration official plus party officials or organizations. We then calculate the percentage of false ("False" plus "Pants on Fire") ratings given the "Pants on Fire" rating.

Why do we do it? Because PolitiFact has never offered an objective means of distinguishing its "False" rating from its "Pants on Fire" rating, we infer that the difference is either substantially or wholly subjective. Assuming the substantial subjectivity of the ratings, we expect that differences in the percentages will help identify PolitiFact's partisan bias, if any.

Here's the updated chart:

What have we learned so far?

We've learned that national PolitiFact after 2007 shows a consistent bias for Democrats/against Republicans. That trend shows poorly on the graph above because this graph includes ratings from PolitiFact's various state operations. Before PolitiFact changed its website making it far less clear which franchise was responsible for what, we kept track of each part of the organization separately. The years from 2010 through 2015 show a moderation of bias thanks to state operations that sometimes were legitimately tough on Democrats. PolitiFact Wisconsin was notably tough on Democrats during that period, for example.

By looking at the total number of various ratings given to the political parties, we've also noted that Republicans (after 2007) receive far more of PolitiFact's bottom two ratings. That effect may stem from Republicans lying more or simply because of bias in story selection and ratings. We've documented enough of the latter two factors to reasonably prefer the second option. That's where the evidence leads.

If, as the available evidence suggests, PolitiFact's "Pants on Fire" rating has no objective basis, "Republicans lie more" carries no objective explanatory value respecting the percentages on our graph.

We've also learned that harsh ratings for both parties are on the decline, in terms of raw numbers. The most obvious explanation for that trend stems from PolitiFact's social media partnerships. If PolitiFact fact checks a politician, revenue consists of donations, grants and ad revenue. But if PolitiFact fact checks something for its social media partners, there's a payday for that. PolitiFact discloses that more than 5 percent of its revenue comes from the social media company Meta. The Chinese social media company TikTok likewise accounts for over 5 percent of PolitiFact's revenue.  

Why doesn't PolitiFact offer more transparency than that regarding its income? Good question, but we don't have an answer free of conjecture.

As for our study of PolitiFact's numbers in 2023, the Republican average fell well below its historic norm, establishing an all-time low for the GOP. PolitiFact's ratings of Democrats pulled their historic average down for the eighth straight year.

A potential weird Trump effect?

The percentages for Republicans haven't really changed much over the years, defying the existence of any Trump effect in terms of increasing Republican dishonesty (in PolitiFact's data, anyway). But the percentages for Democrats have declined noticeably since around 2016 as Trump ascended politically.

Could Trump help explain an increase in Democratic Party honesty?

More likely those changes happen because the makeup of PolitiFact's franchises has shifted over time. State franchises no longer take the edge off the pro-Democrat bias of national PolitiFact. 

PolitiFact's "Pants on Fire" bias in 2022

Back in 2012, we started an ongoing study of PolitiFact's bias particular to its decisions on "False" versus "Pants on Fire" ratings, given that the difference counts to all appearances as "entirely subjective."

This post updates that research with observations about 2022.

The "Pants on Fire" Bias

PolitiFact has never described an objective grounds for deciding between a "False" rating and a "Pants on Fire" rating for its "Truth-O-Meter." Our research approach predicts that PolitiFact's bias will drive a preference for one party over the other in making those decisions. That bias we express as the "PoF Bias number," where 1.0 shows perfect balance between the two parties. A figure below 1.0 shows PolitiFact favoring Republicans and a figure over 1.0 shows PolitiFact favoring Democrats.

 

For 2022, PolitiFact scored its third-highest PoF Bias number since it started in 2007. Of note, these figures include all of PolitiFact's state franchises. Tracking PolitiFact National by itself, as we once did, shows a sharp lean to the left during the 2010-2015 period. State franchises provided the balance shown during that time. The more recent spikes in bias likely stem from a combination of new, more left-leaning franchises and fact checker zeal over President Trump.

The cumulative PoF Bias number stands at a relatively modest 1.33 even with recent left-leaning spikes. So. over its history PolitiFact is 33 percent more likely to give a claim PolitiFact deems false a "Pants on Fire" rating for a Republican compared to a Democrat. That figure for PolitiFact National from 2007 through 2019 was 56 percent.

That variation, by the way, supports the hypothesis that different PolitiFact fact checkers display differing trends in their fact-checking. If, for example, PolitiFact National leans more left than state franchises then increased control over those franchises by National should show increased left-leaning bias.

Trends

We observe a fascinating trend at PolitiFact toward lower numbers of fact checks for politicians. Probably thanks to the lure of social media dollars, PolitiFact's timeline over time shows increased checking of social media claims. That's understandable, as social media reimburse fact-checking partners like PolitiFact for their work.  We noted that trend in a separate dataset including PolitiFact's fact checks of all U.S. politicians. We also see it reflected in this study focused only on "False" and "Pants on Fire" ratings of politicians, though with a notable spike during the 2020 election year.

Why was PolitiFact able to find over 100 false claims from Democrats each year from 2010 through 2012 but unable to crack the peak of 68 in the years since? Though we've noticed PolitiFact claiming politicians have started communicating with greater care, we do not find that explanation plausible without specific supporting evidence. Selection bias likely serves as an adequate explanation. And PolitiFact's numbers show an increased reluctance to rate false statements from Democrats as "Pants on Fire."

In 2022 PolitiFact established a new low rate from Democrats by rating only 6.06 percent of Democratic Party false claims as "Pants on Fire." That edged marks of 6.12 percent for 2018 and 6.38 percent in 2020.

PolitiFact also gave Republicans "Pants on Fire" ratings for the lowest annual rate ever, at 21.7 percent. Of course the rate for Republicans receiving that subjective and severe rating was over three times greater than for Democrats.

Monthly Tracking

We experimented with creation of a monthly chart for 2022's numbers. We expected the chart to show reasonable stable trends in the numbers despite the relatively low number of ratings in 2022. Small datasets should show relatively greater variation with randomized sample sizes. PolitiFact, of course, makes no apparent attempt to randomize its dataset. Selection bias may explain the relatively stable numbers we see from the chart.

Research note: PolitiFact's 'Pants on Fire' bias through October 2022

We continue to update our research project examining PolitiFact's bias in applying its "Pants on Fire" rating.

The project notes that the difference between "False" and "Pants on Fire" ratings on PolitiFact's "Truth-O-Meter" counts as subjective. "False" statements are untrue, while "Pants on Fire" statements are untrue and "ridiculous." "Ridiculous" is a subjective judgment on its face, and PolitiFact has never published a description of the it suggesting otherwise.

Through the end of October 2022, PolitiFact had only given a Democratic Party officeholder/candidate/organization/appointee one "Pants on Fire" rating that counts toward our statistics. Our count omits claims from that attack the claimant's own political party. One of those occurred, published through PolitiFact New York.

As a result, we found PolitiFact nearly six times more likely to subjectively rate (what it views as) false claims from Republicans as "Pants on Fire" compared to such claims from Democrats.

Interestingly, PolitiFact National published its first "Pants on Fire" rating for a Democrat in 2022 not long after we tagged PolitiFact's Editor-in-Chief Angie Drobnic Holan in our tweet about this graph. A graph including the November stats as of today would put the 2022 PoF Bias number near 3.2--not the all-time record but instead solidly among the four highest bias measures taken since 2007.

Of course new ratings may move the numbers greatly by the end of the year.

Readers curious about the trends and details of the chart may wish to look at our explanation for the full chart at the end of 2021.

PolitiFact's "Pants on Fire" bias in 2021

As we noted in our post about the "Pants on Fire" research for 2020, we have changed the way we do the research.

PolitiFact revamped its website in 2020, and the update made it next to impossible to reliably identify which of PolitiFact's various franchises were responsible for a fact check. Instead of focusing on PolitiFact National, it makes more sense to lump all of PolitiFact together. But the new approach has a drawback. The new evaluations represent an apples-to-oranges comparison to the old evaluations.

To deal with that problem, we went back and did PolitiFact's entire history since 2007 using the new method.

With the research updated using the new method, we are now able to compare the new research with the old method.

Spoiler: Using the new method, PolitiFact was 2.66 times more likely to rule a claim it viewed as false as "Pants on Fire" from a Republican than for a Democrat. That's PolitiFact's third-highest bias figure of all time, though PolitiFact National, considered separately, has exceeded that figure at least three times.

 

Method Comparison: New vs. Old 

Our new graph shows the old method, running from 2007 through 2019, along with the new method graphed from 2007 through 2021.

The black line represents the old method. The red line represents the new.

The numbers represent the what we term the "PoF bias number," which is an expression of how much more likely it is that PolitiFact will a claim it regards as false a "Pants on Fire" rating for a Republican over a Democrat. So, for 2009 under the old method (black line), the GOP was 3.14 times more likely to have one of its supposedly false statements rated "Pants on Fire."

As our research has documented, PolitiFact has never offered an objective means of determining the ridiculousness of a claim viewed as false. The "Pants on Fire" rating, to all appearance, has to qualify as a subjective judgment. In other words, the rating represents PolitiFact's opinion.

In 2017, under the old method, the bias number dropped to 0.89, showing a bias against Democrats for that year at PolitiFact National. On average over time, of course, Republicans were significantly more likely to have their false claims regarded as "ridiculous" by PolitiFact.

Notably, the new method (red line) shows a moderating effect on PolitiFact's "Pants on Fire" bias from 2008 through 2014. The red line hovers near 1.00 for much of that stretch. After 2015 the red line tends to run higher than the black line, with the notable exception of 2019.

Explaining the Numbers?

We found two correlations that might help explain the patterns we see in the graphs.

PolitiFact changes over time. From 2007 through 2009, PolitiFact National did nearly every rating. Accordingly, the red and black lines track very closely for those years. But in 2010 PolitiFact added several franchises in addition to PolitiFact Florida. Those franchises served to moderate the PoF bias number until 2015, where we measured hardly any bias at all in the application of PolitiFact's harshest rating.

After 2015, a number of franchises cut way back on their contributions to the PolitiFact "database" and a number ceased operations altogether, such as PolitiFact New Jersey and PolitiFact Tennessee. And in 2016 PolitiFact added eight new state franchises (in alphabetical order): Arizona, Colorado, Illinois, Nevada, New York, North Carolina, Vermont and West Virginia.

The Franchise Shift

We made graphs to help illustrate the franchise shift. PolitiFact has had over 20 franchises over its history, so we'll divide the graph into two time segments to aid the visualization.

First, the franchises from 2010 through 2015 (click for larger view):

We see Florida, Texas, Rhode Island and Wisconsin established as consistent contributors. Tennessee lasts one year. Ohio drops after four years. Oregon drops after five and New Jersey after three.

Next, the franchises from 2016 through 2022 (click for larger view):

I omitted minor contributions from PolitiFact Georgia in 2016 (12) and 2017 (2). The orange bar near the top of 2016 is six states combined (hard to make out in the columns after 2016).

Note that the contributions are skinny, except for the one from Wisconsin. But even Wisconsin cut its output compared to the previous graph. We have a correlation suggesting that the participation of different state franchises impacted the bias measure.

But there's another correlation.

Republicans Lie More! Democrats Lie Less!

Liberals like to explain PolitiFact ratings that look bad for Republicans by saying that Republicans lie more. Seriously, they do that. But we found that spikes--especially recent ones--in the "Pants on Fire" bias measure were influenced by PolitiFact's spiking reluctance to give Democrats a "Pants on Fire" rating.

That correlation popped out when we created a graph showing the percentage of false statement given the "Pants on Fire" rating by party. The graph for Republicans stays pretty steady between 20 and 30 percent. The graph for Democrats fluctuates wildly, and the recent spikes in the bias measure correlate with very low percentages of "Pants on Fire" ratings for Democrats.

As is always the case, our findings support the hypothesis that PolitiFact applies its "Pants on Fire" rating subjectively, with Republicans receiving the bulk of the unfair harm. And in this case Republicans receive the bulk of the unfair harm through PolitiFact's avoidance of rating Democrat claims "Pants on Fire."

Do Democrats lie less? We don't really know. We suspect not, given the number of Democrat whoppers PolitiFact allows to escape its notice (such as this recent gem--transcript). We think PolitiFact's bias explains the numbers better than the idea Democrats lie less.

Notes on the PolitiFact franchise numbers: As we noted from the outset, PolitiFact's revamped website made it all but impossible to identify which franchise was responsible for which fact check. So how did we get our numbers?

We mostly ignored tags such as "Texas" or "Wisconsin" and looked for the names of staffers connected to the partnered newsroom. This was a fallible method because the new-look website departs from PolitiFact's old practice of listing any staffers who helped write or research an article. The new site only lists the first one mentioned from the old lists. And it has long been the case that staffers from PolitiFact National would publish fact checks under franchise banners. So our franchise fact check numbers are best taken as estimates.

PolitiFact's "Pants on Fire" bias--2017 update (Updated)

What tale does the "Truth-O-Meter" tell?

For years, we at PolitiFact Bias have argued that PolitiFact's "Truth-O-Meter" ratings serve poorly to tell us about the people and organizations PolitiFact rates on the meter. But the ratings may tell us quite a bit about the people who run PolitiFact.

To put this notion into practice, we devised a simple examination of the line of demarcation between two ratings, "False" and "Pants on Fire." PolitiFact offers no objective means of distinguishing between a "False" rating and a "Pants on Fire" rating. In fact, PolitiFact's founding editor, Bill Adair (now on staff at Duke University) described the decision about the ratings as "entirely subjective."

Angie Drobnic Holan, who took over for Adair in 2013 after Adair took the position at Duke, said "the line between 'False' and 'Pants on Fire' is just, you know, sometimes we decide one way and sometimes decide the other."

After searching in vain for dependable objective markers distinguishing the "Pants on Fire" rating from the "False" rating, we took PolitiFact at its word and assumed the difference between the two is subjective. We researched the way PolitiFact applied the two ratings as an expression of PolitiFact's opinion, reasoning that we could use the opinions to potentially detect PolitiFact's bias (details of how we sorted the data here).

Our earliest research showed that, after PolitiFact's first year, Republicans were much more likely than Democrats to have a false claim rated "Pants on Fire" instead of merely "False." Adair has said that the "Pants on Fire" rating was treated as a lighthearted joke at first--see this rating of a claim by Democrat Joe Biden as an example--and that probably accounts for the unusual results from 2007.

In 2007, the lighthearted joke year, Democrats were 150 percent more likely to receive a "Pants on Fire" rating for a false statement.

In 2008, Republicans were 31 percent more likely to receive a "Pants on Fire" rating for a false statement.

In 2009, Republicans were 214 percent more likely to receive a "Pants on Fire" rating for a false statement (not a typo).

In 2010, Republicans were 175 percent more likely to receive a "Pants on Fire" rating for a false statement (again, not a typo).

We published our first version of this research in August 2011, based on PolitiFact's first four years of operation.

In 2011, Republicans were 57 percent more likely to receive a "Pants on Fire" rating for a false statement.

In 2012, Republicans were 125 percent more likely to receive a "Pants on Fire" rating for a false statement.

Early in 2013, PolitiFact announced Adair would leave the project that summer to take on his new job at Duke. Deputy editor Angie Drobnic Holan was named as Adair's replacement on Oct. 2, 2013.

In 2013, the transition year, Republicans were 24 percent more likely to receive a "Pants on Fire" rating for a false statement.

Had Republicans started to curb their appetite for telling outrageous falsehoods?

In 2014, Republicans were 95 percent more likely to receive a "Pants on Fire" rating for a false statement.

In 2015, Republicans were 2 percent (not a typo) more likely to receive a "Pants on Fire" rating for a false statement.

In 2016, Republicans were 17 percent more likely to receive a "Pants on Fire" rating for a false statement.

In 2017, Democrats were 13 percent more likely to receive a "Pants on Fire" rating for a false statement.

Had Republicans gotten better than Democrats at reigning in their impulse to utter their false statements in a ridiculous form?

We suggest that our data through 2017 help confirm our hypothesis that the ratings tell us more about PolitiFact than they do about the politicians and organizations receiving the ratings.

Do the data give us trends in political lying, or separate journalistic trends for Adair and Holan?

We never made any attempt to keep our research secret from PolitiFact. From the first, we recognized that PolitiFact might encounter our work and change its practices to decrease or eliminate the appearance of bias from its application of the "Pants on Fire" rating. We did not worry about it, knowing that if PolitiFact corrected the problem it would help confirm the problem existed  regardless of what fixed it.

Has PolitiFact moderated or fixed the problem? Let's look at more numbers.

The "Pants on Fire" bias

From 2007 through 2012, PolitiFact under Adair graded 29.2 percent of its false claims from the GOP "Pants on Fire." For Democrats the percentage was 16.1 percent.

From 2014 through 2017, PolitiFact under Holan graded 26 percent of its false claims from the GOP "Pants on Fire" and 21.9 percent for Democrats.

It follows that under Adair PolitiFact was 81.4 percent more likely to give a "Pants on Fire" rating to a false GOP statement than one for a Democrat. That includes the anomalous 2007 data showing a strong "Pants on Fire" bias against Democrats.

Under Holan, PolitiFact was just 18.7 percent more likely to give a "Pants on Fire" rating to a false GOP statement than one from a Democrat.

Story selection bias

While tracking the percentage of false ratings given a "Pants on Fire" rating, we naturally tracked the sheer number of times PolitiFact issued false ratings (either "False" or "Pants on Fire"). That figure speaks to PolitiFact's story selection.

From 2007 through 2012, PolitiFact under Adair found an average of 55.3 false claims per year from Republicans and 25.8 false claims per year from Democrats. That includes 2007, when PolitiFact was only active for part of the year.

From 2014 through 2017, PolitiFact under Holan found an average of 81 false claims per year from Republicans and 16 false claims per year from Democrats.

Under Holan, the annual finding of false claims by Republicans increased by nearly 58 percent. At the same time, PolitiFact's annual finding of false claims by Democrats fell by 38 percent.

Update Jan. 1, 2018: GOP false claims reached 90 by year's end.

One might excuse the increase for the GOP by pointing to staff increases. But the same reasoning serves poorly to explain the decrease for the Democrats. Likewise, increased lying by Republicans does not automatically mean Democrats decreased their lying.

Did the Democrats as a party tend strongly toward greater truth-telling? With the notable blemish a greater tendency to go "Pants on Fire" when relating a falsehood?

Conclusion

We suggest that changes in PolitiFact's practices more easily make sense of these data than do substantial changes in the truth-telling patterns of the two major U.S. political parties. When Adair stepped down as PolitiFact's editor, a different person started running the "star chamber" meetings that decide the "Truth-O-Meter" ratings and a different set of editors voted on the outcomes.

Changing the group of people who decide subjective ratings will obviously have a substantial potential effect on the ratings.

We suggest that these results support the hypothesis that subjectivity plays a large role in PolitiFact's rating process. That conclusion should not surprise anyone who has paid attention to the way PolitiFact describes its rating process.

Has Holan cured PolitiFact of liberal bias?

We recognized from the first that the "Pant on Fire" bias served as only one measure of PolitiFact's ideological bias, and one that PolitiFact might address. Under Holan, the "Pants on Fire" bias serves poorly to demonstrate a clear ideological bias at PolitiFact.

On the other hand, PolitiFact continues to churn out anecdotal examples of biased work, and the difficulty Holan's PolitiFact has in finding false statements from Democrats compared to Adair's PolitiFact suggests our data simply show something of a trade-off.

When we started evaluating PolitiFact's state operations, such as PolitiFact Georgia, we noticed that lopsided numbers of false statements were often accompanied by a higher percentage of "Pants on Fire" statements from the party receiving many fewer false ratings. We hypothesized a compensatory bias might produce that effect when the fact checkers, consciously or unconsciously, encourage the appearance of fairness.

PolitiFact, after all, hardly needs to grade false Republican statements more harshly to support the narrative that Republicans lie more when it is finding, on average, five times more false statements from Republicans than Democrats.

We doubt not that defenders of PolitiFact can dream up some manner of excusing PolitiFact based on the "fact' that Republicans lie more. But we deeply doubt that any such approach can find a basis in empirical evidence. Subjective rating systems do not count as empirical evidence of the rate of lying.

In addition to empirically justifying the increase in GOP falsehoods, defenders will need to explain the decrease in Democratic Party falsehoods implied in PolitiFact's ratings. Why, with a bigger staff, is PolitiFact having a more difficult time finding false statements from Democrats than it did when Adair was steering the ship?

If Truth-O-Meter data were ostensibly objective, it would make sense to question the reliability of the data given the differing trends we see for PolitiFact under Adair and Holan.

Given PolitiFact's admissions that its story selection and ratings are substantially subjective, it makes sense for the objective researcher to first look to the most obvious explanation: PolitiFact bias. 

 

Notes on the research method

Our research on the "Pants on Fire" bias looks at partisan elected officials or officeholders as well as candidates and campaign officials (including family members participating in the campaign). We exclude PolitiFact ratings where a Republican attacked a Republican or a Democrat attacked a Democrat, reasoning that such cases may muddy the water in terms of ideological preference. The party-on-party exclusions occur rarely, however, and do not likely affect the overall picture much at all.

In the research, we use the term "false claims" to refer to claims PolitiFact rated either "False" or "Pants on Fire." We do not assume PolitiFact correctly judged the claims false.

Find the data spreadsheet here.


Afters

We have completed alternative versions of our charts with the data for Donald Trump removed, and we'll publish those separately from this article at a later time. The number of false claims from Republicans went down from 2015-2017 but with PolitiFact still issuing far more false ratings to Republicans. The "Pants on Fire" percentages were almost identical except for 2016. With Trump removed from the data the Republicans would have set an all-time record for either party for lowest percentage of "Pants on Fire" claims.

These results remain consistent with our hypothesis that PolitiFact's "False" and "Pants on Fire" ratings reflect a high degree of subjectivity (with the former perhaps largely influenced by story selection bias).



Update Dec. 19, 2017: Added intended hyperlink to explanations of the research and the notable Biden "Pants on Fire."
Update Dec. 21, 2017: Corrected date of previous update (incorrectly said Dec. 12), and updated some numbers to reflect new PolitiFact ratings of Donald Trump through Dec. 21, 2017: "13 percent"=>"10 percent",  "87.3 claims per year"=>"80.5 claims per year", "23.8"=>"26.1" and "8.7"="19.2." The original 87.3 and 23.8 figures were wrong for reasons apart from the new data. We will update the charts once the calendar year finishes out. Likewise the 8.7 figure derived in part from the incorrect 23.8.
Update Jan 1, 2017:  Changed "10 percent" back to "13 percent" to reflect updated data for the whole year. "80.5 claims per year" updated to "81 claims per year." We also changed "26.1" to "26" and "8.7" to "18.7." The latter change shows that we neglected to make the "8.7" to "19.2" change we announced in the description of the Dec. 21, 2017 update, for which we apologize.

PolitiFact revises its own history

We remember.

We remember when PolitiFact openly admitted that when Republicans charged that Obamacare cut Medicare it tended to rate such claims either "Half True" or "Mostly False."

PolitiFact has looked askance at bare statements that Obamacare cuts Medicare, rating them either Half True or Mostly False depending on how they are worded.

Nowadays, PolitiFact has reconsidered. It now says it generally rated claims that Obamacare cut Medicare as "Half True."

We did a series of fact-checks about the back-and-forth and generally rated the Republican attacks Half True.

PolitiFact doubtless took the latter position in response to criticism of its claims about Republican "cuts" to Medicaid. PolitiFact has flatly said Trump's budget cuts Medicaid (no caveats) despite the fact that outlays for Medicaid rise every year under the Trump budget. PolitiFact has also joined the mainstream media in attacking the Trump administration for denying it cuts Medicaid, rating those statements "Mostly False" or worse.

Given the context, PolitiFact's fact check of Kellyanne Conway looks like a retrospective attempt to excuse PolitiFact's inconsistency.

Unfortunately for PolitiFact writer Jon Greenberg, the facts do not support his defense narrative. The "Half True" ratings he cites tended to come from joint claims, that 1) Obamacare's Medicare cuts went to 2) pay for the Affordable Care Act. It's completely true that the ACA used Medicare savings to cut the price tag for the legislation. Every version of the CBO's assessment of the Democrats' health care reform bill will confirm it.

When PolitiFact rated isolated GOP claims that Democrats' health care reform cut Medicare, the verdict tended to come in as "Mostly False." It wasn't even close. Keep reading below the fold to see the proof.

So, PolitiFact writer Jon Greenberg either isn't so great at checking his facts, or else he is deliberately misleading his audience. The same goes for the editor, Angie Drobnic Holan.

A reader's take on our "Pants on Fire" research

Elizabeth MacInnis wrote:

Your premise here is that if there are an equal amount of fact checks and Republicans lie more, then there must be fact-checker bias. It couldn't possibly be that Republicans actually do lie more. That's an unobjective analysis. In my opinion, watching both sides closely in each election, Republicans do lie more, but not for the reason that you think. Democrats typically run on a platform of hope and ideas. It's much more subjective. Republicans tend to run on fear and attacks against other candidates (whether you like this or not, it's true). An attack on someone's record is more likely to be proven true or false. Watching all the debates, I consistently hear Republicans say things like "we've had a job killing president," when in reality (as of now) we've had 72 months of private-sector job growth, a record. With consistent, clearly false statements like this, Republicans are doing it to themselves. I believe we need a balanced system with both parties, but in many Americans' opinions (including mine), Republicans have become more and more outrageous in recent years (and no, that is not to say that Democrats are clear of any wrongdoing). The only way, in my opinion, to tame this is to hold them accountable - in fact-checking and votes. I hope both parties learn this lesson before their groups become too fractured.

Point by point:

"Your premise here is that if there are an equal amount of fact checks and Republicans lie more, then there must be fact-checker bias."

No, that's not our premise. Our premise (the one you seem to be talking about) is that if PolitiFact chose its stories only based on its editorial sense of whether the claim is true, then the results would be proportional. So Republicans could have five times more "false" ratings than Democrats but the distribution curve for both parties should appear similar. We don't think PolitiFact uses only its editorial sense in choosing stories.

The true central premise of the "Pants on Fire" research is that PolitiFact offers no objective means of distinguishing between its ratings of "False" and "Pants on Fire."

"It couldn't possibly be that Republicans actually do lie more. That's an unobjective analysis."

Can anyone explain to me how the "Pants on Fire" rating, with no apparent objective measure undergirding its use, contributes any empirical data toward the notion that Republicans lie more? Isn't that notion a sham?

"In my opinion, watching both sides closely in each election, Republicans do lie more, but not for the reason that you think. Democrats typically run on a platform of hope and ideas. It's much more subjective. Republicans tend to run on fear and attacks against other candidates (whether you like this or not, it's true)."

Summing up, then, MacInnis' opinion is true whether I like it or not? Is there any solid evidence supporting that opinion? Any at all?



(skipping some opinion that doesn't interest me so much)

"Watching all the debates, I consistently hear Republicans say things like "we've had a job killing president," when in reality (as of now) we've had 72 months of private-sector job growth, a record."

Remember when the Obama administration was lauding the helpful effects of the $900 billion stimulus bill? Employment was going down, but PolitiFact accepted arguments that unemployment would be even worse without the stimulus bill. Why is it that PolitiFact gives no consideration at all to a parallel principle with respect to claims of job-killing? That's not consistent, is it? We shouldn't make the mistake of thinking that inconsistent methods lead to good fact-checking, should we?

Elizabeth MacInnis, PolitiFact does fact-checking poorly. Don't be fooled.

PolitiFact's subjective "Pants on Fire" ratings tell us about PolitiFact, not about the entities receiving the subjective ratings.


 

The 2015 "Pants on Fire" bias for PunditFact and the PolitiFact states

Earlier this week we published our 2015 update to our study of PolitiFact's bias in applying its "Pants on Fire" rating.

The premise of the research, briefly, is that no objective criterion distinguishes between a "False" rating and a "Pants on Fire" rating. If the ratings are subjective then a "Pants on Fire" rating provides a measure of opinion and nothing more.

In 2015 the states provided comparatively little data. State franchises, with a few exceptions, seem to have a tough time giving false ratings. The state PolitiFact operations also tend to vary widely in the measurement of the "Pants on Fire" bias. PolitiFact Wisconsin's "Pants on Fire" ratings proportionally treat Democrats more harshly than Republicans, for example.

PolitiFact Florida: PolitiFact Florida's data roughly matched those from PolitiFact National and PunditFact. Those three franchises are the most closely associated with one another since all are based in Florida and tend to share writing and editorial staff. In 2015 the PoF bias number was within a range of five hundredths for each. That's so close that it's suspicious on its face. All three gave Republicans more false ratings than Democrats (4.83, 3.50, 2.43).

PolitiFact Georgia: Though PolitiFact Georgia has operated for a good number of years and has in the past provided us with useful data, that wasn't the case in 2015. PolitiFact Georgia's false ratings went to apparently non-partisan claims.

PolitiFact New Hampshire: PolitiFact New Hampshire historically provides virtually nothing helpful in terms of the PoF bias number. But false ratings for Democrats outnumbered false ratings for Republicans (blue numerals indicate that bias).

PolitiFact Rhode Island: PF Rhode Island rated two statements from Democrats "False."

PolitiFact Texas: PF Texas gave Republicans false ratings an astonishing 8 times more than Democrats. But at the same time, PF Texas produced a PoF bias number harming Democrats. The key to both figures? PF Texas only doled out two false ratings to partisan Democrats. Both were "Pants on Fire" ratings. An entire year with no "False" ratings for Democrats? Texas' previous low for "False" ratings was four (twice).

PolitiFact Virginia: PF Virginia achieved perfect neutrality in terms of our PoF bias number. That's the meaning of a 1.00 score. The "Pants on Fire" claims as a percentage of all false claims was equal for Republicans and Democrats.

PolitiFact Wisconsin: PF Wisconsin continued its trend of giving Democrats the short end of the PoF bias measure. That's despite giving Republicans a bigger share than usual of the total false ratings. The 5.00 selection bias number was easily the all-time high for PF Wisconsin, besting the old mark of 1.57 back in 2011.

PunditFact: PunditFact, we should note, produces data we class in our "Group B." PunditFact tends not to rate partisan candidates, officeholders, partisan appointees or party organizations. It focuses more on pundits, as the name implies. We consider group B data less reliable as a measure of partisan bias than the group A data. But we do find it interesting that PunditFact's data profile lines up pretty closely with the most closely associated PolitiFact entities, as noted above. That finding proves consistent with the idea that PolitiFact ratings say something about the viewpoint of the ones giving the ratings.

PolitiFact's "Pants on Fire" bias, 2015 update

It's time again for our annual update to our research on PolitiFact's bias in applying its "Pants on Fire" rating. And the results for 2015 show a surprisingly low bias against Republicans. Read on.

PolitiFact's "Pants on Fire" bias

As we have noted, PolitiFact has failed to ever provide any objective means for distinguishing between its "False" and "Pants on Fire" ratings on its trademarked "Truth-O-Meter." The only difference between the two ratings by PolitiFact's telling is that "Pants on Fire" statements are ridiculous as well as false.

We did an extensive survey of the reasons PolitiFact has given in its stories for its ratings and failed to even find an informal criterion that might pass as objective.

This research project does not focus on whether Republicans simply receive more "Pants on Fire" ratings than Democrats. We look at proportions, not the raw numbers. We look at the percentage of total false ratings ("False" plus "Pants on Fire") and then divide the number of "Pants on Fire" ratings for each party by that number. Comparing the percentages for each party then gives us a number we call the "PoF bias number." And what is the significance of that number?

The PoF bias number shows which political party is more likely to receive a "Pants on Fire" rating. If the difference between "False" and "Pants on Fire" is subjective, as PolitiFact's definitions and our research appear to indicate, the "Pants on Fire" rating shows which party is more likely to receive the subjective "Pants on Fire" rating.

In 2011, the first year we did the study, Republican statements were 57 percent more likely to receive a "Pants on Fire" rating than statements from Democrats. As the chart below shows, 1.57 serves as the corresponding PoF bias number.

In PolitiFact's very first year, Democrats received a higher percentage of "Pants on Fire" ratings. It turned out PolitiFact's founding editor Bill Adair said the "Pants on Fire" rating started out for use on "light-hearted" fact checks. Ever since "Pants on Fire" stopped being a joke at PolitiFact, Republicans have been more likely to have their false statements ruled as "Pants on Fire."

Republicans' false statements were far more likely than Democrats' to receive a "Pants on Fire" rating from 2009 through 2012.

	(PoF	Bias)	(Sel.	Bias)
YEAR	GOP	Democrat	GOP	Democrat
2007		2.50	1.00	1.00
2008	1.31		1.53
2009	3.14		1.48
2010	2.75		3.15
2011	1.57		3.83
2012	2.25		2.52
2013	1.24		1.81
2014	1.95		2.56
2015	1.10		4.83

Chart notes: Columns 2 and 3 deal with the PoF bias number. Columns 3 and 4 deal with the "selection bias number," which is the comparison between Republicans and Democrats of the total number of false claims.


Our results for 2015 proved intriguing.

Republicans' false statements were only 10 percent more likely than Democrats' false statements to receive the "Pants on Fire" rating. In terms of historical patterns, the bias against Republicans in 2015 was minimal.

But given the current mainstream media narrative that the Republicans wantonly lie and do so perhaps the most in history, how should we interpret the data?

The Matter of Interpretation

We've had critics insist that a high PoF bias number against Republicans is best interpreted as a sign that Republicans simply lie more. For those critics, this year's findings present a problem. Is the supposed trend of Republicans lying tailing off?

We'd love to see our critics try to make that argument.

We would suggest two hypotheses to help explain the numbers.

First, we would suggest the numbers do not necessarily need much explanation. Simple regression to the mean may explain the surprisingly low measurement of anti-Republican bias in applying the "Pants on Fire" rating. The PoF Bias number for the past three years, after all, falls very close to the average for the preceding years. However, we looked for an alternative explanation anyway because we should predict that the "Republicans lie more" narrative would make the recent numbers look worse than usual for Republicans if PolitiFact journalists allow ideological bias to affect their ratings.

The Secondary Hypothesis

Our secondary hypothesis came from our past analysis of the PoF bias number for PolitiFact's state operations.

Some of the PolitiFact states had PoF bias numbers favoring Republicans instead of Democrats despite paying relatively little attention to statements from Democrats. We hypothesized some PolitiFact states might be applying a compensatory bias: If a PolitiFact state was grading many Republican statements as "False" compared to very few statements from Democrats, the state PolitiFact might rate the few Democrat statements more harshly to look more fair.

So, we hypothesized that PolitiFact states might go harder on false statements from Democrats to make the large number of false statements from Republicans look less like a sign of journalistic bias.

As a corollary to the secondary hypothesis, we noted with our first publication of our research that PolitiFact staffers could read our research and act to change future outcomes. That possibility does not concern us much, for if the bias in applying the "Pants on Fire" rating stabilizes near a more equitable level, it tends to support our interpretation of the data for PolitiFact's earlier years.

Are Democrats Telling the Truth More and More?

Though our research project focuses on the percentages of "Pants on Fire" ratings compared to the total number of statements PolitiFact views as false (making the number totals unimportant to the PoF bias research), we end up collecting data on the total number of false statements as a matter of course.

Those data do appear to tell an interesting story: PolitiFact National seems to have increasing difficulty in rating Democrats' statements false.

Note that we sift the PolitiFact ratings to obtain the data most likely to accurately measure bias.  We're talking about the "Group A" data described in our first research paper, which consists of ratings of party candidates, elected officeholders, partisan appointees or party organizations. We exclude statements obviously attacking members of one's own party. Still, the shrinking number of false statements for Democrats stands out:

(Numbers exclude party-on-party claims, as where a Democrat attacks another Democrat)

PolitiFact had little difficulty finding false statements from Democrats over its first three years. The low mark during those first three years was 18, when PolitiFact only operated for about half the year. Since that first year the number has declined or stayed even every year save for one. The number of false statements from Democrats skyrocketed from 20 all the way up to 23 in 2011 before resuming its decline.

Why is PolitiFact only finding half the false statements per year it found over the first three years? What explains that decline? Are Democrats edging closer to the truth on the whole? Just once per month, on average, PolitiFact was able to catch a full-fledged partisan Democrat telling a falsehood in 2015?

We doubt Democrats have significantly changed their approach to political speech. We think the explanation has to do with the fact checkers buying the idea that failing to let their bias lead them (they don't see it as bias) results in a "false equivalence" in fact checking. PolitiFact produces results consistent with what we should expect from left-leaning fact checkers.

PunditFact's "Pants on Fire" bias, 2014

We pledged to apply our "Pants on Fire" bias research methods to PolitiFact's "PunditFact" project.

PunditFact is the branch of PolitiFact that looks at and rates statements from pundits. PunditFact uses the same cheesy "Truth-O-Meter" system to which PolitiFact has wedded itself.

Our "Pants on Fire" research project looks at how PolitiFact disproportionately applies its "Pants on Fire" rating.

It's important to note that we don't simply look at the higher numbers of "Pants on Fire" ratings PolitiFact gives to Republicans and conservatives. Nor do we focus on which party receives more "False" ratings. We note that PolitiFact has never provided anything resembling an objective criterion for distinguishing between "False" and "Pants on Fire" ratings. We conclude from the evidence that PolitiFact very probably distinguishes between the two ratings subjectively. From that, we conclude that the proportion of "Pants on Fire" ratings in relation to the total number of false ("False" plus "Pants on Fire") ratings tells us something about the ideology of the people applying the ratings.

PunditFact shows a remarkable skew to the left.

PunditFact's PoF Bias number for 2014 came in at 2.19. That simply means PunditFact was 119 percent more likely to rate a conservative's false statement "Pants on Fire" than a liberal's.

Adding the small amount of data from the end of 2013, when PunditFact was first starting out, we obtain a cumulative figure of 2.57 for the PoF Bias number. So over PunditFact's entire lifespan, conservatives were 157 percent more likely than liberals to have a false statement rated "Pants on Fire."

We also have a hint in our data that PunditFact shows much more interest in statements coming from conservatives.

Of course we're only looking at two categories of statements, so we place no great importance on that aspect of the chart. We note, however, that PolitiFact's oft-stated criterion for choosing its subject matter ("Is that true?") fits just as well (if not better) for the hypothesis of liberal media bias than for the idea that conservatives simply lie more.

It's more natural to question statements that do not fit with what one accepts as true, after all.



Correction Jan 19, 2015: Revised 7th graf to correct description of cumulative totals.

PolitiFact still biased after all these years

It's time for the annual update to our ongoing research project measuring PolitiFact's bias in its application of "Pants on Fire" ratings.

In spite of its frequent publication and pimping of "report cards" showing how various persons and organizations rate on its trademarked "Truth-O-Meter," PolitiFact openly admits that its process for choosing which claims to rate is not scientific. PolitiFact maintains it is making no effort to figure out which persons or groups lie more, albeit at the same time publishing its stories in a way that encourages readers to draw such conclusions based on unscientific data.

We at PolitiFact Bias have helped pioneer the practice of using the fact checkers' rating systems to measure ideological bias in journalism. The "Pants on Fire" bias study was the first we published. It examines differences in how PolitiFact applies its "Pants on Fire" rating compared to its other rating for false statements, "False."

PolitiFact's liberal defenders have a ready defense when it turns out that PolitiFact shows much more enthusiasm for giving "Pants on Fire" ratings to Republicans than it does for Democrats. "Republicans simply tell the biggest lies," they say, or some such.

Not so fast, we say.

One PolitiFact critic insider called PolitiFact's ratings "coin flips." Current PolitiFact editor Angie Drobnic Holan recently described the difference between a "False" rating and a "Pants on Fire" rating:

"We have definitions for all of our ratings. The definition for "False" is the statement is not accurate. The definition for "Pants on Fire" is the statement is not accurate and makes a ridiculous claim. So, we have a vote by the editors and the line between "False" and "Pants on Fire" is just, you know, sometimes we decide one way and sometimes decide the other."

If awarding a claim a "Pants on Fire" rating was based on something objective, we suggest that somebody at PolitiFact could describe how that objective process operates. We've yet to see it in over seven years of observation.

Given PolitiFact's disinclination or inability to reveal any objective basis for the distinction between "False" and "Pants on Fire" statements, we conclude that the difference between the two is substantially or perhaps entirely subjective. So we compare the percentage of all false statements ("False" plus "Pants on Fire") rated "Pants on Fire" by party. It reveals that PolitiFact, after 2007, has consistently given false statements by Republicans "Pants on Fire" ratings at a much higher rate than those from Democrats.

And that, we claim, is an objective measure of ideological bias. The "Pants on Fire" rating amounts to an opinion poll of PolitiFact journalists as to which false ratings are ridiculous. PolitiFact subjectively finds the false claims of Republicans more ridiculous than those of Democrats.

It's not the only measure of ideological bias, of course, and it is subject to some limitations that we've described in earlier publications.

The new data for 2014

Our findings for 2014 proved very consistent with PolitiFact's performance from 2008 through 2013.

The PoF bias number for 2014 was 1.95, meaning a Republican's false statement, as designated by PolitiFact, was 95 percent more likely to receive a "Pants on Fire" rating than a false statement from a Democrat. That's nearly twice as likely.

The selection bias number for 2014 was 2.56, meaning the total number of false statements, "False" and "Pants on Fire" ratings combined, was 156 percent higher for Republicans than for Democrats.

Overall, that means PolitiFact rates more Republican statements false and rates Republican statements "Pants on Fire" at a much higher rate than for Democrats.

PolitiFact continued its recent trend of finding Democrats ever more truthful. Democrats tied their all-time record with only 16 statements found false, and improved on their performance in 2013 with only two of those 16 rated "Pants on Fire."

Republicans had also shown a recent trend toward fewer false ratings since a high mark in 2011, but that reversed in 2014. Still, Republicans posted their lowest percentage of "Pants on Fire" ratings since 2008, the year for which PolitiFact won its 2009 Pulitzer Prize.

The 1.95 PoF Bias number for 2014 boosted PolitiFact's cumulative figure a few hundredths to 1.74. Since its inception, PolitiFact has been 74 percent more likely to rate a false statement as "Pants on Fire" if the claim comes from a Republican instead of a Democrat.

Note

The above figures include only elected or appointed persons or party organizations such as the Democratic National Committee. We call these "Group A" figures and consider them the most reliable group in this study for showing partisan bias at PolitiFact.

We'll soon publish our findings for PunditFact, PolitiFact's effort focused on pundits, which represents "Group B" data for purposes of our research.

PolitiFact editor explains the difference between "False" and "Pants on Fire"

During an interview for a  "DeCodeDC" podcast, PolitiFact editor Angie Drobnic Holan explained to listeners the difference between the Truth-O-Meter ratings "False" and "Pants on Fire":



Our transcript of the relevant portion of the podcast follows, picking up with the host asking why President Barack Obama's denial of a change of position on immigration wasn't rated more harshly (bold emphasis added):

ANDREA SEABROOK
Why wouldn't that be "Pants on Fire," for example?

ANGIE DROBNIC HOLAN
You know, that's an interesting question.

We have definitions for all of our ratings. The definition for "False" is the statement is not accurate. The definition for "Pants on Fire" is the statement is not accurate and makes a ridiculous claim. So, we have a vote by the editors and the line between "False" and "Pants on Fire" is just, you know, sometimes we decide one way and sometimes decide the other. And we totally understand when readers might disagree and say "You rated that 'Pants on Fire.' It should only be 'False.'" Or "You rated that 'False." Why isn't it 'Pants on Fire'?" Those are the kinds of discussions we have every day ...

One branch of our research examines how PolitiFact differentially applies its "Pants on Fire" definition to false statements by the ideology of the subject. Holan's description accords with other statements from PolitiFact regarding the criteria used to distinguish between "False" and "Pants on Fire."

Taking PolitiFact at its word, we concluded that the line of demarcation between the two ratings is essentially subjective. Our data show that PolitiFact National is over 70 percent more likely to give a Republican's false statement a "Pants on Fire" rating than a Democrat's false statement.

We don't necessarily agree with PolitiFact's determinations of what is true or false, of course. What's important to our research is that the PolitiFact editors doing the voting believe it.

Holan's statement helps further confirm our hypothesis regarding the subjective line of demarcation between "False" and "Pants on Fire."

We'll soon publish an update of our research, covering 2014 and updating cumulative totals.

PolitiFact, percent error, partisanship

In last Sunday's post about PolitiFact's math on uninsured Americans, we mentioned the worth of tracking PolitiFact's application of a basic math equation used to calculate error percentages.

To calculate the percent error, one takes the difference between the right figure and the wrong figure, and divides the difference by the right figure.

For example, if we guess Scarlett Johansson weighs 175 lbs and her actual weight is 100 lbs, we take the difference, 75, and divide it by the correct figure, 100.  That gives us 75 percent, so that's the percentage error.  It is incorrect to divide by the incorrect estimate, 175 lbs.  That procedure ends up giving us a figure of about 43 percent, which is the wrong figure for the percentage error even if the calculation is performed correctly.  It's the wrong formula.  Our estimate was off by 75 percent, not 43 percent.

We propose, given the difficulty journalists often display with math problems, that this calculation can serve as an indicator of ideological bias.  Where a journalist performs the wrong equation to the benefit of one political party over the other, we have an evidence of ideological bias.

The Experiment

We used the search string "off by" and the term "percentage" and performed a Google search limited to the politifact.com domain.  We then combed the results for the target results: instances of percentage error calculations.

We found only 14, which is a low number for a study.  Two of the fact checks had two (wrong) calculations in them, but we count each fact check as just one case.

Out of 14 equations, PolitiFact performed the wrong calculation an appalling nine times.  Over half.

Good Fortune for Democrats

Since there are two ways to perform the percentage error calculation, albeit one is the wrong way, we tracked both methods for their correlation with benefit or harm to political parties.  The reason is simple.  One of the calculations will normally minimize the error compared to the other calculation.  If Republicans get the wrong calculation every time the wrong calculation minimizes the error and the right calculation every time it likewise minimizes the error then just counting the number of times the wrong formula helped Republicans doesn't adequately tell the story.  The other half of the story is the uncanny ability to use the right equation when it helps Republicans.

Does anybody sincerely expect PolitiFact to help Republicans?  Do you work for Media Matters or what?

Wrong calculation helped Dem., harmed Rep.	6
Wrong calculation helped Rep., harmed Dem.	3
Right calculation helped Dem, harmed Rep.	3
Right calculation helped Rep., harmed Dem.	0

Each of the nine times PolitiFact used the percentage error concept for a fact check of a Democrat in our sample, the Democrat received the calculation that provided the greatest benefit.  That's not counting the two ambiguous cases. [incorrectly phrased; ignore--ed.]

Where the type of calculation made a noticeable difference in the degree of error, the liberal point of view got the benefit 75 percent of the time.

Three times PolitiFact aided Republicans with the wrong calculation.  One case occurred in a fact check by an intern.  Another case helped out Republican Allen West by a measly 2 percentage points.  The third case helped a Republican criticizing the influence of a special interest group on an election.  In other words, the truth is even worse than the table makes it look.


Correction 5/29/2014:  Allen West received the benefit of two measly percentage points, not Herman Cain as I originally wrote.  Cain occurs in our data as a case of harm from the wrong calculation.


Data notes