Wednesday, September 28, 2016

PolitiFact's presidential "Pants on Fire" bias

PolitiFact Bias has tracked for years a measure of PolitiFact's bias called the "Pants on Fire" bias. The presidential election gives us a fine opportunity to apply this research approach in a new and timely way.

This measure, based on PolitiFact's data, shows PolitiFact's strong preference for Democrat Hillary Clinton over the Republican candidate Donald Trump. When PolitiFact ruled claims from the candidates as false (either "False" or "Pants on Fire"), Trump was 82 percent more likely than Clinton to receive a "Pants on Fire" rating.

Why does this show a bias at PolitiFact? Because PolitiFact offers no objective means of distinguishing between the two ratings. That suggests the difference between the two ratings is subjective. "Pants on Fire" is an opinion, not a finding of fact.

When journalists call Trump's falsehoods "ridiculous" at a higher rate than Clinton's, with no objective principle guiding their opinions, it serves as an expression of bias.


How does the "Pants on Fire" bias measure work?

People often misunderstand our "Pants on Fire" bias measure. Some have suggested that our research simply echoes PolitiFact's implicit finding that Republicans lie more.

That isn't the case. Unless PolitiFact has secret objective criteria it uses to distinguish between a "False" rating and a "Pants on Fire" rating, PolitiFact applies the "Pants on Fire" rating subjectively. PolitiFact's statement of principles and the descriptions from its editors all point toward a subjective difference between the two ratings.

Here's how PolitiFact defines the two ratings:
FALSE – The statement is not accurate.
PANTS ON FIRE – The statement is not accurate and makes a ridiculous claim.
The sole difference between a "False" statement and a "Pants on Fire" statement, by PolitiFact's own description, is that the latter is ridiculous in addition to false.

As noted above, where PolitiFact has no principled means for applying the "Pants on Fire" ruling, the ratings are subjective.

The Clinton/Trump comparison shows that a false rating ("False" or "Pants on Fire") for Trump is 82 percent more likely to result in a "Pants on Fire" rating than for Clinton. This means when Clinton and Trump make a false claims, PolitiFact is 82% more likely to give Trump a "Pants on Fire" instead of a standard "False."

Going by PolitiFact's definitions and descriptions, the difference appears wholly subjective.

The Clinton/Trump bias tracks closely with PolitiFact's overall history for Democrat/Republican bias

Despite all the talk about how this election is the worst ever for political lies, we were somewhat surprised at how closely the Clinton/Trump comparison tracks with the party comparison over PolitiFact's entire history--since 2007.

We updated our spreadsheet for PolitiFact National data on Aug. 24, 2016. With that update, for Democrats the percentage of "Pants on Fire" ratings given to false statements was 17.09 percent--just a little below Clinton's 18.2 percent. For Republicans, the percentage of "Pants on Fire" ratings given to false statements was 28.39 percent. Trump's percentage stands at 33.3 percent.

To help illustrate how closely the presidential election comparison is sticking to PolitiFact's historical tendencies, we offer PolitiFact Wisconsin for comparison. The PolitiFact states vary greatly  compared to one another by our "Pants on Fire" bias measure. PolitiFact Wisconsin shows a bias against Democrats. Cumulatively, Democrats came in at 30 percent "Pants on Fire" from 2010 through the end of 2015. Republicans came in at 20.15 percent over that same span.

These numbers pose a dilemma for PolitiFact's defenders. Are Republicans  more honest than Democrats in Wisconsin by these numbers? Or do the numbers tell us more about how PolitiFact Wisconsin differs from PolitiFact National?

The consistency of PolitiFact National's "Pants on Fire" bias argues that PolitiFact National has sustained a bias against Republicans since at least 2008.

Visit our Research page for more on the "Pants on Fire" research approach.


  1. This comment has been removed by a blog administrator.

  2. Posting for "bah"/"Unknown," who had trouble expressing himself without an F-bomb:

    **Um, that's because he's a ******* liar. If you lie more often, and do it in ridiculously obvious fashion, you should be called out more often.**

    Is that simply your opinion, or does PolitiFact truly operate that way?

    If the former, then the case against PolitiFact's "Pants on Fire" bias rest on layers of flaky croissant-like opinions. Nothing to it.

    If the latter, then I should be able to expect you to make a evidence-based case supporting your point. And PolitiFact should be able to better distinguish between its "False" and "Pants on Fire" ratings with something more detailed and informative than "sometimes we decide one way and sometimes decide the other."

    Until PolitiFact offers some objective criterion for "ridiculous," we do not accept that it surely objective. And neither should you.

  3. Big thank you to "bah" for explaining the objective difference between "false" and "ridiculously false." We will be taking this website down now that you have shown how stupid we are and how scientific PolitiFact's ratings are.

    You are a hero.


Thanks to commenters who refuse to honor various requests from the blog administrators, all comments are now moderated. Pseudonymous commenters who do not choose distinctive pseudonyms will not be published, period. No "Anonymous." No "Unknown." Etc.