"Bias in PolitiFact's ratings:  Pants on Fire vs. False" Aug. 1, 2012

Research update (Updated)  Nov. 6, 2012

Research Update: PolitiFact in 2012  Feb. 1, 2013

Research update: The "Pants on Fire" bias in 2013, plus analysis of the PolitiFact states, Jan. 22, 2014

PolitiFact, percent error, partisanship  May 28, 2014

The "Pants on Fire" bias update for 2014 Jan. 2, 2015

PunditFact's "Pants on Fire" bias, 2014 Jan 19, 2015


  1. If you think Politifacts are bias. Can you please build a website that isn't? What the world needs is facts. You are claiming The facts from Politifacts are Bias. If you have better information you would be doing America a favor.

    1. My attempt at a better mousetrap is Zebra Fact Check, now into its second year of operation.

      No website is entirely free of bias. But PolitiFact is barely one rung above Media Matters on the ladder of objectivity.

      Thanks for commenting.

    2. Ande,

      If your idolization of "Anonymous" is limited to the call for better information, then good.

      If, on the other hand, that idolization has to do with any of the utter nonsense "Anonymous" posted subsequently, then you could do a better job picking a hero.

  2. How can you call any of this research? Your conclusion basically states that conservatives do lie more, but politifact is biased because it doesn't show liberals lying as much. This article was an opinion peice and in no way qualifies as research.

  3. It's research because we objectively collected data and use the data to suggest reasonable conclusions. One of our conclusions is not that conservatives do lie more. That idea, in fact, isn't even in the data, given the selection bias problem.

    You may have the erroneous idea in your head that by "Pants on Fire" PolitiFact is saying a politician is lying. But that's not how the rating is defined, and PolitiFact, as our research shows does not use that concept in applying the rating. Going by PolitiFact's stated definition (which was found by doing research), the only difference between "False" and "Pants on Fire" is that the latter statements are "ridiculous" in addition to being false.

    All I can suggest is that you start from the ground up, looking up the word "research" and then re-reading the research we've posted to try to understand what we've done. If you'd like to make a case based on evidence that we're not doing research, then we'd love to hear it. But your opinion that it isn't research has no grounding in the truth.

  4. Taken from your article, "Do Republicans lie more? Perhaps, but the proportion of “Pants on Fire” ratings should prove about the same for each party regardless of which party lies more if subjective impressions determine the rating." Proves my point entirely. Your sole "argument" is that even though republicans do lie more there should be an even number of "Pants on Fire" rating, which as you stated is slightly subjective. The "false" rating directly before "pants on fire" is not subjective and you do not argue the validity of those classifications. So you are saying that Republicans do in fact lie more and that Politifacts is slightly harsher on the Republican lies coming from chain e-mails, but not on any other source of Repblican fibbery, "The percentages for groups A and B varied only slightly. For group A, Republicans found making false statements received a “Pants on Fire” rating 28.57% of the time and Democrats 16.42 . For group B the percentages were 28.23% and 15.88%, respectively--a very close match. Group C reflected the expected distortion from unreliable chain email claims."
    The reason I call this non-research is you did not evaluate any confounding variables nor did you no anything to prevent them from interfering with your "data," thus all you have are assumptions and opinions based on nothing of relevancy. Oh, and you made up this "PoF bias" number.
    Just to help you out here is the definition of research, "The systematic investigation into and study of materials and sources in order to establish facts and reach new conclusions." Please follow it next time.

  5. "Proves my point entirely."

    It proves you don't comprehend what you're reading.

    We don't say Republicans lie more. We say it's a possibility ("Perhaps").

    We don't say that the "Pants on Fire" rating is "slightly subjective." We say that PolitiFact's description paints it as entirely subjective, and we could find no clear evidence to doubt PolitiFact's own descriptions. And we put some effort into that.

    Nor does it follow that the "false" rating is not subjective. All we know from the "false" rating is that PolitiFact thinks the rating is false. We don't argue the validity because it isn't required to serve the point of our research, and you're not entitled to assume we accept the accuracy of PolitiFact's "False" ratings (fallacy of appeal to silence).

    We're not saying PolitiFact's bias is limited to the "Pants on Fire" bias. In the research on the PoF bias we simply limited our examination of bias to that aspect of PF's bias. In fact, the paper you referenced talks about the potential for PF to apply a compensatory bias.

    We obviously considered confounding variables, such as hidden judgment criteria, and our conclusions perfectly fit the evidence.

    In short, your assessment is marvelous in its carelessness. It's fitting you comment anonymously.

  6. Where in your "research" do you state how you addressed confoundin variables? Merely stating you did does not fit the criteria. In order for something to be considered research it needs to be concerned with validity or else it's just an opinion peice with numbers. I know legitimate science is the bane of most conservatives.

  7. I said we considered confounding variables and gave an example. You should have looked for the example I gave you and worked from there. Next I told you that our conclusions fit the evidence. That's your cue to find a conclusion that we drew that wouldn't be true if one of your confounding variables wasn't accounted for.

    How do you define "valid" if it means something other than drawing conclusions that fit the evidence?

    "I know legitimate science is the bane of most conservatives"

    Right. And don't you just hate it when they engage in ad hominem attacks as they invariably do?

  8. If you feel that my noting the non- empirical nature of research is a personal attack on you then so be it. It is easy to establish another cause for your "findings," conservative robo emails were typically more fictitious and created lies that we're more ridiculous than liberals. This is a claim your opinion piece can't rule out, thus you obviously did not control for confounds.

  9. "If you feel that my noting the non- empirical nature of research is a personal attack on you then so be it."

    Huh? If you think "I know legitimate science is the bane of most conservatives" is you noting the non-empirical nature of our research then so be it. That's quite a stretch, to say the least.

    "It is easy to establish another cause for your "findings," conservative robo emails were typically more fictitious and created lies that we're more ridiculous than liberals."

    We classified email claims separately from claims related to members or representatives of the political parties. So you apparently didn't read the research very carefully before judging it. It's likely you don't know what our findings are, either. Next time quote the finding and the confounding variable that would undermine it. We'll give you a second chance.

    "This is a claim your opinion piece can't rule out, thus you obviously did not control for confounds."

    So if we kept the email claims separate and drew our conclusions from a narrower set of data then we did not control for the extreme nature of email claims? Seriously?

    You get two more chances. If you can't improve your performance markedly we won't host your further comments. You'll have to prove to us you're not paid by the Koch brothers to make liberals look silly.

  10. We're tightening the restrictions on comments. You'll still get your two chances if you register. But no more total access for anonymous hackery.