PolitiFact: "PolitiFact is not biased--here's why" Pt. 1

In an article titled "PolitiFact is not biased--here's why" PolitiFact Editor Angie Drobnic Holan offers four points as evidence PolitiFact is not biased. This series deals with each of the four.

PolitiFact:

1. We fact-check inaccurate statements, not political parties.

We are always on the lookout for bad information that needs correcting. We don’t have any concern about which party it comes from or who says it. If someone makes an inaccurate statement, it gets a negative rating on our Truth-O-Meter: Mostly False, False or Pants on Fire.

If we at PolitiFact Bias were to come up with a story making an assertion, we would certainly try to produce some type of evidence giving palpable evidence in support. We find PolitiFact's article striking for its lack of evidence in support of the claim in the title.

Let's assume for the sake of argument that it's true PolitiFact fact checks inaccurate statements and not political parties. We find both assertions questionable, but we can set that aside for the moment.

What stops a biased fact checker from allowing factors like confirmation bias to guide its selection of fact checks to reflect an ideological bias? This is an obvious objection to the first part of Holan's argument but her article completely fails to acknowledge it. If Holan assumes that PolitiFact has no bias and therefore no confirmation bias can result then her argument begs the question (circular reasoning: PolitiFact is not biased because PolitiFact is not biased).

If Holan isn't using circular reasoning then she's simply not addressing the issue in any relevant way. Fact-checking inaccurate statements and not political parties does nothing to show a lack of bias.

The Elephant in the Room (a pun of foreshadowing)

In early 2011 Eric Ostermeier of the University of Minnesota did a study of PolitiFact's ratings. Ostermeier found Republicans were receiving worse treatment in PolitiFact's ratings. Ostermeier noted that PolitiFact's descriptions of its methodology offered no assurance at all that the skew in its ratings was unaffected by selection bias. In other words, was unrepresentative sampling responsible for making it appear that Republicans lie more?

Ostermeier posed an important question that PolitiFact has never satisfactorily addressed:

The question is not whether PolitiFact will ultimately convert skeptics on the right that they do not have ulterior motives in the selection of what statements are rated, but whether the organization can give a convincing argument that either a) Republicans in fact do lie much more than Democrats, or b) if they do not, that it is immaterial that PolitiFact covers political discourse with a frame that suggests this is the case.

The evidence says PolitiFact's story selection is biased

While developing our own research approaches to PolitiFact's ratings we came up with an observation we say strongly shows PolitiFact guilty of selection bias.

Imagine PolitiFact used only its editorial judgment of whether a statement seemed so false that it was worthy of a fact check and was completely blind to political party and ideology.

We say that regardless of whether one party lies more, the results should prove pretty close to proportional. If 40 percent of PolitiFact's ratings of Republicans come out "Pants on Fire" or "False" then the same should hold true of Democrats. If Republicans lie more that should end up reflected in the number of ratings, not in the proportions.

PolitiFact as much as admitted to selection bias in the early days. PolitiFact founding editor Bill Adair said PolitiFact tried to do a roughly equal number of fact checks for Republicans and Democrats. That makes no less than two criteria for selecting a story, and one of them is not simply whether the statement appeared false. Trying to fact check Republicans and Democrats equally will skew the proportions (unless the parties lie equally and PolitiFact's sample is effectively random).

In Ostermeier's research, Republicans' statements were 39 percent "Pants on Fire" or "False" while Democrats' statements were 12 percent "Pants on Fire" or "False." That's strong evidence of selection bias.

Note: We have not tracked these numbers through the present. Perhaps PolitiFact is closer to rating claims proportionally now than it was in Adair's time. If it is, then PolitiFact could present that as evidence it is blind to ideology when it chooses which claims to check.


Until PolitiFact answers Eric Ostermeier's question it is unsafe to conclude that PolitiFact lacks bias.

What does the "Partisan Selective Sharing" study say about PolitiFact?

A recent study called "Partisan Selective Sharing" (hereafter PSS) noted that Twitter users were more likely to share fact checks that aided their own side of the political aisle.

Duh?

On the other hand, the paper came up in a search we did of scholarly works mentioning "PolitiFact."

The search preview mentioned the University of Minnesota's Eric Ostermeier. So we couldn't resist taking a peek to see how the paper handled the data hinting at PolitiFact's selection bias problem.

The mention of Ostermeier's work was effectively neutral, we're happy to say. And the paper had some surprising value to it.

PSS coded tweets from the "elite three" fact checkers, FactCheck.org, PolitiFact and the Washington Post Fact Checker, classifying them as neutral, beneficial to Republicans or beneficial to Democrats.

In our opinion, that's where the study proved briefly interesting:

Preliminary analysis
Fact-checking tweets
42.3% of the 194 fact-check (n=82) tweets posted by the three accounts in October 2012 contained rulings that were advantageous to the Democratic Party (i.e., either positive to Obama or negative to Romney), while 23.7% of them (n=46) were advantageous to the Republican Party (i.e., either positive to Romney or negative to Obama). The remaining 34% (n=66) were neutral, as their statements contained either a contextualized analysis or a neutral anchor.

In addition to the relative advantage of the fact checks, the valence of the fact-checking tweet toward each candidate was also analyzed. Of the 194 fact checks, 34.5% (n=67) were positive toward Obama, 46.9% (n=91) were neutral toward Obama, and 18.6% (n=36) were negative toward Obama. On the other hand, 14.9% (n=29) of the 194 fact checks contained positive valence toward Romney, 53.6% (n=104) were neutral toward Romney, and 31.4% (n=61) were negative valence toward Romney.

Of course, many have no problem interpreting results like these as a strong indication that Republicans lie more than Democrats. And we cheerfully admit the data show consistency with the assumption that Republicans lie more.

Still, if one has some interest in applying the methods of science, on what do we base the hypothesis that Republicans lie more? We cannot base that hypothesis on these data without ruling out the idea that fact-checking journalists lean to the left. And unfortunately for the "Republicans lie more" hypothesis, we have some pretty good data showing that American journalists tend to lean to the left.

Until we have some reasonable argument why left-leaning journalists do not allow their bias to affect their work, the results of studies like PSS give us more evidence that the media (and the mainstream media subset "fact checkers") lean left while they're working.

The "liberal bias" explanation has better evidence than the "Republicans lie more" hypothesis. As PolitiFact tweeted 126 of the total 194 fact check tweets, a healthy share of the blame likely falls on PolitiFact.


We wish the authors of the study, Jieun Shin and Kjerstin Thorson, had separated the three fact checkers in their results.

Speaking of Selection Bias...

Alan Colmes has now jumped into the discussion (by repeating a Political Wire post) about the Smart Politics review of PolitiFact's selection bias.

In a truly amazing feat, Colmes (and Political Wire) managed to completely ignore the point of Eric Ostermeier's article, presenting the finding that PolitiFact more frequently rates Republicans negatively as proof that Republicans lie more often.

Check out the Colmes headline-

Republicans’ Statements Untrue Three Times As Often As Those Of Democrats

Colmes made absolutely no mention of the fact that Ostermeier was suggesting that the lopsided figures point to selection bias at PolitiFact. This is how Colmes treated the study-

"An analysis by Smart Politics of more than 500 stories during the past year show that statements made by Republican politicians are false three times as often as those made by Democrats."

Representing Ostermeier's piece as the polar opposite of what it is takes either an act of denial or more likely a shoddy job at research.

Compounding the error, Colmes, et al. ignore the focus of the study. Ostermeier's article dealt specifically with the issue of selection bias, i.e. the notion that PolitiFact's editors are more inclined to select outlandish or questionable statements from the GOP. This still leaves the issue of the quality of the ratings themselves untouched. As Ostermeier noted-

"Assuming for the purposes of this report that the grades assigned by PolitiFact are fair (though some would challenge this assumption), there has nonetheless been a great discrepancy regarding which political parties' officials and officeholders receive the top ratings and those that are accused of not telling the truth."

(bold emphasis added)

Colmes' careless posting of misleading information exemplifies how PolitiFact's opinionated brand of journalism can be misrepresented as objective fact, and repeated throughout the echo chamber.



Feb. 16, 2011  Corrected misspelled version of Eric Ostermeier's last name in one of the middle paragraphs.  Also added some new hyperlinks and a definite article in the first sentence.

PFB Smackdown: Kos clueless on Ostermeier study

The reality-based community hasn't taken kindly to Eric Ostermeier's attempts to rescue reality from its liberal bias.

Steve Singiser for the Daily Kos is the latest to take a swipe at the study, writing under the title "Documented proof that Republicans are the biggest liars in politics":

(T)he study (which can be found here) focuses on Politifact itself, charging the nonpartisan analysis done by the St. Petersburg Times fact-checking unit with a systemic bias against Republicans.

Singiser misstates the charge made by Ostermeier.  Ostermeier argues that PolitiFact's presentation suggests Republicans lie more than Democrats, but in scientific terms that conclusion only follows if the conclusions from the data account for selection bias.  But the evidence we have of PolitiFact's selection process offers no assurance against ideological bias in the results.  The stark contrast in the results, Ostermeier says, places a burden of proof on PolitiFact to assure readers that the implicit conclusion is not the result of selection bias.  In other words, make the selection process transparent.  Ostermeier is not charging PolitiFact with "a systemic bias against Republicans."  But he's pointing out that it may well be the case.

After constructing a straw man position for Ostermeier, Singiser proceeds to attack it:

(T)here are other explanations that are equally, if not more, plausible than charging Politifact with grading in bad faith.

True, among them an unconscious bias (good faith, bad methods) among PolitiFact journalists that results in the type of selection bias Ostermeier hypothesizes may account for the disparity in PolitiFact's cumulative grades.  And the latter is an extraordinarily plausible (and parsimonious) explanation.  Singiser would be hard pressed to match it, and that's why Ostermeier argues that PolitiFact should do something to provide a sure foundation for the conclusions it implicitly encourages.

Singiser continues:

For one thing, the study was conducted during a time when the GOP was out of power. The party out of power, it could reasonably be assumed, is going to take more chances with their rhetoric, in an effort to turn the electorate against the party in office.

That's all well and good, but it does nothing to mitigate Ostermeier's real argument.  Remember, PolitiFact grades approximately the same number of statements from figures in both parties.  If PolitiFact used a policy of simply grading chancy rhetoric then one would expect PolitiFact to rate far more statements from one party than the other if the conditions Singiser suggests were to prevail.  And if PolitiFact ends up padding the Democrats' numbers with statements chosen according to a different criterion then we have selection bias by definition.

In short, Singiser's suggested explanation doesn't help.

He doesn't get it.



Feb 17, 2011:  Corrected a recurrent misspelling of Markos Moulitsas' last name (no "z" as it turns out).  Sincere apologies to Mr. Moulitsas.
Jan. 24, 2012:  Double apologies to Markos Moulitsas.  While going back to reference the Kos blog entry I found the link broken.  Shortly after I found an identical version of the post clearly credited to Steve Singiser.

PFB Smackdown: Why grant PolitiFact the presumption of neutrality?

The smackdown is actually courtesy of Eric Ostermeier, publisher of the study highlighted recently here at PolitiFact Bias and originally published at the University of Minnesota's Smart Politics blog.

Ostermeier sniffed out a criticism of his work at mnpACT! by progressive blogger Dave Mindeman.  After noting two of the central facts in Ostermeier's study, that PolitiFact rates about the same number of statements from Democrats as for Republicans and that Republicans get the worst ratings of the two, Mindeman offers his two cents:

Occam's razor. The GOP get the worst ratings because they make the worst statements.

Ostermeier concentrates on making Politifact defend their selection process, but overlooks the facts about the statements themselves. Could it be possible that the Republicans make more outrageous and indefensible assertions?

Politifact is certainly going to be drawn to statements that get the most attention and the more outrageous the statement, the more attention it gets.

Ostermeier promptly addressed the second paragraph by posting in the commentary section:

FYI: this very possibility was in fact addressed in my report:

"One could theoretically argue that one political party has made a disproportionately higher number of false claims than the other, and that this is subsequently reflected in the distribution of ratings on the PolitiFact site."

Mindeman (as DaveM) answered back:

Then why ask title your post with "selection bias"? It would seem that most of the data says there is none.

And Ostermeier responded again:

What data is that? You can't use the data I published from coding PolitiFact's stories itself (noting the site attributes more false statements to the GOP) as proof that the GOP lies more and thus there is de facto no selection bias. That's circular reasoning. Note: My report did not definitively prove there is such bias, but the data published shifts the burden, I would argue, to PolitiFact. And greater transparency in their selection methodology would shine a light on this very question.

Ostermeier is exactly right.  Mindeman apparently wants to entitle PolitiFact to the presumption of neutrality on the issue of selection bias.  But there simply isn't any basis for that presumption.  The presumption would follow if PolitiFact chose its stories at random.  Lacking that, the reader has no good reason to take PolitiFact as a neutral party minus the transparent methodology Ostermeier mentions at the conclusion of his second comment.

Mindeman has the last word on Ostermeier for the moment:

I fail to understand why Politifact has to "prove" anything. They examined political statements that interested them. Most reporters choose their own stories. If they do a good job of reporting the story, do they still have to prove that they have no inherent bias? Maybe I should assume you have an inherent conservative bias because your analysis deals with GOP favorable data?? But I don't, because I think you use the data in a broad enough sense that it tells something regardless of the outcome.

By this time Mindeman is offering some clues that he doesn't understand selection bias.  But the biggest problem in his analysis is actually his attempt to employ Occam's razor (aka the principle of parsimony) to disfavor Ostermeier's hypothesis that PolitiFact displays a selection bias favorable to Democrats.

Occam's razor favors simple explanations, and Mindeman seems to understand it that far.  But it isn't at all clear why he regards a pattern of lying among a large set of entities as a simpler explanation than political bias from a much smaller group of journalists.  Rather than using Occam's razor to legitimately favor a simpler explanation, Mindeman wields it more like a magic wand that produces a supernatural sphere of protection around the ideas he favors.

Smart Politics on selection bias at PolitiFact

Research associate Eric Ostermeier of "Smart Politics" at the University of Minnesota has published a study of PolitiFact's content and finds a disturbing vein of partisanship in PolitiFact's selection bias:

Assuming for the purposes of this report that the grades assigned by PolitiFact are fair (though some would challenge this assumption), there has nonetheless been a great discrepancy regarding which political parties' officials and officeholders receive the top ratings and those that are accused of not telling the truth.

Ostermeier goes on to note PolitiFact's focus on untrue statements by the party out of power--perhaps unusual considering the press considers itself the watchdog of government.

Ostermeier's approach is exactly right in taking PolitiFact's grade groupings as an indication of PolitiFact's selection bias rather than as a measure of candidate truthfulness.

A key chart from the report:

As Hot Air's Ed Morrissey points out, it remains possible to interpret the data to mean that Republicans simply utter more false and "Pants on Fire" statements than Democrats.

But is that the best explanation in terms of the empirical data?

I would again call attention to the divide between the "False" and "Pants on Fire" ratings.  PolitiFact to date has offered no metric for objectively distinguishing one from the other.  This in turn strongly suggests that subjective judgment serves as the ultimate criterion for grading a statement as "Pants on Fire."

 ***

Importantly, Ostermeier also points out that PolitiFact's presentation encourages readers to use the ratings to draw conclusions about the subjects receiving the ratings:

The question is not whether PolitiFact will ultimately convert skeptics on the right that they do not have ulterior motives in the selection of what statements are rated, but whether the organization can give a convincing argument that either a) Republicans in fact do lie much more than Democrats, or b) if they do not, that it is immaterial that PolitiFact covers political discourse with a frame that suggests this is the case.

Feb 11, 2011: Corrected typo affecting the word "Minnesota." My apologies to that fine state.
June 14, 2018: Clarified the third paragraph (first after the quotation) by removing a "that" and an -ed suffix.