Tuesday, October 17, 2017

Can you trust what "Media Bias/Fact Check" says about PolitiFact? (Updated)

(See update at the end)

Somehow we got to the point where it makes sense to talk about Media Bias/Fact Check.

Media Bias/Fact Check bills itself as "The most comprehensive media bias resource." It's run by Dave Van Zandt, making it fair to say it's run by "some guy" ("Dave studied Communications in college" is his main claim to expertise).

We have nothing against "some guy" possessing expertise despite a lack of qualifications, of course. One doesn't need a degree or awards (or audience) to be right about stuff. But is Van Zandt and his Media Bias/Fact Check right about PolitiFact?

Media Bias/Fact Check rates PolitiFact as a "Least-biased" source of information. How does MB/FC reach that conclusion? The website has a "Methodology" page describing its methods:
The method for (rating bias) is determined by ranking bias in four different categories. In each category the source is rated on a 0-10 scale, with 0 meaning without bias and 10 being the maximum bias(worst). These four numbers are then added up and divided by 4. This 0-10 number is then placed on the line according to their Left or Right bias.
This system makes PolitiFact's "Truth-O-Meter" almost look objective by comparison. An 11-point scale? To obtain objectivity with an 11-point scale would require a very finely-grained system of objective bias measures--something that probably nobody on the planet has even dreamt of achieving.

It comes as no surprise that Van Zandt lacks those objective measures:

The categories are as follows (bold emphasis added):
  1. Biased Wording/Headlines- Does the source use loaded words to convey emotion to sway the reader. Do headlines match the story.
  2. Factual/Sourcing- Does the source report factually and back up claims with well sourced evidence.
  3. Story Choices: Does the source report news from both sides or do they only publish one side.
  4. Political Affiliation: How strongly does the source endorse a particular political ideology? In other words how extreme are their views. (This can be rather subjective)
Likely Van Zandt regards only the fourth category as subjective. All four are subjective unless Van Zandt has kept secret additional criteria he uses to judge bias. Think about it. Take the "biased wording" category, for example. Rate the headline bias for "PolitiFact Bias" on a scale of 0-10. Do it. What objective criteria guided the decision?

There is nothing to go on except for one's own subjective notion of where any observed bias falls on the 0-10 scale.

If the scale was worth something, researchers could put the rating system in the hands of any reasonable person and obtain comparable results. Systems with robust objective markers attached to each level of the scale can achieve that. Those lacking such markers will not.

Based on our experience with PolitiFact, we used Van Zandt's system on PolitiFact. Please remember that our experience will not render Van Zandt's system anything other than subjective.

Biased Wording/Headlines: 4
Factual/Sourcing: 3
Story Choices: 4
Political Affiliation: 3

Formula calls for division by 4.
3.5=Left Center Bias

Why is Van Zandt's rating objectively more valid than ours? Or yours?

Here's more of Van Zandt's rating of PolitiFact.
Factual Reporting: VERY HIGH
World Press Freedom Rank: USA 43/180

Notes: PolitiFact.com is a project operated by the Tampa Bay Times, in which reporters and editors from the Times and affiliated media outlets “fact-check statements by members of Congress, the White House, lobbyists and interest groups”. They publish original statements and their evaluations on the PolitiFact.com website, and assign each a “Truth-O-Meter” rating. The ratings range from “True” for completely accurate statements to “Pants on Fire” (from the taunt “Liar, liar, pants on fire”) for false and ridiculous claims. Politifact has been called left biased by Extreme right wing and questionable sources. Our research indicates that Poltifact [sic] is an accurate fact checker and is considered the gold standard for political fact checking. (7/10/2016)

Source: http://www.politifact.com/

Notice the biased language from Van Zandt? Van Zandt only allows that PolitiFact has been called left-leaning by "Extreme right wing and questionable sources." In fact, PolitiFact has been called left-biased by many sources, including the non-partisan Allsides Project.

Van Zandt even has an opt-in poll on his PolitiFact page asking visitors how they rate PolitiFact's bias. Most of the respondents disagree with the site's rating of PolitiFact.

Over 50 percent of Van Zandt's respondents rated PolitiFact biased to the left. Does that mean that all those 2,000+ people were "Extreme right wing" or "questionable sources"?

Note: I voted "Left-Center."

Why is PolitiFact called the "gold standard" for fact checking instead of FactCheck.org, or even Zebra Fact Check? That's a mystery.

The crux of the matter

The temptation of subjective rating scales is obvious, but such scales misinform readers and probably tend to mislead their creators as well.

A rating scale that fails to base its ratings on quantifiable data is worthless. Van Zandt's ratings are worthless except to tell you his opinion.

Opinions about PolitiFact's bias start to have value when backed by specific, quantifiable findings. We've taken that approach for years here at PolitiFact Bias. When we see the biased headline, we write a post about it if it's of sufficient note. When we see the bad reporting, we write a post about it and document PolitiFact's failure with reliable sourcing. When we see PolitiFact skewing its story choices in a way that unfairly harms conservatives (or liberals), we write an article about it. When we see systematic signs of bias in PolitiFact's ratings, we do objective research on it.

We do that because specific examples trump subjective rating scales.

Until Dave Van Zandt adds objective markers to the MB/FC rating scales and justifies every rating with real objective data, take the ratings with a boulder of salt. They're worthless without specific backing data.


On its PolitiFact page, Media Bias/Fact Check links the flawed PolitiFact article we fisked here.

"VERY HIGH" factual reporting.


Update August 11, 2018:

 Dave Van Zandt contacted us on Aug. 9, 2018 to say MB/FC has changed its rating of PolitiFact to "Left-Center." But we can't find any evidence the change occurred so we have no response yet to the supposed change (perhaps Van Zandt's message was simply in error, intending to inform us of a more subtle shift in the rating).

The Internet Archive pretends to have plenty of saves but the only one it shows (when we checked) seems to be from January 2018.

The latest live version contains the following, which seems short of moving PolitiFact to the "Left-Center" category:
Overall, this update reveals a slight leftward shift in Politifact’s fact checking selection, but not enough to move them from the least biased category. (7/10/2016) Updated (D. Van Zandt 7/15/2018)
That seems a bit wishy-washy. Subjectivity can have that effect.


  1. A blog that has an insane hate-boner for Politifact telling me that another source that verifies the credibility of other news sources that so happened to rate Politifact as credible complaining that there wasn't any objective research behind that website, but the author instead points me to blogs from their own website about "credible" research?

    There's definitely no sign of confirmation bias here, none at all lmao.

    1. (deleted and reposted to correct a significant error)

      Jatinder Singh,

      Sorry for the delay publishing your rant. We have updated the email address listed for comments sent to moderation, so this will hopefully be the last such delay.

      As for the content of your post: Our methods (unlike those at PolitiFact and Media Bias Fact Check) do not rely on subjective measures. We build our criticism based on facts and data. We encourage readers to criticize us if we make any mistakes with our facts, data or conclusions drawn.

      Please see our post "PolitiFact Bias is Biased." Cheers.


  2. Totally agree. Politifact is more politileft. They should seriously consider changing the name.

  3. PolitiFARCE ; Media Bias Bash Center

  4. "In fact, PolitiFact has been called left-biased by many sources, including the non-partisan Allsides Project."

    Allsides CEO, John Gable, is actually a conservative and he declares on the site that the site bias is lean right. That's not strictly non-partisan!


    1. pjpw wrote:

      **Allsides CEO, John Gable, is actually a conservative and he declares on the site that the site bias is lean right.**


      The "About" page says Allsides has a politically diverse staff. I don't see the part about leaning right.


      **That's not strictly non-partisan!**

      I suppose you can go to PolitiFact and get strictly non-partisan fact checks from fact-checkers who overwhelmingly (perhaps unanimously) vote Democrat but don't tell you because it's against their principles ...

  5. The truth is that "bias" is a horrible way to think about the validity of a source. Better would be "true" and "false." I can have the most extremist opinions and headlines in existence, but it doesn't matter if my facts and reasoning are valid.

    1. Elect_blob wrote:

      **The truth is that "bias" is a horrible way to think about the validity of a source.**

      We probably agree on this.

      **Better would be "true" and "false."**

      Biased reports aren't always true or false. How would you evaluate, for example, the omission of an important piece of context?

      **I can have the most extremist opinions and headlines in existence, but it doesn't matter if my facts and reasoning are valid.**

      Headlines should suitably reflect the content that follows.

      Here's where we agree: Personal bias is irrelevant to reporting per se. Bias only matter when it is found in the content. In fact-checking, that often manifests itself as an error (false reporting, bad reasoning, omitted context, etc.).

      If a "biased" reporter and a "neutral" reporter write exactly the same words, is the story by the biased reporter more biased than the other story?

    2. Except that it is nearly impossible for a person to completely see/understand all of their own biases, and therefore impossible for them to avoid including bias in their reporting. IOWs, there is no such things as a completely neutral person.

  6. Instead of ranting about your own views on whether they are left or right, why not let their fact checks speak for themselves. Can you prove any of their fact checks to be wrong? Do other members of the IFCN find different results? Do they fail to provide credible sourcing for their facts? Do they rate statements from the left as true even when they are false? Do they rate statements from the right false even if they are true? The answer is no to every one of those questions is no. Your rant against them is biased, they are not.

    1. **Can you prove any of their fact checks to be wrong?**

      You should visit the blog "PolitiFact Bias" sometime. And read it.

  7. I say Kudos to Dave Van Zandt and Media Bias Fact Check. It's a worthy cause and at least he's trying. How about supporting, helping and improving his site?

    1. I tried helping Dave out early on (by warning him not to plagiarize his statement of principles and by pointing out an egregious error it took him a long time to fix). But what's the point? It's just another subjective rating system. It doesn't help anybody except Dave, afaict. Why pitch in to help with that?


Thanks to commenters who refuse to honor various requests from the blog administrators, all comments are now moderated. Pseudonymous commenters who do not choose distinctive pseudonyms will not be published, period. No "Anonymous." No "Unknown." Etc.