Tuesday, October 17, 2017

Can you trust what "Media Bias/Fact Check" says about PolitiFact? (Updated x2)

(See update at the end)

Somehow we got to the point where it makes sense to talk about Media Bias/Fact Check.

Media Bias/Fact Check bills itself as "The most comprehensive media bias resource." It's run by Dave Van Zandt, making it fair to say it's run by "some guy" ("Dave studied Communications in college" is his main claim to expertise).

We have nothing against "some guy" possessing expertise despite a lack of qualifications, of course. One doesn't need a degree or awards (or audience) to be right about stuff. But is Van Zandt and his Media Bias/Fact Check right about PolitiFact?

Media Bias/Fact Check rates PolitiFact as a "Least-biased" source of information. How does MB/FC reach that conclusion? The website has a "Methodology" page describing its methods:
The method for (rating bias) is determined by ranking bias in four different categories. In each category the source is rated on a 0-10 scale, with 0 meaning without bias and 10 being the maximum bias(worst). These four numbers are then added up and divided by 4. This 0-10 number is then placed on the line according to their Left or Right bias.
This system makes PolitiFact's "Truth-O-Meter" almost look objective by comparison. An 11-point scale? To obtain objectivity with an 11-point scale would require a very finely-grained system of objective bias measures--something that probably nobody on the planet has even dreamt of achieving.

It comes as no surprise that Van Zandt lacks those objective measures:

The categories are as follows (bold emphasis added):
  1. Biased Wording/Headlines- Does the source use loaded words to convey emotion to sway the reader. Do headlines match the story.
  2. Factual/Sourcing- Does the source report factually and back up claims with well sourced evidence.
  3. Story Choices: Does the source report news from both sides or do they only publish one side.
  4. Political Affiliation: How strongly does the source endorse a particular political ideology? In other words how extreme are their views. (This can be rather subjective)
Likely Van Zandt regards only the fourth category as subjective. All four are subjective unless Van Zandt has kept secret additional criteria he uses to judge bias. Think about it. Take the "biased wording" category, for example. Rate the headline bias for "PolitiFact Bias" on a scale of 0-10. Do it. What objective criteria guided the decision?

There is nothing to go on except for one's own subjective notion of where any observed bias falls on the 0-10 scale.

If the scale was worth something, researchers could put the rating system in the hands of any reasonable person and obtain comparable results. Systems with robust objective markers attached to each level of the scale can achieve that. Those lacking such markers will not.

Based on our experience with PolitiFact, we used Van Zandt's system on PolitiFact. Please remember that our experience will not render Van Zandt's system anything other than subjective.

Biased Wording/Headlines: 4
Factual/Sourcing: 3
Story Choices: 4
Political Affiliation: 3

Total=14
Formula calls for division by 4.
14/4=3.5
3.5=Left Center Bias

Why is Van Zandt's rating objectively more valid than ours? Or yours?

Here's more of Van Zandt's rating of PolitiFact.
Factual Reporting: VERY HIGH
World Press Freedom Rank: USA 43/180

Notes: PolitiFact.com is a project operated by the Tampa Bay Times, in which reporters and editors from the Times and affiliated media outlets “fact-check statements by members of Congress, the White House, lobbyists and interest groups”. They publish original statements and their evaluations on the PolitiFact.com website, and assign each a “Truth-O-Meter” rating. The ratings range from “True” for completely accurate statements to “Pants on Fire” (from the taunt “Liar, liar, pants on fire”) for false and ridiculous claims. Politifact has been called left biased by Extreme right wing and questionable sources. Our research indicates that Poltifact [sic] is an accurate fact checker and is considered the gold standard for political fact checking. (7/10/2016)

Source: http://www.politifact.com/

Notice the biased language from Van Zandt? Van Zandt only allows that PolitiFact has been called left-leaning by "Extreme right wing and questionable sources." In fact, PolitiFact has been called left-biased by many sources, including the non-partisan Allsides Project.

Van Zandt even has an opt-in poll on his PolitiFact page asking visitors how they rate PolitiFact's bias. Most of the respondents disagree with the site's rating of PolitiFact.


Over 50 percent of Van Zandt's respondents rated PolitiFact biased to the left. Does that mean that all those 2,000+ people were "Extreme right wing" or "questionable sources"?

Note: I voted "Left-Center."

Why is PolitiFact called the "gold standard" for fact checking instead of FactCheck.org, or even Zebra Fact Check? That's a mystery.

The crux of the matter


The temptation of subjective rating scales is obvious, but such scales misinform readers and probably tend to mislead their creators as well.

A rating scale that fails to base its ratings on quantifiable data is worthless. Van Zandt's ratings are worthless except to tell you his opinion.

Opinions about PolitiFact's bias start to have value when backed by specific, quantifiable findings. We've taken that approach for years here at PolitiFact Bias. When we see the biased headline, we write a post about it if it's of sufficient note. When we see the bad reporting, we write a post about it and document PolitiFact's failure with reliable sourcing. When we see PolitiFact skewing its story choices in a way that unfairly harms conservatives (or liberals), we write an article about it. When we see systematic signs of bias in PolitiFact's ratings, we do objective research on it.

We do that because specific examples trump subjective rating scales.

Until Dave Van Zandt adds objective markers to the MB/FC rating scales and justifies every rating with real objective data, take the ratings with a boulder of salt. They're worthless without specific backing data.


Afters

On its PolitiFact page, Media Bias/Fact Check links the flawed PolitiFact article we fisked here.

"VERY HIGH" factual reporting.

Hmm.


Update August 11, 2018:

 Dave Van Zandt contacted us on Aug. 9, 2018 to say MB/FC has changed its rating of PolitiFact to "Left-Center." But we can't find any evidence the change occurred so we have no response yet to the supposed change (perhaps Van Zandt's message was simply in error, intending to inform us of a more subtle shift in the rating).

The Internet Archive pretends to have plenty of saves but the only one it shows (when we checked) seems to be from January 2018.

The latest live version contains the following, which seems short of moving PolitiFact to the "Left-Center" category:
Overall, this update reveals a slight leftward shift in Politifact’s fact checking selection, but not enough to move them from the least biased category. (7/10/2016) Updated (D. Van Zandt 7/15/2018)

That seems a bit wishy-washy. Subjectivity can have that effect.



Update 2, June 6, 2021:


MB/FC updated its rating of PolitiFact to "Left-Center Bias" with an update time-stamped 04/28/2021:

Overall, we rate Politifact Left-Center Biased based on fact checks that tend to be more favorable for the left. We also rate them High for factual reporting and a credible fact-checker that is not without bias. (7/10/2016) Updated (D. Van Zandt (4/28/2021)
Van Zandt's 2021 update vanishes the 2018 update. It would be better to keep the publication and update dates all intact and link each to an appropriate URL at the Internet Archive.

The fact that Van Zandt has come around to our position does not mean that either one of us has rendered an objective judgment of PolitiFact's bias.

In our opinion, if PolitiFact's bias has shifted left since 2018 it was by a fraction. And we would say Van Zandt continues to overestimate PolitiFact's reliability.

23 comments:

  1. A blog that has an insane hate-boner for Politifact telling me that another source that verifies the credibility of other news sources that so happened to rate Politifact as credible complaining that there wasn't any objective research behind that website, but the author instead points me to blogs from their own website about "credible" research?

    There's definitely no sign of confirmation bias here, none at all lmao.

    ReplyDelete
    Replies
    1. (deleted and reposted to correct a significant error)

      Jatinder Singh,

      Sorry for the delay publishing your rant. We have updated the email address listed for comments sent to moderation, so this will hopefully be the last such delay.

      As for the content of your post: Our methods (unlike those at PolitiFact and Media Bias Fact Check) do not rely on subjective measures. We build our criticism based on facts and data. We encourage readers to criticize us if we make any mistakes with our facts, data or conclusions drawn.

      Please see our post "PolitiFact Bias is Biased." Cheers.

      https://www.politifactbias.com/2014/05/politifact-bias-is-biased.html

      Delete
  2. Totally agree. Politifact is more politileft. They should seriously consider changing the name.

    ReplyDelete
  3. PolitiFARCE ; Media Bias Bash Center

    ReplyDelete
  4. "In fact, PolitiFact has been called left-biased by many sources, including the non-partisan Allsides Project."

    Allsides CEO, John Gable, is actually a conservative and he declares on the site that the site bias is lean right. That's not strictly non-partisan!

    ReplyDelete
    Replies

    1. pjpw wrote:

      **Allsides CEO, John Gable, is actually a conservative and he declares on the site that the site bias is lean right.**

      Source?

      The "About" page says Allsides has a politically diverse staff. I don't see the part about leaning right.

      https://www.allsides.com/blog/reasons-hate-allsides

      **That's not strictly non-partisan!**

      I suppose you can go to PolitiFact and get strictly non-partisan fact checks from fact-checkers who overwhelmingly (perhaps unanimously) vote Democrat but don't tell you because it's against their principles ...

      Delete
  5. The truth is that "bias" is a horrible way to think about the validity of a source. Better would be "true" and "false." I can have the most extremist opinions and headlines in existence, but it doesn't matter if my facts and reasoning are valid.

    ReplyDelete
    Replies
    1. Elect_blob wrote:

      **The truth is that "bias" is a horrible way to think about the validity of a source.**

      We probably agree on this.

      **Better would be "true" and "false."**

      Biased reports aren't always true or false. How would you evaluate, for example, the omission of an important piece of context?

      **I can have the most extremist opinions and headlines in existence, but it doesn't matter if my facts and reasoning are valid.**

      Headlines should suitably reflect the content that follows.

      Here's where we agree: Personal bias is irrelevant to reporting per se. Bias only matter when it is found in the content. In fact-checking, that often manifests itself as an error (false reporting, bad reasoning, omitted context, etc.).

      If a "biased" reporter and a "neutral" reporter write exactly the same words, is the story by the biased reporter more biased than the other story?

      Delete
    2. Except that it is nearly impossible for a person to completely see/understand all of their own biases, and therefore impossible for them to avoid including bias in their reporting. IOWs, there is no such things as a completely neutral person.

      Delete
  6. Instead of ranting about your own views on whether they are left or right, why not let their fact checks speak for themselves. Can you prove any of their fact checks to be wrong? Do other members of the IFCN find different results? Do they fail to provide credible sourcing for their facts? Do they rate statements from the left as true even when they are false? Do they rate statements from the right false even if they are true? The answer is no to every one of those questions is no. Your rant against them is biased, they are not.

    ReplyDelete
    Replies
    1. **Can you prove any of their fact checks to be wrong?**

      You should visit the blog "PolitiFact Bias" sometime. And read it.

      Delete
  7. I say Kudos to Dave Van Zandt and Media Bias Fact Check. It's a worthy cause and at least he's trying. How about supporting, helping and improving his site?

    ReplyDelete
    Replies
    1. I tried helping Dave out early on (by warning him not to plagiarize his statement of principles and by pointing out an egregious error it took him a long time to fix). But what's the point? It's just another subjective rating system. It doesn't help anybody except Dave, afaict. Why pitch in to help with that?

      Delete
  8. This is nuts. It doesn't matter how many people who you have rate the website, if you did not use a good method of soliciting your sources. Who are these poeple? Have they read articles from Politifac, are they register dems, reps, ind., green, etc. THis is not a good validation. As a matter of a fact. You sound bias.

    ReplyDelete
    Replies
    1. cindy wrote:

      **This is nuts. It doesn't matter how many people who you have rate the website, if you did not use a good method of soliciting your sources. Who are these poeple? Have they read articles from Politifac, are they register dems, reps, ind., green, etc. THis is not a good validation. As a matter of a fact.**

      So far, it sounds like we agree. Media Bias Fact Check does not appear to have an objective method of evaluation, and it therefore should follow that its conclusions do not serve as a good method of validation regardless of how many staffers run a site through its subjective rubric.

      **You sound bias.**

      This line from you is surprising, given that it looks like we agree.

      Could you elaborate?

      Delete
  9. "PolitiFact Bias exists to expose the defective fact checking apparatus at PolitiFact, with special focus on the problems that help show PolitiFact's marked ideological bias."
    That kind of says it all. If you exist to expose bias, then you start from your own bias. Pot, kettle.

    ReplyDelete
    Replies
    1. **If you exist to expose bias, then you start from your own bias.**

      That doesn't follow. We all start with bias because bias counts as part of the human condition. But bias is something we can often identify using reason and science. So, regardless of any bias we may have, the effective attack is by showing flaws in our work, not by accusing us of bias.

      And that's how we work. We do not say PolitiFact is biased therefore PolitiFact is unreliable. We look PolitiFact to make various kinds of errors and conclude from its pattern of errors that it is unreliable (and biased).

      We appreciate our readers who point out when we make mistakes. It helps us to fix either the mistake or the improper perception of a mistake.

      Delete
    2. The bias being exposed in Politifact is to do with their fact-checking, which is totally different from being focused (which you incorrectly termed "bias") towards groups like them because they act that way

      Delete
  10. Politifact is left of center. Get used to it. Your liberal bias makes objectivity impossible for you.

    ReplyDelete
    Replies
    1. We're not sure to whom you're replying.

      We think Dave was off when he rated PolitiFact in the center. Since we did our original review, Dave moved PolitiFact very slightly left. But PolitiFact leans more than slightly left. So we think Dave underestimates PolitiFact's leftward lean.

      We don't see how it would follow that PolitiFact Bias has a liberal bias that makes objectivity impossible.

      Delete
  11. Apparently the official fact checker at USA Today is using this faux media fact-checking site now. Glad to see them being exposed, not glad they're still around and manipulating Google algorithms to appear at the top of searches.
    https://www.usatoday.com/story/news/factcheck/2021/01/30/who-advises-people-continue-wearing-masks/4314055001/

    ReplyDelete
  12. Thanks for the reminder. I've been meaning to ask that USAToday writer to reconsider treating MB/FC as a serious source.

    ReplyDelete
    Replies
    1. I found a contact address for Chelsea Cox, who has name-dropped MB/FC in at least one of her fact checking articles. I filled her in on a little of the history and suggested that USAToday examine MB/FC before continuing to portray it as a trustworthy source.

      Delete

Thanks to commenters who refuse to honor various requests from the blog administrators, all comments are now moderated. Pseudonymous commenters who do not choose distinctive pseudonyms will not be published, period. No "Anonymous." No "Unknown." Etc.