(See update at the end)
Somehow we got to the point where it makes sense to talk about
Media Bias/Fact Check.
Media Bias/Fact Check bills itself as "The most comprehensive media bias resource." It's run by Dave Van Zandt, making it fair to say it's run by "some guy" ("
Dave studied Communications in college" is his main claim to expertise).
We have nothing against "some guy" possessing expertise despite a lack of qualifications, of course. One doesn't need a degree or awards (or audience) to be right about stuff. But is Van Zandt and his Media Bias/Fact Check right about PolitiFact?
Media Bias/Fact Check rates PolitiFact as a "Least-biased" source of information. How does MB/FC reach that conclusion? The website has a
"Methodology" page describing its methods:
The method for (rating bias) is determined by ranking bias in four different categories. In each category the source is rated on a 0-10 scale, with 0 meaning without bias and 10 being the maximum bias(worst). These four numbers are then added up and divided by 4. This 0-10 number is then placed on the line according to their Left or Right bias.
This system makes PolitiFact's "Truth-O-Meter" almost look objective by comparison. An 11-point scale? To obtain objectivity with an 11-point scale would require a very finely-grained system of objective bias measures--something that probably nobody on the planet has even dreamt of achieving.
It comes as no surprise that Van Zandt lacks those objective measures:
The categories are as follows (bold emphasis added):
- Biased Wording/Headlines- Does the source use loaded words to convey emotion to sway the reader. Do headlines match the story.
- Factual/Sourcing- Does the source report factually and back up claims with well sourced evidence.
- Story Choices: Does the source report news from both sides or do they only publish one side.
- Political Affiliation: How strongly does the source endorse a particular political ideology? In other words how extreme are their views. (This can be rather subjective)
Likely Van Zandt regards only the fourth category as subjective. All four are subjective unless Van Zandt has kept secret additional criteria he uses to judge bias. Think about it. Take the "biased wording" category, for example. Rate the headline bias for "PolitiFact Bias" on a scale of 0-10. Do it. What objective criteria guided the decision?
There is nothing to go on except for one's own subjective notion of where any observed bias falls on the 0-10 scale.
If the scale was worth something, researchers could put the rating system in the hands of any reasonable person and obtain comparable results. Systems with robust objective markers attached to each level of the scale can achieve that. Those lacking such markers will not.
Based on our experience with PolitiFact, we used Van Zandt's system on PolitiFact.
Please remember that our experience will not render Van Zandt's system anything other than subjective.
Biased Wording/Headlines: 4
Factual/Sourcing: 3
Story Choices: 4
Political Affiliation: 3
Total=14
Formula calls for division by 4.
14/4=3.5
3.5=Left Center Bias
Why is Van Zandt's rating
objectively more valid than ours? Or yours?
Here's more of Van Zandt's rating of PolitiFact.
Factual Reporting: VERY HIGH
World Press Freedom Rank: USA 43/180
Notes: PolitiFact.com is a project operated by the Tampa Bay Times, in which reporters and editors from the Times and affiliated media outlets “fact-check statements by members of Congress, the White House, lobbyists and interest groups”. They publish original statements and their evaluations on the PolitiFact.com website, and assign each a “Truth-O-Meter” rating. The ratings range from “True” for completely accurate statements to “Pants on Fire” (from the taunt “Liar, liar, pants on fire”) for false and ridiculous claims. Politifact has been called left biased by Extreme right wing and questionable sources. Our research indicates that Poltifact [sic] is an accurate fact checker and is considered the gold standard for political fact checking. (7/10/2016)
Source: http://www.politifact.com/
Notice the biased language from Van Zandt? Van Zandt only allows that PolitiFact has been called left-leaning by "Extreme right wing and questionable sources." In fact, PolitiFact has been called left-biased by many sources, including the non-partisan
Allsides Project.
Van Zandt even has an opt-in poll on his PolitiFact page asking visitors how they rate PolitiFact's bias. Most of the respondents disagree with the site's rating of PolitiFact.
Over 50 percent of Van Zandt's respondents rated PolitiFact biased to the left. Does that mean that all those 2,000+ people were "Extreme right wing" or "questionable sources"?
Note: I voted "Left-Center."
Why is PolitiFact called the "gold standard" for fact checking instead of FactCheck.org, or even Zebra Fact Check? That's a mystery.
The crux of the matter
The temptation of subjective rating scales is obvious, but such scales misinform readers and probably tend to mislead their creators as well.
A rating scale that fails to base its ratings on quantifiable data is worthless. Van Zandt's ratings are worthless except to tell you his opinion.
Opinions about PolitiFact's bias start to have value when backed by specific, quantifiable findings. We've taken that approach for years here at PolitiFact Bias. When we see the biased headline, we write a post about it if it's of sufficient note. When we see the bad reporting, we write a post about it and document PolitiFact's failure with reliable sourcing. When we see PolitiFact skewing its story choices in a way that unfairly harms conservatives (or liberals), we write an article about it. When we see systematic signs of bias in PolitiFact's ratings, we do
objective research on it.
We do that because specific examples trump subjective rating scales.
Until Dave Van Zandt adds objective markers to the MB/FC rating scales and justifies every rating with real objective data, take the ratings with a boulder of salt. They're worthless without specific backing data.
Afters
On its PolitiFact page, Media Bias/Fact Check links the flawed PolitiFact article we fisked
here.
"VERY HIGH" factual reporting.
Hmm.
Update August 11, 2018:
Dave Van Zandt
contacted us on Aug. 9, 2018 to say MB/FC has changed its rating of PolitiFact to "Left-Center." But we can't find any evidence the change occurred so we have no response yet to the supposed change (perhaps Van Zandt's message was simply in error, intending to inform us of a more subtle shift in the rating).
The Internet Archive pretends to have plenty of saves but the only one it shows (when we checked) seems to be
from January 2018.
The latest live version contains the following, which seems short of moving PolitiFact to the "Left-Center" category:
Overall, this update reveals a slight leftward shift in Politifact’s fact checking selection, but not enough to move them from the least biased category. (7/10/2016) Updated (D. Van Zandt 7/15/2018)
That seems a bit wishy-washy. Subjectivity can have that effect.
Update 2, June 6, 2021:
MB/FC updated its rating of PolitiFact to "Left-Center Bias" with an update time-stamped 04/28/2021:
Overall, we rate Politifact Left-Center Biased based on fact checks that
tend to be more favorable for the left. We also rate them High for
factual reporting and a credible fact-checker that is not without bias.
(7/10/2016) Updated (D. Van Zandt (4/28/2021)
Van Zandt's 2021 update vanishes the 2018 update. It would be better to keep the publication and update dates all intact and link each to an appropriate URL at the Internet Archive.
The fact that Van Zandt has come around to our position does not mean that either one of us has rendered an objective judgment of PolitiFact's bias.
In our opinion, if PolitiFact's bias has shifted left since 2018 it was by a fraction. And we would say Van Zandt continues to overestimate PolitiFact's reliability.
