NTSH: PolitiFact twice misrepresented

This "Nothing To See Here" post looks at two media sources who publicly misrepresented PolitiFact.

We think PolitiFact may be expected protect its reputation from misrepresentations coming from mainstream sources.

We have two recent examples of such misrepresentation.

Thomas Baekdal

Thomas Baekdal bills himself as a new media analyst. Apparently some people take him seriously. But Baekdal tried to support his views public misinformation using data from PolitiFact, data that Baekdal massaged with his own formula, as he explains:

I came up with another system. It's based on a logarithmic scale which works like this:

• We give 'half-true' a value of 1 and center it on the graph.
• We give 'Mostly True' a value of 2, and 'True' a value of 5. The idea here is that we reward not just that something is true, but also that it provides us with the complete picture (or close to it).
• Similarly, we punish falsehoods. So, 'Mostly False' is given a value of -2, and 'False' a value of -5.
• Finally, we have intentional falsehoods, the 'Pants on Fire', which we punish by giving it a value of -10.

Sound reasonable?

No, it doesn't sound reasonable.

It's not reasonable because, regarding the final point, PolitiFact does not define the "Pants on Fire" rating as Baekdal does, and in fact does not take deceitful intent into account in doling out its ratings.

So Baekdal is misinforming people by misrepresenting PolitiFact's rating scale and purpose. Nothing to see here? Should PolitiFact just look the other way? There's an incentive to do that: Baekdal claims the data show PolitiFact's impartiality. PolitiFact might like to preserve that fiction.

Doyle McManus

Doyle McManus works in Washington D.C. as an L.A. Times political columnist. McManus wrote a thinly-veiled "Hillary for President" column published on May 18, 2016. The column, "How brazenly can a candidate lie and get away with it? We're going to find out with Donald Trump," like Baekdal's article, misrepresented PolitiFact:

Trump fibs so often that the fact-checking website Politifact awarded him its 2015 “Lie of the Year” award for his entire body of work, a lifetime achievement award for prevarication.

It's very easy to check PolitiFact's reason for awarding its 2015 "Lie of the Year" to "the campaign misstatements of Donald Trump."

In considering our annual Lie of the Year, we found our only real contenders were Trump’s -- his various statements also led our Readers’ Poll. But it was hard to single one out from the others. So we have rolled them into one big trophy.

To the candidate who says he’s all about winning, PolitiFact designates the many campaign misstatements of Donald Trump as our 2015 Lie of the Year.

The question is, how did McManus and the Times manage to get the facts wrong?

Should PolitiFact let this misinformation stand, or call on its PunditFact unit to fact check McManus?

Is there nothing to see here?


Afters:

It's worth noting that we have taken action to try to correct both errors, contacting Baekdal via Twitter and email and McManus with a comment on his article as well as Twitter. Neither man seems so far inclined to fix his errors. We will update on this point if we discover either has corrected the record.

Thomas Baekdal: 'These graphs also illustrate how impartial PolitiFact is' (Updated)

Danish blogger Thomas Baekdal wrote up a lengthy piece on public misinformation, "The increasing problem with the misinformed," published on March 7, 2016. The piece caught our interest because Baekdal used graphs of PolitiFact data and made some intriguing assertions about PolitiFact, particularly the one quoted in our title. Baekdal said his graphs show PolitiFact's impartiality.

We couldn't detect any argument (premises leading via logical inference to a conclusion) in the article supporting Baekdal's claim, so we wrote to him asking for the explanation.

Then a funny thing happened.

We couldn't get him to explain it, not counting "the data speaks for itself."

So, instead of a post dealing with Baekdal's explanation of his assertion, that his graphs show PolitiFact's impartiality, we'll go over a few points that cause us to doubt Baekdal and his conclusion.

1) Wrong definition

Jeff was the first to respond to Baekdal's article, correctly noting on Twitter that Baekdal misrepresented PolitiFact's "Pants on Fire" rating. Baekdal wrote "(T)hey also have a 6th level for statements where a politician is not just making a false statement, but is so out there that it seems to be intentionally misleading." But PolitiFact never explicitly describes its "Pants on Fire" rating as indicative of deliberate deceit. The closest PolitiFact comes to that is calling the rating "Pants on Fire." That's a point we've criticized PolitiFact over repeatedly. It protests that it doesn't call people liars, but its lowest rating makes that accusation implicitly via association with the popular rhyme "Liar, liar, pants on fire!" Readers sometimes take the framing to heart. Perhaps Baekdal was one of those.

2) An unsupported assertion

Baekdal asserted, as a foundation for his article, "(W)e have always had a problem with the misinformed, but it has never been as widespread as it is today." But he provided no evidence supporting the assertion. Should we risk aggravating our misinformed state by accepting his claim without evidence? Or maybe extend a license for hyperbole?

3) Unscientific approach

Baekdal prepares his readers for his PolitiFact graph presentations by noting potential problems with sample size, but never talks about how using a non-representative sample will undercut generalizations from his data. We think a competent analyst would address this problem.

4) Unscientific approach (2)

Baekdal uses an algorithm to score his PolitiFact data, statistically punishing politicians for telling intentional falsehoods if they received a "Pants on Fire" rating. But PolitiFact never provides affirmative evidence in its fact checks that a falsehood was intentional. The raw data do not show the wrong Baekdal claims to punish with his algorithm. Baekdal punishes others for his own misinterpretation of the data.

5) Unscientific approach (3)

Jeff also pointed out via Twitter that Baekdal accepts (without question) the dependability of PolitiFact's ratings. Baekdal offers no evidence that he considered PolitiFact might have a poor record of accuracy.

Conclusion

What point was Baekdal trying to make with his PolitiFact stats? It looks like he was trying to show the unreliability of politicians and pundits. Why? To highlight concern that people feel more mistrust for the press than for politicians. Baekdal lays a big share of the blame on the press, but apparently fails to realize PolitiFact is guilty of many of the problems he describes in his criticism of the press, such as misleading headlines.

We see no reason to trust Baekdal's assessment of PolitiFact's impartiality, or his assessment of anything else for that matter. His research approach is not scientific, failing to account for reliability of data, the reality of selection bias or alternative explanations of the data. His unwillingness to justify his claims via email did nothing to change our minds.

We continue to extend our invitation to Baekdal to explain how his graphs support PolitiFact's impartiality.


Hat tip to Twitterer @SatoshiKsutra for bringing Baekdal's article to our attention.

Update May 16, 2016:

Baekdal responds:

And this after we did him the favor of not publicly parsing his email responses.

Baekdal has something in common with some of the likewise thick-skinned folks at PolitiFact.