Thomas Baekdal: 'These graphs also illustrate how impartial PolitiFact is' (Updated)

Danish blogger Thomas Baekdal wrote up a lengthy piece on public misinformation, "The increasing problem with the misinformed," published on March 7, 2016. The piece caught our interest because Baekdal used graphs of PolitiFact data and made some intriguing assertions about PolitiFact, particularly the one quoted in our title. Baekdal said his graphs show PolitiFact's impartiality.

We couldn't detect any argument (premises leading via logical inference to a conclusion) in the article supporting Baekdal's claim, so we wrote to him asking for the explanation.

Then a funny thing happened.

We couldn't get him to explain it, not counting "the data speaks for itself."

So, instead of a post dealing with Baekdal's explanation of his assertion, that his graphs show PolitiFact's impartiality, we'll go over a few points that cause us to doubt Baekdal and his conclusion.

1) Wrong definition

Jeff was the first to respond to Baekdal's article, correctly noting on Twitter that Baekdal misrepresented PolitiFact's "Pants on Fire" rating. Baekdal wrote "(T)hey also have a 6th level for statements where a politician is not just making a false statement, but is so out there that it seems to be intentionally misleading." But PolitiFact never explicitly describes its "Pants on Fire" rating as indicative of deliberate deceit. The closest PolitiFact comes to that is calling the rating "Pants on Fire." That's a point we've criticized PolitiFact over repeatedly. It protests that it doesn't call people liars, but its lowest rating makes that accusation implicitly via association with the popular rhyme "Liar, liar, pants on fire!" Readers sometimes take the framing to heart. Perhaps Baekdal was one of those.

2) An unsupported assertion

Baekdal asserted, as a foundation for his article, "(W)e have always had a problem with the misinformed, but it has never been as widespread as it is today." But he provided no evidence supporting the assertion. Should we risk aggravating our misinformed state by accepting his claim without evidence? Or maybe extend a license for hyperbole?

3) Unscientific approach

Baekdal prepares his readers for his PolitiFact graph presentations by noting potential problems with sample size, but never talks about how using a non-representative sample will undercut generalizations from his data. We think a competent analyst would address this problem.

4) Unscientific approach (2)

Baekdal uses an algorithm to score his PolitiFact data, statistically punishing politicians for telling intentional falsehoods if they received a "Pants on Fire" rating. But PolitiFact never provides affirmative evidence in its fact checks that a falsehood was intentional. The raw data do not show the wrong Baekdal claims to punish with his algorithm. Baekdal punishes others for his own misinterpretation of the data.

5) Unscientific approach (3)

Jeff also pointed out via Twitter that Baekdal accepts (without question) the dependability of PolitiFact's ratings. Baekdal offers no evidence that he considered PolitiFact might have a poor record of accuracy.

Conclusion

What point was Baekdal trying to make with his PolitiFact stats? It looks like he was trying to show the unreliability of politicians and pundits. Why? To highlight concern that people feel more mistrust for the press than for politicians. Baekdal lays a big share of the blame on the press, but apparently fails to realize PolitiFact is guilty of many of the problems he describes in his criticism of the press, such as misleading headlines.

We see no reason to trust Baekdal's assessment of PolitiFact's impartiality, or his assessment of anything else for that matter. His research approach is not scientific, failing to account for reliability of data, the reality of selection bias or alternative explanations of the data. His unwillingness to justify his claims via email did nothing to change our minds.

We continue to extend our invitation to Baekdal to explain how his graphs support PolitiFact's impartiality.


Hat tip to Twitterer @SatoshiKsutra for bringing Baekdal's article to our attention.

Update May 16, 2016:

Baekdal responds:

And this after we did him the favor of not publicly parsing his email responses.

Baekdal has something in common with some of the likewise thick-skinned folks at PolitiFact.