Thursday, October 29, 2015

Poynter.org: "Fact-checking the fact-checkers"

The Poynter Institute, the school for journalists that owns the Tampa Bay Times and PolitiFact, helped create an international organization for fact checkers. It appointed Alexios Mantzarlis, who sustains a connection with Italy's Pagella Politica, to lead the organization.

Mantzarlis published a fine exhortation to fact checkers on Oct. 28, 2015. Fact checkers, he said, need to self-correct aggressively, transparently and consistently.

That's a great and welcome message, but we have a quibble or two with some of Mantzarlis' content.

After his article, Mantzarlis' announced he was offering full disclosure, reminding readers of his association with Pagella Politica, which he had mentioned in his article:
Full disclosure: I was Managing Editor of Pagella Politica before joining Poynter and I remain on the board of that editorial project. I don’t intend for this fact to affect my coverage in any way, but I will specify it for transparency purposes any time I cover the Italian fact-checkers on this site.
Now that Mantzarlis works at Poynter, he should similarly note Poynter.org's relationship with PolitiFact when he mentions PolitiFact in his articles.

And speaking of PolitiFact, Mantzarlis mentioned the Poynter-owned fact checkers in his article:
A fact-checking organization should strive to do more than update its corrected article. If the error sheds a light on a flawed editorial practice, it should change that practice and write about it. Letting readers decide whether the organization has a good track record or not is also useful: PolitiFact, for example, groups all the fact-checks it has corrected.
Mantzarlis could accurately say PolitiFact publishes a policy stating it groups all the fact-checks it has corrected. But we've documented corrections that PolitiFact does not mention on its corrections page, and we have no reason to think PolitiFact has improved on that record.

A recent PunditFact (PolitiFact) fact check involving Sean Hannity will serve to illustrate.

Hannity said the Obama administration said it would allow 250,000 refugees into the United States. PunditFact ruled Hannity's claim "Pants on Fire" and proclaimed in a headline that Hannity had been snookered by a hoax website.

After the fact check appeared, Hannity contested the ruling, saying he was talking about administration announcements that it would increase the number of refugees taken into the United States this year and the next two years. The administration's numbers totaled 255,000 over three years.

Hannity's explanation jibes with the timing of his broadcast, so that means PolitiFact's headline was almost certainly false and unfounded.

So did PunditFact run a correction?

No, not at all.

PunditFact did a new version of the article and archived the old version. PunditFact makes no admission of any error of any kind. PunditFact gives itself a journalistic Mulligan.

How are people supposed to use an incomplete corrections page to judge the accuracy of a fact checker? An incomplete record of corrections leads to the same type of error that happens when readers try to judge candidates based on their "Truth-O-Meter" report cards.

Judge PolitiFact in part based on what it will not admit as a correction.


Afters

On a related noted, I tried on Oct. 29, 2015 to respond to Mantzarlis' article on the Poynter.org website. As of 6 p.m. the website had logged no responses to the article.

Perhaps my message will appear in time.
Mr. Mantzarlis, you make an excellent point about the importance of corrections in fact checking. But I'd suggest adding some context may help magnify the importance of your good advice.

1) Now that you work for Poynter, it's a good idea to mention Poynter's relationship with PolitiFact when you cover PolitiFact, just as you mention your former relationship with Pagella Politica.

2) Your summary of PolitiFact's policy does match what PolitiFact's policy statement. But what does PolitiFact do in practice? A recent fact check of Sean Hannity serves to illustrate how PolitiFact fudges below the best set of standard practices.

Hannity said the U.S. was planning on admitting 250,000 refugees. PolitiFact concluded Hannity had fallen for information at a hoax website and stated that conclusion as a fact in its headline. Hannity later offered a plausible explanation for his figure. PolitiFact posted a new version of the article and archived the old one without acknowledging any error at all.

Guess what? This case does not appear where PolitiFact supposedly groups all the fact checks it has corrected.

"The correction should include what caused the mistake and what led to the revision."

Yes! PolitiFact jumped to a conclusion (in the headline) based on weak evidence. Hannity's partly to blame for not responding to PolitiFact with his explanation. But that shouldn't excuse PolitiFact for printing its hypothesis as a fact. And isn't running a false headline worthy of a correction?

The new version of the fact check also has an issue with its handling of statistics. The article uses refugee data since 1980 to suggest that recent refugee numbers reflect business as usual at the State Department. The article leaves the impression that admitting about 83,000 refugees is about standard. PolitiFact says Hannity's 255,000 over three years is "about 6,000 refugees more than the recent annual average." But reporters should know better than to use averages where a list of numbers fluctuates widely. That's the case here. The median figure is about 72,000 since 1980 and the refugee numbers are particularly low in the post-9/11 era. So the proposed refugee numbers are substantially higher than the "recent annual" average and PolitiFact's readers will be none the wiser unless they take the trouble to look at the linked data.

http://www.politifact.com/pund...

I wish you great success in your new role.


Correction Dec. 29, 2017: Changed "we was" to "he was." We don't always post a correction notice when fixing obvious typos, but this time seemed like a good exception.