Sunday, January 22, 2012

Bill Adair describes selection bias at PolitiFact

We've noted before, along with Eric Ostermeier, that PolitiFact's self-described methods of choosing stories amount to a recipe for selection bias.  The Parker Report recently published an interview with PolitiFact editor Bill Adair that describes more of the same:
How do you decide which reporters take on which claims? How many people check PolitiFact’s facts before publishing? In other words, what is the process? 

Here’s how it works: Every day, our interns look through transcripts, campaign videos, news coverage and interviews for factual claims. The editors review them and choose the claims to fact-check based on whether they are timely, newsworthy and whether people will wonder if the claim is true. That is our biggest criteria for selecting a claim to check — to satisfy people’s curiosity.
Most of the time, our reporters choose the claims they want to check. They do the research and write the article, which typically takes a day.

The articles, which include a recommended Truth-O-Meter rating, are then edited by one of our editors and then reviewed by a three-editor panel. The panel makes the final decision on the rating.

Though Adair does not mention it in this interview, PolitiFact also solicits fact check ideas from readers (see image below).

(clipped from

If those who send in story suggestions are predominantly liberal or predominantly conservative, this feature will tend to skew story selection toward one political pole or the other.

Likewise, in Adair's description we have two factors that will help introduce partisan bias.  First, if the interns' political biases influence their choices of stories to send to the editors, then the system again will tend to skew story selection toward one political pole or the other.  Second, if the editors use their own sense of curiosity in choosing which statements to rate we have another case where ideology may influence the selection process left or right.

The three-editor panel that chooses the final "Truth-O-Meter" rating will by its nature tend to preserve a majoritarian ideological bias in PolitiFact's ratings.  Suppose the panel has two conservatives and one liberal.  In any rating where ideology serves as an influence, the two conservatives could consistently outvote the liberal.

Interviewer Erik Parker did not ask Adair any particularly tough questions (at least it doesn't appear on the record). Maybe next time.

No comments:

Post a Comment