Thursday, June 21, 2012

Bill Adair still ignoring the poison of selection bias

In an otherwise okay article about adapting journalism to the Internet, PolitiFact editor Bill Adair ended up making his usual pitch for pretending that journalists' selection bias does not affect fact check stories:
At PolitiFact, we’ve created two new forms. Instead of traditional articles, our Truth-O-Meter fact-checks are a new form that allows you to see a politician’s report card, to see all fact-checks on a subject or see all the Pants on Fire ratings. We can make larger journalistic points through the automatic tallies and summaries of our work.

We’ve done the same thing with the Obameter and the other meters we use for tracking campaign promises. The unit of journalism is the promise and then we write updates and rate whether the promise is kept or broken. The promises also get tallied so you can see how the politician is doing.
PolitiFact's "two new forms" of journalism encourage readers to accept a non-scientific process as science.  PolitiFact and Bill Adair continually sell the falsehood that its candidate report cards serve as a useful guide for judging whom to support in an election.

He's not entirely wrong.

PolitiFact's left-leaning bias makes its report cards primarily useful to liberals.

For review, here's how it works.

PolitiFact writes fact check stories its left-of-center staff thinks are relevant and interesting.  Left-of-center staffers find stories favoring their left-of-center views more interesting than others.  This naturally results in more fact checks of conservatives since left of center staffers question conservative ideas more than those jibing with their own.  Should PolitiFact experience concerns that its fact checking looks overly focused on conservative claims, it picks a few claims by liberals to help balance the ledger--claims that were in the first place less likely to warrant a fact check by left-of-center lights.  As a result, to no one's surprise, liberals tend to fare better on the report cards.  Liberals feel no surprise because of confirmation bias.  Conservatives feel no surprise because they've learned to expect biased reporting from journalists.

PolitiFact's defenders like to suggest that conservatives perceive a non-existent bias.  The simplest explanation, some of them say, is that conservatives simply lie more.

That response ignores two absolutely obvious points. 

Number one:  Journalists tend to lean to the political left of the general population.  The difference is not small.

Number two:  The story selection process--indeed, PolitiFact's entire process--might as well be designed to reinforce liberal bias.  There's no check on selection bias, and the three-editor panels PolitiFact uses to assign grades help ensure that the liberal majority in the newsroom decides the ratings by popular vote on any occasions where the voting isn't done only by liberal judges.

PolitiFact's process requires those who understand its workings to suspend disbelief when looking at its report cards and collected numbers.

No comments:

Post a Comment