National Press Foundation:
How do you avoid selection bias?Shorter Bill Adair: "We don't avoid selection bias."
There are many things that go into deciding what we are going to choose. We try to be timely, we try to stay on top of the news and we try to have balance so we check people from both parties. That can be challenging though, because if you have eight voices speaking up in a Republican primary and only one Democratic incumbent – naturally you have eight times the number of statements being made on the Republican side than on the Democratic side. We try to check roughly the same number of claims by Democrats as we do for Republicans, but we have to go where the claims are and lately there have been more made by Republicans. In terms of avoiding selection bias, I think the key is to be guided by what serves the reader. Once you get past claims selections, our fact-check process is entirely driven by journalistic and independent assessment.
The guy just doesn't get it.
Selection bias is simply the non-random selection of subject matter. Adair has described two different methods of story selection. First, PolitiFact tries to sniff out suspicious-sounding claims. Second, PolitiFact tries to pick out roughly equal numbers of claims by Democrats and Republicans.
That's a recipe for producing selection bias, not a method for minimizing it.
Jeff adds: Adair's misdirection may placate PolitiFact's sympathetic followers, but his excuse that "naturally you have eight times the number of statements being made on the Republican side than on the Democratic side" because of the current campaigning doesn't pass the laugh test. That may be a plausible reason if PolitiFact limited it's fact checking only to current presidential contenders, but that's not the case.
PolitiFact checks statements by Congressmen, aging pop singers, television personalities, Senate conferences, and chain emails. Are right leaning Congressmen making more statements because of the upcoming election? Are conservative pop stars issuing statements at an 8-1 ratio to their liberal counterparts? No.
And as we've explained before, simply choosing an equal number of statements by opposing parties is an insufficient method to avoid selection bias. Adair knows this, and his evasion on the issue would be embarrassing if it wasn't so insulting.