Past questions about the dangers of selection bias should have amply prepared PolitiFact editors to answer this sort of question.
PolitiFact Ohio editor Robert Higgs of the Cleveland Plain Dealer took a shot at addressing the issue of balance in comments to the Plain Dealer's reader representative, Ted Diadium:
I asked Bob Higgs, the editor who oversees the PolitiFact Ohio operation, if he deliberately tries for balance:Even if PolitiFact Ohio applies its evaluation standards consistently to all its stories, balanced treatment need not result. In fact, it probably won't result.
"The belief is that if we apply the same constructive standards to all claims, we'll end up treating all sides fairly," he said. "Some of the state operations (there are 10 in the PolitiFact organization), as well as the national operation, do not tally the rankings at all."
Higgs admits that he does tally up the results by party (which shows them remarkably even), "but only to see after the fact how we've done."
It won't result because selection bias will occur without active steps taken to avoid the problem.
Select nine stories likely to make Democrats look bad while selecting one that will likely made Republicans look good will not achieve balance regardless of applying identical standards in the evaluation--not that we at PolitiFact Bias believe PolitiFact applies its standards consistently.
Newsflash for Higgs: Every writer and editor at PolitiFact likely has a sense of how the stories break down by party. The "remarkably even" count that results at PolitiFact Ohio helps prove the point.
PolitiFact markets its stats as candidate report cards and the like, but the real value of PolitiFact's numbers comes from the insights we obtain into PolitiFact's behavior--not the behavior of those featured in the stories.
Correction: Changed the first of two consecutive instances of "the" to a "from" in the concluding paragraph. Hat tip to Jeff Dyberg for catching the the error.
There aren't any steps that can be taken to eliminate selection bias, or any other cognitive bias for that matter. The effects can be mitigated by working in groups where the individual biases of the members are not well aligned. But no group can be constructed in a manner that fully overcomes cognitive biases.ReplyDelete
On the contrary, random story selection would eliminate bias in the selection of stories. Using identical evaluation techniques would likewise eliminate ideological bias, if we take for granted identical standards for using words in the stories, at least to the point of making bias undetectable.ReplyDelete
It's never our point that PolitiFact eliminate selection bias. Our point is that PolitiFact admit to selection bias. PolitiFact's failure to disclose the effects of selection bias ends up misleading its readers.