PFB Semi-Smackdown: Kossack "Brash Equilibrium"

Ordinarily, we use the "PFB Smackdown" feature to critique liberal criticisms of PolitiFact.  This "Semi-Smackdown" deals with something different, a misrepresentation of this blog along with yet another irresponsible attempt to use the ratings of mainstream fact checkers to draw conclusions about the persons whose statements they rate.

Our pseudonymous subject goes by "Brash Equilibrium."  Brash goes through the trouble of adding Kessler's Pinocchios together with PolitiFact's "Truth-O-Meter" ratings and then calculates confidence intervals for various sets of ratings, based on the apparent assumption that the selection of stories is essentially random.

And there's this (bold emphasis added):

If instead we believed like a moderate conservative that the true comparison was reversed - that is, if we believed that Obiden spewed 17% more malarkey than Rymney - then it suggests that the fact checkers's [sic] average bias is somewhere between 16% and 54% for the Democrats, with a mean estimated bias of 34%.

It seems unrealistic to me that PolitiFact and The Fact Checker are on average that biased against the Republican party, even subconsciously. So while I think it's likely that bias could inflate the difference between the Republicans and Democrats, I find it much less likely that bias has reversed the comparison between the two tickets. Of course, these beliefs are based on hunches. Unlike politifactbias.com's rhetoric and limited quantitative analysis, however, it is based on good estimates of the possible bias, and our uncertainty in it.

It's hard to believe Brash put much time into any investigation of our rhetoric and analysis, considering his estimates ignore one of our favorite pet peeves, selection bias.  It's a waste of time calculating confidence intervals if the data set exhibits a significant degree of selection bias.  Giving our site more than a cursory read should have informed Brash on that point.

Our case against PolitiFact is based on solid survey data showing a left-of-center ideological tendency among journalists, an extensive set of anecdotes showing mistakes that more often unfairly harm conservatives and our own study of PolitiFact's bias based on its ratings.

Our study does not have a significant selection bias problem.

Brash's opinion of PolitiFact Bias consists of an assertion without any apparent basis in fact.

Brash:

We need more large-scale fact checking institutions that provide categorical rulings like The Fact Checker and PolitiFact. The more fact checker rulings we have access to, the more fact checker rulings we can analyze and combine into some (possibly weighted) average.

How often have we said it?

Lacking a control for selection bias, the aggregated ratings tell us about PolitiFact and The Fact Checker, not about the subjects whose statements they grade.

We need fact checkers who know how to draw the line between fact and opinion.  And critics who know enough to whistle a foul when "fact checkers" cross the line and conflate the two.



Correction Oct 28, 2012, 11:30 a.m.:  The second paragraph was originally published in this post minus its true beginning, "It seems unrealistic to me that."