PFB Semi-Smackdown: Kossack "Brash Equilibrium"

Ordinarily, we use the "PFB Smackdown" feature to critique liberal criticisms of PolitiFact.  This "Semi-Smackdown" deals with something different, a misrepresentation of this blog along with yet another irresponsible attempt to use the ratings of mainstream fact checkers to draw conclusions about the persons whose statements they rate.

Our pseudonymous subject goes by "Brash Equilibrium."  Brash goes through the trouble of adding Kessler's Pinocchios together with PolitiFact's "Truth-O-Meter" ratings and then calculates confidence intervals for various sets of ratings, based on the apparent assumption that the selection of stories is essentially random.

And there's this (bold emphasis added):

If instead we believed like a moderate conservative that the true comparison was reversed - that is, if we believed that Obiden spewed 17% more malarkey than Rymney - then it suggests that the fact checkers's [sic] average bias is somewhere between 16% and 54% for the Democrats, with a mean estimated bias of 34%.

It seems unrealistic to me that PolitiFact and The Fact Checker are on average that biased against the Republican party, even subconsciously. So while I think it's likely that bias could inflate the difference between the Republicans and Democrats, I find it much less likely that bias has reversed the comparison between the two tickets. Of course, these beliefs are based on hunches. Unlike politifactbias.com's rhetoric and limited quantitative analysis, however, it is based on good estimates of the possible bias, and our uncertainty in it.

It's hard to believe Brash put much time into any investigation of our rhetoric and analysis, considering his estimates ignore one of our favorite pet peeves, selection bias.  It's a waste of time calculating confidence intervals if the data set exhibits a significant degree of selection bias.  Giving our site more than a cursory read should have informed Brash on that point.

Our case against PolitiFact is based on solid survey data showing a left-of-center ideological tendency among journalists, an extensive set of anecdotes showing mistakes that more often unfairly harm conservatives and our own study of PolitiFact's bias based on its ratings.

Our study does not have a significant selection bias problem.

Brash's opinion of PolitiFact Bias consists of an assertion without any apparent basis in fact.

Brash:

We need more large-scale fact checking institutions that provide categorical rulings like The Fact Checker and PolitiFact. The more fact checker rulings we have access to, the more fact checker rulings we can analyze and combine into some (possibly weighted) average.

How often have we said it?

Lacking a control for selection bias, the aggregated ratings tell us about PolitiFact and The Fact Checker, not about the subjects whose statements they grade.

We need fact checkers who know how to draw the line between fact and opinion.  And critics who know enough to whistle a foul when "fact checkers" cross the line and conflate the two.



Correction Oct 28, 2012, 11:30 a.m.:  The second paragraph was originally published in this post minus its true beginning, "It seems unrealistic to me that." 

PFB Smackdown x2: Daily (Kos) double

Unfortunately it remains far easier to locate poor criticism of PolitiFact than it is to locate good criticism.

First up for PFB Smackdown, "Hunter" from the Daily Kos. Hunter thinks that PolitiFact blew its rating of Mitt Romney's claim that he never voted for a Democrat if a Republican was on the ballot.

Hunter:

(P)eople had a wee bit of a problem with this, because the context was Romney's vote for Democrat Paul Tsongas in the 1992 Democratic presidential primary. The Republicans certainly were having a primary that day as well: The incumbent president, George H.W. Bush, was running against Pat Buchanan. Now we can all look back now and have a good laugh at permanent cable news fixture Pat Buchanan taking on the incumbent president, but they both were certainly "on the ballot" in the 1992 primaries. So Mitt's completely making stuff up on this one—his critics have got him dead to rights.

One should not ignore the fact that the Democratic and Republican presidential primaries are two different elections.  There was no Republican on the ballot for the Democratic presidential primary.  Given the context, it is overpoweringly obvious that Romney was saying that he votes for Republicans whenever Republicans are on the ballot against Democrats, as with a general election.  The critics have to ignore one of the primary salient facts to have Romney "dead to rights."  We agree on the point that PolitiFact blew the rating, though for different reasons.  The context makes his statement true, assuming the Tsongas case is the worst would-be exception.

Next up, "dcg2," also writing for the Daily Kos, noticed a supposed trend with PolitiFact's front page material:

I took a quick look at Politifact's home page, and -- surprise, surprise... found two more quick examples of their front page of taking statements from Democrats that they admit are true, but calling them half-true anyway. Just a quick look at the Politifact's front page shows even more outrages -- all against Democrats ...

The author included no list of stories in his post, so it's hard to verify his claim to some extent, but it seems likely dcg2 was somehow able to ignore PolitiFact's flubs of Obama's milk regulations claim, Romney's voting claim and Mitch Daniels claim about the percentage of those under 30 years of age not going to work for the day ("Because many of them were in school"!).

PolitiFact announced in the summer of 2011 that it would start grading statistical statements in light of arguments regarding cause and effect.  Critics like these two Kossacks simply ignore relevant data.  And that makes for poor critiques.