PFB Semi-Smackdown: Kossack "Brash Equilibrium"

Ordinarily, we use the "PFB Smackdown" feature to critique liberal criticisms of PolitiFact.  This "Semi-Smackdown" deals with something different, a misrepresentation of this blog along with yet another irresponsible attempt to use the ratings of mainstream fact checkers to draw conclusions about the persons whose statements they rate.

Our pseudonymous subject goes by "Brash Equilibrium."  Brash goes through the trouble of adding Kessler's Pinocchios together with PolitiFact's "Truth-O-Meter" ratings and then calculates confidence intervals for various sets of ratings, based on the apparent assumption that the selection of stories is essentially random.

And there's this (bold emphasis added):

If instead we believed like a moderate conservative that the true comparison was reversed - that is, if we believed that Obiden spewed 17% more malarkey than Rymney - then it suggests that the fact checkers's [sic] average bias is somewhere between 16% and 54% for the Democrats, with a mean estimated bias of 34%.

It seems unrealistic to me that PolitiFact and The Fact Checker are on average that biased against the Republican party, even subconsciously. So while I think it's likely that bias could inflate the difference between the Republicans and Democrats, I find it much less likely that bias has reversed the comparison between the two tickets. Of course, these beliefs are based on hunches. Unlike politifactbias.com's rhetoric and limited quantitative analysis, however, it is based on good estimates of the possible bias, and our uncertainty in it.

It's hard to believe Brash put much time into any investigation of our rhetoric and analysis, considering his estimates ignore one of our favorite pet peeves, selection bias.  It's a waste of time calculating confidence intervals if the data set exhibits a significant degree of selection bias.  Giving our site more than a cursory read should have informed Brash on that point.

Our case against PolitiFact is based on solid survey data showing a left-of-center ideological tendency among journalists, an extensive set of anecdotes showing mistakes that more often unfairly harm conservatives and our own study of PolitiFact's bias based on its ratings.

Our study does not have a significant selection bias problem.

Brash's opinion of PolitiFact Bias consists of an assertion without any apparent basis in fact.

Brash:

We need more large-scale fact checking institutions that provide categorical rulings like The Fact Checker and PolitiFact. The more fact checker rulings we have access to, the more fact checker rulings we can analyze and combine into some (possibly weighted) average.

How often have we said it?

Lacking a control for selection bias, the aggregated ratings tell us about PolitiFact and The Fact Checker, not about the subjects whose statements they grade.

We need fact checkers who know how to draw the line between fact and opinion.  And critics who know enough to whistle a foul when "fact checkers" cross the line and conflate the two.



Correction Oct 28, 2012, 11:30 a.m.:  The second paragraph was originally published in this post minus its true beginning, "It seems unrealistic to me that." 

PFB Smackdown: Dylan Otto Krider vs. Jon Cassidy (Updated)

Dylan Otto Krider's orbit around truth-hustler Chris Mooney helped bring him into direct conflict with Ohio Watchdog writer Jon Cassidy this week.  We've featured some of Cassidy's PolitiFact critiques here at PolitiFact Bias, and our review of Cassidy's longer article for Human Events is pending.

Krider, like Mooney, believes that statistics from PolitiFact indicating that Republicans receive harsher ratings than Democrats help show that Republicans simply have a more cavalier attitude toward telling the truth.  Cassidy's Human Events story challenged that interpretation and prompted a story in reply from Krider.

Krider's central point carries partial merit.  He challenges Cassidy's headline with his own:  "Does PolitiFact say Republicans lie nine times more? Really?"

That answer to that question is "no," but Krider used specious reasoning to reach the conclusion.  Examples follow.

PFB Smackdown: Mediaite's Tommy Christopher (Updated/Corrected)

We use the PFB Smackdown feature to critique the worst of the left's best critiques of PolitiFact.


I get another excuse to say the political Left's critiques of PolitiFact are generally poor, thanks to Mediaite's political editor and White House correspondent Tommy Christopher.

Take it away, Christopher:

Political ads have been an especially hot topic this week, with surrogates from both presidential campaigns alternately citing, and arguing with, vaunted fact-checking outfits like the Pulitzer Prize-winning Politifact. Although controversial rulings have eroded the magic of such efforts, it is worth noting that, by Politifact’s numbers, Republican presidential candidate Mitt Romney is 58% more likely to lie than President Obama.

What magic?

PolitiFact's ratings have always drawn well-deserved criticism, from Bill Adair's brain-dead analysis of Joe Biden's hyperbole through last week's continuation of PolitiFact's series of misdirections about effective tax rates.  Why is it worth noting PolitiFact's comparison of Romney to Obama after we add the problem of selection bias to PolitiFact's inability to apply consistent standards or even achieve a reasonable minimum standard of quality?

Other than the fact that it might serve Christopher's politics, that is?  It's hard not to notice that both of Christopher's examples of supposed "controversial rulings" allegedly caused unfair harm to the Left.

It doesn't take many blown calls to produce an 58 percent difference between two individuals' "Truth-O-Meter" report cards, nor does it take much selection bias to produce that type of difference.

In short, contrary to Christopher's suggestion, aggregating PolitiFact's ratings is a useless exercise for purposes other than evaluating PolitiFact.

We have at least two examples of the latter so far:


"Selection Bias? PolitiFact Rates Republican Statements as False at 3 Times the Rate of Democrats"
"Bias in PolitiFact’s ratings: Pants on Fire vs. False"

The utility of its "report cards" stands as one of PolitiFact's most spectacular lies.  Don't buy it.


Update Aug. 14, 2012:  

Tommy Christopher tweets in response:

@PolitiFactBias Before you "smack" anyone "down," you ought to learn some math. It was a 17-point difference, not 8. Romney 46% Obama 29%

Christopher has a point in that the numbers I used were incorrect.  The passage was intended from the first to read "a 58 percent difference," and with this update that reading shows above.  My primary mistake was in failing to see a typographical error instead of a math error when I did yesterday's correction.

The change in percentage does not significantly affect the thrust of the criticism of Christopher's claim.  Without a control on selection bias,--and there is no good evidence of any such control--using PolitiFact's ratings other than to find out things about PolitiFact just doesn't make sense.

Will Christopher address that point or allow the typographical error to serve as a red herring covering for his mushy thinking?


Correction Aug. 13, 2012:  Math mistake:  Was:  "an 8 percent difference."  Is:  "an 8 percentage point difference."  Apologies for the error.
Correction correction Aug. 14, 2012:  See update above. 
Corrected update Au. 14, 2012:   Thanks to Jeff Dyberg for pointing out that I had incorrectly posted the original intent as "a 58 percentage point difference.  Rather, the intended figure was as Christopher expressed it, as a 58 percent difference.

 

PFB Smackdown: Rachel Maddow (again) and White House baby screening

It does seem that MSNBC host Rachel Maddow has gone a tad unhinged over PolitiFact.

Maddow's latest complaint about PolitiFact stems from a "Mostly False" rating PolitiFact gave to a pro-life/anti-abortion group over its claim that the White House recognizes the existence of unborn babies for security purposes.

Maddow's problem seems to come from the fact that she doesn't recognize that PolitiFact's rating system weights toward the middle ratings because of compound statements and underlying arguments. 

Watch:



Maddow seems inches from screaming (3:05) "It's got to be either true or false!"

Clearly Maddow believed the "Mostly False" rating from PolitiFact was inflated.  But she made her case with cherry-picked information.  And her choices weren't so surprising when PolitiFact did much the same thing, though not quite to the extreme Maddow took it.

This is the key line from the White House e-mail, which neither Maddow nor PolitiFact saw fit to mention (from the National Right to Life press release):

"We have received a number of calls regarding how to enter security information for a baby that has not yet been born," Shafer wrote.

The PolitiFact telling picks up with Shafer's next sentence (bold emphasis added):

The release was a response to an early-morning email from the White House Visitors Office detailing how to record the personal information of babies still in utero.

"Crazy as it may sound, you MUST include the baby in the overall count of guests in the tour. It’s an easy process," Visitors Office director Ellie Schafer wrote to congressional staffers, specifying that nine zeros should be filled in for the infant’s Social Security Number.

Using just the second sentence, it's very easy to charge National Right to Life with distorting the meaning of the email newsletter.  But with the addition of the preceding sentence it is plain that Shafer is giving instructions on entering security information for a baby that has not yet been born.  The first sentence helps illuminate why Shafer says it sounds crazy to include the (unborn) baby in the overall count.  The PolitiFact version at least introduces the quotation with a helpful paraphrase ("detailing how to record the personal information of babies still in utero").

The White House instruction to provide security information for the unborn baby, clear in the White House email, was the fulcrum on which the NRTL built its implication of White House hypocrisy.

But we need to clarify which fact PolitiFact chose to check.

(clipped from PolitiFact.com)

The top portion of the PolitiFact image is a suitably accurate representation of the NRTL newsletter, and in light of the information already discussed, it can't be entirely false that the White House recognizes unborn babies in terms of its security procedures.  It sent out instructions to help expectant mothers fill out security information for babies not yet born.

The bottom portion of the PolitiFact image tends to mislead.  Saying the White House "screens" unborn babies conveys the impression that the babies have enough history to warrant some type of security threat, and implies that some unborn babies may not make the cut.  The NRTL newsletter doesn't mention anything about that type of screening and does not imply it.  On the contrary, the newsletter appears to take for granted that the White House asks for the information for the purpose of providing security for the visitors, including babies born or otherwise.

I'll go out on a limb and assume that if the White House security team has information about a pregnant woman visiting the White House on a day the White House is attacked, rescue efforts will take into account the fetus and take special action to help ensure its survival.  That's in line with the purpose of the NLRB press release.  There's a touch of hypocrisy in the policy and the NLRB newsletter doesn't overplay that angle.  If "Mostly False" isn't the correct rating then it should be higher than that.  Indeed, the NLRB's newsletter handled the truth more carefully than either Maddow or PolitiFact in this case.

Maddow's rant fits with a pattern of low-quality criticism of PolitiFact from the Left.

PFB Smackdown: Lawrence O'Donnell, Rachel Maddow and Tommy Christopher

Uh-oh!  Liberals are once again scandalized by a PolitiFact fact check!

MSNBC's Lawrence O'Donnell appeared in the following political ad:




PolitiFact looked into O'Donnell's claim about critics calling the GI Bill "welfare" and ruled it "Mostly False."   The fact check does have some problems.

PolitiFact went easy on O'Donnell

The fact check contains a huge error.  PolitiFact overlooks the fact that O'Donnell is making an equivocal argument.  O'Donnell stresses that the GI Bill was an education program.  But when PolitiFact pressed MSNBC to support O'Donnell's claim, the latter responded by providing criticisms that almost exclusively aimed at unemployment benefits that were part of the bill.  O'Donnell's argument is a bait-and-switch.

PolitiFact claims to take such abuse of context into account.  Some of the "Truth-O-Meter" grades, in fact, carry clear signs of the perils of making a claim with limited context.

Rachel Maddow

PFB Smackdown: Rachel Maddow (Updated)

We agree with Rachel Maddow up through about the 55 second mark.  Yes, PolitiFact is bad, and PolitiFact is so bad at fact checking that it doesn't deserve frequent citations as a trustworthy source. 

After that, our level of agreement starts to drop.

Sen. Rubio (R-Fla.) stated that most Americans are conservative and went on to argue the point based on attitudes toward the labels "conservative" and "liberal."

Maddow ignores the context of Rubio's remarks and attacks it using survey data about the way Americans self-identify politically.

Maddow is supposed to be ultra smart.  So how come she can't figure out that Rubio's statement isn't properly measured against self-identification numbers?

It appears that Maddow uncritically followed PolitiFact's approach to judging Rubio's accuracy.  The self-identification numbers serve as interesting context, but it's perfectly possible for 100 percent of Americans to self-identify as "liberal" yet reasonably classify as majority conservative.  That's because people can have inaccurate perceptions of their location on the political spectrum.

So, was Rubio correct that the majority of Americans are conservative?  That depends on his argument.  Rubio didn't cite surveys about self-identification.  He used a method concerned with attitudes toward the respective labels.  One can argue with the method or the application of the method, but using an inappropriate benchmark doesn't cut it.

When you ask people which party they lean toward, the independents split up so that the country is almost evenly divided. For the year of 2011, Gallup reported that 45 percent of Americans identified as Republicans or leaned that way, while 45 percent identified as Democrats or leaned that way.

Is "Republican" the same label as "conservative"?  No, of course not.

PolitiFact came close to addressing Rubio's point by looking at the political leanings of moderates, but fell short by relying on the wrong label along with the self-identification standard.  Maddow's approach was even worse, as she took Rubio's comment out of context and apparently expected PolitiFact to do the same thing.

Meanwhile, PolitiFact defends itself with the usual banalities:

“Our goal at PolitiFact is to use the Truth-O-Meter to show the relative accuracy of a political claim,” Adair explained. “In this case, we rated it Mostly True because we felt that while the number was short of a majority, it was still a plurality. Forty percent of Americans consider themselves conservative, 35 percent moderate and 21 percent liberal. It wasn’t quite a majority, but was close.”

“We don’t expect our readers to agree with every ruling we make,” he continued.

Pretty weak, isn't it?


Update 2/19/2012:

With a hat tip to Kevin Drum of Mother Jones (liberal mag), we have survey data that help lend support to Marco Rubio (as well as to my argument in his defense):

(click image to enlarge)

1)  The survey, from Politico and George Washington University, is limited to likely voters.
2)  The poll essentially forces likely voters to choose between "liberal" and "conservative."
3)  A plurality of those surveyed (43 percent) lean Democrat or self-identify as Democrat.
4)  Despite the plurality of Democrats in the survey sample, 61 percent identify as conservative ("Very conservative" or "Somewhat conservative").

PFB Smackdown x2: Daily (Kos) double

Unfortunately it remains far easier to locate poor criticism of PolitiFact than it is to locate good criticism.

First up for PFB Smackdown, "Hunter" from the Daily Kos. Hunter thinks that PolitiFact blew its rating of Mitt Romney's claim that he never voted for a Democrat if a Republican was on the ballot.

Hunter:

(P)eople had a wee bit of a problem with this, because the context was Romney's vote for Democrat Paul Tsongas in the 1992 Democratic presidential primary. The Republicans certainly were having a primary that day as well: The incumbent president, George H.W. Bush, was running against Pat Buchanan. Now we can all look back now and have a good laugh at permanent cable news fixture Pat Buchanan taking on the incumbent president, but they both were certainly "on the ballot" in the 1992 primaries. So Mitt's completely making stuff up on this one—his critics have got him dead to rights.

One should not ignore the fact that the Democratic and Republican presidential primaries are two different elections.  There was no Republican on the ballot for the Democratic presidential primary.  Given the context, it is overpoweringly obvious that Romney was saying that he votes for Republicans whenever Republicans are on the ballot against Democrats, as with a general election.  The critics have to ignore one of the primary salient facts to have Romney "dead to rights."  We agree on the point that PolitiFact blew the rating, though for different reasons.  The context makes his statement true, assuming the Tsongas case is the worst would-be exception.

Next up, "dcg2," also writing for the Daily Kos, noticed a supposed trend with PolitiFact's front page material:

I took a quick look at Politifact's home page, and -- surprise, surprise... found two more quick examples of their front page of taking statements from Democrats that they admit are true, but calling them half-true anyway. Just a quick look at the Politifact's front page shows even more outrages -- all against Democrats ...

The author included no list of stories in his post, so it's hard to verify his claim to some extent, but it seems likely dcg2 was somehow able to ignore PolitiFact's flubs of Obama's milk regulations claim, Romney's voting claim and Mitch Daniels claim about the percentage of those under 30 years of age not going to work for the day ("Because many of them were in school"!).

PolitiFact announced in the summer of 2011 that it would start grading statistical statements in light of arguments regarding cause and effect.  Critics like these two Kossacks simply ignore relevant data.  And that makes for poor critiques.

PFB Smackdown: "WisFactCheck" and Wisconsin job creation

We at PolitiFact Bias emphasize in part sets of anecdotes as an evidence of PolitiFact's ideological bias.  Where the combined sets of anecdotes soft on liberals and those harsh on conservatives outweigh their opposites, a solid (though short of absolute proof) evidence of bias results.

Since the comparison between the sets of anecdotes affects the case for bias at PolitiFact, the "PFB Smackdown" feature takes the attempts by liberals to show a right-leaning bias at PolitiFact and exposes the errors therein.

Which brings us to "WisFactCheck":

Another day, another outrageous claim from the right-leaning Politifact.

The latest eyerolling claim from Politifact is that Sen. Lena Taylor's claim that Walker "has not created at single job" was not accurate:

We at PFB admit that PolitiFact's work exhibits problems that harm both conservatives and liberals at times.  And, of course, we expect that the legitimate examples of the former exceed those for the latter.  So what's the evidence in this case?

PFB Smackdown: The TPM critique of PolitiFact

Joshua Marshall of Talking Points Memo joins the fray with his own critique of PolitiFact.  Marshall's critique, like that of Jonathan Chait reviewed here earlier, focuses on PolitiFact's grading of a Democratic Party ad attacking Republican policy on Medicare.  And even more specifically, Marshall criticizes the portions of a recent post at Columbia Journalism Review (also reviewed here) that mostly defended PolitiFact's reasoning while questioning the rating system itself.

TPM:

Did Republicans vote to 'end Medicare'?

Err, not really. As already mentioned, Republicans did not, as the ad suggests, vote to end Medicare. Rather, they voted--in the lower house--for a plan that would change Medicare, were it to reach the president's desk and be signed into law. Which it won't. The ad mentions none of this, instead leaving its bold claim hanging like a piñata for PolitiFact's batsmen.

'End Medicare' is the heart of the question. And as I've already repeatedly noted, ending a program that functions in one way (single-payer guaranteed medical insurance regardless of health status) with one that works in a fundamentally different manner (provide limited subsidies for private insurance which would quite possibly not exist for many seniors) and doesn't provide anything like the same service by any definition counts as 'ending' the program regardless of whether you give the latter program the same label. But look at the reasoning in the excerpt. One of the reasons the claim isn't 'true' apparently is because only the House voted for it so far, not the Senate. And the President would still have to sign it. And he probably won't. So since it likely won't become law in this Congress, House Republicans aren't even really voting to do it.

By that standard, Bernie Sanders doesn't really support single-payer because it's never going to become law.

 Let's look at Marshall's reasoning regarding the reasoning from PolitiFact and CJR.

"End Medicare" may be the heart of the question as Marshall says, but "vote to" is a big part of the claim as well--probably the bigger part of the claim since it constitutes the action supposedly taken without which "end Medicare" doesn't happen.

Marshall argues that Medicare under Ryan would be "fundamentally different" from Medicare, but isn't clear that private Medicare insurance is "fundamentally different" than Medicare single-payer insurance, contrary to what Marshall seems to contend.  Both, after all, qualify as insurance.  And suppose we cut Medicare funding under the present system to 10 percent of what Ryan's plan would pay.  Would Marshall argue that Medicare ceases to be Medicare at some point on the money/benefit continuum?  Or did Medicare stop being Medicare the moment it started paying for hip replacements?  Marshall doesn't delve into the principle, leaving his argument akin to a "slippery slope" argument.  Medicare is fundamentally a health insurance program for the elderly.  The name, in fact, is meant to express that idea and when it was coined Medicare was a far more limited program than it is today.  Marshall then extends the argument by suggesting that Medicare under Ryan "doesn't provide anything like the same service by any definition," but that's just hyperbole if he isn't simply lying. Marshall's case ultimately rests on his opinion.

Marshall also attacks the CJR for reasoning the bill will not reach the Senate.  But he apparently misunderstands the situation.  The bill the House passed is non-binding.  So even if the Senate passed it and the president signed it (which never happens with concurrent resolutions), it wouldn't carry the force of law.  Binding bills do that.

Marshall's argument is ridiculous.  I suppose he can try to blame Joel Meares of CJR for not explaining the details, but I consider Meares within his rights to assume a decent level of civics knowledge for his readers.


April 26, 2011:  Changed "Democrat Party" to "Democratic Party" in the opening paragraph.

PFB Smackdown: The Providence Phoenix rises to aid Politifact--sort of

The Providence Phoenix provided a would-be defense of PolitiFact Rhode Island--but in the end the defense hurts the mainstream media's version of Media Matters.

The story, by David Scharfenberg, opens with the misleading title "PolititFact Faces Its Conservative Critics."  Scharfenberg's story included, I have yet to see PolitiFact meaningfully engage its conservative critics.  The closest we get to that is stuff like Bill Adair waving off criticisms from the right by pointing out that PolitiFact is also criticized from the left.  No doubt the critics from the left are eligible for the same meaningless defense--except that PolitiFact seems concerned enough about offending its left-tilted base to the point of defending itself from the criticisms of Arianna Huffington and Rachel Maddow.  For contrast, a major Wall Street Journal editorial had its existence noted without addressing its content.

Scharfenberg moves on to a critique of PolitiFact written by Willam A. Jacobson (and linked at PolitiFact Bias).

Scharfenberg's conclusion is curious:

Any of the ratings from half-true to mostly true to true would have been in order. For the ProJo to find "pants on fire" itself deserves a "pants on fire" rating.

PolitiFact, you have a problem.

I find Jacobson's critique less than persuasive. The ProJo may be guilty of examining McKay's statement a bit too literally. But for McKay to claim that Whitehouse labeled any Rhode Islander who didn't agree with him on health care reform a "white supremacist" is a pretty serious distortion.

While Scharfenberg apparently disagrees with Jacobson that McKay may have warranted a "Half True" or higher on the "Truth-O-Meter," he apparently concurs with Jacobson that the "Pants On Fire" rating was unjustified.

Scharfenberg subsequently sharpens the point:

Still, when I read the PolitiFact entry, I couldn't help but wonder: is parsing this sort of political theater, this sort of obvious hyperbole, the best use of PolitiFact's time?

PolitiFact claims it grants license for hyperbole:

In deciding which statements to check, we ask ourselves these questions:

• Is the statement rooted in a fact that is verifiable? We don’t check opinions, and we recognize that in the world of speechmaking and political rhetoric, there is license for hyperbole.

Fact check that?

PFB Smackdown: Kos clueless on Ostermeier study

The reality-based community hasn't taken kindly to Eric Ostermeier's attempts to rescue reality from its liberal bias.

Steve Singiser for the Daily Kos is the latest to take a swipe at the study, writing under the title "Documented proof that Republicans are the biggest liars in politics":

(T)he study (which can be found here) focuses on Politifact itself, charging the nonpartisan analysis done by the St. Petersburg Times fact-checking unit with a systemic bias against Republicans.

Singiser misstates the charge made by Ostermeier.  Ostermeier argues that PolitiFact's presentation suggests Republicans lie more than Democrats, but in scientific terms that conclusion only follows if the conclusions from the data account for selection bias.  But the evidence we have of PolitiFact's selection process offers no assurance against ideological bias in the results.  The stark contrast in the results, Ostermeier says, places a burden of proof on PolitiFact to assure readers that the implicit conclusion is not the result of selection bias.  In other words, make the selection process transparent.  Ostermeier is not charging PolitiFact with "a systemic bias against Republicans."  But he's pointing out that it may well be the case.

After constructing a straw man position for Ostermeier, Singiser proceeds to attack it:

(T)here are other explanations that are equally, if not more, plausible than charging Politifact with grading in bad faith.

True, among them an unconscious bias (good faith, bad methods) among PolitiFact journalists that results in the type of selection bias Ostermeier hypothesizes may account for the disparity in PolitiFact's cumulative grades.  And the latter is an extraordinarily plausible (and parsimonious) explanation.  Singiser would be hard pressed to match it, and that's why Ostermeier argues that PolitiFact should do something to provide a sure foundation for the conclusions it implicitly encourages.

Singiser continues:

For one thing, the study was conducted during a time when the GOP was out of power. The party out of power, it could reasonably be assumed, is going to take more chances with their rhetoric, in an effort to turn the electorate against the party in office.

That's all well and good, but it does nothing to mitigate Ostermeier's real argument.  Remember, PolitiFact grades approximately the same number of statements from figures in both parties.  If PolitiFact used a policy of simply grading chancy rhetoric then one would expect PolitiFact to rate far more statements from one party than the other if the conditions Singiser suggests were to prevail.  And if PolitiFact ends up padding the Democrats' numbers with statements chosen according to a different criterion then we have selection bias by definition.

In short, Singiser's suggested explanation doesn't help.

He doesn't get it.



Feb 17, 2011:  Corrected a recurrent misspelling of Markos Moulitsas' last name (no "z" as it turns out).  Sincere apologies to Mr. Moulitsas.
Jan. 24, 2012:  Double apologies to Markos Moulitsas.  While going back to reference the Kos blog entry I found the link broken.  Shortly after I found an identical version of the post clearly credited to Steve Singiser.

PFB Smackdown: Ron Paul proves that PolitiFact isn't biased?

Ken Winters (on FaceBook) gave voice to one common type of defense of PolitiFact's objectivity:

Of the politicians checked by Politifact for more than 10 statements, the most honest one is Ron Paul and the most dishonest one is Michele Bachman (both are GOP). So anyone calling Politifact biased, needs to fact-check their own thinking.

I'll stick with the reply I offered on FaceBook:

You need to consider PF ratings in terms of selection bias (PF does not randomly choose its topics). It is completely consistent with a liberal bias to rate an oddball Republican (Paul) highly against mainstream Republican competition. Anyone who thinks that Paul's average rating on the "Truth-O-Meter" somehow gets PolitiFact off the hook from the charge of bias needs to fact check their own thinking.

Evaluations such as Winters' rely on dubious assumptions like the following:

• selection bias as to subject doesn't matter
• selection bias as to charitable/uncharitable interpretation doesn't exist
• selection bias as to charitable interpretation doesn't matter
• PolitiFact doesn't make a significant number of mistakes in its ratings

Winters went on to make a valid point in a subsequent post:  It's hard to back up the charge of bias empirically.

But it's just as hard to back up the claim of fairness empirically.