The Weekly Standard: "Damned Lies and ‘Fact Checking’ (cont.)"

The Weekly Standard has a follow up to Mark Hemingway's story earlier this month focusing on the foibles of fact checking (link to our review).

The update, under the title "Damned Lies and 'Fact Checking' (cont.)," is mostly subscriber-only content, though the whole of it is available for preview at present on the Standard's "The Scrapbook" main page. 

Without giving too much away, this nugget both serves as an appropriate tease and a colorful summary of the story:

It’s high time liberal pundits figured out that there’s more going on in this fact-checking bordello than raucous piano music. If they’d been paying attention, they would have long ago stopped patronizing these journalistic houses of ill repute.

Slate: "PolitiFact Weirdly Unable to Discuss Facts"

PolitiFact's recent spat with its liberal readership base has led to the publication of quite a few stories that echo criticisms recurrent in the posts we publish and link at PolitiFact Bias.

Slate's Dave Weigel, famously/formerly of the Journolist, has another such:

After this week, plenty of pundits are well and done with the national version of PolitiFact. The local versions? They're great. I was actually pretty fond of how one of them debunked an ad that misued [sic] one of my quotes, attributing it to a candidate, in 2010. Alas, PolitiFact Editor Bill Adair has committed the main site to a factually dubious "Lie of the Year" claim. PolitiFact claims that it's a "lie" to say that the Path to Prosperity ends Medicare. ActualFacts tell us that this is not a lie.

Adair responds to the critics in the worst possible way.

At a Republican campaign rally a few years ago, I asked one of the attendees how he got his news.

"I listen to Rush and read NewsMax," he said. "And to make sure I'm getting a balanced view, I watch Fox."

We're starting with an anoymous [sic] quote from a straw man that Adair met once?

Weigel continues to expand on Adair's defense, noting that it does nothing to address substantive criticisms.

Adair's response matches the customary pattern at PolitiFact, with the possible exception of the explanation PolitiFact offered after one of its criticisms of Rachel Maddow likewise offended liberal sensibilities.  The sad thing is that it took so long for so many liberals to see it.  Apparently it's easy to overlook the problem so long as conservatives have to deal with the bulk of the harm.

Though we hardly agree with Weigel about the quality of PolitiFact's state franchises (the jury's still out on most of them), his main point is well taken and the post is worth reading.

PolitiFact would gain credibility if it answered substantive criticisms with well-reasoned rebuttals. 

Claiming the critics suffer from some type of echo-chamber syndrome that prevents them from understanding PolitiFact's greatness is not a well reasoned rebuttal.  Rather, it is an ad hominem fallacy.  Readers are not well served with that type of response.

Jeff adds: Weigel continues with a curious new pattern we've noticed with liberal writers describing PolitiFact. What used to be a ubiquitary reference to PolitiFact's Pulitzer (which served to inform the reader of their unquestionable credibility and authority) is no longer worth the extra space to mention. 

Forbes: "PolitiFact and the Traditional Journalism Trap"

After a focused effort to publish my own response to Bill Adair's thin defense of PolitiFact's choice for its "Lie of the Year," I ran across a similar item in Forbes by John McQuaid.

McQuaid and I found two key areas of agreement.

First, the response story from PolitiFact editor Bill Adair was born of conceit:

The whole PolitiFact ruckus has the feel of traditional newspaper journalism (despite the new-ish fact-checking approach) whipsawed by forces it cannot grasp. Traditional newspaper journalism wades boldly into the public square wielding its post-Watergate, “objective” approach and finds itself besieged. And so it concludes: from nasty anonymous comments to partisan sniping to political debates that are never resolved, the public square sucks.

Second, PolitiFact is unable or unwilling to adequately explain and defend its argument for "Lie of the Year":

Today, if you make a Big Statement, people will come after you. Yes, some (most?) will be hacks and fools. But some will be smart, and they will demolish you. Your Pulitzer Prize will not protect you. So you should be prepared to defend yourself and your statement. That means wading into the public square not only with facts, but with arguments and a grasp of the subtleties of the issue at hand. This is, on the whole, a good thing. Readers can affirm or object. Commentators can comment. And fact-checkers can defend and elaborate on their decisions.

That Politifact is apparently unable to understand the necessity of this, and may not even possess the vocabulary or self-awareness needed to do it, suggests it has big problems ahead.

McQuaid's article is worth a read.  It's short and occurs in two parts (1,2).

McQuaid does not touch on the issue of PolitiFact's bias problem.  But his comments touch one of the key sources of media bias.  Superficial knowledge of the subject tends to increase the role of ideological bias in reporting.  That's why citing expert sources may lead to problems where the experts disagree.  The journalist isn't likely to settle an issue debated by the experts on the topic.  Yet bias may lead the journalist to prefer one expert assessment over another.

Bill Adair: You who criticize us are in an echo chamber chamber chamber chamber

PolitiFact editor Bill Adair served up some delicious irony with his recent defense of PolitiFact's 2011 "Lie of the Year" selection.

That selection was Democrats' claim that Republicans voted to end Medicare.  Liberals and progressives far and wide have condemned the selection, and we at PolitiFact Bias share a degree of sympathy with offended liberals since there is some (not much) truth in the claim.

The deluge of port side criticism has prompted yet another one of PolitiFact's nearly content-free rebuttals under the headline "Fact-checking in the Echo Chamber Nation."

Adair seems blissfully unaware that he's inside the echo chamber.

Adair:

At a Republican campaign rally a few years ago, I asked one of the attendees how he got his news.

"I listen to Rush and read NewsMax," he said. "And to make sure I'm getting a balanced view, I watch Fox."

My liberal friends get their information from distinctly different sources — Huffington Post, Daily Kos and Rachel Maddow. To make sure they get a balanced view, they click Facebook links — from their liberal friends.

Adair just told us that he's positioned within an echo chamber oriented left.  He hears opinions from the right when he's out reporting. But to hear what the left is saying he can just hang out with his friends.  A truly centrist Bill Adair may be expected to have discussions with a conservative friend to draw from in writing his story.

Adair:

This is life in our echo chamber nation. We protect ourselves from opinions we don't like and seek reinforcement from like-minded allies.

Bear in mind Adair just finished hinting that his list of friends is predominantly (if not exclusively) liberal.

If Adair isn't in the echo chamber shoulder-to-shoulder with those he criticizes, then it's more akin to a liberal echo chamber duplex with one common living area.

Adair:

The paradox of the Internet age is that never before have we had access to more ideas and different thoughts. And yet, many of us retreat into comfy parlors where everyone agrees and the other side is always wrong. Each side can manufacture its truths and get the chorus to sing along.

PolitiFact had its latest brush with the Echo Chamber Nation this week. We gave our Lie of the Year to the Democrats' claim that the Republicans "voted to end Medicare." That set off a firestorm in the liberal blogosphere, with many saying that claim was not actually wrong. We've received about 1,500 e-mails about our choice and only a few agreed with us.

Adair borrows a page from President Obama's book of rhetorical tricks.  Sure, "many of us" insist on surrounding ourselves with like-minded opinions.  But Adair's problematic audience response probably comes more from those who expose themselves to contrary opinion yet do not have the ability and/or inclination to sift through the clash of ideas to figure out what's wrong or right from either side.

And blame falls on PolitiFact on this point.  PolitiFact often fails to make a clear case in favor of its decisions, and its 2011 "Lie of the Year" is another good example. Observe Adair's method of treating substantial criticisms in response to the "Lie of the Year" selection:

Some of the response has been substantive and thoughtful. The critics said we ignored the long-term effects of Rep. Paul Ryan's plan and that we were wrong to consider his privatized approach to be Medicare. In their view, that is an end to Medicare.

We've read the critiques and see nothing that changes our findings. We stand by our story and our conclusion that the claim was the most significant falsehood of 2011. We made no judgments on the merits of the Ryan plan; we just said that the characterization by the Democrats was false.

You just can't blame the outraged liberals for finding this type of response unsatisfactory.  Adair appears to admit that they have a point.  And then tells them with no reason why--unless it's sufficient to claim non-specific support from Annenberg Fact Check or the Washington Post fact checker--that there's no reason to change the ruling.  .

Adair:

We got other silly comments from readers who declared we were "a tool" of the Republicans, Fox News and the Koch brothers. Their reaction is typical these days. To paraphrase George W. Bush, you're either with us, or against us.

In reality, fact-checking is growing and thriving because people who live outside the partisan bubbles want help sorting out the truth. PolitiFact now has nine state sites run by news organizations around the country that employ more than 30 full-time journalists for fact-checking. We've inspired many copycat sites around the nation and roughly a dozen in other countries.

Adair says the extremist reactions are "typical."  And in almost the next breath he claims that fact checking is thriving because of the people living outside the partisan bubbles.  The atypical ones account for PolitiFact's success?   Why, if that's the case, did PolitiFact not receive greater email support for its "Lie of the Year" selection?  Is it that hard for Adair to see the writing on the wall from within his echo chamber?

On the whole, Adair's defense is elitist and defensive. The PolitiFact staff is enlightened, thank you very much.  If you don't like their "Lie of the Year" selection then there are plenty of potential readers who live outside the echo chamber.  And it would be nice if a few of those readers would send in some supportive emails (hint, hint).

It seems Adair doesn't know his audience.


Addendum:

One more area where PolitiFact needs to clean up its act:

Adair:

Some of our critics wrongly attributed our choice to our readers' poll and said we were swayed by a lobbying campaign by Ryan. But our editors made the choice and the poll was not a factor.

Um--how do we know the poll was not a factor?  Because Adair says so?

Free advice for Adair:  If you want to be able to claim with confidence that the poll plays no role in the editors' selection then keep the editors ignorant of the poll numbers until they're finished making their choice.  And if you do it that way then you can write your defense like this:

Some of our critics wrongly attributed our choice to our readers' poll and said we were swayed by a lobbying campaign by Ryan. But we shield the editors from the poll data to ensure that it will not affect our decision.

Doesn't that sound a lot better?  More convincing?


Jeff adds: For us PolitiFoes, Adair's airing of grievances was a Festivus Miracle. Adair implies (as does the entire premise of the PolitiFact operation) that he is somehow immune from the echo chamber, and if you don't trust him, it's because you're in too deep to notice. You're not objective enough to see the emperor's non-partisan clothes. I also have issues with his you-a-culpa over the examples he provides. His GOP rally attendees and liberal friends are straw men all day long. Whatever the political inclinations of his acquaintances may be they are irrelevant to whether or not PolitiFact gives it to us straight.

It's also interesting to note the tone of the article when compared to the dispassionate text of Adair's response to 2010's Lie of the Year criticism. We've noticed PolitiFact responds much more aggressively to anger from the left than from the right and this article adheres to that theme.

Perhaps the injuries inflicted by friendly fire were deep, but it's doubtful Adair corralled any sheep back into the flock with this ill-advised tantrum.

Apoplectic Now: The Aneurysm of the Year

That massive popping sound you heard on Tuesday was the collective hearts and minds of liberals across America bursting as they witnessed their favorite source of smug validation betray them. PolitiFact editors played their pre-selected card and announced the Democrats' claim that Republicans voted to "end Medicare" as the Lie of the Year for 2011.

What could go wrong?

The wrath unleashed on PolitiFact went far and wide as hysterical condemnations and inordinate smiting piled up on the left side of the Internet. The High Priest of Haute Liberals himself, Paul Krugman, sounded the death knell in his subtly titled article "Politifact, R.I.P." in which he described PolitiFact as "useless and irrelevant." Talking Points Memo called the decision a "sham", and Steve Benen at Washington Monthly called the decision "indefensible" in his article "PolitiFact ought to be ashamed of itself." The list goes on and on and on (and on.).  The formerly ubiquitous mention of PolitiFact's Pulitzer that was previously announced as a badge of credibility is suspiciously absent in these articles.  

But for long-time PolitiFact critics like us, few things in life have been as entertaining as the epidemic hysteria witnessed over at PolitiFact's Facebook page. Check out this sample of outbursts posted on various Facebook threads throughout the week. (Names have been removed to protect the aggrieved):

"I've awarded Politifact the Steaming, Festering Turd of The Year Award for this one. Your credibility has been flushed."

"Politifact, you're either being bought off by the right wing echo machine or you're scared of them."

[The Pauline Kael Trophy goes to:] "This has been voted by everyone I know,including myself as the stinkiest,lamest,most cowardly decision of the year!"

"You let Fox News choose your Lie of the Year, didn't you."

"Embarrasing."

[This guy may be on to something:]"Maybe, we just gave a group of idiots too much credit to begin with simply because the bore the name 'Politifact.'"

"PolitiFact's "Lie of the Year" is pretty good.......... for me to poop on."

"Noting your selective ignorance of objective facts, I am now forced to ignore you as a reference source. Unfortunate, but I am only interested in objective, FULL, analysis of facts." [Which is what I considered you when you were confirming my opinions.]

"So now we know Politifact is as bought as the politicians they scrutinize."

[Murderers!:] "Presumably, politifact also believes that if someone kills another person that it is not murder if they kill them slowly with a slow acting poison. Such lame and disreputable analysis and logic is incomprehensible for an organization wishing to claim some skill and reputation at factchecking."

[From the 'Paul Ryan stuffed the ballot box' conspiracy:] "The mere product of lobbying. Hey politifact way to bend over and take it. Hope you had on lipstick so atleast you looked good doing it."

[The Jews!:] "How many shekels did you guys get for that choice?"

"Did you guys get purchased by Newscorp?"

"Another election stolen. Dislike."

"What a bummer, I trusted Politifact implicitly until this." [Spencer Pratt responds]

[Baby, Don't Go Award:] "If you guys can do something to win back your credibility after this outrageous and outlandish ruling, then I may be back. Right now, though, I'm unliking this page and deleting the bookmarks I have to your website."

"Either you fire your editorial board and give yourself a pants on fire or just close up shop."


"God you guys are stupid."

Hell hath no fury like a liberal scorned.

PolitiFans fell into one of a few groups. Some accused PolitiFact of being a tool of the GOP.  Others claimed Paul Ryan sabotaged the vote by his email campaign (unaware that the readers poll is not the same as the editors' pick). Most simply said the claim was true, and that determining what constitutes the "end of Medicare" is an issue of semantics that falls outside the scope of objective values. That's a fair point, and it's one we've chronicled a number of times, including last years Lie of The Year. So where have all the indignant liberals been since PolitiFact's inception? Affixing varying degrees of "fact" to obvious hyperbole and opinion has been PolitiFact's shtick all along. For the left to become unhinged now betrays their own selective bias. In short: PolitiFact served its purpose as neutral, objective arbiters of fact, as long as they were validating liberal axioms.

To illustrate this point, check out this Jonathan Chait article (with some, uh, minor edits in bold):

The Patient Protection and Affordable Care Act would very dramatically change health care.

...

Is that “a government takeover?” Well, it’s a matter of opinion. At some point, a change is dramatic enough that it is clearly a government takeover. If you proposed to replace a voluntary, free market system with a plan that mandated everyone purchase health insurance and the government dictated what patients and ailments insurance companies had to cover and what to charge, I would hope Politfact would concede that this would be “a government takeover,” even if you call the new mandates “a free market solution.” On the other hand, small tweaks could not accurately be called “a government takeover.” Between those two extremes, you have gray areas where you can’t really say with certainty whether a change is radical enough to constitute a takeover.


Does ObamaCare indeed establish a government takeover? I would argue no. But it’s obviously a question of interpretation, not fact. And the whole problem with Politifact’s “Lie of the Year” is that it doesn’t grasp this distinction. Politifact doesn’t even seem to understand the criteria for judging whether a claim is a question of opinion or a question of fact, let alone whether it is true.

Obviously, Chait's unedited piece argued that whether or not Ryan's plan did in fact end Medicare was a matter of interpretation (and ironically it mirrors the Wall Street Journal's op-ed about last years LOTY). We tend to agree with this criticism. And to be fair to Chait, he's called PolitiFact out for being harsh toward the GOP before. But the mountain of new criticism of the Lie of the Year, and PolitiFact's operation in general seems to be a few years late. Like all of PolitiFact's betrayed lovers this week, the reaction to the sudden realization that PolitiFact operates as a biased actor with motivations less noble than honest determination of facts is comical and disingenuous to everyone who's seen it for years. The irony for us is it took PolitiFact's calculated attempt to appear even-handed for the liberals to rise up in revolt.

The Medicare claim was the winner from the outset. Just take a look at its competitors. The reality is that Jon Kyl's abortion claim, Michelle Bachmann's vaccine statement, and Debbie Wasserman Schultz's rant about Jim Crow laws were hardly repeated outside of PolitiFact's circles. They were minor blurbs that barely lasted on the news cycles and had no place being in the running for statements "that played the biggest role in the national discourse." For all the gnashing of teeth about the winner, somehow PolitiFact managed to protect Team Democrat from any unflattering press about legitimate nationally popular issues like Solyndra or "Fast and Furious." The ten finalists were carefully selected, with an eye on the Medicare claim to be the winner. And anyone that thought they would select a GOP claim for the third year in a row ignored the reality that PolitiFact is a political animal with a brand to protect and an impartial image to uphold.

In the end it's hard to determine the final estimate of the damage PolitiFact has caused with its overwhelmingly liberal readership. We've seen smaller scale exodus whenever they've gone after Jon Stewart that had only short term effects. Whatever the case, conservatives would be wise to avoid finding anything redeeming in this temporary respite from the partisans at PolitiFact. As we've explained before, the shoddy standards PolitiFact employs will inevitably hit both sides of the aisle, but the liberal fishbowl of the newsroom will ultimately cause them to come down against the right much more often.

The 2011 Lie of the Year selection does little to diminish PolitiFact's aura of liberal bias. If anything, it exemplifies the selection bias and inherent flaws of their operation that have made it so unreliable in the first place. Whether this is PolitiFact's demise as a tool of liberal validation, or if it bolsters their claims that "upsetting both sides proves they're doing it right", for us at least, it's been a fun week to be watching.


Bryan adds:

Count me among those naive enough to believe that PolitiFact would pick three consecutive Republican claims as "Lie of the Year" depending on the material under consideration.

Jeff notes: I was correct in predicting the winner would go against the left, but my final pick (Obama hasn't raised taxes) was wrong. I suspect that had PolitiFact followed my advice there would be much less turmoil among the ranks. It's hard to imagine liberals being too upset about PF confirming Obama raised taxes.

The meaning of PolitiFact's "Lie of theYear" for 2011

I wonder whether this award will have those conservatives blasting politically-motivated “fact check” operations rethinking that criticism?

--Ed Morrissey, Hot Air blog

Fact check critics who base their criticism on a completely consistent pattern of wronging only one party or ideological position should take Morrissey's argument to heart.

As I have written repeatedly, a significant ideological bias does not require all the harm to hit one side and all the benefit to accrue to the opposite side.  In scientific terms, a simple majority of cases favoring one ideology over another indicates an ideological bias (after taking the margin of error into account).  Two out of three "Lie of the Year" awards going to conservatives, for example, fits well with the hypothesis of liberal bias.  Granted, three out of three makes an even better case.

What, if anything, does the 2011 "Lie of the Year" mean with respect to the issue of media bias?

Answer:  probably not much.

One liberal media hypothesis, as expressed by economist/political hack Paul Krugman:

(T)he people at Politifact are terrified of being considered partisan if they acknowledge the clear fact that there’s a lot more lying on one side of the political divide than on the other. So they’ve bent over backwards to appear “balanced” — and in the process made themselves useless and irrelevant.

Krugman's charge is plausible if we simply take him to mean that PolitiFact carries a consciousness of the effect on its brand of, for example, choosing a Republican claim as its "Lie of the Year" for 85 years straight.  We'll table discussion of Krugman's evidence supporting a "clear fact that there's a lot more lying on one side of the political divide than the other."

At the bottom line, the criticisms of the 2011 "Lie of the Year" from the left are no better than the right's criticisms of the 2009 and 2010 "Lie of the Year" winners.  The latter linked story helped earn Joseph Rago a Pulitzer Prize.  This year's award is no different than those in the past except that the left got hit instead of the right.  And, of course, the apoplectic response from the left creates such a contrast to the right's past reactions that Karl of Patterico's Pontifications offers the following:

PolitiFact’s most useful function may be in triggering an analysis of the overwrought reactions of these progressive crybabies. 

The left is largely content with PolitiFact so long as conservatives take the worst of it.  If not, well, the sky is falling and PolitiFact loses all credibility.  Or something like that.

Krugman's hypothesis is an unlikely explanation for this year's "Lie of the Year" selection.  The pressure to pick a lie of the left was probably subtle and semiconscious.  Why?  Because PolitiFact already carries very little credibility with conservatives, Ed Morrissey notwithstanding.  PolitiFact has angered its main demographic without much hope of building trust in a potential audience of largely suspicious conservatives.

If PolitiFact gains nonpartisan credibility with this move, the effect is primarily in-house:  The journalists reinforce their own belief in their fairness and objectivity with moves like this one.

PolitiFact probably misjudges its audience.  The net effect will be decreased overall trust in the brand.  Sure, the staff can take solace in the absurd notion that criticism from partisans on either side shows their even handedness.

It doesn't work that way.

Stay tuned, because PolitiFact Bias will soon roll out objective research supporting our position that PolitiFact manifests a significant bend to the left.



Correction Sept. 5, 2017: Very belatedly effected the change from "James Rago" to "Joseph Rago" in the seventh paragraph. RIP Joseph Rago.

New feature: The (annotated) Principles of PolitiFact and the Truth-O-Meter

Jeff and I continually run across cases where PolitiFact applies its standards unevenly, contradicts them or simply ignores them.  But simply mentioning it on a case by case basis doesn't quite carry the impact to the reader as we might hope.  After all, most readers don't happen to look at every case we mention.

To help communicate the degree to which PolitiFact fails to keep to its principles (and, to be sure, to give us an added opportunity to express our snarky sides), we added a new page:  The (annotated) Principles of PolitiFact.  The page takes the text of PolitiFact's "Principles of PolitiFact and the Truth-O-Meter" and adds our commentary regarding the application--or lack thereof--of those principles.

We'll be adding scads of links to provide examples of the failures we point out.

The page will always be a work in progress, reflecting our growing body of work examining PolitiFact.

Forbes: "How to Fix Fact-Checking"

It's gratifying to see journalists and pundits piling on PolitiFact for all the reasons PFB preaches.  The latest to jump on is Forbes magazine with a story by John McQuaid titled "How to Fix Fact-Checking."

McQuaid uses the recent Weekly Standard story by Mark Hemingway as his jumping-off point:

The Weekly Standard deplores fact-checking – the journalistic efforts, by PolitiFact and others, to vet what politicians and others in the public eye say and call out lies and half-truths. So much that Standard editor Mark Hemingway is trying to knock down the whole fact-checking enterprise, arguing it’s a liberal media scam.

McQuaid doesn't buy into Hemingway's suggestion that fact checkers intentionally skew to the left.  But he grants that the fact checking biz does have problems (bold emphasis added):

Here’s the thing, though. The Standard piece offers up some genuine examples of faulty fact-checking in service of its tendentious argument. The problem with fact-checking is not that it’s a liberal media plot. The problem is that fact-checking – like everything – is sometimes a lazy, half-assed business. If fact-checking is as important as it claims, its practitioners need to acknowledge its problems and fix them.

We argue that the tendency of journalists to lean left translates into a tendency for the preponderance of "half-assed" fact checking to harm conservatives and benefit liberals.  McQuaid is probably right that the bias isn't intentional--but often it's so bad that one is hard pressed to detect the difference between intentional and unintentional bias.

Read it all.  The snippets above come from Page 1, while Page 2 contains McQuaid's prescriptions for the tainted industry.