Wednesday, August 22, 2012

The circus inside PolitiFact's "Star Chamber"

I suspect that many people think, as I originally did, that PolitiFact selects its "Truth-O-Meter" ratings through something like an objective process.

Andrew Phelps of the Nieman Journalism Lab recently sat in on PolitiFact's formerly private deliberations and produces much the picture I have come to expect (pun not intended) during my years of increasing skepticism.

Adair doesn't reveal his politics.  Who put that Obama cutout back there?

WASHINGTON — PolitiFact’s “Star Chamber” is like Air Force One: It’s not an actual room, just the name of wherever Bill Adair happens to be sitting when it’s time to break out the Truth-O-Meter and pass judgment on the words of politicians. Today it’s his office.

Three judges preside, usually the same three: Adair, Washington bureau chief of the Tampa Bay (née St. Petersburg) Times; Angie Drobnic Holan, his deputy; and Amy Hollyfield, his boss.
"Star Chamber" aptly describes the secretive nature of the judges' meeting. PolitiFact staffers sometimes talk about what goes on in the meetings, but PolitiFact readers get no "report card" on the voting records of the fact check judges.

Jeff and I have repeatedly criticized PolitiFact's process for its institutionalization of PolitiFact's group ideology. Phelps' descriptions and transcripts bring our worst nightmares to life as the judges make their decisions with no apparent grounding in objective data. Phelps featured the following transcript early in his story:
Hollyfield: Is there any movement for a Pants on Fire?

Adair: I thought about it, but I didn’t feel like it was far enough off to be a Pants on Fire. What did you think, Lou?

Jacobson: I would agree. Basically it was a case I think of his staff blindly taking basically what was in Drudge and Daily Caller. Should they have been more diligent about checking the fine print of the poll? Yes, they should have. Were they being really reckless in what they did? No. It was pretty garden-variety sloppiness, I would say. I don’t think it rises to the level of flagrancy that I would think of a Pants on Fire.

Adair: It’s just not quite ridiculous. It’s definitely false, but I don’t think it’s ridiculous.
  1. Hollyfield tests for support of the "Pants on Fire" rating she apparently wishes to promote.
  2. Adair didn't "feel" the claim went that far. How far is too far?
  3. Writer Jacobson (not one of the judges) also offers his vote in terms of opinion: He doesn't "think" it's flagrantly false. What's the objective measure for "flagrant"?
Perhaps editor Angie Drobnic Holan, whose opinion was missing from this exchange, carried the torch for objective standards during the meeting. But don't bet on it. The portion of the conversation Phelps provides smacks of exactly the type of subjectivity hypothesized in PolitiFact Bias' initial research study into PolitiFact's bias.

Like the original Court of Star Chamber, PolitiFact’s Truth-O-Meter rulings have always been secret. The Star Chamber was a symbol of Tudor power, a 15th-century invention of Henry VII to try people he didn’t much care for.

...The site’s basic idea — rate the veracity of political statements on a six-point scale — has modernized and mainstreamed the old art of fact-checking.
Granted, I put together statements from Phelps a few sentences apart, but regardless of that one is still struck by the segue from PolitiFact's parallel to the 15th century "Star Chamber" to the praising of PolitiFact for modernizing fact checking. Indeed, the abandonment of transparency occurs as one of PolitiFact's most distinctive innovations in the fact checking business.

The secret voting serves the same purpose as the secrecy about staff members' voting history. PolitiFact does not want its readers taking the fact checkers' biases into account. The fact checkers doubtless assure themselves of their neutrality as nearby two-dimensional cardboard Obamas smile approvingly at their work.

The truth is that fact-checking, and fact checkers, are kinda boring. What I witnessed was fair and fastidious; methodical, not mercurial. (That includes the other three (uneventful) rulings I watched.) I could uncover no evidence of PolitiFact’s evil scheme to slander either Republicans or Democrats. Adair says he’s a registered independent. He won’t tell me which candidate he voted for last election, and he protects his staff members’ privacy in the voting booth. In Virginia, where he lives, Adair abstains from open primary elections. Revealing his own politics would “suggest a bias that I don’t think is there,” Adair says.
It's nice that Phelps didn't see any obvious bias, but who is he kidding? The PolitiFact staff knew he was observing them, didn't they? I think probably PolitiFact doesn't deliberately slant its fact checks, but Phelps offers the thinnest of reassurances on that count, particularly since the tone of his story suggests he shares the port side slant so common in modern journalism.

We could probably mine Phelps' story for a week's worth of material.  Maybe we will.

All credit to Jeff for spotting the President Obama cardboard figure in Adair's office that turns the statement about staffers' voting history into an absolute howler.

But don't get the idea that Adair is biased or anything. We wouldn't want that.

Jeff adds:

This ode to the majesty of PolitiFact's Echo Star Chamber is jaw-droppingly awful. Phelps' inability to untangle the contradictions in front of his own eyes was painful to read. For example, he describes the Truth-O-Meter as "simple, fixed, unambiguous." Unfortunately, Phelps never reconciles these concrete terms with the subjective nomenclature of the actual ratings. He writes:
“Pants on Fire,” a PolitiFact trademark reserved for claims it considers not only false but absurd.
Phelps never shares PolitiFact's unequivocal, dispassionate standard for "absurd." And the ratings process Phelps describes as "fair and fastidious; methodical", ultimately boils down to "What did you think, Lou?"

But fear not PolitiFans. PolitiPhelps assures us that he "could uncover no evidence of PolitiFact’s evil scheme to slander either Republicans or Democrats." This is the same Andrew Phelps who once asked "[W]hy is George Bush such a flaming moron?" The man who described Maureen Dowd as "The person who best captures my feelings about our miraculously awful [Bush] administration" is confident Bill Adair gives it to us straight.

It's worth noting that Phelps, despite mentioning the left's outrage over the recent LOTY rating, as well as Rachel Maddow's outbursts, was unable to link to the any of the persistent, and numerous PolitiFact rebukes from the right. It's been our experience that PolitiFact's attacks from the left are very often lame, but they get the links in Phelps' piece.

If there are actual scientific, objective standards applied to PolitiFact's ratings, Phelps failed to report them. Phelps' flattering prose aside, the article shows that PolitiFact's system is really just a bunch of coworkers asking each other if they are having a "movement."


  1. The more I read through Politifact, the more I am convinced that it is a political hack for Obama and Company/ Their definition of their "Truth-O-Meter" is rather subjective - meaning that if a Democrat does;t tell the full truth, Politifact will either find a truth in a portion of the context, or interpret what the person saying the context "Actually Meant." I find it quite comical. But the irony of the whole thing is that Liberals hate facts - and here is a Liberal entity deciding what is and isn;t a fact. It's like allowing the fox to guard the hen house. Pretty pathetic!

  2. While the difference between "pants on fire" and "false" or "mostly false" and "half true" is most definitely subjective, you have to actually read each individual article about the fact-checking in order to get the meat of the issue, and that part is actually very objective. Instead of reading about Politifact, try actually reading their articles. The "truth-o-meter" is a gimmick to draw people in, but as with everything you have to spend the time to read in detail to learn anything not just looking at the blurb. They are far from bias, grading Obama about half and half in the several true and several false categories.

  3. Joe,

    I don't read every PolitiFact story. But I do read many of them, particularly when the subject suggests to me that liberally biased journalists will have trouble doing a proper job on the story. The reporting from PolitiFact often falls well below the minimum standards for fact checking.

    1. It does seem like there was a preconceived opinion by the fact checker in the cuts to Medicare. That point probably shouldn't have even been touched by Politifact because it depends largely on assumptions and predictions of the future, and not actual facts. They may sometimes fall short of fact checking but compared to most 21st century news outlets they are far above the bar and I think they do a great service by exposing misleading statements by both liberals and conservatives. As far as being biased, that blog relates only about 10 articles on grading Politfact in the past 4 months and they are all graded as F's, what about the other hundreds of fact checks they have done.

  4. Joe,

    That blog (mine, as it happens) repeatedly emphasizes that I review what I wish (selection bias) and grade harshly. I caution readers not to use the grades except as my judgment of the journalism involved. I grade harshly because I think accuracy and thoroughness are important in fact checking. The relatively low number of ratings occurs because I review PF as a part-time vocation that competes with other interests. I could easily find more flawed stories by dedicating more time to the pursuit. The greater number of flaws, especially the truly boneheaded ones, tend to harm Republicans and/or benefit Democrats. That's the definition of ideological bias in practice. Kessler and Annenberg are both biased yet they are head and shoulders better than PolitiFact. Those two aren't quite worthy of their own critical website. PolitiFact could power several by itself.