Ranting and Rating: Why PolitiFact's Numbers Don't Add Up

Crossposted from Bewz, Newz, 'n' Vewz.

"I've given you a decision to make,
Things to lose, things to take,
Just as she's about ready to cut it up, She says:
  "Wait a minute, honey, I'm gonna add it up."

-Violent Femmes

One of the more common methods of using PolitiFact's findings is to add up total ratings and form a conclusion based on the data. In its simplest form, this is when someone looks at, for example, 100 PolitiFact ratings, 50 from Republicans and 50 from Democrats, then adds up who received more trues and who had more falses, and concludes from that total who is more credible. The reality is that a collection of PolitiFact's ratings provides far more information about the ideological bias of PolitiFact's editors than it does about the people they check.

One of the reasons this flawed method is so popular is that PolitiFact frequently promotes it as part of its shtick. Whether it's the ubiquitous report cards, or the iPhone app with its absurd Truth Index (described as a "Dow Jones Industrial Average of truth"), PolitiFact implicitly tells readers they can simply click a link to find out the credibility of a particular politician. Like most diet pills and get-rich-quick schemes, it's snake oil science and complete junk.

The most obvious flaw with this method is selection bias. There's simply no way for PolitiFact, or anyone for that matter, to check every statement by every politician. This means PolitiFact needs to have some sort of random selection process in order to ensure their sample reflects the wide variety of political statements being made, as well as the politicians making them. Without a random process, editors and writers might investigate statements that pique their own ideological interests. And how does PolitiFact choose its subjects?

"We choose to check things we are curious about. If we look at something and we think that an elected official or talk show host is wrong, then we will fact-check it."

Ruh-roh.

This "things we're curious about" method may explain why Senate hopeful Christine O'Donnell (Rep-RI) garnered four PolitiFact ratings, while the no less comical Senate hopeful Alvin Greene (Dem-SC) received none. 

Officially, PolitiFact only checks claims that are the "most newsworthy and significant." (Unless it's about Obama getting his daughters a puppy. Or baseball). PolitiFact also has a penchant for accepting reader suggestions. Anyone visiting PolitiFact's Facebook page is aware that their fans overwhelmingly reside on the left side of the political spectrum. If PolitiFact asks 50,000 liberals what statements to check, guess what? Statements about Fast and Furious won't be as popular as, say, Sarah Palin.

It's also important to consider the source of the statement being rated. For example, when Barack Obama made the claim that preventative health care is an overall cost saver, and Republican David Brooks wrote a column explaining Obama is wrong; PolitiFact gave a True to Brooks. This spares Obama a demerit in his column* while granting Republicans an easy True. Another example of this source selection is evident in the rating about $16 muffins for a Department of Justice meeting. Despite the claim being made in an official Inspector General report and being repeated by several media outlets, including the New York Times and PolitiFact partners NPR and ABC news, PolitiFact hung a False rating around Bill O'Reilly's neck. PolitiFact refrained from judging the nominally liberal media outlets--and the source of the claim--all while burdening O'Reilly with a negative mark in his file.

One of the most overlooked problems with analyzing a tally of the ratings is the inconsistent application of standards PolitiFact employs in different fact checks. Even if one was to assume PolitiFact used a random selection process and assigned its ratings to the appropriate source, we still have a problem when subjects aren't checked according to the same set of standards. For example, Politifact rated Obama "Half True" when he made a claim about the rates certain taxpayers pay. His claim only earned that rating when PolitiFact considered the amount their employers contributed to the employees' tax burden. Almost simultaneously, they labeled Herman Cain Mostly False in a similar claim specifically because he used the same formula. A cursory analysis of total ratings fails to detect this disparate treatment. When considering such flexible guidelines, the "report cards" don't seem like such a credible evaluation.

Ultimately, the sum of PolitiFact's ratings tells us far more about what interests PolitiFact's editors and readers than it does about the credibility of any individual politician. With so many flaws in their process, and such a minute sample size in a vast ocean of statements, conclusions about a subject's overall honesty should be considered dubious. We recognize that this flawed process will undoubtedly affect liberal politicians as well. However, it's our contention that the personal bias of the editors and writers will harm those on the right side of the aisle more often and more dramatically than those on the left.

Adding up PolitiFact's ratings in an attempt to analyze a person's or party's credibility produces misleading results. Until PolitiFact includes a check for selection bias and establishes and adheres to transparent and objective standards, an analysis based on their cumulative work is ill-founded at best, and grossly inaccurate at worst.




*PolitiFact did eventually give Obama a False after he repeatedly made the claim, but still spared several high profile Democrats for the same statement. Unlike perennial fact check favorites like the jobs created by the stimulus or Obama's birth certificate, PolitiFact seems to think the issue isn't worth revisiting.


Update (4/8/2012)-Shortly after (because?) we published this post PolitiFact did revisit Obama's  "Preventative Care Saves Money" claim, and gave him another False. Fair enough. But there's still plenty of other claims that give Obama a pass while simultaneously earning their "We give good grades to the GOP too!" chits that the original point still stands.-Jeff



(2/1/2012): Corrected Alvin Greene's state and name.

The Weekly Standard: PolitiFact Can’t Get Its Story Straight on Romneycare and Abortion

Does the truth have a shelf-life?

Jeffrey H. Anderson, writing at the Weekly Standard, takes the fact-finding DeLorean all the way back to 2007 to highlight PolitiFact's conflicting ratings on RomneyCare's coverage of abortions. Before you read Anderson's article, check out the graphics for the remarkably dissimilar PolitiFact articles:

In 2007, PolitiFact says RomneyCare covers abortions:

Image clipped from PolitiFact.com

In 2012, the issue isn't so clear:

Image clipped from PolitiFact.com

Anderson notes:

[The Gingrich rating] sounds reasonable enough — except that the 2007 PolitiFact verdict directly refutes it. 

At first glance the statements have just enough wiggle room between them to possibly have different ratings. But Anderson's article explains there's not enough to justify different ratings..

Also observe how PolitiFact presented Newt's statement. Newt claimed "Romney signed government-mandated health care with taxpayer-funded abortions." Notice PolitiFact's first question: "Did Mitt Romney make taxpayer funded abortion the law of the land?" The difference is that abortion's status as a covered procedure prior to Romney enacting the legislation is independent of Gingrich's claim. The context of Gingrich's ad was that Romney was sympathetic to abortion rights issues. Whether or not abortion was covered by taxpayers under existing Massachusetts law is irrelevant to Gingrich's point. The fact that Romney helped perpetuate the taxpayer funding is enough to make Gingrich's underlying argument accurate.

We'd also like to point out that for a Republican plan, RomneyCare has been the subject of several favorable articles, and even a ridiculous push poll, at PolitiFact. The cynical reader might surmise the kid glove treatment has something to do with RomneyCare's similarity to ObamaCare. Nah, that couldn't be it.




(1/31/2012) Corrected quote of Gingrich's PF statement. No change in context-Jeff

PFB Smackdown x2: Daily (Kos) double

Unfortunately it remains far easier to locate poor criticism of PolitiFact than it is to locate good criticism.

First up for PFB Smackdown, "Hunter" from the Daily Kos. Hunter thinks that PolitiFact blew its rating of Mitt Romney's claim that he never voted for a Democrat if a Republican was on the ballot.

Hunter:

(P)eople had a wee bit of a problem with this, because the context was Romney's vote for Democrat Paul Tsongas in the 1992 Democratic presidential primary. The Republicans certainly were having a primary that day as well: The incumbent president, George H.W. Bush, was running against Pat Buchanan. Now we can all look back now and have a good laugh at permanent cable news fixture Pat Buchanan taking on the incumbent president, but they both were certainly "on the ballot" in the 1992 primaries. So Mitt's completely making stuff up on this one—his critics have got him dead to rights.

One should not ignore the fact that the Democratic and Republican presidential primaries are two different elections.  There was no Republican on the ballot for the Democratic presidential primary.  Given the context, it is overpoweringly obvious that Romney was saying that he votes for Republicans whenever Republicans are on the ballot against Democrats, as with a general election.  The critics have to ignore one of the primary salient facts to have Romney "dead to rights."  We agree on the point that PolitiFact blew the rating, though for different reasons.  The context makes his statement true, assuming the Tsongas case is the worst would-be exception.

Next up, "dcg2," also writing for the Daily Kos, noticed a supposed trend with PolitiFact's front page material:

I took a quick look at Politifact's home page, and -- surprise, surprise... found two more quick examples of their front page of taking statements from Democrats that they admit are true, but calling them half-true anyway. Just a quick look at the Politifact's front page shows even more outrages -- all against Democrats ...

The author included no list of stories in his post, so it's hard to verify his claim to some extent, but it seems likely dcg2 was somehow able to ignore PolitiFact's flubs of Obama's milk regulations claim, Romney's voting claim and Mitch Daniels claim about the percentage of those under 30 years of age not going to work for the day ("Because many of them were in school"!).

PolitiFact announced in the summer of 2011 that it would start grading statistical statements in light of arguments regarding cause and effect.  Critics like these two Kossacks simply ignore relevant data.  And that makes for poor critiques.

Props to Micah Cohen

Micah Cohen, in a piece appearing at Nate Silver's portion of the New York Times, provides an excellent example for PolitiFact to follow in presenting its candidate report cards.

Cohen's summary of PolitiFact data on the Republican field of candidates very prominently featured the following (from a post Cohen wrote back in September of 2011):

PolitiFact only looks at statements that pique its interest. Here’s how Bill Adair, the editor of PolitiFact, described the process: “We select statements that we think readers will be curious about. If someone hears a claim and wonders, ‘Is that true?’ then it’s something that we’ll check.”

In other words, if Mitt Romney says the sky is blue, PolitiFact doesn’t bother grading the statement as true. So there is a sampling bias at play here. Accordingly, the following numbers should be interpreted with caution. They aren’t perfect indicators of the honesty of each candidate, and conclusions like “Candidate X lies the most” or “Candidate Y is the most truthful” should probably not be drawn from the data.

Perfect.

PolitiFact, take note.



(1/31/2012) Jeff adds: A commenter to this post brings up an excellent point, and it's one I happen to agree with. Cohen deserves credit for at least recognizing some of the flaws, and including a disclaimer. The fact that it happened in the NYT is also worthy of note. In my view, however, it's unfortunate that Cohen immediately disregards his own reservations about the legitimacy of the report cards and goes on, in detail, to provide an analysis of them anyway. A colored chart with percentages provides an undeserved air of scientific authenticity. I'm not impressed.

The flaws with PolitiFact's report cards go deeper than Cohen implied. We've published a new post that explains these problems in detail. You can read it here.

Liberals late to the party on PolitiFact

As expected, PolitiFact's 2011 "Lie of the Year" selection did a good bit of damage to PolitiFact's reputation on the left.  President Obama's 2012 State of the Union speech produced a claim that again has some liberals crying foul.  The Daily Kos and the Huffington Post both published entries condemning PolitiFact's "Half True" ruling on Obama's claim that the private sector jobs increased by 3 million in 22 months.

Jared Bernstein:

I ask you, why do they go where they go? Because of this:

In his remarks, Obama described the damage to the economy, including losing millions of jobs "before our policies were in full effect." Then he describe [sic!] the subsequent job increases, essentially taking credit for the job growth. But labor economists tell us that no mayor or governor or president deserves all the claim or all the credit for changes in employment.

Really? That's it? That makes the fact not a fact? I've seen some very useful work by these folks, but between this and this, Politifact just can't be trusted. Full stop.

(what's with the exclamation point after the "sic," Bernstein?)

Was PolitiFact blatantly unfair to Obama?

Not necessarily. PolitiFact pledged in July of 2011 to take credit and blame more into account for statistical claims.  PolitiFact, in the segment Bernstein quoted, made a decent case that Obama was giving credit to his policies.

Fortunately for the crybabies of the left, PolitiFact promptly caved on this one, revising the ruling to "Mostly True."  The rationale for the change is weaker than the justification for the original ruling:

EDITOR’S NOTE: Our original Half True rating was based on an interpretation that Obama was crediting his policies for the jobs increase. But we've concluded that he was not making that linkage as strongly as we initially believed and have decided to change the ruling to Mostly True.

That editor's note doesn't give readers any concrete information at all justifying the new ruling.  It doesn't take Obama's phrasing into account in any new way, doesn't acknowledge any misinterpretation of Obama's words and doesn't reveal new information unavailable for the earlier ruling.  In short, it looks like a judgment call all the way, where PolitiFact arbitrarily (if we don't count the criticism from the left) decided to give Obama the benefit of the doubt.

The critics on the left, meanwhile, remain apparently oblivious to the another ruling from the State of the Union speech where Obama received an undeserved "True" rating. 

And where were they when Sarah Palin could have used their defense for her true claim about defense spending as a percentage of GDP?

We have a PFB research project planned to address this general issue of technically true claims.


Addendum:

PolitiFact editor Bill Adair has once again come forth to explain PolitiFact's ruling and change of mind:

Lou, deputy editor Martha Hamilton and I had several conversations about the rating. We wrestled with whether it deserved a Half True or a Mostly True and could not reach a conclusion. We decided that it would depend on how directly Obama linked the jobs numbers to his policies.

What criteria were used to determine how directly Obama linked the jobs numbers to his policies?

Adair:

Lou, Martha and I had another conversation about the rating and whether it should be Half or Mostly True. At various points, each of us switched between Half and Mostly True. Each of us felt it was right on the line between the two ratings (unfortunately, we do not have a rating for 5/8ths True!).

We brought another editor, deputy government & politics editor Aaron Sharockman, into the conversation and he too was on the fence. Finally, we decided on Half True because we thought Obama was implicitly crediting his own policies for the gains.

How was Obama's statement "right on the line"?  What criteria placed it there?  What criteria might have moved it one way or the other?

An item like this from Adair is precisely where we should expect a detailed explanation if there is any detailed explanation.

There's essentially nothing.

We get the report of disagreement and vacillation and none of the specific reasons in favor of one rating over the other, except for the implied admission that at least one person making the determination had a change of heart leading to a reversal of the rating.

If that sounds subjective on PolitiFact's part, it probably is.

Sublime Bloviations: "PolitiFlub: Udderly confused by EPA milk regs"

 Crossposted from Sublime Bloviations.


This.
just.
doesn't.
look.
very.
easy.
to.
reconcile:

(link to story at PolitiFact.com)

That's President Obama from yesterday's State of the Union Address.

(clipped from PolitiFact.com)

The latter rating came from PolitiFact Virginia almost a full year ago.

On the face of it, one can imagine a reconciliation of the two rulings.  But it's doubtful if you've looked at the one for Obama after evaluating the one for Griffith.

The best part of it is that PolitiFact may have flubbed both rulings.  The EPA was leaning toward an exemption for homogenized milk.  But the exemption would not have covered raw milk, which should have left Griffith's claim at least partly true.  And if raw milk received no exemption from the EPA then Obama's claim is approximately half true as well.  A raw milk spill would still be treated just like an oil spill even after President Obama supposedly eliminated the rule that required a milk spill to be treated like an oil spill.

Welcome to the wonderful world of PolitiFact fact checking.


Update:  Looks like the EPA did get around to exempting all milk products from the rule. Griffith was still partially correct at worst, and President Obama did not eliminate an EPA rule.  Rather, Obama's EPA exempted milk from a rule that remains in effect.  And Obama's statement obscures the administration's vacillating actions on the issue:

"The Obama Administration pulled back the rule in January of 2009, then reissued it in November, and to large degree it was the same rule," Schlegel said.

Jeff adds (1/26/2012): This is more accurately a "Bryan adds" because I simply wanted to highlight something Bryan wrote on PolitiFact's Facebook page regarding this ruling. I think it exemplifies the "gimme" True's that PolitiFact grants to Obama:

"[T]he Bush administration had created an exemption for milk as it was leaving office. That exemption was one of those that Obama pulled back for re-examination when he took office. So the only reason he had the chance to save us from the EPA's application of oil spill rules to milk was by preventing Bush from fixing it first."

Bill Adair describes selection bias at PolitiFact

We've noted before, along with Eric Ostermeier, that PolitiFact's self-described methods of choosing stories amount to a recipe for selection bias.  The Parker Report recently published an interview with PolitiFact editor Bill Adair that describes more of the same:

How do you decide which reporters take on which claims? How many people check PolitiFact’s facts before publishing? In other words, what is the process? 

Here’s how it works: Every day, our interns look through transcripts, campaign videos, news coverage and interviews for factual claims. The editors review them and choose the claims to fact-check based on whether they are timely, newsworthy and whether people will wonder if the claim is true. That is our biggest criteria for selecting a claim to check — to satisfy people’s curiosity.
 
Most of the time, our reporters choose the claims they want to check. They do the research and write the article, which typically takes a day.

The articles, which include a recommended Truth-O-Meter rating, are then edited by one of our editors and then reviewed by a three-editor panel. The panel makes the final decision on the rating.

Though Adair does not mention it in this interview, PolitiFact also solicits fact check ideas from readers (see image below).

(clipped from http://www.politifact.com/)

If those who send in story suggestions are predominantly liberal or predominantly conservative, this feature will tend to skew story selection toward one political pole or the other.

Likewise, in Adair's description we have two factors that will help introduce partisan bias.  First, if the interns' political biases influence their choices of stories to send to the editors, then the system again will tend to skew story selection toward one political pole or the other.  Second, if the editors use their own sense of curiosity in choosing which statements to rate we have another case where ideology may influence the selection process left or right.

The three-editor panel that chooses the final "Truth-O-Meter" rating will by its nature tend to preserve a majoritarian ideological bias in PolitiFact's ratings.  Suppose the panel has two conservatives and one liberal.  In any rating where ideology serves as an influence, the two conservatives could consistently outvote the liberal.

Interviewer Erik Parker did not ask Adair any particularly tough questions (at least it doesn't appear on the record). Maybe next time.

Spreading the Word

Big thanks to Right Wing News and Ace of Spades for the links this week.

John Hawkins provided a link to us on his site Right Wing News, and Ace Re-Tweeted a mention from one Wayne Austin.

Highlighting the bias and flawed standards at PolitiFact hasn't always been as popular as it is now. When we started PFB we did so in order to offer readers a collection point of disconnected and hard to find criticisms of the supposedly objective fact checkers.

Slowly but steadily the evidence is piling up and PolitiFact is no longer as trusted a source as they once were. We'd like to think we had something to do with encouraging the skepticism, but we wouldn't have been able to do it without our fans promoting our blog.

Thanks to people like Austin, and outlets like Right Wing News, Ace, Legal Insurrection and Big Hollywood, the word is spreading.

Thanks for your support and we look forward to being your source for evidence of and links to evidence of PolitiFact's liberal bias and journalistic failings.