Left Jab: "Politifact's Fuzzy Case Against Bernie Sanders on Pentagon Spending"

An article from the Huffington Post gets some good pokes in against PolitiFact's less-than-clear reasoning in a fact check of Democratic presidential candidate Bernie Sanders.

We're not convinced of the author's main point. He says the mainstream media have no interest in real discussions of the military budget.

But he makes a pretty good case that the fact check of Sanders is short on facts:

What is the core of PolitiFact's argument? Sanders' claim is "mostly false" not because we have good reason to believe that some very different number is much more likely to be correct - like 20%, or 50%, or 90% - but because the share of the Pentagon budget which goes into fighting ISIS and international terrorism is a fundamentally unknowable fact, like how God passed the time before creating the world. The question is intrinsically outside the scope of human knowledge.

And what does PolitiFact do with questions outside the scope of human knowledge? Why, fact check them, of course!

These types of fact checks occur often on PolitiFact's pages. More often Republicans are the victims of this shoddy approach to fact-checking. But we don't shy away from sharing good examples showing PolitiFact treating Democrats or progressives unfairly.




Update 1/20/16 2129PST: Added links to Huffington Post and PolitiFact articles in first paragraph- Jeff

PolitiFact's "Pants on Fire" bias, 2015 update

It's time again for our annual update to our research on PolitiFact's bias in applying its "Pants on Fire" rating. And the results for 2015 show a surprisingly low bias against Republicans. Read on.

PolitiFact's "Pants on Fire" bias

As we have noted, PolitiFact has failed to ever provide any objective means for distinguishing between its "False" and "Pants on Fire" ratings on its trademarked "Truth-O-Meter." The only difference between the two ratings by PolitiFact's telling is that "Pants on Fire" statements are ridiculous as well as false.

We did an extensive survey of the reasons PolitiFact has given in its stories for its ratings and failed to even find an informal criterion that might pass as objective.

This research project does not focus on whether Republicans simply receive more "Pants on Fire" ratings than Democrats. We look at proportions, not the raw numbers. We look at the percentage of total false ratings ("False" plus "Pants on Fire") and then divide the number of "Pants on Fire" ratings for each party by that number. Comparing the percentages for each party then gives us a number we call the "PoF bias number." And what is the significance of that number?

The PoF bias number shows which political party is more likely to receive a "Pants on Fire" rating. If the difference between "False" and "Pants on Fire" is subjective, as PolitiFact's definitions and our research appear to indicate, the "Pants on Fire" rating shows which party is more likely to receive the subjective "Pants on Fire" rating.

In 2011, the first year we did the study, Republican statements were 57 percent more likely to receive a "Pants on Fire" rating than statements from Democrats. As the chart below shows, 1.57 serves as the corresponding PoF bias number.

In PolitiFact's very first year, Democrats received a higher percentage of "Pants on Fire" ratings. It turned out PolitiFact's founding editor Bill Adair said the "Pants on Fire" rating started out for use on "light-hearted" fact checks. Ever since "Pants on Fire" stopped being a joke at PolitiFact, Republicans have been more likely to have their false statements ruled as "Pants on Fire."

Republicans' false statements were far more likely than Democrats' to receive a "Pants on Fire" rating from 2009 through 2012.

	(PoF	Bias)	(Sel.	Bias)
YEAR	GOP	Democrat	GOP	Democrat
2007		2.50	1.00	1.00
2008	1.31		1.53
2009	3.14		1.48
2010	2.75		3.15
2011	1.57		3.83
2012	2.25		2.52
2013	1.24		1.81
2014	1.95		2.56
2015	1.10		4.83

Chart notes: Columns 2 and 3 deal with the PoF bias number. Columns 3 and 4 deal with the "selection bias number," which is the comparison between Republicans and Democrats of the total number of false claims.


Our results for 2015 proved intriguing.

Republicans' false statements were only 10 percent more likely than Democrats' false statements to receive the "Pants on Fire" rating. In terms of historical patterns, the bias against Republicans in 2015 was minimal.

But given the current mainstream media narrative that the Republicans wantonly lie and do so perhaps the most in history, how should we interpret the data?

The Matter of Interpretation

We've had critics insist that a high PoF bias number against Republicans is best interpreted as a sign that Republicans simply lie more. For those critics, this year's findings present a problem. Is the supposed trend of Republicans lying tailing off?

We'd love to see our critics try to make that argument.

We would suggest two hypotheses to help explain the numbers.

First, we would suggest the numbers do not necessarily need much explanation. Simple regression to the mean may explain the surprisingly low measurement of anti-Republican bias in applying the "Pants on Fire" rating. The PoF Bias number for the past three years, after all, falls very close to the average for the preceding years. However, we looked for an alternative explanation anyway because we should predict that the "Republicans lie more" narrative would make the recent numbers look worse than usual for Republicans if PolitiFact journalists allow ideological bias to affect their ratings.

The Secondary Hypothesis

Our secondary hypothesis came from our past analysis of the PoF bias number for PolitiFact's state operations.

Some of the PolitiFact states had PoF bias numbers favoring Republicans instead of Democrats despite paying relatively little attention to statements from Democrats. We hypothesized some PolitiFact states might be applying a compensatory bias: If a PolitiFact state was grading many Republican statements as "False" compared to very few statements from Democrats, the state PolitiFact might rate the few Democrat statements more harshly to look more fair.

So, we hypothesized that PolitiFact states might go harder on false statements from Democrats to make the large number of false statements from Republicans look less like a sign of journalistic bias.

As a corollary to the secondary hypothesis, we noted with our first publication of our research that PolitiFact staffers could read our research and act to change future outcomes. That possibility does not concern us much, for if the bias in applying the "Pants on Fire" rating stabilizes near a more equitable level, it tends to support our interpretation of the data for PolitiFact's earlier years.

Are Democrats Telling the Truth More and More?

Though our research project focuses on the percentages of "Pants on Fire" ratings compared to the total number of statements PolitiFact views as false (making the number totals unimportant to the PoF bias research), we end up collecting data on the total number of false statements as a matter of course.

Those data do appear to tell an interesting story: PolitiFact National seems to have increasing difficulty in rating Democrats' statements false.

Note that we sift the PolitiFact ratings to obtain the data most likely to accurately measure bias.  We're talking about the "Group A" data described in our first research paper, which consists of ratings of party candidates, elected officeholders, partisan appointees or party organizations. We exclude statements obviously attacking members of one's own party. Still, the shrinking number of false statements for Democrats stands out:

(Numbers exclude party-on-party claims, as where a Democrat attacks another Democrat)

PolitiFact had little difficulty finding false statements from Democrats over its first three years. The low mark during those first three years was 18, when PolitiFact only operated for about half the year. Since that first year the number has declined or stayed even every year save for one. The number of false statements from Democrats skyrocketed from 20 all the way up to 23 in 2011 before resuming its decline.

Why is PolitiFact only finding half the false statements per year it found over the first three years? What explains that decline? Are Democrats edging closer to the truth on the whole? Just once per month, on average, PolitiFact was able to catch a full-fledged partisan Democrat telling a falsehood in 2015?

We doubt Democrats have significantly changed their approach to political speech. We think the explanation has to do with the fact checkers buying the idea that failing to let their bias lead them (they don't see it as bias) results in a "false equivalence" in fact checking. PolitiFact produces results consistent with what we should expect from left-leaning fact checkers.

It's a fact! It's from a Republican! It's "Pants on Fire"!

Computer access problems stopped us from joining the vanguard in trashing  PolitiFact's recent "Pants on Fire" rating given to Republican presidential candidate Marco Rubio for saying Reagan's strength compared to Carter's weakness led to the release of hostages on the day Reagan was inaugurated in early 1981.

PolitiFact:

Fortunately, Power Line and Red State admirably filled the breach.

Power Line faults PolitiFact for citing the controversial Gary Sick as an expert historical source, then skewers PolitiFact for simply ignoring the evidence supporting Rubio:

Anyway, Politifact continues:

Instead, the Iranians had tired of holding the hostages, and that the administration of Jimmy Carter did the legwork to get the hostages released.

They got tired of it, you see. Riiiight. Okay, if you’re done being convulsed with laughter on the floor, let’s recall what the Washington Post editorial page (!!) had to say about the matter on January 21, 1981:

“Who doubts that among Iran’s reasons for coming to terms now was a desire to beat [Reagan] to town?”

And who doubts that Politifact and other “fact checkers” are too clueless to grasp Rubio’s argument that your reputation in the world counts for something—especially with your enemies.

Red State noted:

The release did coincide with Reagan’s inauguration. Any critique of Rubio’s statement must include an very solid bit of proof that the two events were disconnected. As a matter of fact, the negotiations that led to the release of the hostages were not even signed until January 19, 1981. If as Gary Sick states, it was that the Iranians were afraid of having to start all over again with Reagan then why was the release not effected earlier. While the Reagan administration, rightfully, had nothing to do with the negotiations it is utter lunacy to assert that Reagan’s election did not have a demonstrable effect.

Power Line and Red State do a nice job in pointing out the holes in PolitiFact's version of the events surrounding the hostage crisis. But we would add to their criticisms the point that PolitiFact also pulled its all-too-typical creative straw man technique on Rubio.

Where Rubio staked out the very defensible position that Iran cut its deal with the president from which it thought it would get the better deal, PolitiFact implies Rubio claimed that Reagan's inauguration caused the release of the hostages:

We flagged Rubio’s comment as a misleading framing of history. Reagan’s inauguration in 1981 may have coincided with the release of the hostages, but historians say it did not cause it.

Is that how PolitiFact framed the issue when it contacted its select panel of experts?

Regardless, this looks like a case of PolitiFact non-transparently interviewing a half-dozen experts and then declaring an expert consensus where no such consensus exists in reality.

Via the United States Institute for Peace (bold emphasis added):

Ronald Reagan was sworn into office on January 20, 1981, just as Iran released 52 Americans held hostage at the U.S. Embassy in Tehran for 444 days. The timing was deliberate. The young revolutionary regime did not want the hostages freed until after Jimmy Carter, who had supported the shah and allowed him into the United States, left office. At the same time, Tehran wanted to clear the slate in the face of a new Republican administration that had vowed to take a tougher stand on terrorism and hostage-taking.

 So totally nothing to do with Reagan.

Whatever.

Here's predicting PolitiFact will do what it usually does when confronted with a strong critique from the right: Nothing.

PolitiFact's Policy Plinko: What Rules Get Applied Today?

Yesterday Bryan wrote a piece noting PolitiFact ignores their own policies in favor of subjective whim and it's easy to find evidence supporting him. PolitiFact's application of standards resembles the game of Plinko, wherein they start off at one point, but can bounce around before they reach a final ruling. The notable difference between the two is that Plinko is much less predictable.

In 2012 former editor Bill Adair announced a new policy at PolitiFact that they would begin taking into account a person's underlying argument when determining a rating for a numbers claim. That new policy turned out to be bad news for Mitt Romney:

In Romney's case, PolitiFact says we need to look beyond the numbers and observe the historical context to find the truth:

The numbers are accurate but quite misleading....It's a historical pattern...not an effect of Obama's policies.

There is a small amount of truth to the claim, but it ignores critical facts that would give a different impression. We rate it Mostly False.

Romney's numbers are accurate, but, gee golly, PolitiFact needs to investigate in order to find out the meaning behind them so no one gets the wrong impression.

Thankfully for Democrats, just a few months later PolitiFact was back to dismissing the underlying argument and was simply performing a check of the numbers:

Some underlying arguments are more equal than others. Contrary to the Romney rating, PolitiFact chose to ignore the implication of the claim:

Our ruling

Clinton’s figures check out, and they also mirror the broader results we came up with two years ago. Partisans are free to interpret these findings as they wish, but on the numbers, Clinton’s right. We rate his claim True.

PolitiFact suddenly has no interest in whether or not the statistics are misleading. They're just here to make sure the numbers check out and all you partisans can decide what they mean.

Sometimes...

Look, kids! Wheel-O-Standards has come all the way back around! And just in time to hit the conservative group the Alliance Defending Freedom:

The organization does not provide mammograms at any of its health centers...

So Mattox is correct, by Planned Parenthood’s own acknowledgement, that the organization does not provide mammograms...

Federal data and Planned Parenthood’s own documents back up the claim from the Alliance Defending Freedom.

That puts the claim in the realm that won’t make either side happy: partially accurate but misleading without additional details. We rate the claim Half True.

We're back to numbers being accurate but misleading! In this rating PolitiFact finds the number of Planned Parenthood facilities licensed to perform mammograms (zero) is accurate, but after editorially judging the statistic gives the wrong impression, PolitiFact issues a rating based on the underlying argument. Because "fact checker" or something.

Why muck up such a great narrative just for the sake of applying consistent standards?

PolitiFact: Point? What underlying point?

In January 2012 PolitiFact's founding editor Bill Adair famously--at least we've tried to make it famous--assured PolitiFact's readers that the most important aspect of a numerical claim is the underlying point:

About a year ago, we realized we were ducking the underlying point of blame or credit, which was the crucial message. So we began rating those types of claims as compound statements. We not only checked whether the numbers were accurate, we checked whether economists believed an office holder's policies were much of a factor in the increase or decrease.

Today PolitiFact rated a State of the Union address claim from President Obama that the private sector created jobs every month since the Affordable Care Act, otherwise known as "Obamacare," went into effect.

PolitiFact's rating? "True."

And what about that underlying argument, eh?

There’s room for argument over what the growth would have looked like absent the health care law, but Obama’s statistic is on target. We rate this claim True.

So ... these days it's like "Why mess up Obama's "True" rating by unnecessarily complicating things? We have a deadline!"

We don't mean to imply any kind of meaningful policy change since PolitiFact supposedly altered its policy under Adair. So far as we can tell, PolitiFact employs a subjective set of rules in reaching its "Truth-O-Meter" ratings. That subjectivity will always take precedence over whatever changes PolitiFact makes to its stated policies.

Note (Jan. 14, 2016): Be sure to check out Jeff D's follow up to this post, PolitiFact's Policy Plinko: What Rules Get Applied Today? Jeff gives some examples of the inconsistency talked about in the above post.

True statements ruled "Mostly False"

What happens when PolitiFact finds that a statement is literally true?

That issue was brought up indirectly when Jeff retweeted economist Howard Wall. Wall had tweeted:

.@jamestaranto I was an expert in Politifact once. Sole purpose of the interview was to illicit a few words to show pre-existing conclusion.

— Howard Wall (@GatewayEcon) January 10, 2016

We looked up the story where Wall was quoted as an expert. It was a fact check of Mitt Romney from the 2012 presidential election. The Romney campaign said women had suffered the greatest job losses under Obama, implying Obama's leadership had been bad for women.

PolitiFact ruled the claim "Mostly False."

The Romney campaign pushed back. PolitiFact looked at the issue again and ruled the claim "Mostly False." But at the same time, PolitiFact said "The numbers are accurate but quite misleading."

Don't blame the Romney campaign. It probably operated under the assumption that PolitiFact's definitions for its "Truth-O-Meter" ratings mean something.

Taking PolitiFact's definitions literally, the lowest rating one should receive for a literally true claim is "Mostly True." Once below that level, the definitions start talking about "partially true" statements that give a misleading impression ("Half True") and "The statement contains some element of truth" but ignores facts that could give a different impression ("Mostly False").

What's our point? We've always said PolitiFact's ratings reveal more about PolitiFact than they do about the entities receiving the ratings. It's a scandal that social scientists keep their eyes closed to that. Search PolitiFact's ratings for claims it says are literally true. Note the rating given to the claim. Then take a look at the ideology of the entity making the claim.

There's your evidence of journalistic bias by fact checkers.

This is an important issue. If social scientists aren't looking at it, it suggests they don't care.

Why wouldn't they care?



Jeff Adds: We highlighted a Mark Hemingway critique of PolitiFact's Romney claim back in 2012 that is still worth a read. It would seem little has changed at PolitiFact since then.



Update 0956PST 1/12/2016: Added "Jeff Adds" portion - Jeff

PolitiFact dances the "can"-"can" on Obama's gun speech

PolitiFact's recent fact check of a statement from President Obama's January 5, 2015 gun control speech ought to give us a week's worth of PolitiFact Bias material.

PolitiFact looked at the president's claim that violent felons can buy guns through the Internet without a background check, ruling it "Mostly True."

We'll kick things off with perhaps the most obvious error:

Some readers seemed to think Obama was suggesting such transactions were legal. We don’t see that in Obama’s comments. (The grammarians at PolitiFact would note that Obama said "can," not "may.") To be clear, such a transaction would be illegal. What Obama said is that such transactions are possible.

We're champions of charitable interpretation, but this one doesn't fly. Why not? Because it leaves no reason for Mr. Obama to single out the Internet. If the issue is simply whether it's possible to buy guns illegally--yes it's possible over the Internet, in person or in any imaginable scenario. The interpretation PolitiFact chose is effectively meaningless. Is Obama going to pass an executive order that criminals cannot potentially disobey? No. That's ridiculous. No matter what Obama orders, criminals "can" make gun purchases bypassing the order under PolitiFact's interpretation.

Violent criminals and others can even buy guns across state lines though PolitiFact explicitly says they cannot. That's the type of absurdity to which PolitiFact's wordplay leads.

It's a little early in the year for a worst fact check of 2016, but we've got an early contender with this one.

It's very likely we'll have more to say about this one.

"Not a lot of reader confusion" in San Luis Obispo

The Tribune of San Luis Obispo, California, published a reader's letter touching on our favorite topic, PolitiFact. The letter provides yet more evidence of reader confusion about PolitiFact's candidate report cards. PolitiFact editor Angie Drobnic Holan insists that little reader confusion exists about the report cards.

A Dec. 13 New York Times column (“All politicians lie. Some lie more than others”) indicated that the percentages of claims in these categories for candidates Ben Carson, Donald Trump and Ted Cruz are 84 percent, 76 percent and 66 percent, respectively.

The reader doesn't bother to say whether he believes those percentages reasonably extrapolate to generalizations about the candidates. But there was hardly a need for that, thanks to the headline provided by the Tribune:

GOP presidential front-runners lie the most

Do they?

Well, the editors of the Tribune read it in The New York Times via the editor of Pulitzer Prize-winning PolitiFact, so it must be true, right?

That's PolitiFact and the mainstream press running their own deceptive political messages while flying the banner of truth overhead.