The PolitiFact response to the Romney story pushback

PolitiFact doesn't just entertain with its incompetent and unfair fact check stories.  It also entertains with its response to criticism. 

The most popular response of all is the "turtle."  Just don't respond and wait for the criticism to die off.

The second method involves acknowledging the criticism, followed by ignoring it as though it makes no difference.

PolitiFact editor Bill Adair uses a variant of the second method in responding to criticism of a recent fact check of Republican presidential candidate Mitt Romney:  Seize on some minor point in the criticism and act like the minor point nullifies the criticism (second method plus the minor point, in other words).

Hilariously, PolitiFact takes its critic, Dylan Byers, out of context for purposes of its response.  Note the relevant portion of Byers' critique:

I've reached out to Jacobson to see how many experts PolitiFact spoke with, and to ask for his reaction to Bruscino's post. There's a fair chance that PolitiFact spoke to other experts who saw things differently, but its hard to imagine how Bruscino, with all that detailed analysis, could be relegated to a minority view.

Bruscino was one of the experts PolitiFact cited in the Romney story.  He objected to PolitiFact's methods and conclusion in a blog post earlier this week at Big Tent.

Check out the headline over PolitiFact's response:

'There's a fair chance PolitiFact spoke to other experts' -- yes, 13 others

Yes, ladies and gentlemen, our esteemed fact checkers lead off with a fine example of quote mining.

Commentary: "PolitiFact's Pants on Fire"

Another PolitiFact-cited expert has joined Tom Bruscino in dumping on PolitiFact over a recent fact check on Republican presidential candidate Mitt Romney.

This time it's Ted R. Bromund, writing in Commentary magazine.  Like Bruscino, Bromund pans PolitiFact for using leading questions and for issuing a ruling with which he cannot agree.  But Bromund goes further in noting that PolitiFact performed its fact check based on a straw man version of Romney's point (bold emphasis added):

Jacobson sums up Romney’s contention as being: “The U.S. military has been seriously weakened compared to what it was 50 and 100 years ago.” Since the Truman Doctrine of 1947 is as good a marker as any of the moment when the U.S. assumed the global security responsibilities that formerly belonged to Britain, the fact that our Air Force is smaller and older than it has been at any point since that date might give immediately give cause for concern.

But the obvious point of Romney’s statement was not that the U.S. military of today would lose a war to the U.S. military of 1917 or 1947. It was that the margin of U.S. “military superiority” – i.e., its relative strength versus potential and actual adversaries – is at risk if defense spending declines, as President Obama plans for it to do. The question is not whether the U.S.’s “military posture is in any way similar to that of its predecessors in 1917 or 1947”: it is whether the U.S.’s margin of superiority over other actors, taking contextual factors into account, is better or worse than it was in previous eras.

Bromund weakens his case some with his characterization of PolitiFact's argument in the story.  It seems the argument was more that the current state of the armed forces as described by Romney remains the world's superior military force, though PolitiFact does use some language easily taken to support his interpretation.  But Bromund scores a huge hit in noting that PolitiFact stuffed Romney's argument full of straw.  Just have a look at the manipulated quotation PolitiFact used in its deck material:

The U.S. military is at risk of losing its "military superiority" because "our Navy is smaller than it's been since 1917. Our Air Force is smaller and older than any time since 1947." 

PolitiFact uses the above to represent Romney's argument. True, the story goes on to quote Romney accurately, including his argument that cuts in defense spending might lead to the U.S. losing military superiority.  But the story treats Romney's argument in accordance with the inaccurate hybrid quotation/paraphrase:

(A) wide range of experts told us it’s wrong to assume that a decline in the number of ships or aircraft automatically means a weaker military. Quite the contrary: The United States is the world’s unquestioned military leader today, not just because of the number of ships and aircraft in its arsenal but also because each is stocked with top-of-the-line technology and highly trained personnel.

Thanks to the development of everything from nuclear weapons to drones, comparing today’s military to that of 60 to 100 years ago presents an egregious comparison of apples and oranges.

It's remarkable that PolitiFact's summary omits Romney's mention of cuts in defense spending.


PolitiFact has stepped in it again and has again reacted by publishing a pedantic rebuttal along the lines of the one it used in response to criticism of its 2011 Lie of the Year.  The former will receive a review here in due time. (Update 1/22/12 See review here).

Additional reading:

Power Line stays on the story, providing additional valuable material by publishing the text of Bromund's reply to PolitiFact writer Louis Jacobson after Jacobson's initial inquiry about the Romney issue.

Politico took note of the story with a piece by Dylan Byers.  Byers's story drew most of PolitiFact's ire in the rebuttal mentioned above, so reading it provides excellent background material.

The Huffington Post even gets in on the act with solid article.  The story provides the context of Romney's remarks with a partial debate transcript.


Also see the my latest post in the "Piquing PolitiFact" category at Sublime Bloviations, which features the text of an email message I sent to PolitiFact asking for PolitiFact to fact check one of its own claims.



Update (1/22/2012): Added link to Bruscino article, link to PF response review, and minor spelling corrections-Jeff

Future of Capitalism: "Two More Obama Switcheroos"

I'm a regular reader of Ira Stoll's Future of Capitalism, though it's not where I typically troll for PolitiFact rebukes. So I was surprised last week to catch a brief mention of our factastic friends in the otherwise non-PolitiFact related piece.

At issue is the announcement that the Obama administration seeks to reduce the number of armed services personnel by 490,000 troops over the next decade. Stoll happens to be keeping his own mini-ObamaMeter and points out that this reduction contradicts an Obama campaign promise that he "supports plans to increase the size of the Army by 65,000 troops and the Marines by 27,000 troops."

It takes Stoll a whopping two sentences to expose PolitiFact's sophistry:

The Politifact Web site absurdly rates this as a "promise kept," explaining, "Obama said nothing about keeping the higher levels indefinitely." How cynical can you get?

Not only is it cynical, it's also an example of PolitiFact's alternating standards. Note how PolitiFact dealt with Rand Paul's claim that the "average federal employee makes $120,000 a year. The average private employee makes $60,000 a year." Paul received a False rating because, as PolitiFact explained:

"Most people hearing that would assume he was talking about salary alone, but  he was talking about total compensation, including benefits such as retirement pay and paid holidays."

In Obama's case, PolitiFact gives him a Promise Kept because of the literal (and cynical) interpretation of his words. Paul, on the other hand, is deemed false not because of what he said, but because of what PolitiFact thinks people would assume he meant.

If that's not good enough, check out this recent rating on Mitt Romney's claim that "More Americans have lost their jobs under Barack Obama than any president in modern history..."

Romney’s claim is accurate if you count from every president’s first day in office to his final day -- by those standards, Obama is indeed the only president since World War II to have presided over a net job loss.

Don't worry. Despite finding the literal claim true, PolitiFact invented arbitrary standards to justify finding Romney's claim Mostly False. No "said nothing about" treatment for poor ol' Mitt.

One more?

You can read Bryan's in-depth take down of this rating on Sarah Palin, but I'll give you a hint on how it ends:

Although she's technically correct, the numbers are wildly skewed by tiny, non-industrialized countries. We find her claim Barely True.

The examples are, if I can avoid being taken literally, endless. PolitiFact's vacillating guidelines on when to literally interpret something can provide solid evidence of bias. And, to be fair, PolitiFact doesn't always take Obama literally:

So overall, the poll numbers support Obama’s general point, but they don’t fully justify his claim that "the American people for the most part think it’s a bad idea." Actually, in most of the polls just a plurality says that. On balance, we rate his statement Mostly True.

Doh!

In case readers infer that the above conflicting standards have been cherry picked, it's important to note that every one of the examples provided, including the Promise Kept cited by Stoll, was written by PolitiFacter Louis Jacobson. So we can rule out different approaches from different authors.

Until PolitiFact develops objective standards and adheres to them, the political bias of the writers and editors will be exposed with each contradictory rating.

Although Future of Capitalism isn't a usual destination for PolitiFact articles, I encourage readers to read the whole thing and enjoy Stoll's brilliant writing.


Bryan adds:

Bill Adair offered this in mid-2011:

"You're right that we have not always been consistent on our ratings for these types of claims. We've developed a new principle that is reflected in the Axelrod ruling and should be our policy from now on. The principle is that statistical claims that include blame or credit like this one will be treated as compound statements, so our rating will reflect 1) the relative accuracy of the numbers and 2) whether the person is truly responsible for the statistic.)"

That's probably good news for "Bush's fault" in the past and bad news for "Obama's fault" moving forward.


Jeff adds: Bryan is correct that PolitiFact has "updated" their principles. But we've shown, repeatedly,  that PolitiFact hasn't changed in practice subsequent to publication of Adair's new and improved standard. And Adair's update still fails to provide a transparent, let alone objective, method to weigh the various components of a compound statement.

Big Tent: "A PolitiFact Example"

Blogger and PolitiFact-cited expert Tom Bruscino supplies a partial insider's look at the PolitiFact process along with a critique of the finished work of which he was a part in his post "A PolitiFact Example."

PolitiFact writer Louis Jacobson asked Bruscino for his assessment of Mitt Romney's claim that the U.S. Navy is at its smallest since 1947.

Bruscino found Jacobson's questions leading:

Jacobson did a remarkable bit of research in a very short period of time. However, I did think his questions to me were leading. Remember, Mr. Jacobson asked "(2) What context does this ignore (changing/more lethal technology, changed geopolitical needs, etc)?," which both assumes and implies to the interviewees that Romney ignored those specific contexts.

And after registering some surprise at Jacobson's use of apparently non-objective descriptors of Romney, Bruscino demurs from PolitiFact's "Pants on Fire" ruling:

My opinion, for what it is worth, is that since Romney's base statement was factually accurate when it came to most numerical metrics, it would seem that he could be given credit for a half-truth, even if the context complicates the matter.

Do read Bruscino's entire post, which is particularly valuable since it provides yet another look at the style of inquiry used by PolitiFact journalists.  The commentary thread is also well worth reading.

Hat tip to Power Line blog.  Visit Power Line also for a parallel review I'd have been better off copying rather than writing up my own.



Jeff adds: I first saw this rating yesterday, and couldn't help but notice it provided another example of PolitiFact's alternating standards. Check out how PolitiFact presented this article on their Facebook page:

Image from http://www.facebook.com/politifact

Notice that Romney is spreading ridiculous falsehoods because he "ignores quantum leaps in technology and training."

Poor Mitt. If only he had made this statement back in 2009 when PolitiFact's standards were much different:

We agree that the two cars are totally different. But Obama was careful in the way he phrased his statement: "The 1908 Model T earned better gas mileage than a typical SUV sold in 2008."  As long as you don't consider any factors other than mileage, he's right. We rate his statement Mostly True.

You see, Obama is rated only for his literal statement, while ignoring quantum leaps in technology that make the Model T "totally different." Romney suffers from additional qualifiers that PolitiFact throws in to the mix.

The similarities between the two ratings don't end there. Here's a bit from the Obama/Model T rating:

So technically Obama is right.


But his implication is that we haven't gotten more fuel efficient in 100 years. And that's a reach.
...

...Model Ts reached top speeds of only 40 miles an hour. They guzzled motor oil, about a quart a month. The original tops were made of canvas, and they had no heating or cooling systems. They also had none of the safety features of modern cars: no bumpers, no air bags, no seat belts, no antilock breaks [sic].

The cars had large, skinny wheels to more easily clear the obstacles on rocky, rutted roads. Corner them too fast and they could tip over. And if you crashed, the windshield would usually shatter into sharp, jagged pieces that could slice you to ribbons.

"The government would not allow anyone to sell Model Ts today because they're so unsafe," Casey said. "It's a car that no one would use on a regular basis today. It's not a fair comparison."

Here's similar text from the Romney rating:

This is a great example of a politician using more or less accurate statistics to make a meaningless claim. Judging by the numbers alone, Romney was close to accurate.

...

Thanks to the development of everything from nuclear weapons to drones, comparing today’s military to that of 60 to 100 years ago presents an egregious comparison of apples and oranges. Today’s military and political leaders face real challenges in determining the right mix of assets to deal with current and future threats, but Romney’s glib suggestion that today’s military posture is in any way similar to that of its predecessors in 1917 or 1947 is preposterous.

Obama: Technically correct, as long as you don't consider any other factors, but a reach. Mostly True.

Romney: Close to accurate, meaningless, egregious, glib, preposterous. Pants on Fire.

Bruscino is right to point out the terms used to describe Romney's statement are more appropriate for the editorial page as opposed to an objective determination of facts. And once again, we're left to wonder why different guidelines are used for different people.

Update (1/19/2012 1921 pst) Jeff adds: Speaking of glib and preposterous, this part of the rating just caught my eye:

A wide range of experts told us it’s wrong to assume that a decline in the number of ships or aircraft automatically means a weaker military. Quite the contrary: The United States is the world’s unquestioned military leader today, not just because of the number of ships and aircraft in its arsenal but also because each is stocked with top-of-the-line technology and highly trained personnel.

The first problem is obvious. Romney never claimed that a reduction in the number of ships or aircraft automatically meant a weaker military.  Actually, Romney was citing examples in support of his overall claim (that continued cuts in defense spending will eventually lead to a weaker force). Jacobson's second sentence is a howler. "Quite the contrary" to what? The fact that the U.S. is the world's supreme military force is totally irrelevant to whether or not it's on the path to becoming weaker. If Warren Buffet loses a million dollars on a bad deal, the fact that he's still the richest guy in the room does not negate the fact that he's also a million dollars poorer. And just like Romney claimed in his statement, Buffet simply cannot continue to cut bad deals if he is going to remain the richest guy in the room.

Anchor Rising: "Cicilline Gifted Another Mostly True From Politifact -- Seriously?"

Patrick Laverty and the Anchor Rising blog give us yet another quality criticism of PolitiFact.

PolitiFact just makes it way too easy.

Ok sure, the funding level was "close". It was in the 90s and it was more than the previous administration. However, as Politifact themselves often say, that isn't what Cicilline said. He said it was at 100% all but two years. It was there for all but six years. That's a big difference.

So the issue really speaks to Politifact's credibility, if they have much left. They are, at best, inconsistent with their rulings especially when it comes to Congressman Cicilline. This is the same newspaper that int 2010 endorsed Cicilline for Congress, in part due to his fiscal management of Providence. 

 PolitiFact's justification for the ruling?

Cicilline, in his off-the-cuff statement, mixed up where the years with the lowest contribution fell. But he made it clear a few times that he was citing figures from memory, so we’ll give him some leeway and rule Mostly True.

It's good to know that one can obtain some leeway if one is citing figures from memory.  At least when PolitiFact decides to grant such leeway (I'm not finding other examples of this kind of treatment).  If you're working from memory then the same set of facts can become more true than otherwise.  Like magic.

Looks like we can add a new wrinkle to the Principles of PolitiFact.

Mark Hemingway and Glenn Kessler on NPR

Mark Hemingway, who wrote a key critique of modern fact-checking operations back in December, appeared with the Washington Post's fact checker, Glenn Kessler, for a radio interview on NPR.  It's worth either listening to it or reading the transcript, but one particular section deserves special attention:

CONAN: Here's an email from Noreen(ph). I don't understand that the - that since - excuse me. I don't understand the idea that since PolitiFact demonstrates that Republicans lie three times as often as Democrats mean it's biased. Maybe Republicans actually do lie that much more. The idea that you have to have an even number of lies reported for Democrats and Republicans in order to be considered not biased is ridiculous. One side could lie way more than the other. And by trying to make them even, you are distorting fact. Is simple numerical balance an indication of nonpartisanship?

KESSLER: No. I don't look at them that way, and, as I said, I don't really keep track of, you know, how many Democrats or how many Republicans I'm looking at until, you know, at the end of the year, I count it up. My own experience from 30 years covering Washington and international diplomacy and that sort of thing is there's - both Democrats and Republicans will twist the truth as they wish if it somehow will further their aims. I mean, no one is pure as a driven snow here. And I've often joked that if I ever write an autobiography, I'm going to title it "Waiting for People to Lie to Me."

(SOUNDBITE OF LAUGHTER)

CONAN: That's something reporters do a lot. Mark Hemingway?

HEMINGWAY: Why - I think I said when I even brought this up. I mean, you know, I don't think that, you know, that, you know, numerical selection is indicative of, you know, bias per se. I just think that it's highly suspicious. When it's three to one, you know, if it were 60-40, you know, whatever, yeah, sure, you know? But when it's three to one, you start getting things where, you know, you start wondering about, you know, why the selection bias.

Hemingway's December article was quite valuable, but he missed an opportunity to explain an important aspect of Eric Ostermeier's examination of PolitiFact's story selection.

PolitiFact rated about the same number of politicians from each party.  Yet one party received significantly worse "Truth-O-Meter" ratings.   The key inference behind Ostermeier's study was the expectation that a party-blind editorial selection process should be expected to choose the same types of stories for both parties.  The results, then, if Republicans really do lie more, would show approximately the same distribution of ratings but with one party more heavily represented in the total number of stories.  The approximately even number of stories for each group throws the monkey wrench in Noreen's reasoning.

It would have been good if Hemingway had explained that during the broadcast.

As a side note, it's interesting that Kessler likewise ends up writing approximately as many stories about Democrats as about Republicans.  Run the numbers for Kessler as Ostermeier did for PolitiFact and perhaps the tendencies look alike. The obvious reason for focusing on PolitiFact instead of Kessler is PolitiFact's far greater volume of material.



(1/12/12) Jeff adds: There's a flaw that is often overlooked when discussing the "add 'em up" style of interpreting PolitiFact's ratings, and that's the issue of the quality of the fact checks themselves. Assuming PolitiFact actually adheres to an objective formula for avoiding selection bias, and then rates 50 statements from the left and 50 from the right, it still wouldn't disprove an ideological lean.

Take for example the different standards used when PolitiFact rated similar statements from Herman Cain and Barack Obama. Both included the employer's portion of payroll taxes in their respective calculations, but in Cain's case PolitiFact downgraded him for it, while in Obama's case it pushed him higher up the rating scale. And this still doesn't take into account the dishonest tactic of inventing statements out of thin air.

It may be interesting to review the tallies of who gets what ratings and discuss the merits of the numbers. Ultimately though it's the alternating standards that will offer the best evidence of PolitiFact's liberal bias.


(1/19/2012) Jeff adds: An additional flaw with adding up PolitiFact's ratings is the fact that PolitiFact chooses who to give the rating to.

When Obama claimed that preventative health care "saves money", and David Brooks said he's wrong, PolitiFact gave a True to Brooks. This serves the dual purpose of sparing Obama a False on his "report card" that PolitiFact likes to shill so often, while also providing cover in the "we give Republicans True's too!" sense.

When PolitiFact rated the oft-repeated, and false, claim about $16 muffins at DOJ event, PolitiFact could have given the rating to NPR, the New York Times, or even (gasp!) PolitiFact partner ABC News. Instead, they chose to burden Bill O'Reilly with the falsehood, despite the original claim coming from a government report.

It's these types of shenanigans that will always distort a ratings tally.

Update/clarification (1/14/2012):

Added "for a radio interview on NPR" to the first sentence.

James Taranto: "Bad-Faith Journalism"

The Wall Street Journal once again weighs in on PolitiFact following its "Lie of the Year" selection.

This cycle, James Taranto hits PolitiFact for its wrongheaded approach to fact checking:

We're not as troubled as Ponnuru is by the effect of PolitiFact, and the "fact checking" genre it exemplifies, on politics. We'd argue instead that it has a baneful effect on journalism.

Taranto's complaint about PolitiFact has much to do with the rationale Michael F. Cannon used to make his decision to withhold from PolitiFact the expert opinions it solicits from time to time.

Though Taranto goes a bit easy on PolitiFact for its past "Lie of the Year" failures, it's worth reading every word.

PolitiFact 2011: A review

Crossposted from Sublime Bloviations.

PolitiFact Bias has now spent approximately a full year highlighting criticisms of the PolitiFact fact checking brand.

Our hopes that PolitiFact would improve its performance in light of outside criticism have gone largely unfulfilled.  Perhaps the biggest improvement was the reconciliation of two differing definitions of the "Half True" rating, but that modest accomplishment occurred without any announcement or acknowledgment at all from PolitiFact.  By contrast, PolitiFact wrote extensively about its momentous change in calling its fourth rating from the top "Mostly False" rather than "Mostly True" even though the definition remained the same.

Here's a rundown of the issues that should keep discerning readers from trusting PolitiFact:

1)  PolitiFact persistently ignores the effects of selection bias.  It simply isn't plausible that editors who are very probably predominantly liberal will choose stories of interest on a neutral basis without some systematic check on ideological bias.  PolitiFact, for example, continues to publish candidate report cards as though selection bias has no effect on the report card data.

2)  PolitiFact continues to publish obviously non-objective stories without employing the journalistic custom of using labels like "commentary," "opinion" or even "news analysis."  Readers are implicitly led to believe that stories like an editorial "Lie of the Year" selection are objective news stories.

3)  PolitiFact continues to routinely apply its principles of analysis unevenly, as with its interpretation of job creation claims (are the claims assumed to refer to gross job creation or net job creation?).

4)  PolitiFact has yet to shake its penchant for snark.  Snark has no place in objective reporting (see #2 above).  Unfortunately, PolitiFact treats it like a selling point instead of a weakness, and PolitiFact's intentional use of it has apparently influenced Annenberg Fact Check to follow suit.

There is a silver lining.  PolitiFact's methods produce perhaps the best opportunity yet to objectively measure mainstream media bias.  Some of those projects will be published at PolitiFact Bias over the coming year, with the study specifics available through Google Docs.