True statements ruled "Mostly False"

What happens when PolitiFact finds that a statement is literally true?

That issue was brought up indirectly when Jeff retweeted economist Howard Wall. Wall had tweeted:

.@jamestaranto I was an expert in Politifact once. Sole purpose of the interview was to illicit a few words to show pre-existing conclusion.

— Howard Wall (@GatewayEcon) January 10, 2016

We looked up the story where Wall was quoted as an expert. It was a fact check of Mitt Romney from the 2012 presidential election. The Romney campaign said women had suffered the greatest job losses under Obama, implying Obama's leadership had been bad for women.

PolitiFact ruled the claim "Mostly False."

The Romney campaign pushed back. PolitiFact looked at the issue again and ruled the claim "Mostly False." But at the same time, PolitiFact said "The numbers are accurate but quite misleading."

Don't blame the Romney campaign. It probably operated under the assumption that PolitiFact's definitions for its "Truth-O-Meter" ratings mean something.

Taking PolitiFact's definitions literally, the lowest rating one should receive for a literally true claim is "Mostly True." Once below that level, the definitions start talking about "partially true" statements that give a misleading impression ("Half True") and "The statement contains some element of truth" but ignores facts that could give a different impression ("Mostly False").

What's our point? We've always said PolitiFact's ratings reveal more about PolitiFact than they do about the entities receiving the ratings. It's a scandal that social scientists keep their eyes closed to that. Search PolitiFact's ratings for claims it says are literally true. Note the rating given to the claim. Then take a look at the ideology of the entity making the claim.

There's your evidence of journalistic bias by fact checkers.

This is an important issue. If social scientists aren't looking at it, it suggests they don't care.

Why wouldn't they care?



Jeff Adds: We highlighted a Mark Hemingway critique of PolitiFact's Romney claim back in 2012 that is still worth a read. It would seem little has changed at PolitiFact since then.



Update 0956PST 1/12/2016: Added "Jeff Adds" portion - Jeff

Right Wisconsin: "Kooyenga Rips PolitiFact Rating" (Updated)

State representative and C.P.A Dale Kooyenga responds via Right Wisconsin to PolitiFact's "Mostly False" rating of his statement that the legislature has used GAAP accounting principles for the past two years.

We're not professional accountants, but it looks like Kooyenga has a good case that PolitiFact Wisconsin's Dave Umhoefer gave his statement an unreasonable interpretation.

I have spoken to dozens of CPAs about your rating - they all agree your rating is simply wrong.  I would be happy to arrange a meeting with several CPAs from academia and the private sector to resolve the misinformation you printed in today's paper.  Ironically, your rating is simply wrong and I hope you are willing to revisit this matter. 

Read the whole thing, and then go get yourself some popcorn.

Update 9/27/2013

The MacIver Institute's MacIver News Service updates the story with a news video:

Big Journalism: "Politispin: 'Fact-Checkers' Mislead on GOP Leaders' Favorable Unemployment Numbers"

Big Journalism's Tony Lee posted an article yesterday that sets the tone with the first line:

The purportedly unbiased Politifact will go to great lengths to help Democrats.

That's old news around here, but like the liberals who endlessly parrot PolitiFact's spin, we appreciate confirmation bias as much as the next guy. 

So what's all the hubbub about? PolitiFact Rhode Island's rating of GOP gubernatorial candidate John Robitaille's claim that "Unemployment rate dropped in every state that elected a Republican gov. in 2010." Robitaille based his claim on a report done by Robert Elliott. Lee critiques PolitiFact's dance moves:

In a remarkable twisting of facts and logic, Politifact concedes Elliott’s two points are true before somehow rendering those points to be “Half True.”

This isn't the first time in recent memory where PolitiFact found ways to determine accurate figures weren't worthy of a True rating. This rating also adds to the list of experts left with a bad taste in their mouth after dealing with PolitiFact:

“That type of spin would be expected of, say, the Democratic Governors Association, but not a supposedly ‘objective’ and ‘nonpartisan’ news organization that claims to be the official arbiter of the truth,” Elliott told Breitbart News. “It is the insidious nature of PolitiFact's bias that makes them so loathsome.”

Elliot's statements by themselves make Lee's article a must-read. But Lee sweetens the pot by highlighting PolitiFact's use of extraneous evidence to cloud the issue they were ostensibly reviewing:

Politifact then goes on to compare the unemployment rates of the states that simply elected a governor who was from a different party from the predecessor’s, which is a completely different analysis than Elliott’s, which is what Politifact was supposed to be “fact checking.”

When doing that analysis resulted in Republican governors still reducing the unemployment rate faster than Democratic governors, Politifact decided to compare the unemployment numbers of the Republican predecessors in states that elected Democrats and Democratic predecessors in states that elected Republicans. Only then -- when not even comparing the current crop of governors or the past two years, which was the basis of Elliott’s analysis -- was Politifact able to find something they could use to say Democrats (predecessors) were slightly better than Republicans (predecessors) at reducing the unemployment rate.

Lee nails the point, and this reminds me of PolitiFact's treatment of Laura Ingraham's claim about RomneyCare's unpopularity with national voters. In that case, PolitiFact based their entire rating on statistics only from Massachusetts when Ingraham was probably talking about all 50 states. It appears that when PolitiFact doesn't like the initial outcome, they find new facts to throw into the mix until they reach their desired outcome.

The most hilarious part of the rating was that PolitiFact not only conceded, but confirmed Elliot's numbers:

Considering the unemployment rate has fallen in 49 states in the last year, that’s stretching the statistic pretty thin.

We find Robitaille’s claim "is partially accurate but leaves out important details or takes things out of context," our definition of Half True.

If the unemployment rate fell in 49 states, by definition Robitaille's claim is true. Using PolitiFact's logic, if Robitaille had claimed "The sun rose in every state that elected a GOP governor", he'd only be rated Half-True because he left out important details. It's nonsense. And it's not fact checking.

This is the first time we've noticed Tony Lee, but if this installment at Big Journalism is any indication, we're looking forward to highlighting his work in the future. Head over to Breitbart and read the whole thing. There's plenty more to this smackdown.


Bryan adds:

Matthew Hoy of Hoystory also takes issue with PolitiFact's rating of Robitaille.

One might cut PolitiFact a break for trying to take credit into account for the sake of its ratings if the effort was evenly applied and didn't force PolitiFact to largely ignore the definitions it established for its ratings.

What do I think of PolitiFact's execution?  I'd borrow a line from legendary football coach John McKay:  "I'm in favor of it."

Michael F. Cannon: "Strike Three for PolitiFact"

Better late than never, we take note of Cato's Michael F. Cannon's take on PolitiFact's 2011 "Lie of the Year":

 The annual unveiling of its “Lie of the Year” award garners PolitiFact more attention than anything else. Hopefully, it will garner so much attention that people will recognize this award, which is supposed to improve political discourse, instead degrades it.

PolitiFact’s past three Lies of the Year have been about health care.  Not one of them was a lie.

Cannon has boycotted PolitiFact because of the subjectivity of its "Lie of the Year" selections, turning down requests that he contribute is expert opinion to various stories.

We join others in calling for an expanded boycott of PolitiFact:  Don't cite PolitiFact as a dependable source of information.  Pressure it to reform by dumping the silly and misleading "Truth-O-Meter" and finding additional ways to separate objective reporting from news analysis and outright opinion.

Cannon's commentary is crisp and to-the-point.  Do read it all, and visit his links to appreciate the history of his principled boycott.

Commentary: "PolitiFact's Pants on Fire"

Another PolitiFact-cited expert has joined Tom Bruscino in dumping on PolitiFact over a recent fact check on Republican presidential candidate Mitt Romney.

This time it's Ted R. Bromund, writing in Commentary magazine.  Like Bruscino, Bromund pans PolitiFact for using leading questions and for issuing a ruling with which he cannot agree.  But Bromund goes further in noting that PolitiFact performed its fact check based on a straw man version of Romney's point (bold emphasis added):

Jacobson sums up Romney’s contention as being: “The U.S. military has been seriously weakened compared to what it was 50 and 100 years ago.” Since the Truman Doctrine of 1947 is as good a marker as any of the moment when the U.S. assumed the global security responsibilities that formerly belonged to Britain, the fact that our Air Force is smaller and older than it has been at any point since that date might give immediately give cause for concern.

But the obvious point of Romney’s statement was not that the U.S. military of today would lose a war to the U.S. military of 1917 or 1947. It was that the margin of U.S. “military superiority” – i.e., its relative strength versus potential and actual adversaries – is at risk if defense spending declines, as President Obama plans for it to do. The question is not whether the U.S.’s “military posture is in any way similar to that of its predecessors in 1917 or 1947”: it is whether the U.S.’s margin of superiority over other actors, taking contextual factors into account, is better or worse than it was in previous eras.

Bromund weakens his case some with his characterization of PolitiFact's argument in the story.  It seems the argument was more that the current state of the armed forces as described by Romney remains the world's superior military force, though PolitiFact does use some language easily taken to support his interpretation.  But Bromund scores a huge hit in noting that PolitiFact stuffed Romney's argument full of straw.  Just have a look at the manipulated quotation PolitiFact used in its deck material:

The U.S. military is at risk of losing its "military superiority" because "our Navy is smaller than it's been since 1917. Our Air Force is smaller and older than any time since 1947." 

PolitiFact uses the above to represent Romney's argument. True, the story goes on to quote Romney accurately, including his argument that cuts in defense spending might lead to the U.S. losing military superiority.  But the story treats Romney's argument in accordance with the inaccurate hybrid quotation/paraphrase:

(A) wide range of experts told us it’s wrong to assume that a decline in the number of ships or aircraft automatically means a weaker military. Quite the contrary: The United States is the world’s unquestioned military leader today, not just because of the number of ships and aircraft in its arsenal but also because each is stocked with top-of-the-line technology and highly trained personnel.

Thanks to the development of everything from nuclear weapons to drones, comparing today’s military to that of 60 to 100 years ago presents an egregious comparison of apples and oranges.

It's remarkable that PolitiFact's summary omits Romney's mention of cuts in defense spending.


PolitiFact has stepped in it again and has again reacted by publishing a pedantic rebuttal along the lines of the one it used in response to criticism of its 2011 Lie of the Year.  The former will receive a review here in due time. (Update 1/22/12 See review here).

Additional reading:

Power Line stays on the story, providing additional valuable material by publishing the text of Bromund's reply to PolitiFact writer Louis Jacobson after Jacobson's initial inquiry about the Romney issue.

Politico took note of the story with a piece by Dylan Byers.  Byers's story drew most of PolitiFact's ire in the rebuttal mentioned above, so reading it provides excellent background material.

The Huffington Post even gets in on the act with solid article.  The story provides the context of Romney's remarks with a partial debate transcript.


Also see the my latest post in the "Piquing PolitiFact" category at Sublime Bloviations, which features the text of an email message I sent to PolitiFact asking for PolitiFact to fact check one of its own claims.



Update (1/22/2012): Added link to Bruscino article, link to PF response review, and minor spelling corrections-Jeff

Big Tent: "A PolitiFact Example"

Blogger and PolitiFact-cited expert Tom Bruscino supplies a partial insider's look at the PolitiFact process along with a critique of the finished work of which he was a part in his post "A PolitiFact Example."

PolitiFact writer Louis Jacobson asked Bruscino for his assessment of Mitt Romney's claim that the U.S. Navy is at its smallest since 1947.

Bruscino found Jacobson's questions leading:

Jacobson did a remarkable bit of research in a very short period of time. However, I did think his questions to me were leading. Remember, Mr. Jacobson asked "(2) What context does this ignore (changing/more lethal technology, changed geopolitical needs, etc)?," which both assumes and implies to the interviewees that Romney ignored those specific contexts.

And after registering some surprise at Jacobson's use of apparently non-objective descriptors of Romney, Bruscino demurs from PolitiFact's "Pants on Fire" ruling:

My opinion, for what it is worth, is that since Romney's base statement was factually accurate when it came to most numerical metrics, it would seem that he could be given credit for a half-truth, even if the context complicates the matter.

Do read Bruscino's entire post, which is particularly valuable since it provides yet another look at the style of inquiry used by PolitiFact journalists.  The commentary thread is also well worth reading.

Hat tip to Power Line blog.  Visit Power Line also for a parallel review I'd have been better off copying rather than writing up my own.



Jeff adds: I first saw this rating yesterday, and couldn't help but notice it provided another example of PolitiFact's alternating standards. Check out how PolitiFact presented this article on their Facebook page:

Image from http://www.facebook.com/politifact

Notice that Romney is spreading ridiculous falsehoods because he "ignores quantum leaps in technology and training."

Poor Mitt. If only he had made this statement back in 2009 when PolitiFact's standards were much different:

We agree that the two cars are totally different. But Obama was careful in the way he phrased his statement: "The 1908 Model T earned better gas mileage than a typical SUV sold in 2008."  As long as you don't consider any factors other than mileage, he's right. We rate his statement Mostly True.

You see, Obama is rated only for his literal statement, while ignoring quantum leaps in technology that make the Model T "totally different." Romney suffers from additional qualifiers that PolitiFact throws in to the mix.

The similarities between the two ratings don't end there. Here's a bit from the Obama/Model T rating:

So technically Obama is right.


But his implication is that we haven't gotten more fuel efficient in 100 years. And that's a reach.
...

...Model Ts reached top speeds of only 40 miles an hour. They guzzled motor oil, about a quart a month. The original tops were made of canvas, and they had no heating or cooling systems. They also had none of the safety features of modern cars: no bumpers, no air bags, no seat belts, no antilock breaks [sic].

The cars had large, skinny wheels to more easily clear the obstacles on rocky, rutted roads. Corner them too fast and they could tip over. And if you crashed, the windshield would usually shatter into sharp, jagged pieces that could slice you to ribbons.

"The government would not allow anyone to sell Model Ts today because they're so unsafe," Casey said. "It's a car that no one would use on a regular basis today. It's not a fair comparison."

Here's similar text from the Romney rating:

This is a great example of a politician using more or less accurate statistics to make a meaningless claim. Judging by the numbers alone, Romney was close to accurate.

...

Thanks to the development of everything from nuclear weapons to drones, comparing today’s military to that of 60 to 100 years ago presents an egregious comparison of apples and oranges. Today’s military and political leaders face real challenges in determining the right mix of assets to deal with current and future threats, but Romney’s glib suggestion that today’s military posture is in any way similar to that of its predecessors in 1917 or 1947 is preposterous.

Obama: Technically correct, as long as you don't consider any other factors, but a reach. Mostly True.

Romney: Close to accurate, meaningless, egregious, glib, preposterous. Pants on Fire.

Bruscino is right to point out the terms used to describe Romney's statement are more appropriate for the editorial page as opposed to an objective determination of facts. And once again, we're left to wonder why different guidelines are used for different people.

Update (1/19/2012 1921 pst) Jeff adds: Speaking of glib and preposterous, this part of the rating just caught my eye:

A wide range of experts told us it’s wrong to assume that a decline in the number of ships or aircraft automatically means a weaker military. Quite the contrary: The United States is the world’s unquestioned military leader today, not just because of the number of ships and aircraft in its arsenal but also because each is stocked with top-of-the-line technology and highly trained personnel.

The first problem is obvious. Romney never claimed that a reduction in the number of ships or aircraft automatically meant a weaker military.  Actually, Romney was citing examples in support of his overall claim (that continued cuts in defense spending will eventually lead to a weaker force). Jacobson's second sentence is a howler. "Quite the contrary" to what? The fact that the U.S. is the world's supreme military force is totally irrelevant to whether or not it's on the path to becoming weaker. If Warren Buffet loses a million dollars on a bad deal, the fact that he's still the richest guy in the room does not negate the fact that he's also a million dollars poorer. And just like Romney claimed in his statement, Buffet simply cannot continue to cut bad deals if he is going to remain the richest guy in the room.

Michael F. Cannon repeats/clarifies: "PolitiFact Just Asked Me to Be a Source, Again. I Declined, Again."

Michael F. Cannon of the CATO Institute, in the past used by PolitiFact as an expert source, started boycotting PolitiFact based on its weighing in on opinion.

Cannon gave us an update on July 8 indicating that PolitiFact had called on him for expertise and he had declined to provide it.

On Oct. 12 Cannon posted a similar message clarifying his statement that he "declined to help" PolitiFact:

The whole purpose of my PolitiFact boycott is to help them.

We try to emphasize the same attitude at PolitiFact Bias.  A PolitiFact aware of and able to check its tendency toward ideological bias can properly serve a useful and essential role in our constitutional republic.

But PolitiFact's failure to address errors such as its flip-flop on the percentage of the payroll tax paid by employees and its blind support of Alan Grayson's ridiculous argument serve to discourage those of us hoping to see PolitiFact rise to a higher standard of fact checking.

Michael F. Cannon: "PolitiFact Just Called, Again. I Declined to Help, Again."

Short and to the point, Cato Institute's Michael J. Cannon reminds us that his boycott of PolitiFact continues.  As he notes in his title, PolitiFact sometimes seeks his services as an expert source.  Follow the links to see why Cannon no longer cooperates with PolitiFact.

Pressroom Buzz @ Inside Milwaukee: "No Comment"

Jeff tipped me off to a blog post at Inside Milwaukee's "Pressroom Buzz."  At first I thought it was fodder for a PFB Smackdown feature.  But with respect to any kind of credible charge of a bias against the left at PolitiFact Wisconsin there's just no "there" there.

The blog post, by Erik Gunn, discloses the decision of some Democratic Party politicians in Wisconsin to stop cooperating with PolitiFact, a la Cato Institute's Michael F. Cannon.

The decision by the Democrats seems a bit more self-serving and a bit less principled than Cannon's--at least to me. Your mileage may vary:

The party doesn’t intend to enforce a uniform stance against PolitiFact among Democratic candidates, and (Democratic Party of Wisconsin's communications director Graeme) Zielinski says officials also didn’t plan any sort of big campaign to publicize the party’s decision to stop cooperating with the feature’s reporters. “We’re not going to make a big deal of it,” Zielinski says. “We just think there’s no utility in dealing with them anymore.”

To help illustrate the DPW's discontent with PolitiFact Wisconsin, Gunn tosses in a couple of recent complaints about PFW's impartiality.  But they're both duds.

The first is barely long enough to qualify as a blog post.  The criticism amounts to an assertion that a PolitiFact story about a voter ID issue can't be taken seriously, not counting linked material of dubious value.  If the author's serious about being taken seriously then he won't farm out the argument.

The second blog post at least invests a good number of words, but it amounts to a complaint about selection bias.  Selection bias is ubiquitous at PolitiFact.  Lone examples mean nothing or next to nothing. Larger sets of data however, such as the collection we present on this blog, may carry some weight.

Gunn also obtained a pair of recent examples from Zielinski.  Those were marginally more compelling than the blogged complaints:

PolitiFact labeled as “Pants on Fire” the statement by Democratic state Rep. Sandy Pasch – who is challenging Republican Sen. Alberta Darling in a recall election – that Paul Ryan’s plan to change Medicare to a voucher system with a fixed government contribution “would end health care for our seniors.”

 And:

PolitiFact labeled as “Pants on Fire” a Democratic Party assertion that windows at the state Capitol were bolted in response to protests in February, and said that party Chairman Mike Tate didn’t respond to PolitiFact inquiries.

We share some sympathy for any figure who receives a "Pants on Fire" rating from PolitiFact, since the rating by definition cannot be justified objectively.  But for political entities for which I could easily assign a political position the Republicans and conservatives had 12 Pants on Fire ratings compared to 15 for the Democrats and progressives.  The difference, I expect, is borderline for statistical significance.

Granted, in comparison to other PolitiFact operations the Democrats have received harsher grading in Wisconsin. As a result the Wisconsin branch has received the brunt of the "PolitiFact has a right-wing bias" complaints. That doesn't necessarily say anything about an ideological bias at PolitiFact Wisconsin.

Despite the weak examples, Gunn does have some good points to make.  They just don't add up to an argument for an anti-liberal bias.  Rather, they resonate with some of the criticisms of PolitiFact's journalistic model made standard here at PolitiFact Bias.

For example:

PolitiFact’s critics have a point, but it’s a bit more complicated than partisan bias. The larger problem remains in its simplistic rating system and especially the incendiary “Pants on Fire” category, which doesn’t appear to be consistently applied.

It's hard for PolitiFact to apply it consistently when PolitiFact's statement of principles describes the rating category in subjective terms.

Gunn also noted what I've referred to as PolitiFact's snark:

Finally, there’s an air of smugness in the tone of many PolitiFact pieces that condescends not just to the objects of criticism but to readers. I have a hunch that tone alone accounts for some of the hostility toward the operation, from whatever corner.

Where smugness of tone occurs as a result of opinionated language, readers can be forgiven for their impression that the journalists responsible for the story have compromised their objectivity.

While journalists overall have a ideological tilt to the left, of course they are not monolithically liberal.  We at PFB welcome careful criticism of PolitiFact from the left--such little as there is at present--confident in the expectation that when all the facts are in the evidence will show PolitiFact predisposed to favoring the left on balance.  Significantly so.

Michael F. Cannon: "Why I'm Boycotting PolitiFact"

Michael F. Cannon of the CATO Institute has often served PolitiFact as an expert source.  Today he explained why he no longer cooperates with PolitiFact in that capacity:

Reporters at PolitiFact.com have used me as a resource half a dozen times or so when fact-checking something someone said about health care reform. Sometimes we disagree about where the truth lies, but I’ve always been happy to help. That changed recently, and I should let PolitiFact’s reporters know why.

read it all

Cannon's boycott is conditional:   He will lift it after he is satisfied that PolitiFact has addressed problems with its past two "Lie of the Year" selections.

I'll quote one more important passage, though Cannon's column ought to be read completely through:

Some conservatives think PolitiFact is a left-wing outfit. I don’t think that’s true, and I have defended PolitiFact against that charge. I believe that PolitiFact’s reporters are earnestly doing their best to get at the truth. But there’s a tension between that belief and these errors. Whether PolitiFact recognizes and addresses that tension will tell us a lot about PolitiFact.

I share Cannon's assessment.  It is extraordinarily unlikely that PolitiFact intends to do its fact checking any way other than impartially.  But on the other hand it's hard to reconcile that premise with some of the things PolitiFact puts to print and with some of the errors PolitiFact refuses to acknowledge.

I predict that Cannon will not end up lifting his boycott.  PolitiFact is too invested in its model, given its relative success, to risk tampering with it.

But I'd love to be proved wrong.  Just like Cannon wouldn't mind lifting his boycott under the right conditions.