Wednesday, January 23, 2019

Not a Lot of Reader Confusion XII

We say that PolitiFact's graphs and charts, including its PunditFact collections of ratings for news networks, routinely mislead readers. But PolitiFact Editor Angie Drobnic Holan says there isn't much reader confusion.

Today on its Facebook Page, PolitiFact continued its custom of highlighting one of its politician "report cards."

In keeping with our expectation that PolitiFact's report cards mislead the PolitiFact audience, we found the following:

We're not making the commenters impossible to verify, but we're blacking out the names and emphasizing that nobody should harass these people over their comments in any way. Somebody already pointed out a problem with their view, so there's not even a need to do that. Just leave it alone and allow this to stand as the latest proof that PolitiFact has its head in the sand over the way its charts mislead its audience. That's using the charitable assumption that PolitiFact isn't deliberately deceiving its audience.

As we have endlessly pointed out, the non-scientific method PolitiFact uses to sample the universe of political claims make statements about the overall percentages of false statements unreliable.

PolitiFact knows this but declines to make it clear to readers with the use of a consistent disclaimer.

The type of response we captured above happens routinely.

Correction May 3, 2019: We did XI twice! Fixed the title to reflect the more accurate count.

Thursday, January 17, 2019

PolitiFact's Heart Transplant

In the past we have mocked PolitiFact's founding editor Bill Adair for saying the "Truth-O-Meter" is the "heart of PolitiFact."

We have great news. PolitiFact has given itself a heart transplant.

PolitiFact's more recent (since May of 2018) self-descriptions now say that fact-checking is the heart of PolitiFact:

That's a positive move we applaud, while continuing to disparage the quality of PolitiFact's fact checks.

It was always silly to call a subjective sliding-scale Gimmick-O-Meter the heart of PolitiFact (even if it was or remains true).

The new approach at least represents improved branding.

Now if PolitiFact could significantly upgrade the quality of its work ...

Post-publication update: Added hotlinks to the first paragraph leading to past commentary on PolitiFact's heart.

Tuesday, January 15, 2019

PolitiFact and the Contradiction Fiction

We consider it incumbent on fact checkers to report the truth.

PolitiFact's struggles in that department earn it our assessment as the worst of the mainstream fact checkers. In our latest example, PolitiFact reported that President Donald Trump had contradicted his claim that he had never said Mexico would pay for the border wall with a check.

We label that report PolitiFact's contradiction fiction. Fact checkers should know the difference between a contradiction and a non-contradiction.

PolitiFact (bold emphasis added):
"When during the campaign I would say ‘Mexico is going to pay for it,’ obviously, I never said this, and I never meant they're going to write out a check," Trump told reporters. "I said they're going to pay for it. They are."

Later on the same day while visiting the border in Texas, Trump offered the same logic: "When I say Mexico is going to pay for the wall, that's what I said. Mexico is going to pay. I didn't say they're going to write me a check for $20 billion or $10 billion."

We’ve seen the president try to say he never said something that he very much said before, so we wondered about this case.

Spoiler: Trump has it wrong.

We found several instances over the last few years, and in campaign materials contradicting the president’s statement.
PolitiFact offers three links in evidence of its "found several instances" argument, but relies on the campaign material for proof of the claimed contradiction.

We'll cover all of PolitiFact evidence and show that none of it demonstrated that Mr. Trump contradicted himself on this point. Because we can.

Campaign Material

PolitiFact made two mistakes in trying to prove its case from a Trump Campaign description of how Trump would make Mexico pay for the border wall. First, it ignored context. Second, it applied spin to one of the quotations it used from the document.

PolitiFact (bold emphasis added):
"It's an easy decision for Mexico: make a one-time payment of $5-10 billion to ensure that $24 billion continues to flow into their country year after year," the memo said.

Trump proposed measures to compel Mexico to pay for the wall, such as cutting off remittances sent from undocumented Mexicans in the U.S. via wire transfers.

Then, the memo says, if and when the Mexican government protested, they would be told to pay a lump sum "to the United States to pay for the wall, the Trump Administration will not promulgate the final rule, and the regulation will not go into effect." The plan lists a few other methods if that didn’t work, like the trade deficit, canceling Mexican visas or increasing visa fees.
We placed bold emphasis on the part of the memo PolitiFact mentioned but ignored in its reasoning.

If the plan mentions methods to use if Mexico did not fork over the money directly, then how can the memo contradict Trump's claim he did not say Mexico would pay by check? Are fact checkers unable to distinguish between "would" and "could"? If Trump says Mexico could pay by check he does not contradict that claim by later saying he did not say Mexico would pay by check.

And what's so hard to understand about that? How can fact checkers not see it?

To help cinch its point, PolitiFact quotes from another section of the document, summarizing it as saying Mexico would pay for the wall with a lump sum payment: "(Mexico) would be told to pay a lump sum 'to the United States to pay for the wall'"). Except the term "lump sum" doesn't occur in the document.

There's reason for suspicion any time a journalist substitutes for the wording in the original document, using only a partial quotation and picking up mid-sentence. Here's the reading from the original:
On day 3 tell Mexico that if the Mexican government will contribute the funds needed to the United States to pay for the wall ...
We see only one potential justification for embroidering the above to make it refer to a "lump sum." That's from interpreting "It's an easy decision for Mexico: make a one-time payment of $5-10 billion to ensure that $24 billion continues to flow into their country year after year" as specifying a lump sum payment. We think confirmation bias would best explain that interpretation. It's more reasonable to take the statement to mean that paying for the wall once and having it over with is an obvious choice when it helps preserve a greater amount of income for Mexico annually after that. And the line does not express an expectation of a lump-sum payment but instead the strength (rightly or wrongly) of the bargaining position of the United States.

In short, PolitiFact utterly failed to make its case with the example it chose to emphasize.

... And The Rest

 (these are weak, so they occur after a page break)

Monday, January 7, 2019

Research shows PolitiFact leans left: The "Pants on Fire" bias

In 2011 PolitiFact Bias started a study of the way PolitiFact employs its "Pants on Fire" rating.

We noted that PolitiFact's definitions for "False" and "Pants on Fire" ratings appeared to differ only in that the latter rating represents a "ridiculous" claim. We had trouble imagining how one would objectively measure ridiculousness. PolitiFact's leading lights appeared to state in interviews that the difference in the ratings was subjective. And our own attempt to survey PolitiFact's reasoning turned up nothing akin to an empirically measurable difference.

We concluded that the "Pants on Fire" rating was likely just as subjective as PolitiFact editors described it. And we reasoned that if a Republican statement PolitiFact considered false was more likely than the same type of statement from a Democrat to receive a "Pants on Fire" rating we would have a reasonable measure of ideological bias at PolitiFact.

Every year we've updated the study for PolitiFact National. In 2017, PolitiFact was 17 percent more likely to give a Democrat a "Pants on Fire" rating for a false statement. But the number of Democrats given false ratings was so small that it hardly affected the historical trend. Over PolitiFact's history, Republicans are over 50 percent more likely to receive a "Pants on Fire" rating for a false claim than a Democrat.


After Angie Drobnic Holan replaced Bill Adair as PolitiFact editor, we saw a tendency for PolitiFact to give Republicans many more false ("False" plus "Pants on Fire") ratings than Democrats. In 2013, 2015, 2016 and 2017 the percentage was exactly 25 percent each year. Except for 2007, which we count as an anomaly, that percentage marked the record high for Democrats. It appeared likely that Holan was aware of our research and leading PolitiFact toward more careful exercise of its subjective ratings.

Of course, if PolitiFact fixes its approach to the point where the percentages are roughly even, this powerfully shows that the disparities from 2009 through 2014 represent ideological bias. If one fixes a problem it serves to acknowledge there was a problem in need of fixing.

In 2018, however, the "Pants on Fire" bias fell pretty much right in line with PolitiFact's overall history. Republicans in 2018 were about 50 percent more likely to receive a "Pants on Fire" rating for a claim PolitiFact considered false.

The "Republicans Lie More!" defense doesn't work

Over the years we've had a hard time explaining to people why it doesn't explain away our data to simply claim that Republicans lie more.

That's because of two factors.

First, we're not basing our bias measure on the number of "Pants on Fire" ratings PolitiFact doles out. We're just looking at the percentage of false claims given the "Pants on Fire" rating.

Second, our research provides no reason to believe that the "Pants on Fire" rating has empirical justification. PolitiFact could invent a definition for what makes a claim "Pants on Fire" false. PolitiFact might even invent a definition based on some objective measurement. And in that case the "Republicans lie more!" excuse could work. But we have no evidence that PolitiFact's editors are lying when they tell the public that the difference between the two ratings is subjective.

If the difference is subjective, as it appears, then PolitiFact's tendency to more likely give a Republican's false statement a "Pants on Fire" rating counts as a very clear indicator of ideological bias.

To our knowledge, PolitiFact has never addressed this research with public comment.

Thursday, January 3, 2019

PolitiFact's 10 Worst Fact Check Flubs of 2018

The worst of the mainstream fact checkers, PolitiFact, produced many flawed fact checks in 2018. Here's our list of PolitiFact's 10 worst fact check flubs from 2018.

10 PolitiFact Wisconsin's Worry-O-Meter

Republican Leah Vukmir challenged Democrat Tammy Baldwin for one of Wisconsin's senate seats in 2018. Vukmir challenged Baldwin's willingness to take a hard line on terrorism by saying Baldwin was more worried about "the mastermind of 9/11" than supporting Trump's nominee head of the CIA.

How does a fact checker measure worry?

No worries! PolitiFact Wisconsin claimed to have looked for signs Baldwin worried about Khalid Sheik Mohammed and didn't find anything. So it rated Vukmir's claim "Pants on Fire." PolitiFact Wisconsin skillfully circumnavigated Vukmir's clearly implied reference to a key reason Democrats opposed Trump's nominee, Gina Haspel: She had followed orders to implement enhanced interrogation techniques, including waterboarding. Mohammed was one of those to whom the technique was applied. Those are not the kinds of dots a fact checker like PolitiFact can connect.

Current immigration policy costs as much as $300 billion according to one study

The White House published an infographic claiming immigration policy costs the government money--as much as $300 billion according to one study. PolitiFact examined the question and found that it was "Half True" because it supposedly left out important details, like "U.S.-born children with at least one foreign-born parent are among the strongest economic and fiscal contributors, thanks in part to the spending by local governments on their education." No, really. That's critical missing context in PolitiFact's book. At least in this case.

Trump says senior White House official who said North Korean summit would be impossible to keep does not exist

The New York Times reported a "senior White House official" said U.S. summit with North Korea would be impossible to keep on its original date. It turned out the official didn't quite say that, but instead said words to the effect that keeping the original date would prove extremely difficult.

When Trump tweeted that the the source did not exist, PolitiFact fact checked the claim. In doing so, the fact checkers set aside the idea that Trump was saying no senior White House official had made the claim attributed by the Times. The fact checkers concluded that Trump's tweet was "Pants on Fire" false because the person to whom the Times attributed its dubious paraphrase was a real person.

We count this as a classic example of uncharitable interpretation.

Sen. Ted Cruz claims he has consistently opposed government shutdowns

After Sen. Ted Cruz (R-Texas) said he has consistently opposed government shutdowns, PolitiFact rated his claim "Pants on Fire" because Cruz joined a failed vote against cloture on a bill that would have ended a government shutdown. PolitiFact said Cruz had failed his own test for supporting a shutdown: Cruz said shutdowns happen when senators vote to deny cloture on a funding bill. But obviously on a failed cloture vote the shutdown does not occur even though some senators voted to deny cloture on a funding bill. PolitiFact tried to make Cruz look like a hypocrite by taking his statement out of context.

PolitiFact claims Trump was wrong that a civilian in the room with Omar Mateen might have prevented the Pulse nightclub massacre

(Via Zebra Fact Check) When President Trump tweeted that a civilian with a gun in the room with Mateen might have prevented or reduced the casualties from the Pulse nightclub shooting, PolitiFact ruled the claim "False."

But PolitiFact made an incoherent case for its ruling. Trump was stating a counterfactual scenario, that if a civilian with a gun had been in the room with Mateen then the killing might have been prevented. PolitiFact argued, in effect, that the police detective doing guard duty in the Pulse parking lot counted as the civilian in the room with Mateen and had no effect on the outcome. A person in a parking lot is not the same as a person in the room, we say. And we see no grounds for the implication that the detective in the parking lot failed to contribute to a better outcome compared to having no armed guard in the parking lot.

PolitiFact determines 4.1 percent GDP growth objectively not "amazing."

After President Trump went to Twitter to declare 4.1 percent GDP growth rate "amazing," PolitiFact fact checked the claim and determined it "False." The Weekly Standard took note of PolitiFact's factual interest in a matter of opinion.

PolitiFact often fails to follow its principle against fact-checking opinion or hyperbole, and this case serves as an excellent example.

Does the European Union export cars to the U.S. by the millions?

After President Trump claimed the EU exports cars to the United States by the millions, PolitiFact interpreted the claim to refer specifically (and separately) to Mercedes and BMW vehicles (Trump mentioned both in his tweet) or to Germany in particular. Additionally, PolitiFact assumed that the rate of imports had to exceed 1 million per year to make Trump's claim true. That's despite the fact Trump specified no timed rate.

PolitiFact found its straw man version of Trump's claim "False." As a matter of fact, the number of cars manufactured in the European Union and exported to the United States exceeded 1 million from 2014 through 2016 (the latest numbers when Trump tweeted). Definitely false, then?

3 Did added debt in 2017 exceed cumulative debt over the United States' first 200 years in terms of GDP?

When MSNBC host Joe Scarborough said the Trump administration had added more debt than was accumulated in the nation's first 200 years, PolitiFact fact checked the claim. It was true, PolitiFact found, in terms of raw dollars. But experts told PolitiFact that the debt as a measure percentage of GDP serves as the best measure. So PolitiFact incorrectly interpreted the total accumulated debt in 2017 as added debt and proclaimed that Scarborough's claim checked out in terms of percentage of GDP.

Scarborough received a "Mostly True" rating for a claim that was incorrect in terms of GDP--what PolitiFact reported as the most appropriate measure.

Making this one even better, PolitiFact declined to fix the problem after we pointed it out to them.

2 PolitiFact decides who built what

After the right-leaning Breitbart news site published a fact check announcing that immigrants did not build Fall River, Massachusetts ("mostly false," according to that fact check), PolitiFact published a fact checking finding the Breitbart fact check "False." PolitiFact and Breitbart reported that established residents of Fall River built factories. Immigrants came to work in the factories. We agree with Breitbart that it does not make sense to withhold all credit from the people who built the factories.

1 PolitiFact flip-flops on its 2013 "Lie of the Year"

Republican senatorial candidate Josh Hawley (R-Mo.) tagged his opponent, Democrat Claire McCaskill, with chiming in on PolitiFact's 2013 "Lie of the Year"--the promise that Americans would be able to keep their existing health care plans under the Affordable Care Act.

Hawley accurately summarized PolitiFact's reasoning for its 2013 award. Obama's promise was emphatic that people would not lose their existing plans. Yet millions received cancellation notices in 2013 from insurance companies electing to simply drop potentially grandfathered plans. That led to Obama's promise sharing the "Lie of Year." PolitiFact baselessly claimed that when Hawley said people lost their plans he was sending the message that the millions of people completely lost insurance instead of simply losing the plans they preferred.

Happy New Year!

Correction Jan 3, 2018: Did a strikethrough correction, changing "measure of GDP" to "percentage of GDP"