Are PolitiFact's "report cards" misleading?

One of our recurrent themes at PolitiFact Bias concerns the misleading nature of PolitiFact's "report cards." PolitiFact admits it does not perform its fact checks on a scientific basis, particularly in that story choices do not occur randomly. Despite that, it's utterly common for PolitiFact to publish a "report card" story encouraging readers to consider a candidate's "report card."

The latest such asks readers to consider the record of just-announced presidential candidate and former Secretary of State Hillary Clinton.

The reader comments from PolitiFact's Facebook page offer us a window into the degree of deception PolitiFact achieves with its "report card" stories.

We'll omit the names to save these individuals unnecessary embarrassment.

"When you compare the overall honesty of Democrats vs. Republicans, it's no wonder that some Republicans believe that fact checking web sites are liberally biased. It's an easier explanation then the reality that Republicans tend to lie more."

"Better than Ted Cruz."

"Comparison: Clinton, true or mostly true = 48%; Rand Paul, true or mostly true = 15%; Ted Cruz, true or mostly true = 8% (from PolitiFact archives)."

"I would love to see a chart comparison against the other candidates. Paul's record would be a joke, his pants have been on fire so much the fire department had to move into a spare room"

"Still not a Hillary fan, but at least she's more honest that the right wing."

"Not a bad record."

Many more after the break! 

It's tweezers for Senator Paul

Tweezers or tongs?

Will PolitiFact take just part of a statement into account (tweezers), or will it focus on the whole of the statement (tongs)?

PolitiFact fact checked a statement from Sen. Rand Paul (R-Ky.)

Here's Sen. Rand Paul (R-Ky.) announcing his bid for the presidency:

It seems to me that both parties and the entire political system are to blame.

Big government and debt doubled under a Republican administration.

And it’s now tripling under Barack Obama’s watch. President Obama is on course to add more debt than all of the previous presidents combined.

Paul's first sentence in the above quotation looks like opinion. That leaves three great potential fact checks. First, that the debt doubled under "a Republican administration" (George W. Bush). Second, the debt is now tripling under President Barack Obama. Third, Obama is on course to add more debt than all of the previous presidents combined.

PolitiFact feints as though it will cover the two claims about multiplying the debt. But the "Half True" rating suggests that PolitiFact only rated the claim about Obama tripling the debt. PolitiFact concludes:

Paul said, "Debt doubled" under Bush "and now it’s tripling under Barack Obama’s watch."

This statement is confusing. A person could easily interpret it to mean that debt has tripled since Obama took office -- which would be incorrect. Paul, on the other hand, said that it means debt today, under Obama, is triple what it was when Bush’s term started. 

...

From one not-so-obvious angle, Paul's numbers are correct. But because the statement could so easily be interpreted in another, less accurate way, we rate it Half True.

PolitiFact found Paul was accurate about the doubling of the debt under a Republican administration. So if his statement about the debt tripling under Obama was completely false, combining the true and false statements averages out to "Half True." But PolitiFact doesn't say Paul was wrong about the tripling of the debt, only that it was wrong if taken in the supposedly obvious way, that the debt tripled starting from the time Obama took office. So why isn't Paul's claim "Mostly True"?

Spoiler: PolitiFact rigs the game.

The "Half True" rating doesn't fit. The context of Paul's statement makes clear he's criticizing Democrats and Republicans. But the clincher is Paul's claim that Obama is on course to add more debt than all previous presidents combined.

Looked at in the simplest way, the way people are likely to understand it, the debt from year to year represents the debt of all previous presidents combined. Most added debt, but a few, like Calvin Coolidge, produced a surplus.

If Obama had nearly tripled the debt since he took office then he's not "on course" to add more debt than all previous presidents combined. He'd have done it once already with a good shot at doing it a second time.

Taken properly in context, the only sensible meaning of Paul's statement is the one he gave: He was talking about Obama tripling the debt in the sense of taking the next step past Bush's doubling of the debt.

PolitiFact, unsurprisingly, did not quote Paul's statement about Obama adding more debt than all the presidents preceding him combined. Leaving out important context helps PolitiFact apply its tweezers treatment.

Fact checkers shouldn't blame politicians when people interpret their statements incorrectly or stupidly. Fact checkers should explain the correct or most sensible interpretation to help those people understand it correctly.

But-but-but ... Experts!

I can imagine the PolitiFact apologist noticing that in our comparison between PolitiFact's alligator attack and shark attack stories that experts only complained about the silliness of the alligator attack comparison.

So, logically, it's not PolitiFact Florida's fault that the alligator attack comparison was silly while the shark attack comparison wasn't silly.

So there! Reality just has a liberal bias, and stuff.

Well, I covered this angle when I wrote up the alligator attack story at Zebra Fact Check.

Interpreting the comparison is not rightly the job of the criminology expert, nor the alligator attack expert. Those experts properly inform as to the number of attacks by concealed-carry permit holders, or the number of alligator attacks. It is the expert on English communications (if needed) that rightly evaluates the comparison. And in this case I challenge any expert on English to draw a principled distinction between PolitiFact Florida's stories on alligator and shark attacks. Both compare rare but dramatically different things. Both involve a number for which we don't have reliable statistics.

On the issue of experts pointing out the silliness of the comparison, the real question is why experts didn't call both comparisons silly.

Perhaps it was random variation.

Perhaps PolitiFact Florida didn't feel the need to interview a panel of experts about voter fraud.

Perhaps the experts carry their own political bias.

We think a fact checker should be able to make the call on a literary comparison that demands no particular English or logical expertise. And with respect to the alligator attack and shark attack stories, the call should be the same for both.

I'll reiterate what I wrote at Zebra Fact Check: Fact checkers should not allow experts to decide an issue outside their area of expertise.

Sharks & Alligators

I spent time at Zebra Fact Check last week dissecting PolitiFact Florida's effort to fact check whether the first 10 years of Florida's concealed-carry gun law produced twice as many alligator attacks as attacks by concealed-carry permit holders.

One aspect of that study deserves special attention here at PolitiFact Bias.

PolitiFact Florida couldn't find the information it needed to check the alligator claim. But it found a couple of experts who thought it was silly to compare gun attacks to alligator attacks, and the supposed silliness of the comparison found its way into the "Mostly False" rating. Health News Florida reviewed the rating with PolitiFact's Amy Hollyfield:

Since there's no source of comprehensive data for attacks by gun license holders, experts told PolitiFact Florida that it’s not very meaningful to compare alligator bites to the misuse of firearms.

“One of them told us it’s more than silly to compare bites to bullets,” Hollyfield said.

We couldn't avoid thinking about PolitiFact Florida's 2012 fact check of whether shark attacks outnumber cases of voter fraud. PolitiFact chose the measure: "cases" considered for prosecution by the state of Florida instead of "cases" as individual instances of fraud.

PolitiFact Florida admitted its measure was imperfect:

While the shark attack figures are cut and dry (sorry!), the voter fraud numbers are not. There could be more cases than we know about, involving more people. The numbers may not represent total voter fraud cases, as those could be handled by local supervisors and state attorneys.

Without considering the silliness of comparing voter fraud to shark attack and undeterred by the lack of good data on voter fraud, PolitiFact Florida ruled it "Mostly True" that shark attacks occur more frequently than voter fraud.

Obviously inconsistent? Yeah. Coincidentally, the point liberals like, that voter fraud occurs rarely, gets a pass. The claim conservatives like, that concealed-carry permit holders very rarely use their guns to attack others, gets the harsh rating.

That's PolitiFact.  That's bias.

Tweezers: PolitiFact and the Indiana boycott

Sometimes PolitiFact focuses on one part of a statement. Sometimes PolitiFact spreads its focus to cover the whole of a statement. We use the "Tweezers or Tongs" tag for posts where we draw a contrast involving PolitiFact's choice of focus.

On April 2, 2015, PolitiFact published a story looking at a statement from Red State's Erick Erickson. Erickson wrote about Indiana's version of the Religious Freedom Restoration Act and the willingness of some on the left to punish Indiana economically for passing the legislation.

PolitiFact lays out the basics, we provide the bold emphasis:

In a column for conservative grassroots site RedState.com, editor-in-chief Erick Erickson criticized business owners and people on the left who say the law will allow anyone to cite religious belief in refusing to serve gays and lesbians. Erickson’s opening sentence hones in on Apple chief executive Tim Cook for what he sees as hypocritical business practices.

"To recap: Tim Cook (please, please click this link) and the left are happy to do business in countries that stone to death or otherwise jail gay people, but will not do business with Indiana," Erickson wrote, "which merely passed a law insisting that the ‘free exercise’ clause of the first amendment be on the same legal footing in courts as the ‘free speech’ clause of the first amendment."

Obviously Erickson wasn't talking only about Apple CEO Tim Cook. He mentioned "the left." And Erickson has a point that some on the left took action to cut back business dealings with Indiana:

Companies, celebrities and even local and state governments have come out in opposition to Indiana's controversial "religious freedom" bill. Several have even cancelled plans to do business in the state, citing the potential for discrimination against gays and lesbians.

PolitiFact's story contains mention of only one boycott: the one PolitiFact says Erickson said was coming from Apple. Tweezers.

PolitiFact claimed Erickson was making the point that Cook was acting hypocritically by boycotting Indiana while continuing to do business with nations like Iran. But if that was really Erickson's point then why did he dedicate only one line in his entire column to that point? Tweezers.

Erickson was making a broader point about the reaction by some on the left. He accurately characterized reaction of some on the left in threatening Indiana with economic sanctions. And in the process he made it sound like Apple had committed to a concrete set of such sanctions against Indiana. That's where PolitiFact's tweezers came in, for Cook had simply written a column criticizing Indiana's RFRA law.

But here's the hole in PolitiFact's fact check: Did Apple have any sponsored events occurring in Indiana that it might have cancelled, like some other companies had done? If not, is it safe to assume Apple would not have joined some other companies in canceling such events?

The Apple convention "MacWorld/iWorld" took place in the middle of March earlier this year. In San Francisco. A lost opportunity to teach Indiana a lesson?

What did PolitiFact do wrong, if anything?

As we pointed out, if Erickson was making a point about a real Apple boycott of Indiana, he had plenty of opportunity to mention Apple specifically. But he did not. He lumped Cook in with the boycott, which was somewhat misleading, but Erickson was setting the stage for a general criticism of the left's intolerance to resistance of its mainstreaming of homosexuality. And Cook's a fair example to match with the point of Erickson's column. PolitiFact missed the point of the column and the reference to Cook, using its tweezers to ding Erickson while not even acknowledging the reality of the boycott threatened by companies aside from Apple.

Tweezers.

Impartial tweezers?

Nah. PolitiFact's trick is to often treat parallel statements from liberals or Democrats with tongs. Sure, part of the statement is false, but part of it is true! So, "Half True" or something!



Footnote:

The Erickson response: Erickson defended his column by saying sometimes a tweet is just a tweet. PolitiFact gleefully made light of that excuse, noting that Erickson's column was not a tweet. Note to Erickson: What PolitiFact did was ridiculous, but you need to do better than that.

Correction: Struck "Florida" from the title, as PolitiFact National was responsible for the Red State fact check, not PolitiFact Florida.

Fact checking the PolitiFact way

Okay, kids, today we're going to look at how to fact check the PolitiFact way. We'll look at how to tell whether President Obama's signature health care reform bill, known as the ACA or "Obamacare," was a job killer.

It's okay to leak your findings at the beginning of the fact check:

Predictions about the health care law were a dime a dozen back in 2010. Supporters contended that virtually everyone around the country would soon have access to affordable insurance. Opponents said the law would cost a fortune by adding to the national debt and killing jobs.

Actually, none of those things have happened.

Next, provide evidence supporting your findings:

One of the warnings that the law’s opponents issued repeatedly in the months leading up to passage was that the health care law would kill jobs. In 2011, Republicans titled the repeal legislation they were pursuing the "Repealing the Job-Killing Health Care Law Act." But independent studies didn’t back up the claims that the law would end up reducing employment, so PolitiFact has rated such statements False.

Some might question whether it's appropriate to judge the ACA's effect on jobs by simply looking at employment numbers. Just assume that such people are biased and that their criticism will be drowned out by those who agree with your judgment.

Even though pointing to your own past work without getting into specifics ought to be plenty to convince people, it's okay to repeat your evidence for emphasis:

In the years since passage, employment in both the United States and Florida has been on an upward trajectory as the economy has recovered from a recession.

What more proof could anyone need? There we have it: the facts. Because the total number of jobs increased and the unemployment rate decreased, therefore the ACA had no negative effect on the number of jobs.


Seriously, how can PolitiFact take itself seriously?

More PolitiMath, featuring PunditFact and Jalen Ross

In the past, we've pointed out the tendency at PolitiFact to make mistakes on basic math (and the tendency of Democrats to receive benefits from those mistakes).

Today PunditFact gave us another example, though this one they fixed pretty quickly when they learned something was amiss. Therefore our focus will fall not on the fact that a liberal, student Jalen Ross, gained the benefit from PunditFact's temporary misstatement of fact, but on the PolitiMath (the relationship, or lack thereof, between percentage error and PolitiFact's ratings) involved in keeping a "Mostly True" rating for Ross after fixing the mistake.

Ross attempted to use evidence from a scientific experiment to support the idea that racism remains a prominent problem in the United States. But he did not properly use the study's findings, as PunditFact belatedly pointed out:

Ross erred slightly in his exact wording. While white-sounding names spurred 50 percent more callbacks than the ones with black-sounding names, black-sounding names were 33 percent less likely to get responded to.

The slight error in Ross' exact wording ended up exaggerating the discrimination against so-called "black-sounding names" by about 50 percent ((50-33)/33).

So Ross used bad math and ended up exaggerating the key figure by 50 percent.

Let's compare what happened to Republican Mitch McConnell when he used questionable math and exaggerated his key figure by 37 percent or more (bold emphasis added):

In any event, the lowest estimate of Bush's war spending through 2008 that is even remotely defensible is $808 billion. Tack that onto the $132 billion cost of Katrina and you get $940 billion for the wars and Katrina.

That's well over the expenditures expected from the Democrats' stimulus and children's health insurance bills, which total $686 billion once tax cuts are subtracted. Even if we included the cost of the tax cuts (for a total of $818 billion), he would still be wrong because that's less than the $940 billion that uses a more accurate cost of the war spending.

PolitiFact's ruling on McConnell's claim? "False."

Hmmm. Apparently McConnell had no valid underlying point that the stimulus bill cost a great deal of money.

Inconsistent.

PolitiFact & Israel, thousand words edition

The left-wing bloggers at PolitiFact used Israeli Prime Minister Netanyahu's address to Congress to throw out more clickbait: Come see all our fact checks involving Israel!

We found their choice of photographs intriguing. After all, nothing says "objective and nonpartisan" like people with bloody hands wearing Netanyahu masks, right?

The image comes from PolitiFact's Facebook page.