Wednesday, May 27, 2015

Bernie Sanders, PolitiFact, PolitiMath

We do a "PolitiMath" evaluation of PolitiFact's fact checks where numerical errors ought to have a powerful bearing on PolitiFact's "Truth-O-Meter" ratings. We're interested in how percentage error impacts the differences between "Pants on Fire," "False," "Mostly False" and so on.

An older item on Sen. Bernie Sanders (I, Vt.) caught our eye in the midst of one of PolitiFact's bogus "report card" stories. Sanders said the United States spends twice as much per capita on health care than any other nation on earth.

PolitiFact found Sanders was off:
According to the 2009 edition of WHO's World Health Statistics report, which uses figures from 2006, health care spending in the United States — both public- and private-sector — amounted to $6,719 per capita. Ranking next were Luxembourg and Monaco at $6,506 and $6,353 per capita, respectively. All told, either 11 or 15 countries told the WHO they spent more than $3,360 per capita, the point at which the United States no longer doubles their spending. (We provide two possible figures here because the WHO offers both raw figures and statistics adjusted for currency valuations.) The other nations that rank near the top with the United States include Austria, Belgium, Canada, Denmark, France, Germany, Iceland, Ireland, the Netherlands, Norway, Sweden and Switzerland, in addition to tiny Malta and San Marino.
We'd have gone to bat for Sanders if only OECD nations were counted. The United States spends more per capita than its nearest rival by over 50 percent, which is the reasonable floor for rounding up to a "twice as much" claim. But the United States only spends 3.3 percent more per capita than Luxembourg, which is a pretty far cry from 50 percent.

Sanders exaggerated the truth by at least 1,400 percent, using the figures from Luxembourg as the counterexample to his claim.

PolitiFact's rating of Sanders? "False."

Tuesday, May 26, 2015

The awe-inspiring importance of place

On May 22, 2015, PolitiFact Wisconsin published a fact check of the National Republican Senatorial Committee. It seems the NRSC accused Democratic candidate for Wisconsin senator, Russ Feingold, of announcing his candidacy in California.

Whoa, said PolitiFact Wisconsin. That's "Pants on Fire" false!

We admit the ruling amused us. After all, PolitiFact is the non-partisan fact checker that gave then-candidate Barack Obama a "Mostly True" for the claim his uncle helped liberate Auschwitz.

No, we're not kidding.

Auschwitz was liberated by the Soviet Union, not by Allied forces. PolitiFact gave Obama the "Mostly True" rating since he did have an "uncle" (in a loose-but-legitimate sense of the term) who was with the Allied forces that liberated a concentration sub-camp called Ohrdruf.

The NRSC was ineligible for that kind of break despite saying Feingold was a Stanford (California) professor and a failed 2010 candidate:
"In what may be the oddest move of the campaign season thus far, Stanford Professor and failed 2010 candidate Russ Feingold announced his Wisconsin Senate run today from … California."
Funny how that works.

Thursday, May 7, 2015

PolitiFact Wisconsin: It's false until somebody fact-checks it

We've long registered our objections to PolitiFact's fallacious "burden of proof" criterion for political claims.

PolitiFact Wisconsin gives us a fantastic example of that flawed method with its fact check of Republican presidential candidate Ben Carson. Carson highlighted problems with advancement in the black community by saying there are more blacks involved with the criminal justice system than with higher education.

PolitiFact Wisconsin decided to evaluate claim and rated it "False." But of course there's a problem with the rating. PolitiFact Wisconsin found it had poor data with which to work:
(T)here is only one solid figure -- 75,000 black males ages 18 to 24 in prison. We’re not aware of any recent counts of the black males in that age group who were arrested, in jail, or on probation or parole at a particular time.
PolitiFact Wisconsin emphasized that relatively low solid figure in its summary paragraph:
Carson did not provide evidence that backs his claim. The latest federal figures we found show 75,000 black males in that age group who were in prison in 2013 and in the range of 690,000 to 779,000 who were in college. We are not aware of any recent figures for the number of black males ages 18 to 24 arrested, in jail, or on probation or parole at any particular time.

If figures do surface, we’ll re-evaluate this item, but we rate Carson’s claim False.
Perhaps it makes sense if the number of blacks ages 18 to 24 in college outnumber those involved with the criminal justice system 779,000 to 75,000. But that number comparison is rigged against Carson. PolitiFact Wisconsin acknowledges Carson's claim on unknown numbers of young blacks arrested, in jail, on probation or on parole.

And that's the Achilles' heel of PolitiFact Wisconsin's fact check. It collected enough information to enable rough estimates of those categories.

Was Carson's claim plausible?


We start our estimate by noting the percentage of blacks ages 18 to 24 in the federal prison population was fairly high: PolitiFact said the number of about 14 percent of the total black population.

We aim to create a conservative estimate, erring on the side of caution, so we'll assume that just 10 percent of blacks in the other categories fall in the 18 to 24 age range.

Arrests


PolitiFact Wisconsin noted that one individual might be arrested more than once. Still, the fact checkers gave the number 3 million for arrests of adult blacks in 2012. Ten percent of 3 million gives us a figure of 300,000, but to help account for multiple arrests we'll cut that number in half and use 150,000.


In Jail


PolitiFact Wisconsin gave a figure of 261,500 for jail inmates in mid-2013. Ten percent of that figure gives us about 26,000.

On Probation or Parole


PolitiFact Wisconsin said 4.75 million people of all races were on probation, parole or other supervision in 2013. The Bureau of Justice Statistics, PolitiFact's source, says blacks account for 30 percent of that number. That gives us 1.4 million, and 10 percent of 1.4 million comes to 140,000.

Totals

Combined with the 75,000 prison population PolitiFact Wisconsin used, our conservative estimate comes to 391,000--about half of PolitiFact Wisconsin's peak figure for black college enrollment. Based on our estimate, we think it's very unlikely Carson's claim exaggerates the truth by more than 100 percent, probably exaggerates it by substantially less than 100 percent and perhaps doesn't exaggerate at all.

For comparison, PolitiFact Georgia recently gave a "Mostly False" rating to a claim exaggerated by over 200 percent.

Do these numbers potentially support Carson's underlying point about the upward mobility of young black males? For some reason, PolitiFact Wisconsin did not deem that point worth considering.

Shall fact checkers rate claims "False" if they are difficult to settle? We think that's the wrong method. We also think fact checkers err by selectively ignoring politicians' underlying arguments. Either consider the underlying argument every time or never consider the underlying argument. Fairness demands it.

Monday, May 4, 2015

The Federalist: PunditFact: A Case Study In Fact-Free Hackery

Sean Davis, writing for The Federalist, gives us one of the most detailed critiques of PolitiFact we've seen in some time. Davis wrote to defend an article he wrote on the Clinton Foundation's finances, which PolitiFact's PunditFact rated after Rush Limbaugh repeated the information.

Davis starts strong and keeps it up through the end of his devastating story:
If you like liberal ideologues who label inconvenient facts as “false,” then you’ll love PunditFact. Why? Because PunditFact just declared that a demonstrably true fact was false because, according to PunditFact, the factual claim “ignores critical facts that would give a different impression.” Facts are strange that way. They do tend to give an impression of the truth, even if some people find that impression discomfiting.
As usual, we see special value where Davis shows his interactions with PolitiFact writer Louis Jacobson. Davis takes pains to point out to Jacobson that Clinton Foundation activities don't fit what people typically understand as a charity. That point never surfaces in PunditFact's fact check, despite the fact that it represents Davis' (and probably Limbaugh's) underlying point.

PunditFact and GDP revisited

In a post from May 2, 2015 we looked at an error on GDP rankings that PunditFact ruled "Pants on Fire." Showtime's Jim Lampley, speaking about the boxing match between Floyd Mayweather and Manny Pacquiao, said the revenue from the fight exceeded the GDP of 29 countries. A commenter on PolitiFact's Facebook page pointed out a parallel error PunditFact included in its fact check:


None of the 50 states produce a GDP below $400 million. PunditFact made a more severe error, in terms of exaggeration, than Lampley did. If even one state had a GDP below $400 million then PolitiFact exaggerated by 3,700 percent compared to Lampley's 383 percent exaggeration.

Lampley gets a 500+ word article highlighting his error, more-or-less permanently archived at PolitiFact. PunditFact, in contrast, gets the following treatment from PolitiFact:
Correction: All 50 U.S. states have a gross state product in excess of $400 million. An earlier version of this fact-check was incrorrect [sic] on this point.
We think this correction serves pretty effectively to sweep PunditFact's mistake under the rug. Yes, PunditFact implicitly admits suggesting at least one state had a gross state product less than $400 million. But the correction obscures the nature of the error, particularly its magnitude.

PolitiFact maintains of habit of incomplete transparency with its corrections.


Transparent correction May 6, 2015:  We put "greater" in the next-to-last paragraph when we meant "less." We substituted "less" for "greater" to correct the mistake.

Sunday, May 3, 2015

PunditFact's PolitiMath on the GDP of 29 countries

We do "PolitiFact" stories to examine how PolitiFact's ratings correlate to percentage error. Claims where the ratings seem based purely or mainly on the degree of error serve as the best case studies. PunditFact gives us a great study example with its article on the claim that a boxing match would generate more revenue than the GDP of 29 different countries.

PunditFact ruled that claim "Pants on Fire," finding only six countries with a GDP lower than that predicted for the fight: $400 million.

Jim Lampley's figure of 29 exaggerates PunditFact's total by 383 percent. That substantial error, we suppose, justifies the "Pants on Fire" rating.

On the other hand, PunditFact gave Cokie Roberts a "Half True" rating for a claim she exaggerated by over 9,000 percent. PunditFact gave Roberts credit for her underlying point, that the risk of getting murdered in Honduras is greater than for New York City.

Apparently Lampley has no valid underlying point that the Mayweather-Pacquiao fight would generate a great deal of revenue.

You be the judge.


Update May 3, 2015

While researching and wondering how Lampley ended up with 29 countries producing a GDP under $400 million, we noticed a perhaps-coincidental statistic: The World Bank's 2013 GDP rankings have 29 countries with a GDP above $400 billion.


Lampley's claim may have started with this statistic. After mixing up millions with billions and mistaking the top of the list for the bottom, Lampley's claim makes perfect sense, in a way.


Correction May 4, 2015: Fixed spelling of "Pacquiao."

Thursday, April 23, 2015

PolitiFact Georgia, PolitiMath and the gender pay gap

On April 22, 2015, PolitiFact Georgia found it "Mostly False" that women make only 78 percent of what men make for doing the same work.

PolitiFact Georgia reported that the claim, from a Buzzfeed video, based its claim on statistics that, among other faults, did not bother to ensure that the men and women were doing the same work.

At Zebra Fact Check I've published an in-depth treatment of the way mainstream fact checkers mishandle the gender pay gap. But here we'll look narrowly at how PolitiFact Georgia applies a "Mostly False" rating to a gross exaggeration. Our "PolitiMath" stories explore the relationship between percentage error and PolitiFact's ratings, so PolitiFact Georgia's story makes a good subject.

PolitiFact's highest estimate of the wage gap after controlling for the type of job and some other factors was about 7 percent:
(T)he American Association of University Women that controlled for college major, occupation, age, geographical region, hours worked and more, and found there was still a 7 percent wage gap between male and female college grads a year after graduation.
Using that high-end estimate, the Buzzfeed video exaggerated by no less than 214 percent. There's precedent for liberals receiving ratings of "Mostly False" or better for exaggerations that large and larger. On the other hand, PolitiFact Wisconsin gave a state Democrat a "False" rating for an exaggeration of 114 percent.

At least we know Buzzfeed's exaggeration is not the largest to receive a rating of "Mostly False" or higher.

If anyone can find a statement from a Republican or conservative where a figure exaggerated by more than 100 percent received a rating of "Mostly False" or higher from PolitiFact, we'd love to hear about it. We haven't turned up anything like that yet.