Thursday, May 7, 2015

PolitiFact Wisconsin: It's false until somebody fact-checks it

We've long registered our objections to PolitiFact's fallacious "burden of proof" criterion for political claims.

PolitiFact Wisconsin gives us a fantastic example of that flawed method with its fact check of Republican presidential candidate Ben Carson. Carson highlighted problems with advancement in the black community by saying there are more blacks involved with the criminal justice system than with higher education.

PolitiFact Wisconsin decided to evaluate claim and rated it "False." But of course there's a problem with the rating. PolitiFact Wisconsin found it had poor data with which to work:
(T)here is only one solid figure -- 75,000 black males ages 18 to 24 in prison. We’re not aware of any recent counts of the black males in that age group who were arrested, in jail, or on probation or parole at a particular time.
PolitiFact Wisconsin emphasized that relatively low solid figure in its summary paragraph:
Carson did not provide evidence that backs his claim. The latest federal figures we found show 75,000 black males in that age group who were in prison in 2013 and in the range of 690,000 to 779,000 who were in college. We are not aware of any recent figures for the number of black males ages 18 to 24 arrested, in jail, or on probation or parole at any particular time.

If figures do surface, we’ll re-evaluate this item, but we rate Carson’s claim False.
Perhaps it makes sense if the number of blacks ages 18 to 24 in college outnumber those involved with the criminal justice system 779,000 to 75,000. But that number comparison is rigged against Carson. PolitiFact Wisconsin acknowledges Carson's claim on unknown numbers of young blacks arrested, in jail, on probation or on parole.

And that's the Achilles' heel of PolitiFact Wisconsin's fact check. It collected enough information to enable rough estimates of those categories.

Was Carson's claim plausible?

We start our estimate by noting the percentage of blacks ages 18 to 24 in the federal prison population was fairly high: PolitiFact said the number of about 14 percent of the total black population.

We aim to create a conservative estimate, erring on the side of caution, so we'll assume that just 10 percent of blacks in the other categories fall in the 18 to 24 age range.


PolitiFact Wisconsin noted that one individual might be arrested more than once. Still, the fact checkers gave the number 3 million for arrests of adult blacks in 2012. Ten percent of 3 million gives us a figure of 300,000, but to help account for multiple arrests we'll cut that number in half and use 150,000.

In Jail

PolitiFact Wisconsin gave a figure of 261,500 for jail inmates in mid-2013. Ten percent of that figure gives us about 26,000.

On Probation or Parole

PolitiFact Wisconsin said 4.75 million people of all races were on probation, parole or other supervision in 2013. The Bureau of Justice Statistics, PolitiFact's source, says blacks account for 30 percent of that number. That gives us 1.4 million, and 10 percent of 1.4 million comes to 140,000.


Combined with the 75,000 prison population PolitiFact Wisconsin used, our conservative estimate comes to 391,000--about half of PolitiFact Wisconsin's peak figure for black college enrollment. Based on our estimate, we think it's very unlikely Carson's claim exaggerates the truth by more than 100 percent, probably exaggerates it by substantially less than 100 percent and perhaps doesn't exaggerate at all.

For comparison, PolitiFact Georgia recently gave a "Mostly False" rating to a claim exaggerated by over 200 percent.

Do these numbers potentially support Carson's underlying point about the upward mobility of young black males? For some reason, PolitiFact Wisconsin did not deem that point worth considering.

Shall fact checkers rate claims "False" if they are difficult to settle? We think that's the wrong method. We also think fact checkers err by selectively ignoring politicians' underlying arguments. Either consider the underlying argument every time or never consider the underlying argument. Fairness demands it.

Monday, May 4, 2015

The Federalist: PunditFact: A Case Study In Fact-Free Hackery

Sean Davis, writing for The Federalist, gives us one of the most detailed critiques of PolitiFact we've seen in some time. Davis wrote to defend an article he wrote on the Clinton Foundation's finances, which PolitiFact's PunditFact rated after Rush Limbaugh repeated the information.

Davis starts strong and keeps it up through the end of his devastating story:
If you like liberal ideologues who label inconvenient facts as “false,” then you’ll love PunditFact. Why? Because PunditFact just declared that a demonstrably true fact was false because, according to PunditFact, the factual claim “ignores critical facts that would give a different impression.” Facts are strange that way. They do tend to give an impression of the truth, even if some people find that impression discomfiting.
As usual, we see special value where Davis shows his interactions with PolitiFact writer Louis Jacobson. Davis takes pains to point out to Jacobson that Clinton Foundation activities don't fit what people typically understand as a charity. That point never surfaces in PunditFact's fact check, despite the fact that it represents Davis' (and probably Limbaugh's) underlying point.

PunditFact and GDP revisited

In a post from May 2, 2015 we looked at an error on GDP rankings that PunditFact ruled "Pants on Fire." Showtime's Jim Lampley, speaking about the boxing match between Floyd Mayweather and Manny Pacquiao, said the revenue from the fight exceeded the GDP of 29 countries. A commenter on PolitiFact's Facebook page pointed out a parallel error PunditFact included in its fact check:

None of the 50 states produce a GDP below $400 million. PunditFact made a more severe error, in terms of exaggeration, than Lampley did. If even one state had a GDP below $400 million then PolitiFact exaggerated by 3,700 percent compared to Lampley's 383 percent exaggeration.

Lampley gets a 500+ word article highlighting his error, more-or-less permanently archived at PolitiFact. PunditFact, in contrast, gets the following treatment from PolitiFact:
Correction: All 50 U.S. states have a gross state product in excess of $400 million. An earlier version of this fact-check was incrorrect [sic] on this point.
We think this correction serves pretty effectively to sweep PunditFact's mistake under the rug. Yes, PunditFact implicitly admits suggesting at least one state had a gross state product less than $400 million. But the correction obscures the nature of the error, particularly its magnitude.

PolitiFact maintains of habit of incomplete transparency with its corrections.

Transparent correction May 6, 2015:  We put "greater" in the next-to-last paragraph when we meant "less." We substituted "less" for "greater" to correct the mistake.

Sunday, May 3, 2015

PunditFact's PolitiMath on the GDP of 29 countries

We do "PolitiFact" stories to examine how PolitiFact's ratings correlate to percentage error. Claims where the ratings seem based purely or mainly on the degree of error serve as the best case studies. PunditFact gives us a great study example with its article on the claim that a boxing match would generate more revenue than the GDP of 29 different countries.

PunditFact ruled that claim "Pants on Fire," finding only six countries with a GDP lower than that predicted for the fight: $400 million.

Jim Lampley's figure of 29 exaggerates PunditFact's total by 383 percent. That substantial error, we suppose, justifies the "Pants on Fire" rating.

On the other hand, PunditFact gave Cokie Roberts a "Half True" rating for a claim she exaggerated by over 9,000 percent. PunditFact gave Roberts credit for her underlying point, that the risk of getting murdered in Honduras is greater than for New York City.

Apparently Lampley has no valid underlying point that the Mayweather-Pacquiao fight would generate a great deal of revenue.

You be the judge.

Update May 3, 2015

While researching and wondering how Lampley ended up with 29 countries producing a GDP under $400 million, we noticed a perhaps-coincidental statistic: The World Bank's 2013 GDP rankings have 29 countries with a GDP above $400 billion.

Lampley's claim may have started with this statistic. After mixing up millions with billions and mistaking the top of the list for the bottom, Lampley's claim makes perfect sense, in a way.

Correction May 4, 2015: Fixed spelling of "Pacquiao."

Thursday, April 23, 2015

PolitiFact Georgia, PolitiMath and the gender pay gap

On April 22, 2015, PolitiFact Georgia found it "Mostly False" that women make only 78 percent of what men make for doing the same work.

PolitiFact Georgia reported that the claim, from a Buzzfeed video, based its claim on statistics that, among other faults, did not bother to ensure that the men and women were doing the same work.

At Zebra Fact Check I've published an in-depth treatment of the way mainstream fact checkers mishandle the gender pay gap. But here we'll look narrowly at how PolitiFact Georgia applies a "Mostly False" rating to a gross exaggeration. Our "PolitiMath" stories explore the relationship between percentage error and PolitiFact's ratings, so PolitiFact Georgia's story makes a good subject.

PolitiFact's highest estimate of the wage gap after controlling for the type of job and some other factors was about 7 percent:
(T)he American Association of University Women that controlled for college major, occupation, age, geographical region, hours worked and more, and found there was still a 7 percent wage gap between male and female college grads a year after graduation.
Using that high-end estimate, the Buzzfeed video exaggerated by no less than 214 percent. There's precedent for liberals receiving ratings of "Mostly False" or better for exaggerations that large and larger. On the other hand, PolitiFact Wisconsin gave a state Democrat a "False" rating for an exaggeration of 114 percent.

At least we know Buzzfeed's exaggeration is not the largest to receive a rating of "Mostly False" or higher.

If anyone can find a statement from a Republican or conservative where a figure exaggerated by more than 100 percent received a rating of "Mostly False" or higher from PolitiFact, we'd love to hear about it. We haven't turned up anything like that yet.

Tuesday, April 21, 2015

No 2015 Pulitzer for PolitiFact

Though PolitiFact uses its 2009 Pulitzer Prize to burnish its credibility, we've argued that PolitiFact won in 2009 thanks to technical innovation. With that innovation now old news, our view of PolitiFact's win anticipated that PolitiFact would be unlikely to keep winning Pulitzer Prizes.

With the 2015 winners announced, PolitiFact was shut out for the sixth straight year. Nor did PolitiFact make the list of finalists in any category.

No matter! PolitiFact will run "Winner of the Pulitzer Prize" forever at the top of its website.

We'll celebrate PolitiFact's failure with a rerun of the "Hitler finds out" video we put together for last year's Pulitzer Prize announcements.

We predict a similar outcome in 2016.

Saturday, April 18, 2015

Breitbart News and Eric Wemple on fact checkers

The April 17 installment of Breitbart News' interview with the Washington Post's Eric Wemple features a section on fact checking. John Nolte, the Breitbart News interviewer, let Wemple turn the tables on which was conducting the interview, but on the bright side Nolte brings makes points Wemple probably wouldn't touch.

No. 1 highlight :
EW: Here are my thoughts on that specifically, and I told Glenn the same thing: You are saying that the Washington Post disproportionately targeted Republicans, and that’s fine. My only point is that I don’t think anyone can expect politicians from any party to lie at an even rate.

BNN: That wasn’t my approach with the Washington Post, though. My argument wasn’t that Kessler was calling more Republicans liars, my issue was that, by 2-to-1, Republican statements were chosen for the fact check treatment.
Credit Nolte with correcting Wemple's straw man. But the problem with the Fact Checker and PolitiFact comes at the point story selection and truth ratings intersect. Neither one tells the whole story by itself.

Nolte follows up by emphasizing the subjectivity of the ratings. Wemple offers a counter of sorts: That criticism also comes from the left.

From this point in the interview, Wemple stops giving his take and the rest of the fact checking section has Wemple prompting Nolte for his take. There's no admission from Wemple that the ratings are subjective. Wemple's abandonment of the issue leaves an implicit "the fact checkers are criticized from both sides so they must be doing something right" argument.
BNN: When PolitiFact fact checks a quip from Ted Cruz about Iran celebrating “Hate America Day” — everyone knows what he means, but they still call him a liar. It just goes too far. A subjective decision is made to get literal so Cruz can be called a liar. Kessler did this one once where Romney said Obama had never gone to Israel — and that was a fact. But Romney got Pinocchios because Kessler made a subjective decision to make certain context relevant. That’s an opinion column, not a fact check.

EW: PolitiFact is big on context too. Like the time Rachel Maddow went crazy on them. In a State of the Union speech, Obama took credit for creating so many jobs and PolitiFact said that wasn’t entirely true because those jobs were not created as a result of his policies.

BNN: Exactly!

EW: You think she’s right about that.

BNN: I think she’s dead right about that. That’s a subjective decision to bring in subjective context. Put it on the opinion pages.

EW: And you think that cudgel is used more often against Republicans.

BNN: Much more often.
Do you agree that the ratings are subjective, Eric Wemple? If so, then what does that say about claims that harsher ratings of Republicans show that Republicans simply lie more?

The meat of this issue comes from the fact checkers' framing of political truth-telling: Republicans lie more. But the fact checkers' methods remove the foundation for the frame.