Wednesday, June 1, 2016

Something rotten in PolitiFact Missouri

It's not the bad reporting, it's the cover up.

Okay, it's both.

On May 18, 2016, PolitiFact Missouri published a fact check of the gender pay gap issue. Democratic gubernatorial candidate Chris Koster said closing Missouri's gender pay gap would gain $9 billion for Missouri women. PolitiFact Missouri botched the fact check, calculating that Missouri women would only gain $7.5 billion, not $9 billion. Koster received a "Mostly True" rating.

It read like this at the time:
The bias related portion of the gap could be as much as $7.5 billion. That’s a lot of money, but it isn’t the $9 billion Koster claimed. When we’ve rated such claims before, statements that speak broadly about a wage gap, regardless of the underlying factors, get some benefit of the doubt.

Koster’s claim greatly oversimplifies a very complex situation, but the size of the gap is real. We rate this claim Mostly True.

Now it reads like this (bold emphasis added):
The bias related portion of the gap could be as much as $1.7 billion. That’s a lot of money, but it isn’t the $9 billion Koster claimed. When we’ve rated such claims before, statements that speak broadly about a wage gap, regardless of the underlying factors, get some benefit of the doubt.

Koster’s claim greatly oversimplifies a very complex situation, but the size of the gap is real. We rate this claim Mostly True.

A key figure in the story changed from $7.5 billion to $1.7 billion. Koster's exaggeration, by percentage, went from 20 percent to 429 percent.  The new version of the story carries no correction notice, and the rating remains "Mostly True."

How did we get here? What went wrong at PolitiFact Missouri?

Spoiler: The present version of PolitiFact Missouri's fact check remains far from accurate.


As soon as I read the PolitiFact Missouri fact check, I knew the $7.5 billion figure was suspect.

In 2014 I wrote a critical review of the way fact checkers handled gender wage gap claims. The fact checkers, especially PolitiFact, tend to ignore the lowest and best-researched estimates of discrimination's share of that gap, particularly ignoring the portion of a study sponsored by the Bush administration that said the gap from discrimination could shrink nearly to zero with all non-discrimination factors considered. It's good to hide that kind of information if you're a fact-checker prone to advocacy against society's obvious gender pay discrimination, I suppose.

I wrote to the writer and editor of the PolitiFact Missouri fact check. I also posted about it. But my criticism did not properly pinpoint PolitiFact Missouri's error.

I had made a mistake. And here is that admission again, as an example to fact checkers everywhere:

I made a mistake. And I wrote it down and sent it in a message to PolitiFact Missouri. More than once, in fact.

I tripped over this line in PolitiFact Missouri's fact check:
A study from the Institute for the Study of Labor, an economic research institute based in Bonn, Germany, shows the unexplained wage gap in the United States falls somewhere between 8 percent and 18 percent of the total earnings difference, if the figures are adjusted for additional factors.
As I noted above, I knew going in that the unexplained portion of the gender pay gap, taking the great number of researched explanations into account, is somewhere around 0-5 percent. If PolitiFact Missouri was correctly representing the research, then a raw gender pay gap of 23 percent was between 8 and 18 percent unexplained. Using the high-end estimate would yield a difference in Missouri of $1.7 billion--the figure now appearing in the fact check.

I looked at the research paper PolitiFact had referenced, and it seemed to me that it considered enough pay gap explanations to ensure a finding in keeping with other research I had read. I concluded that PolitiFact Missouri had accurately represented the research. That's where I went wrong. And I went wrong again, later, but we'll get to that.

The research paper, to be fair to PolitiFact Missouri, does not spell out its estimate of the unexplained pay gap with perfect clarity:
A decrease in the unexplained gap over the 1980s contributed to the robust convergence in the gender wage gap over that decade, with the unexplained gap falling sharply from 21-29% in 1980 to 8-18% by 1989.  However, the unexplained gap did not fall further subsequently, remaining in this range over the succeeding 20 years.
It's possible to take the paragraph to mean that the unexplained pay gap falls somewhere between 8 and 18 percent in 2010. I think that is how PolitiFact Missouri took it. And that is how I took it (mistake No. 2) until I got around to looking at Table 4:

Table 4 from the research paper makes clear that the 18 percent pay gap was derived from the "Human Capital Specification" section of the table. Note that with more variables taken into account for the "Full Specification" section the "Total Unexplained Gap" shrinks from 85.2 percent to 38 percent.  That corresponds closely to 18 percentage points or 8 percentage points of a 21 percent raw pay gap.

The upshot is this: The paper isn't really describing a range of uncertainty in the pay gap. It is describing the difference between one type of estimate (Human Capital Specification) and another type of estimate (Full Specification). But other estimates account for factors not included in this study's "Full Specification."

PolitiFact could have used the 38 percent figure from Table 4 to correct Koster's $9 billion figure. That would have shown some understanding of the research paper, and would have yielded a figure of $3.61 billion.

Unfortunately, that fix falls far short of rehabilitating a severely deficient fact check.

1) Using an 8 percent unexplained gap amounts to cherry-picking when PolitiFact Missouri's source list includes sources offering lower figures for the unexplained gap
2) PolitiFact Missouri's explanation of the 8 and 18 percent figures is wrong
3) PolitiFact uncritically accepts Koster's assumption that women would gain the pay gap difference. The gap could also close by men losing pay.
4) Assuming the unexplained gap entirely amounts to gender discrimination when doing the math does Koster a big favor
5) PolitiFact's policies call for a correction notice, at minimum (a correction appears above the story in the Missourian). We would recommend redoing the fact check from scratch.
6) Omitting any mention of the lower unexplained gap estimates shortchanges readers
7) PolitiFact Missouri misquotes Koster's tweet (incorrectly repeating a word pair)
8) It's "caption," not "captain" (the version of the fact check in the Missourian does not make this error)

Do we need any more proof that PolitiFact's ratings are an exercise in subjectivity? The only objective finding in the article was Koster's "$9 billion" exaggeration. It was "Mostly True" when PolitiFact Missouri thought the truth was $7.5 billion, and still "Mostly True" when PolitiFact thought the truth was $1.7 billion.

We can't imagine that using the still-flawed $3.61 billion figure would make the least bit of difference to PolitiFact. PolitiFact Missouri seems determined to give Koster a "Mostly True" rating regardless of what the supposed facts show.

No comments:

Post a Comment