Wednesday, March 28, 2018

How PolitiFact Fights Its Reputation for Anti-conservative Bias

This week we ran across a new paper with an intriguing title: Everyone Hates the Referee:How Fact-Checkers Mitigate a Public Perception of Bias.

The paper, by Allison Colburn, pretty much concludes that fact checkers do not know how to fight their reputation for bias. Aside from that, it lets the fact checkers describe what they do to try to seem fair and impartial.

The paper mentions PolitiFact Bias, and we'll post more about that later. We place our focus for this post on Colburn's October 2017 interview of PolitiFact Editor Angie Drobnic Holan. Colburn asks Holan directly about PolitiFact Bias (Colburn's words in bold, following the format from her paper):
I'm just kind of curious, there's the site, What are what are your thoughts on that site?

That seems to be one guy who's been around for a long time, and his complaints just  seem to be that we don't have good, that we don't give enough good ratings, positive ratings to conservatives. And then he just kind of looks for whatever evidence he can find to support that point.

Do you guys ever read his stuff? Does it ever worry you?

He's been making the same complaint for so long that it has tended to become background noise, to be honest. I find him just very singularly focused in his complaints, and he very seldom brings up anything that I learn from.

But he's very, you know, I give him credit for sticking in there. I mean he used to give us, like when he first started he would give us grades for our reporting and our editing. So it would be like grades for this report: Reporter Angie Holan, editor Bill Adair. And like we could never do better than like a D-minus. So it's just like whatever. What I find is it's hard for me to take critics seriously when they never say we do anything right. Sometimes we can do things right, and you'll never see it on that site.
We could probably mine material from these answers for weeks. One visit to our About/FAQ page would prove enough to treat the bulk of Holan's misstatements. Aside from the FAQ, Jeff's tweet of Rodney Dangerfield's mug is the perfect response to Holan's suggestion that PolitiFact Bias is "one guy."

The Holan interview does deliver some on the promise of Colburn's paper. It shows how Holan tries to marginalize PolitiFact's critics.

I summed up one prong of PolitiFact's strategy in a post from Jan 30, 2018:
Ever notice how PolitiFact likes to paint its critics as folks who carp about whether the (subjective) Truth-O-Meter rating was correct?
In that post, I reported on how Holan bemoaned the fact that PolitiFact critics do not offer factual criticisms of its fact checks, preferring instead to quibble over its subjective ratings.
If they're not dealing with the evidence, my response is like, ‘Well you can say that we're biased all you want, but tell me where the fact-check is wrong. Tell me what evidence we got wrong. Tell me where our logic went wrong. Because I think that's a useful conversation to have about the actual report itself.
Holan says my (our) criticism amounts to a call for more positive ratings for conservatives. So we're just carping about the ratings, right? Holan's summation does a tremendous disservice to our painstaking and abundant research pointing out PolitiFact's errors (for example).

In the Colburn interview Holan also says she has trouble taking criticism seriously when the critic doesn't write articles complimenting what PolitiFact does correctly.

We suppose it must suck to find one's self the victim of selection bias. We suppose Holan must have a tough time taking seriously, given its policy against publishing fact checks showing a politician spoke the truth without misleading.

The hypocrisy from these people is just too much.

Exit question: Did Holan just not know what she was talking about, or was she simply lying?


For what it's worth, we sometimes praise PolitiFact for doing something right.

Correction March 31, 2018: We erred by neglecting to include the URL linking to Colburn's paper. We apologize to Allison Colburn and our readers for the oversight.

Tuesday, March 20, 2018

PolitiFact's apples and oranges make Zinke a liar?

Fact checkers often mete out harsh ratings to politicians who employ apples-to-oranges comparisons to make their point.

Mainstream media fact checkers often find themselves immune from the principles they use to find fault with others, however.

Consider PolitiFact's March 19, 2018 fact check of Interior Secretary Ryan Zinke.

Zinke said a Trump administration proposal "is the largest investment in our public lands infrastructure in our nation's history."

PolitiFact found that Civilian Conservation Corps program under President Franklin Roosevelt would far exceed the proposed Trump administration spending if adjusted for inflation:
The CCC’s director wrote in 1939 that it had cost $2 billion; that was two-thirds of the way through the program’s life. And according to a Park Service study, the annual annual cost per CCC enrollee was $1,004 per year. If you assume that the average tenure of the CCC’s 3.5 million workers was about a year, that would produce a cumulative cost around $3 billion.

Such calculations "sound right — millions of young men, camps to house them, food and uniforms, and they were paid," said Steven Stoll, an environmental historian at Fordham University.

Once you factor in inflation, $3 billion spent in the 1930s would be the equivalent of about $53 billion today — about three times bigger than even the fully funded Trump proposal.
When a spokesperson for the Trump administration pointed out that the CCC included lands controlled at the state and local level, PolitiFact brushed the objection aside (bold emphasis added):
Interior Department spokeswoman Heather Swift pointed out that the CCC "also incorporated state and local land."

It’s true that the CCC created more than 700 state parks and upgraded many others, in addition to its efforts on federally owned land. Ultimately, though, the point is moot: Zinke didn’t say the proposal is the largest investment in federal lands infrastructure. He said "public lands infrastructure," and state and local parks count as "public lands."
The key to the "False" rating PolitiFact gave Zinke comes entirely from its insistence that Zinke's statement covers all public lands.

But the context, which PolitiFact reported but ignored, clearly shows Zinke was talking specifically about spending on federal lands (bold emphasis added):
"The president is a builder and the son of a plumber, as I am," Zinke told the Senate Energy & Natural Resources Committee. "I look forward to working with the president on restoring America's greatness through a historic investment of our public lands infrastructure. This is the largest investment in our public lands infrastructure in our nation's history. Let me repeat that, this is the largest investment in our public lands infrastructure in the history of this country."

Zinke specified that he was referring to the president's budget proposal, which would create a fund to provide "up to $18 billion over 10 years for maintenance and improvements in our national parks, our national wildlife refuges, and Bureau of Indian Education funds."
We note a pair of irreconcilable problems with PolitiFact's reasoning.

If Zinke had claimed the CCC spending was greater than the spending proposed by the Trump administration, he would be guilty of using an apples-to-oranges comparison. Why? Because the scope of the two spending programs varies at a fundamental level.

Any would-be comparison between spending on federal lands only and spending on federal, state and local lands qualifies as an apples-to-oranges comparison.

If Zinke's statement was interpreted in keeping with his comments on the scope of the spending--kept to "federal lands"--then PolitiFact simply elected to avoid doing the appropriate fact check. That is, measuring the CCC spending on federal lands against the proposed Trump administration spending on federal lands. Apples-to-apples.

PolitiFact bases its fact check on the apples-to-oranges comparison: CCC spending on federal, state and local parks against proposed Trump administration spending on federal lands only.



Not hardly.

Afters: Multiple flaky layers

In its fact check PolitiFact stresses the enormity of CCC spending under Roosevelt by expressing it as a percentage of the federal budget. And compares that to the tiny percentage of the total budget taken up by Trump's proposed spending.


Has the federal budget increased over time (try as a percentage of GDP)? Medicare? Medicaid? Hello?

PolitiFact loves it some apples and oranges.

Not a Lot of Reader Confusion IX

We say that PolitiFact's graphs and charts, including its PunditFact collections of ratings for news networks, routinely mislead readers. But PolitiFact Editor Angie Drobnic Holan says there isn't much reader confusion.

In a March 19, 2018 column published by The Hill, liberal radio talk show host Bill Press declared that we can't believe what President Donald Trump says.

What evidence did Press have to support his claim?


Press (bold emphasis added):
Trump tells so many lies, so often, that not even Politifact, the Pulitzer Prize-winning, nonpartisan fact-checking website can keep score. But they do the most thorough job of anybody. Since he launched his 2016 campaign, Politifact has evaluated more than 500 assertions made by candidate and president Donald Trump, and they’ve rated an astounding 69 percent of them as “false,” “mostly false,” or, the worst category, “liar, liar, pants on fire.” Think about that. On any given day, you know that seven out of ten things Donald Trump says are not true!
We at PolitiFact Bias have endlessly pointed out that even if one assumes that PolitiFact did its fact checks without mistakes that selection bias and subjective application of its rating system make claims like the one in bold ridiculous.

Claims mirroring the one above happen routinely, yet PolitiFact denies the evidence ("not a lot of reader confusion") and continues publishing its misleading charts and graphs with no explanatory disclaimer.

We can think of two primary potential explanations.
  • PolitiFact truly doesn't see the abundance of evidence even though we jump up and down calling attention to it
  • PolitiFact is deliberately misleading people
 We invite readers to provide alternative possibilities in the comments section.

Sunday, March 18, 2018

PolitiFact: It's 'Half True' and 'Mostly True' that President Obama doubled the debt

Twitterer Ely Brit (@RealElyBritt) tweeted out this comparison of past PolitiFact ratings on March 17, 2018:

For the Trump fact check, PolitiFact came down hard on The Donald for placing blame too squarely on President Obama when Congress controls the federal government's purse strings.

On the other hand, it's hard to see how Sen. Paul eases up on placing the blame, unless he gets a bipartisan pass for blaming President Bush for the earlier debt increase.

Apart from that, neither Trump nor Paul received a "True" rating because Congress shares the blame for government spending.

Right, PolitiFact?

O-kay, then.

Correction 3/18/2018: Corrected transposed misspellings of Ely Brit's name. Our apologies to Brit. 
Correction 3/18/2018: Commenter YuriG pointed out that I (Bryan) used "deficit" in the headline, conflicting with the content of the post. Changed "deficit" to "debt" in the headline. Our thanks to YuriG for taking the time to point out the problem.

Monday, March 12, 2018

PolitiFact's Jolly problem

PolitiFact hired David Jolly at some point in February, with the hire date depending on whether his job title was "reader representative" or "Republican guest columnist."

PolitiFact said the hire was intended to build trust in PolitiFact across party lines. We've viewed the experiment with justified skepticism. And Jolly's work so far as the "Republican guest columnist" only solidifies our skepticism.

Jolly's first guest column was published March 2, 2018. We noted that Jolly used that column to address a subject tied to his own advocacy of "common sense gun control." We doubted many PolitiSkeptical conservatives would hear their voices in that column. We judged that Jolly was using his position at PolitiFact to essentially write an op-ed about one of his pet political issues.

Jolly's March 7, 2018 column followed that pattern.

Instead of critiquing PolitiFact, Jolly used his column to attack the target of a PolitiFact fact check. The target of that fact check? President Donald Trump.

Jolly (bold emphasis added):
As the nation continues to debate which gun policies might provide for the safety of our schools and communities, PolitiFact demonstrated in a single column the critical importance fact-checkers serve in both informing the American public as well as holding politicians and advocates on both sides of the debate accountable for their assertions.
Jolly played his gun control theme song in the background again and again we ask: How does Jolly's approach to his columns build trust among conservatives skeptical of PolitiFact? Does he think that just having a Republican say something like "PolitiFact is right" will budge the needle of partisan mistrust?

We'll go out on a limb and predict that approach has a snowball's chance in hell of working.

Conservative mistrust in PolitiFact stems primarily from two factors:
  • Conservatives see PolitiFact turning a blind eye to conservative arguments
  • PolitiFact commonly makes errors of fact and logic damaging to conservatives
Jolly's column serves as an example of both problems, despite his willingness to identify as a Republican.

Jolly reinforces PolitiFact's left-leaning bias

Jolly lauded PolitiFact for rating "False" the claim that an armed civilian might have stopped the "Pulse" nightclub shooting in Orlando (bold emphasis added):
"You take Pulse nightclub," Trump said. "If you had one person in that room that could carry a gun and knew how to use it, it wouldn’t have happened, or certainly to the extent that it did."
The problem for both the president and his theory is that an armed officer and 15-year veteran of the Orlando Police Department, Adam Gruler, was actually working security at Pulse that fateful night, and indeed engaged the shooter directly with gunfire. Forty-nine people still lost their lives.
PolitiFact rightly rated as False the president's statement that an armed security guard could have saved those 49 victims.
The conservative would have read the PolitiFact fact check looking for evidence of fair treatment of the mainstream conservative point of view. That view is absent, and when Jolly fails to notice its absence and celebrates PolitiFact's fact check in spite of that, the conservative cannot take Jolly's columns any more seriously than the fact check in the first place.

The problem with the fact check

What's the problem with PolitiFact's fact check?

The fact check pretends that the armed guard working security in the Pulse parking lot counts as "one person in that room that could carry a gun and knew how to use it."

Is the Pulse parking lot inside the Pulse nightclub?

Shouldn't that make a difference for non-partisan fact checkers and GOP columnists alike?

Note PolitiFact's description of armed guard Anthony Gruler's involvement in the Pulse incident (bold emphasis added):
The Justice Department in 2017 released a nearly 200-page report detailing the Orlando police response to the shooting. Here’s the report’s account of Gruler’s initial confrontation with Mateen:
"Outside, in the Pulse parking lot, (Gruler), who was working extra duty at the club — to provide outside security and to provide assistance to security personnel inside the club if needed — heard the shots that were being fired; at 2:02:17 a.m., he broadcast over the radio, 'Shots fired, shots fired, shots fired,' and requested additional officers to respond.

"The detective told the assessment team that he immediately recognized that his Sig Sauer P226 9mm handgun was no match for the .223 caliber rifle being fired inside the club and moved to a position that afforded him more cover in the parking lot. Two patrons attempted to flee through an emergency exit on the south side of the club. When the detective saw the suspect shoot them, he fired at the suspect."
According to an Orlando Police Department report, additional officers arrived on the scene about a minute after Gruler’s call for backup was broadcast. A second backup officer arrived about a minute after that.
 PolitiFact's conclusion, sadly, serves as an adequate summary of its argument:
Talking about the Pulse nightclub shooting, Trump said, "If you had one person in that room that could carry a gun and knew how to use it, it wouldn’t have happened, or certainly to the extent that it did."

An armed, off-duty police officer in uniform was at the club during the shooting, and exchanged gunfire with the shooter, who managed to kill 49 people.

We rate this False.
Notice that PolitiFact says Gruler was "at the club," not "outside the club."

The "False" rating gets its premise from the fiction that Gruler was in the same room with Mateen while the latter was murdering club patrons. Jolly, PolitiFact's voice of the GOP, signs on with that falsehood.

In this case the deception comes from PolitiFact and Jolly. Not from Trump.

PolitiFact demonstrated nothing false in Trump's statement, yet pinned a "False" rating on his statement.

Jolly cheered PolitiFact's work, missing the key discrepancy in its fact check.

That's all kinds of wrong.

Saturday, March 3, 2018

The first critique from PolitiFact's Republican "guest columnist" David Jolly (Updated)

In early February, PolitiFact tabbed two former Washington D.C. politicians as Democrat and Republican "guest columnists." PolitiFact said the hires were intended to raise its credibility across partisan lines.

We looked the the first column from the Democrat back on Feb. 12 and judged it pointless crap.

Republican David Jolly published his first critique on March 2, 2018. And though we were aware of charges Jolly counts as an "MSNBC Republican," we were still a bit surprised at first blush when Jolly used his column to apparently suggest PolitiFact had rendered a too-harsh judgment on a Democrat's claim about gun violence.

It's worth noting that Jolly has been busy lately promoting "common sense" gun control legislation, so maybe it was just too tempting to resist tying that political cause to his role at PolitiFact.

PolitiFact gave a gun violence statistic from Rep. Ted Deutch (D-Fla) a rating of "Mostly False." Jolly appeared to think Deutch deserved a better fate (bold emphasis added)
Rep. Deutch is said to have relied on statistics provided by The Century Foundation, a left-leaning think tank, which plainly assessed there had been "an increase of over 200%" in mass shootings since the expiration of the assault weapons ban. PolitiFact, however, rated Deutch’s statement Mostly False, largely on the basis that the commentary upon which the congressman relied was substantively challenged by other experts in the field. Presumably, had Deutch said in the town hall, "According to The Century Foundation, mass shootings went up 200% in the decade after the assault weapons ban expired," PolitiFact would have found the Congressman’s statement True on its face, while ruling the findings of The Century Foundation Mostly False.

Moreover, had PolitiFact evaluated Deutch’s statement simply on the numbers, there is ample evidence in the PolitiFact article to support a ruling of Mostly True.
Jolly's critique hits at PolitiFact's standard operating procedure. Yes, of course PolitiFact's ratings oversimplify complexities. Will writing a column pointing that out for the umpteenth time change anything?

Is that the extent of Jolly's point?  That's what his conclusion suggests:
In this case, a congressman’s statement seems to have been ruled Mostly False on two primary factors — his citing a credible think tank’s commentary on gun violence statistics, and a drawn inference by fact-checkers that may or may not have been intended in the congressman’s statement. Neither makes PolitiFact’s ruling right or wrong, but it reflects the enormous challenge faced by politicians, fact-checkers and ultimately voters in today’s political environment.
Newsflash for David Jolly: The fact checkers do not see the subjectivity of their ratings as a problem. The problem, in their eyes, is that readers do not place enough trust in fact checkers. And they recruited their "guest columnists" to help build bipartisan trust in their subjective judgments.

Jolly's critique is about as deep as Barbie's "Math is hard" critique of high school and as such works better as a soft-sell version of his gun control op-ed that we linked above. But we like it for the fact that its premise arguably strikes against the fundamental subjectivity of the PolitiFact approach.

Expect a short run for this PolitiFact experiment if Jolly's future work similarly attacks the PolitiFact premise.


We learned about Jolly's column through PolitiFact's Facebook page, where it featured a link to his column.

PolitiFact's main page at does a pretty awesome job of burying these columns. The reader would have to scroll near the bottom of the page and see the "Inside the Meters" section in the right sidebar.

It's literally the last element on the right sidebar, and there's no way to directly reach the "Inside the Meters" posts using the navigation buttons at the top of the PolitiFact website.

Afters 2

Unsurprisingly, PolitiFact uses an "even the Republican thinks we were too tough on the Democrat" frame in its Facebook announcement of his column:

See why he said "there is ample evidence ... to support a ruling of "Mostly True."
PolitiFact left out enough context of Jolly's column to earn its own "Half True" (bold emphasis added):
Moreover, had PolitiFact evaluated Deutch’s statement simply on the numbers, there is ample evidence in the PolitiFact article to support a ruling of Mostly True.
Sometimes PolitiFact judges simply on the numbers, sometimes it doesn't. No objective, nonpartisan principle appears to guide the decision.

We suggest PolitiFact's manipulation of Jolly's statement misleads its audience about Jolly's point. Jolly's overall point, albeit delivered weakly, was the difficulty of making the ratings (thanks to subjectivity). PolitiFact spins that into the Republican saying PolitiFact was too hard on the Democrat.

PolitiFact's spin helps stuff Jolly's column into a frame that assists PolitiFact's purpose of fluffing up its own credibility: They're so nonpartisan! They gave a Democrat a "Mostly False" when a Republican said he should have had a "Mostly True!!"

Afters 3

Exit question for David Jolly:

PolitiFact sees your role as a guest columnist as a way for it to help build its own credibility. It has said as much in public statements. Describe the difference between the way you see your role as a guest columnist and the way PolitiFact sees it.

Update: Jeff adds

Cherry-picking the source of a claim

One might assume PolitiFact's newest critic would know something about PolitiFact, until you read PolitiFact's newest critic's newest PolitiFact critique.

Jolly's gripe, and his solution to it, betray a gross ignorance of the way his employer works.
"PolitiFact would have found the Congressman’s statement True on its face, while ruling the findings of The Century Foundation Mostly False."
Jolly finds fault with PolitiFact giving the False rating to Rep. Deutch, when Deutch was simply repeating a claim made by a (Jolly's words) "credible think tank." He argues the rating should have gone to the think tank.

That's nice, but it's also a problem spelled out on these pages for nearly a decade. One of the ways  PolitiFact shows its liberal lean comes from its choice of the source for a claim. We've called that "attribution bias."

In 2011, when The New York Times, ABC, and NPR all repeated a bogus number published in an official Inspector General report regarding $16 muffins, PolitiFact added a "False" rating to conservative Bill O'Reilly's "report card." 

In 2012, when Time, CNN, and the New York Post published a faulty stat from a dubious research firm, it was conservative media group American Crossroads that earned the demerit on its PolitiFact scorecard.

More recently in 2017 PolitiFact ignored a widely publicized liberal talking point espoused by numerous talking heads in the national media for months, and then quickly jumped in to deem it "False" within a day of Donald Trump repeating it.

Though in Deutch's case PolitiFact gave the rating to a Democrat instead of the think tank responsible for the stat, it still exemplifies PolitiFact's subjective, cherry-picking process for assigning its ratings. That Jolly seems to think his observation is novel only highlights his own naivety about PolitiFact.

The critique overall?

 Jolly's overall critique comes off as compelling as warm beer. It's almost as if he quickly cobbled together the column immediately after being criticized for not producing a critique during the first month of PolitiFact's new "readers advocate" feature.

We're not sure what value Jolly adds to the operation here, unless being duped into being a real life version of PolitiFact's lame "we get it from both sides!" defense is his goal. He didn't provide any insight or commentary that hasn't been offered in a better form on this blog for the last several years. We think having an in-house, right-leaning critic at PolitiFact is a good idea and think it could improve its work immensely. So far it appears Jolly is not the vehicle for the improvement.

In any event, congratulations to Jolly for finding a way to get his first critique of PolitiFact from the right to dovetail so nicely with his conservative efforts to support ... gun control.

Thursday, March 1, 2018

Top five reasons why David Jolly did not critique PolitiFact in February (Updated)

Update March 2, 2018

We've been curious, especially in light of the gun debate's current popularity, what has happened to PolitiFact's experiment involving "reader advocates." Particularly we noted the absence of Republican David Jolly. We published a post yesterday 
[March 1--bww] addressing this curiosity and included a healthy dose of snark. 

Mr. Jolly has since responded via Twitter that his silence was the result of personal matters, including a death in the family and threats he's been receiving.

Our sarcasm was intended to mock the same problems we typically do with PolitiFact. While there's nothing offensive in anything we wrote, we've decided to remove the post in deference to Mr. Jolly's personal circumstances.

We'll continue to use snark to highlight PolitiFact's dubious operation, but in this instance we don't see the value in unnecessary jokes in light of Mr. Jolly's response.
(above by Jeff D.)
Update continued:

Instead of deleting the post or preserving it offsite via the Internet Archive, we're putting it below a page break and changing the text to white (on a white background) to make it mostly invisible to all except the insatiably curious.

Some blue highlights from embedded URLs will remain visible.