Friday, May 19, 2017

What "Checking How Fact Checkers Check" says about PolitiFact

A study by doctoral student Chloe Lim (Political Science) of Stanford University gained some attention this week, inspiring some unflattering headlines like this one from Vocativ: "Great, Even Fact Checkers Can’t Agree On What Is True."

Katie Eddy and Natasha Elsner explain inter-rater reliability

Lim's research approach somewhat resembled research by Michelle A. Amazeen of Rider University. Amazeen and Lim both used tests of coding consistency to assess the accuracy of fact checkers, but the two reached roughly opposite conclusions. Amazeen concluded that consistent results helped strengthen the inference that fact-checkers fact-check accurately. Lim concluded that inconsistent fact-checker ratings may undermine the public impact of fact-checking.

Key differences in the research procedure help explain why Amazeen and Lim reached differing conclusions.

Data Classification

Lim used two different methods for classifying data from PolitiFact and the Washington Post Fact Checker. She converted PolitiFact ratings to a five-point scale corresponding to the Washington Post Fact Checker's "Pinocchio" ratings, and she divided ratings into "True" and "False" groups using the line between "Mostly False" and "Half True" as the barrier between true and false statements.

Amazeen opted for a different approach. Amazeen did not try to reconcile the two different rating systems at PolitiFact and the Fact Checker, electing to use a binary system that counted every statement rated other than "True" or "Geppetto check mark" as false.

Amazeen's method essentially guaranteed high inter-rater reliability, because "True" judgments from the fact checkers are rare.  Imagine comparing movie reviewers who use a five-point scale but with their data divided up into great movies or not-great movies. A one-star rating of "Ishtar" by one reviewer would show agreement with a four-star rating of the same movie by another reviewer. Disagreements only occur when one reviewer gives five stars while the other one gives a lower rating.

Professor Joseph Uscinski's reply to Amazeen's research, published in Critical Review, put it succinctly:
Amazeen’s analysis sets the bar for agreement so low that it cannot be taken seriously.
Amazeen found high agreement among fact checkers because her method guaranteed that outcome.

Lim's methods provide for more varied and robust data sets, though Lim experienced the same problem Amazeen found in that two different fact-checking organizations only rarely check the same claims. Both researchers used relatively small data sets.

The meaning of Lim's study

In our view, Lim's study rushes to its conclusion that fact-checkers disagree without giving proper attention to the most obvious explanation for the disagreement she measures.

The rating systems the fact checkers use lend themselves to subjective evaluations. We should expect that condition to lead to inconsistent ratings. When I reviewed Amazeen's method at Zebra Fact Check, I criticized it for applying inter-coder reliability standards to a process much less rigorous than social science coding.

Klaus Krippendorff, creator of the K-alpha measure Amazeen used in her research, explained the importance of giving coders good instructions to follow:
The key to reliable content analyses is reproducible coding instructions. All phenomena afford multiple interpretations. Texts typically support alternative interpretations or readings. Content analysts, however, tend to be interested in only a few, not all. When several coders are employed in generating comparable data, especially large volumes and/or over some time, they need to focus their attention on what is to be studied. Coding instructions are intended to do just this. They must delineate the phenomena of interest and define the recording units to be described in analyzable terms, a common data language, the categories relevant to the research project, and their organization into a system of separate variables.
The rating systems of PolitiFact and the Washington Post Fact Checker are gimmicks, not coding instructions. The definitions mean next to nothing, and PolitiFact's creator, Bill Adair, has called PolitiFact's determination of Truth-O-Meter ratings "entirely subjective."

Lim's conclusion is right. The fact checkers are inconsistent. But Lim's use of coder reliability ratings is, in our view, a little like using a plumb line to measure whether a building has collapsed due to earthquake. The tool is too sophisticated for the job. The "Truth-O-Meter" and "Pinocchio" rating systems as described and used by the fact checkers do not qualify as adequate sets of coding instructions.

We've belabored the point about PolitiFact's rating system for years. It's a gimmick that tends to mislead people. And the fact-checking organizations that do not use a rating system avoid it for precisely that reason.

Lucas Graves' history of the modern fact-checking movement, "Deciding What's True: The Rise of Political Fact-Checking in American Journalism," (Page 41) offers an example of the dispute:
The tradeoffs of rating systems became a central theme of the 2014 Global Summit of fact-checkers. Reprising a debate from an earlier journalism conference, Bill Adair staged a "steel-cage death match" with the director of Full Fact, a London-based fact-checking outlet that abandoned its own five-point rating scheme (indicated by a magnifying lens) as lacking precision and rigor. Will Moy explained that Full Fact decided to forgo "higher attention" in favor of "long-term reputation," adding that "a dodgy rating system--and I'm afraid they are inherently dodgy--doesn't help us with that."
Coding instructions should provide coders with clear guidelines preventing most or all debate in deciding between two rating categories.

Lim's study in its present form does its best work in creating questions about fact checkers' use of rating systems.

Sunday, May 14, 2017

PolitiFact and robots.txt (updated)

We were surprised earlier this week when our attempt to archive a PolitiFact fact check at the Internet Archive failed.

Saving a page to the Internet Archive has served as one of the standard methods for keeping record of changes at a website. PolitiFact Bias has often used the Internet Archive to document PolitiFact's mischief.

Webmasters have the option of instructing search engines to skip indexing content at a website through use of a "robots.txt" instruction. Historically, the Internet Archive has respected the presence of a robots.txt prohibition.

PolitiFact apparently decided to start using a limiting robots.txt recently. As a result, it's likely that none of the archived links will work for a time, either at PolitiFact Bias or elsewhere.

The good news in all of this? The Internet Archive is likely to start ignoring the robots.txt instruction in the very near future. Once that happens, PolitiFact's sketchy Web history will return from the shadows back into the light.

PolitiFact may have had a legitimate reason for the change, but our extension of the benefit of the doubt comes with a big caveat: The PolitiFact webmaster could have created an exception for the Internet Archive in its robots.txt instruction. That oversight creates an embarrassment for PolitiFact, at minimum.

Update May 18, 2017:

This week the Internet Archive Wayback Machine once again functioned properly in saving Web pages at Links at to archived pages likewise function properly.

We do not know at this point whether PolitiFact created an exception for the Internet Archive (and others), or whether the Internet Archive has already started ignoring robots.txt. PolitiFact has made no announcement regarding any change, so far as we can determine.

Friday, April 7, 2017

PolitiFact fixes fact check on Syrian chemical weapons

When news reports recently appeared suggesting the Syrian government used chemical weapons, it presented a problem for PolitiFact. As noted by the Daily Caller, among others, PolitiFact said in 2014 it was "Mostly True" that 100 percent of Syrian chemical weapons were removed from that country.

If the Syrian government used chemical weapons, where did it get them? Was it a fresh batch produced after the Obama administration forged an agreement with Russia (seriously) to effect removal of the weapons?

Nobody really knows, just like nobody truly knew the weapons were gone when PolitiFact ruled it "Mostly True" that the weapons were "100 percent gone." (screen capture via the Internet Archive)

With public attention brought to its questionable ruling with the April 5, 2017 Daily Caller story, PolitiFact archived its original fact check and redirected the old URL to a new (also April 5, 2017) PolitiFact article: "Revisiting the Obama track record on Syria’s chemical weapons."

At least PolitiFact didn't make its old ruling simply vanish, but has PolitiFact acted in keeping with its commitment to the International Fact-Checking Network's statement of principles?
We publish our corrections policy and follow it scrupulously. We correct clearly and transparently in line with our corrections policy, seeking so far as possible to ensure that readers see the corrected version.
And what is PolitiFact's clear and transparent corrections policy? According to "The Principles of PolitiFact, PunditFact and the Truth-O-Meter" (bold emphasis added):

When we find we've made a mistake, we correct the mistake.

  • In the case of a factual error, an editor's note will be added and labeled "CORRECTION" explaining how the article has been changed.
  • In the case of clarifications or updates, an editor's note will be added and labeled "UPDATE" explaining how the article has been changed.
  • If the mistake is significant, we will reconvene the three-editor panel. If there is a new ruling, we will rewrite the item and put the correction at the top indicating how it's been changed.
Is the new article an update? In at least some sense it is. PolitiFact removed and archived the fact check thanks to questions about its accuracy. And the last sentence in the replacement article calls the article an "update":
In the days and weeks to come, we will learn more about the recent attacks, but in the interest of providing clear information, we have replaced the original fact-check with this update.
If the new article counts as an update, we think it ought to wear the "update" tag that would make it appear on PolitiFact's "Corrections and Updates" page, where it has yet to appear (archived version).

And we found no evidence that PolitiFact posted this article to its Facebook page. How are readers misled about the original fact check supposed to encounter the update, other than by searching for it?

Worse still, the new article does not even appear on the list for the "The Latest From PolitiFact." What's the excuse for that oversight?

We believe that if PolitiFact followed its corrections policy scrupulously, we would see better evidence that PolitiFact publicized its admission it had taken down its "Mostly True" rating of the claim of an agreement removing 100 percent of Syria's chemical weapons.

Can evidence like this stop PolitiFact from receiving "verified" status in keeping the IFCN fact checkers' code?

We doubt it.

It's worth mentioning that PolitiFact's updated article does not mention the old article until the third paragraph. The fact that PolitiFact pulled and archived that article waits for the fifth paragraph, nearly halfway through the update.

Since PolitiFact's archived version of the pulled article omits the editor's name, we make things easy for our readers by going to the Internet Archive for the name: Aaron Sharockman.

PolitiFact's "star chamber" of editors approving the "Mostly True" rating likely included Angie Drobnic Holan and Amy Hollyfield.

Sunday, April 2, 2017

Angie Drobnic Holan: "Find news organizations that have a demonstrated commitment to the ethical principles of truthfulness, fairness, independence and transparency."

PolitiFact, thy name is Hypocrisy.

The editors of PolitiFact Bias often find themselves overawed by the sanctimonious pronouncements we see coming from PolitiFact (and other fact checkers).

Everybody screws up. We screw up. The New York Times screws up. PolitiFact often screws up. And a big part of journalistic integrity comes from what you do to fix things when you screw up. But for some reason that concept just doesn't seem to fully register at PolitiFact.

Take the International Fact-Checking Day epistle from PolitiFact's chief editor Angie Drobnic Holan:
Find news organizations that have a demonstrated commitment to the ethical principles of truthfulness, fairness, independence and transparency. (We adhere to those principles at PolitiFact and at the Tampa Bay Times, so if you’re reading this, you’ve made a good start.)
The first sentence qualifies as great advice. The parenthetical sentence that follows qualifies as a howler. PolitiFact adheres to principles of truthfulness, fairness and transparency?

We're coming fresh from a week where PolitiFact published a fact check that took conservative radio talk show host Hugh Hewitt out of context, said it couldn't find something that was easy to find, and (apparently) misrepresented the findings of the Congressional Budget Office regarding the subject.

And more to the issue of integrity, PolitiFact ignores the evidence of its failures and allows its distorted and false fact check to stand.

The fact check claims the CBO finds insurance markets under the Affordable Care Act stable, concluding that the CBO says there is no death spiral. In fact, the CBO said the ACA was "probably" stable "in most areas." Is it rightly a fact checker's job to spin the judgments of its expert sources?

PolitiFact improperly cast doubt on Hewitt's recollections of a New York Times article where the head of Aetna said the ACA was in a death spiral and people would be left without insurance:
Hewitt referred to a New York Times article that quotes the president of Aetna saying that in many places people will lose health care insurance.

We couldn’t find that article ...
We found the article (quickly and easily). And we told PolitiFact the article exists. But PolitiFact's fact check still makes it look like Hewitt was wrong about the article appearing in the Times.

PolitiFact harped on the issue:
In another tweet, Hewitt referenced a Washington Post story that included remarks Aetna’s chief executive, Mark Bertolini. On the NBC Meet the Press, Hewitt referred to a New York Times article.
We think fact checkers crowing about their integrity and transparency ought to fix these sorts of problems without badgering from right-wing bloggers. And if they still won't fix them after badgering from right-wing bloggers, then maybe they do not qualify as "organizations that have a demonstrated commitment to the ethical principles of truthfulness, fairness, independence and transparency."

Maybe they're more like liberal bloggers with corporate backing.

Correction April 3, 2017: Added a needed apostrophe to "fact checkers job."

Thursday, March 30, 2017

Hewitt v. PolitiFact: Two facts clearly favor Hewitt

Over the past week, conservative radio host Hugh Hewitt claimed on Sunday the health insurance industry has entered a "death spiral," PolitiFact rated Hewitt's claim "False" and Hewitt had PolitiFact Executive Director Aaron Sharockman on his radio show for an hour long interview (transcript here).

Aside from the central dispute over the "death spiral," where PolitiFact's work arguably commits a bifurcation fallacy, we have identified two areas where Hewitt has the right of the argument. PolitiFact has published (and superficially defended) false statements in both areas.

The New York Times article

PunditFact (PolitiFact):
Hewitt referred to a New York Times article that quotes the president of Aetna saying that in many places people will lose health care insurance.

We couldn’t find that article, but a simple remark on how premiums are rising and insurers are leaving the marketplace is not enough evidence to meet the actuarial definition of a death spiral.
We found the article in just a few minutes (it was dead easy; see hit No. 5). Two quotations from it will show it matches the content, other than the term "president," that Hewitt described on his Sunday television appearance.

Aetna CEO Mark Bertolini has pronounced the ACA's health insurance markets in a "death spiral."

This story has been corrected to show that consumers have reduced options, not that some consumers have no health care options.
So we have the "death spiral" comment from the head of AETNA that Hewitt described, as well as the dire statement that some people have no options, though that part was reported in error in the AP story that appeared in the Times.

During his radio interview, Sharockman tried to pin on Hewitt PolitiFact's failure to find the described Times story and flatly said the article was not in the Times:
AS: You said the president of Aetna. It’s the chairman and CEO, and it was not in the New York Times, as you also know. It was originally probably in the Wall Street Journal.
The article was in The New York Times, and we have informed PolitiFact writer Allison Graves (via Twitter) and Sharockman (via email).

We expect ethical journalists to make appropriate corrections.

Where does the CBO stand on the "death spiral"?

The mainstream media widely interpreted the CBO report addressing President Trump's health care proposal as a judgment the ACA has not entered a "death spiral."

PolitiFact did likewise in its fact check of Hewitt:
CBO, independent analysis: No death spiral
Others have also concluded that the Affordable Care Act is not in a death spiral. The nonpartisan Congressional Budget Office, as part of its recent analysis of the GOP legislation, described the Affordable Care Act as stable.
Though PolitiFact did not link to the CBO report in its fact check (contrary to PolitiFact's statement of standards), we believe the claim traces to this CBO report, which contains this assessment (bold emphasis added):

Stability of the Health Insurance Market

Decisions about offering and purchasing health insurance depend on the stability of the health insurance market—that is, on having insurers participating in most areas of the country and on the likelihood of premiums’ not rising in an unsustainable spiral. The market for insurance purchased individually (that is, nongroup coverage) would be unstable, for example, if the people who wanted to buy coverage at any offered price would have average health care expenditures so high that offering the insurance would be unprofitable. In CBO and JCT’s assessment, however, the nongroup market would probably be stable in most areas under either current law or the legislation.
Note the CBO report does not call the insurance market "stable" under the ACA. Instead it projects that insurance markets will probably remain stable in most areas. Assuming PolitiFact has no better support from the CBO than the portion we have quoted, we find PolitiFact's version a perversion of the original. The CBO statement leaves open the possibility of a death spiral.

Sharockman stood behind the fact check's apparent spin during his appearance on the Hugh Hewitt Show:
AS: Hugh, you’re misleading the listeners.

HH: …is that we have gone from 7…

AS: You’re misleading the listeners. The same CBO report that you’re quoting said that the markets are stable whether it’s the AHCA…
Again, unless Sharockman has some version of a CBO report different from what we have found, we judge that Sharockman and PolitiFact are misleading people about the content of the report.

We used email to point out the discrepancy to Sharockman and asked him to provide support for his and PolitiFact's interpretation of the CBO report.

We will update this article if we receive a response from Sharockman that includes such evidence.

Tuesday, March 28, 2017

Hugh Hewitt v. PolitiFact (Power Line Blog)

Via Power Line blog, the liberal bloggers at PolitiFact tangle with conservative radio show host Hugh Hewitt:

The issue: During a television appearance, Hewitt said the ACA is in a death spiral. PolitiFact did its usual limited survey of experts and ruled Hewitt's statement "False."

Part 1: PolitiFact Strikes Hugh Hewitt

A favorite part:
Allison Graves evaluates Hugh’s assertion regarding the Obamacare death spiral for PolitiFact. She defines the question in a manner that tends to belie Hugh’s assertion, cites some relevant authorities and rates Hugh’s assertion False.

I think this is a question on which reasonable minds can disagree, depending on how the question is framed. I would rate Graves’s judgment False in implying the contrary.
Part 2: Pol[i]tiFact strikes Hugh Hewitt (2)

A favorite part (quotation of Hewitt):
PolitiFact is a liberal-agenda-driven group of classically lefty “journalists” masquerading as a non-partisan evaluators of arguments. In this case their defense of their “journalism” rests on a partial and biased recounting of a 10:20 a.m. Meet the Press roundtable discussion, one which omits my stated acknowledgment of a differing argument therein, and their discounting of the expert testimony of a major insurance company president, along with a Sunday afternoon three-hour “deadline” window for response following a perfunctory email to a booker of a show that runs Monday through Friday, when the host is himself online and answering a journalists’ questions and comments.
To this we would add that PolitiFact's story misrepresents a Congressional Budget Office report.

PolitiFact cited the CBO in support of its finding that the ACA is not in a death spiral:
The nonpartisan Congressional Budget Office, as part of its recent analysis of the GOP legislation, described the Affordable Care Act as stable.
PolitiFact failed to link to the CBO in this fact check, but the source wasn't hard to find. The tough part was figuring out why PolitiFact added its own spin to the CBO's view (bold emphasis added):

Stability of the Health Insurance Market>

Decisions about offering and purchasing health insurance depend on the stability of the health insurance market—that is, on having insurers participating in most areas of the country and on the likelihood of premiums’ not rising in an unsustainable spiral. The market for insurance purchased individually (that is, nongroup coverage) would be unstable, for example, if the people who wanted to buy coverage at any offered price would have average health care expenditures so high that offering the insurance would be unprofitable. In CBO and JCT’s assessment, however, the nongroup market would probably be stable in most areas under either current law or the legislation.

Under current law, most subsidized enrollees purchasing health insurance coverage in the nongroup market are largely insulated from increases in premiums because their out-of-pocket payments for premiums are based on a percentage of their income; the government pays the difference. The subsidies to purchase coverage combined with the penalties paid by uninsured people stemming from the individual mandate are anticipated to cause sufficient demand for insurance by people with low health care expenditures for the market to be stable.

Under the legislation, in the agencies’ view, key factors bringing about market stability include subsidies to purchase insurance, which would maintain sufficient demand for insurance by people with low health care expenditures, and grants to states from the Patient and State Stability Fund, which would reduce the costs to insurers of people with high health care expenditures. Even though the new tax credits would be structured differently from the current subsidies and would generally be less generous for those receiving subsidies under current law, the other changes would, in the agencies’ view, lower average premiums enough to attract a sufficient number of relatively healthy people to stabilize the market.
Is it defensible journalistic practice to leave out the "probably" and "most areas" caveats in the CBO report?

Something tells us that if PolitiFact caught a Republican omitting that kind of information, it would result in a rating of "Half True" or worse. Assuming the Republican wasn't making a point that a liberal would like, of course.


We just finished listening to PolitiFact's Aaron Sharockman spending an hour on the Hugh Hewitt Show. Sharockman reaffirmed the paraphrase of the CBO we quoted above. When a transcript becomes available, we will look at whether Sharockman magnified the distortion from the original fact check.

Thursday, March 23, 2017

Rorschach context

It seems as though the liberal bloggers (aka "mainstream fact checkers") at PolitiFact treat context like a sort of Rorschach inkblot, to interpret as they see fit.

What evidence prompts these unkind words? The evidence runs throughout PolitiFact's history, but two recent fact-checks inspired the imagery.

The PolitiFact Florida Lens

In our previous post, we pointed out the preposterous "Mostly True" rating PolitiFact Florida gifted on a Florida Democrat who equated the raw gender wage gap with the gender wage gap caused by sex discrimination. The fact checkers did not interpret words uttered in context, "simply because she isn't a man," as an argument that the raw wage gap was entirely the result of gender discrimination. Perhaps it wasn't specific enough, like saying the difference in pay occurred despite doing the same work ("Mostly False")?

Whatever the case, PolitiFact opted not to accept a crystal clear clue that it was checking a claim that mirrored one it had previously rated "Mostly False" and rated the similar claim "Mostly True."

The PolitiFact California Lens

A recent fact check from PolitiFact California makes for a jarring contrast with the PolitiFact Florida item.

California Lt. Governor Gavin Newsom tweeted that Republican Jason Chaffetz had treated the cost of an iPhone to the cost of health care "as if the 2 are the same." Newsom was making the point that health care costs more than an iPhone, so saying the two are the same misses the mark by a California mile.

But did Chaffetz say the costs are the same?

First let's look at how the PolitiFact California lens processed the evidence, then we'll put that evidence together with some surrounding context.

PolitiFact California:
We also examined Newsom’s final claim that Chaffetz had compared the iPhone and health care costs "as if they are the same."

Chaffetz’ comments, particularly his phrase "Americans have choices. And they’ve got to make a choice," leave the impression that obtaining health care is as simple as sacrificing the purchase of a smartphone.
It's worth noting at the outset that PolitiFact California's key evidence doesn't mention the iPhone and does not even imply any type of cost comparison. The only way to adduce Chaffetz's quotation as evidence of a price comparison would have to come from the context of Chaffetz's remarks. And a fact-checker ought to explain to readers how that works, unless the fact checker can count on his audience sharing his ideological bias.

Chaffetz (as quoted at length in the PolitiFact California fact check; bold emphasis added):
"Well we're getting rid of the individual mandate. We're getting rid of those things that people said they don't want. And you know what? Americans have choices. And they've got to make a choice. And so, maybe rather than getting that new iPhone that they just love and they want to go spend hundreds of dollars on that, maybe they should invest it in their own health care. They've got to make those decisions for themselves."
Chaffetz in no way offers anything approaching a clear suggestion that the cost of an iPhone equals the cost of health care or health insurance. His words about people having choices come right after he says the health care bill would eliminate the individual mandate. After that comes the mention of an iPhone costing "hundreds of dollars" that one might instead invest in health care. In context, the statement is just one example of a great number of choices one might make about paying for health care.

The PolitiFact California lens (like magic!) turns Chaffetz's words conveniently into what is needed to say the Democrat said something "Mostly True."

It's the bias, stupid.

We have PolitiFact Florida ignoring clear context to give a Democrat a more favorable rating than she deserves. We have PolitiFact California finding clear evidence from the context where none exists to give a Democrat a more favorable rating than he deserves.

Point out the absurdity to PolitiFact (as we did for the PolitiFact Florida flub) and somebody from the Tampa Bay Times will read the critique and no changes to the article will result.
How are they able to repeatedly overlook problems like these?

The simplest explanation? Because they're biased. Biased to the left. Biased to trust their own work (despite the incongruity with other PolitiFact fact checks!). And Dunning-Kruger out the wazoo.

Clarification: March 27, 2017: Added link to the PolitiFact California fact check of Gavin Newsom.