PolitiFact's "many" problems

On Nov. 3, 2016 we brought some focus to the "Mostly False" rating PolitiFact gave Donald Trump for saying many Americans were paying more for health care than for their mortgage or rent.

PFB co-editor Jeff D. today reminded me about an "Afters" section I added to a post from Sept. 1, 2016:

PolitiFact exaggerated the survey evidence supposedly supporting Clinton by claiming "many" teachers blamed Trump for increasing bullying and harassment:

Many of these teachers, unsolicited, cited Trump’s campaign rhetoric and the accompanying discourse as the likely reason for this behavior.

The Zebra Fact Check investigation suggests PolitiFact was misled about the number of teachers saying Trump was responsible for increasing bullying or harassment. Out of almost 2,000 teachers participating in the survey, 849 answered the question about bullying or biased language and of those 123 mentioned Trump. A fraction of those placed any kind of blame on Trump for anything. We would generously estimate that 25 teachers blamed Trump for something (not necessarily bullying or harassment) in answering that question. This implies that, to PolitiFact, "many" can be less than 1.25 percent of 2,000.

That's right. The hypocritical liberal bloggers at PolitiFact said "many" teachers cited Trump's rhetoric as the likely reason for bullying and/or harassing behavior. PolitiFact shoveled that to its readers as a fact, though the data showed a small fraction of the surveyed teachers offered that opinion. Then. about a month later, PolitiFact said Trump's statement about health care costs was "Mostly False."

Could PolitiFact be right in both cases?

That seems like a stretch.

Wrong in both cases?

That's more likely.

Another day, another deceptive PolitiFact chart

On election day, PolitiFact helpfully trotted out a set of its misleading "report card" graphs, including an updated version of its comparison between Democrat Hillary Clinton and Republican Donald Trump.

What is the point of publishing such graphs?

The graphs make an implicit argument to prefer the Democratic Party nominee in the general election. See how much more honest she is! Or, alternatively, see how the Republican tells many falsehoods!

The problem? This is the same PolitiFact deception we have pointed out for years.

The chart amounts to a political ad, making the claim Clinton is more truthful than Trump. But to properly support that conclusion, the underlying data should fairly represent typical political claims from Clinton and Trump--the sort of thing scientific studies capture by randomizing the capture of data.

In the same vein, a scientific study would allow for verification of its ratings. A scientific study would permit this process by using a carefully defined set of ratings. One might then duplicate the results by independently repeating the fact check and reaching the same results.

Yet none of that is possible with these collected "Truth-O-Meter" ratings.

Randomly selected stories aren't likely to grip readers. So editors select the fact-checks to maximize reader interest and/or serve some notion of the public good.

So much for a random sample.

And trying to duplicate the ratings through following objective scientific procedure counts as futile. PolitiFact editor Bill Adair recently confirmed this yet again with the frank admission that "the decision about a Truth-O-Meter rating is entirely subjective."

So much for objectively verifying the results.

PolitiFact passes off graphs of its opinions as though they represent hard data about candidate truthfulness.

This practice ought to offend any conscientious journalist, and that should go double for any conscientious fact-checking journalist.

We have called for PolitiFact to include some type of disclaimer each time it publishes this type of item. Such disclaimers happen only on occasion. The example embedded in this post contains no hint of a disclaimer.

Wonder why Republicans and Trump voters do not trust mainstream media fact-checking?

Take a look in the mirror, PolitiFact.

PolitiFact founder Bill Adair: "Lord knows the decision about a Truth-O-Meter rating is entirely subjective"

 What have we said for years? PolitiFact's "Truth-O-Meter" rating system is hopelessly subjective.

And now, thanks to an interview by freelance journalist Michael Schulson, PolitiFact's founding editor has made perhaps his clearest statement yet confirming that key charge against PolitiFact (bold emphasis added):

[Michael Schulson]

But there is some subjectivity baked into the process, in terms of which claims you check, and where you draw the line between statements of opinion and statements of fact. Objective journalists are still making subjective choices.

[Bill Adair]

Oh, absolutely. But they always have!

I think that transparency is key. You need to have your own guidelines on how you select what you fact-check.

But yeah, we’re human. We’re making subjective decisions. Lord knows the decision about a Truth-O-Meter rating is entirely subjective. As Angie Holan, the editor of PolitiFact, often says, the Truth-O-Meter is not a scientific instrument.

How often does PolitiFact's Angie Drobnic Holan say the "Truth-O-Meter" is not a scientific instrument? Not nearly enough for our tastes. PolitiFact announces new candidate report card updates and comparisons by the week. But a Google search on Nov. 5, 2016 for "Truth-O-Meter" and "scientific instrument" drew only seven pages of hits. And a good number of those were not directly related to PolitiFact. Many of the rest were duplicates of this page (38 hits).

Would it alter PolitiFact's impact if every "report card" or report card story it published wore the disclaimer that the "Truth-O-Meter" ratings are subjective?

We think it would. Plus, doing so would represent an important step toward full transparency for PolitiFact.

So why don't they do it?

Note: Those who read Schulson's interview of Adair may wish to also read my response.

Update Oct. 24, 2024: The old link to Schulson's interview of Adair wasn't working, so I replace it with a working link. The Internet Archive Wayback Machine is malfunctioning today, or else I'd provide an archived version.

The Daily Caller: "PolitiFact Used Doctored Clinton Foundation Memo On Its HIV/AIDS Program"

The conservative website The Daily Caller has wound up in a bit of a feud with PolitiFact. The Caller ran an article criticizing the Clinton Foundation. PolitiFact did a fact check of the Caller in response. And the Caller has responded blow for blow.

This one'll leave a mark:

High-ranking officers with the Clinton Foundation gave PolitiFact a doctored version of a 2008 memo lauding its HIV/AIDS program presumably to defend against congressional charges that the charity distributed ‘watered down’ drugs to poor patients on the African continent, according to new information acquired by The Daily Caller News Foundation Investigative Group.

The altered memo went to Politifact Sept. 21, 2016, three days after TheDCNF published a story entitled, “Clinton Foundation AIDS Program Distributed ‘Watered-Down’ Drugs To Third World Countries.” (RELATED: EXCLUSIVE: Clinton Foundation AIDS Program Distributed ‘Watered-Down’ Drugs To Third World Countries)

The Caller also achieved a minor miracle by getting PolitiFact's Aaron Sharockman to respond.

Steps were reportedly taken to verify the authenticity of the Clinton Foundation document, Aaron Sharockman, executive director of PolitiFact, claimed in a statement to TheDCNF.

He told TheDCNF that the memo “was provided to us by the Clinton Foundation in response to our questions,” adding that PolitiFact “verified its authenticity through emails sent and received at that time.”

Read the whole thing here.



Correction/clarification Nov. 6, 2016: "The Daily Caller has wound up a bit of a feud"=>"The Daily Caller has wound up in a bit of a feud"

PolitiFact: "Mostly False" that many are paying more for health care than for mortgage or rent

Sometimes reading a PolitiFact fact check is like being whisked off to Wonderland for a conversation with the Mad Hatter.

Case in point: Donald Trump says that many Americans are paying more for health care than to pay their rent or mortgage for the first time in history. PolitiFact finds it "Mostly False."

This rating drew our attention right away because of the ambiguity of Trump's claim. How does a fact-check go about making a truth distinction about "many instances"? How many is "many"? And if pinning down "many" poses a challenge, how does one go from that challenge to finding out whether it's happening for the first time in history?

After getting into the text of the fact check, it was a matter of trying to control laughter over the way PolitiFact approached the problem.

Fact, motivation, and PolitiFact's inconsistency

One of the oldest legislative tricks involves introducing a bill that will not pass so that one party can slam the member of the opposing party for not supporting one part of the bill.

We've seen the Democrats use that technique to terrific effect with the "Violence Against Women Act." And Republicans do the same type of thing to Democrats.

A Democrat or a Republican might have motivations behind their opposition that undercut the message their opponents try to use against them.

But does PolitiFact treat these same types of campaign ads the same way for both parties?

It sure doesn't look like it.

PolitiFact Missouri today graded a Republican claim in this category "Half True."

Note how PolitiFact Missouri justifies its conclusion (bold emphasis added):

Greitens says Koster voted against a 2007 bill requiring the state to pay for rape victims’ medical exams.In reality, the bill did more than that.

Koster says he objected to wording that made it possible for convicted murderers to be granted parole by claiming they were victims of domestic abuse. Koster said the language made it possible for murderers to manufacture evidence to be released before the completion of their sentence.

Greitens is cherry-picking one part of the legislation to paint his opponent as soft on domestic abuse. We rate his claim Half True.

As we noted back in August, PolitiFact Florida gave a "True" rating to Democrat Patrick Murphy when he made a parallel claim about his Republican opponent:

And note how PolitiFact Florida justifies its conclusion:

Murphy said Rubio "voted against the bipartisan Violence Against Women Act."

Rubio voiced support for the original law, but he and some Republicans in both the Senate and House opposed certain provisions added to the bill pertaining to spending and federal oversight. Rubio voted against the bill in 2012 and 2013, but it passed with bipartisan support the second time.

Even though he had clearly stated his reasons why, Rubio still voted nay. We rate Murphy’s statement True.

Both cases feature the same type of deception, and PolitiFact's fact checkers take note of the deception in both cases. But the Republican gets a "Half True" rating while the Democrat gets a "True" rating.

This type of example isn't atypical. It's just another day at the office for PolitiFact's left-leaning fact checkers.


Afters

It's worth pointing out that our previous post shows PolitiFact Wisconsin using essentially this same illicit ad technique against Senator Ron Johnson (R-Wisc.).

The next year, Johnson voted against a Senate amendment to affirm that human activity significantly contributes to climate change.

While all but one senator supported an earlier amendment affirming the existence of climate change, only five Republicans this time voted to acknowledge there is a human impact. The amendment, seen as a symbolic effort by the Democrats to force GOP senators to take a position, failed 50 to 49 (it required a 3/5 majority).

PolitiFact Wisconsin saw nothing wrong with using Johnson's opposition to the amendment as a solid evidence that Johnson thinks humans have no role in climate change even though the amendment did not narrowly address that issue.

Q: What's the difference between PolitiFact and the Democratic Party?
A: The Democratic Party doesn't claim to be nonpartisan.

More PolitiFact climate change shenanigans, featuring PolitiFact Wisconsin

One of PolitiFact's more reliable bends to the left occurs on the issue of climate change. The arbiters of truth, for example, class Republicans as climate change deniers if they do not go on record affirming man-made climate change. So much for PolitiFact's burden of proof criterion, right?

Hypocrites.

This related example comes from PolitiFact Wisconsin, checking on a claim from Senate candidate Russ Feingold that his Republican opponent does not believe humans contribute to climate change.

PolitiFact Wisconsin's approach to the fact check resembles the incompetent methods used by other iterations of the PolitiFact family. A Zebra Fact Check critique of PolitiFact's past fact check foreshadows the problems with PolitiFact Wisconsin's fact check of Feingold:

First, interpret an unclear statement according to a more clear statement by the same source.  Second, in judging what a person thinks in the present place greater weight on more recent statements.

PolitiFact Wisconsin does not apply these commonsense principles.

PolitiFact Wisconsin's evidences, in chronological order

2010
Johnson: "I absolutely do not believe in the science of man-caused climate change. It’s not proven by any stretch of the imagination."

2014

"There are other forces that cause climate to change," Johnson told Here and Now’s Robin Young. "So climate does change and I don’t deny that man has some effect on that. It certainly has a great deal of effect on spoiling our environment in many different ways."

But Johnson softened his view as soon as the next sentence: "I’ve got a very open mind, but I don’t have the arrogance that man can really do much to affect climate."

2015
Johnson votes against a proposed amendment to a bill touching the Keystone Pipeline. The amendment would have described the sense of the Senate on the issue of anthropogenic climate change, including the ideas that humans "significantly" affect the climate and that climate change increases the severity of extreme weather events (such as hurricanes).

2015

"Man-made global warming remains unsettled science. World-renowned climate experts have raised serious objections to the theories behind these claims. I believe it is a bad idea to impose a policy that will raise taxes on every American, will balloon energy prices and will hurt our economic competiveness (sic) – especially on such uncertain predictions."

2016

"Listen, man can affect the environment; no doubt about it," he said. "The climate has always changed, it always will. … The question is, how much does man cause changes in our environment, changes in our climate, and what we could possibly even do about it?"

Assessing PolitiFact Wisconsin's evidences

Following the principles mentioned above, Johnson's clearest statements on humans having some role in climate change comes from the 2014 and 2016 quotations. In 2014, Johnson said he does not deny humans have a role in climate change. In 2016, Johnson said humans "clearly" have a role in changing the climate. Johnson's clearest statements on the subject directly contradict Feingold's claim.

Our principles also guide us toward giving a preference to more recent statements. Therefore, we consider the 2015 climate change amendment for some sign that Johnson denied a human role in climate change.

Is there a worse proof of a legislator's specific views on a topic than their willingness to vote in favor of a "sense of the Senate" amendment? Particularly when that amendment does not feature language narrowly tailored to suit the question?

Would Johnson have voted in favor of the amendment if he believed there was good evidence that undefined "climate change" causes an increase in severe weather events? Who knows? We don't. But if you're PolitiFact Wisconsin you can simply assume the answer is "no" and call it fact-checking.

PolitiFact concludes:

Johnson did not support a Senate amendment to acknowledge a man-made role in climate change and expressed skepticism each of the few times he acknowledged humans might contribute. He has acknowledged at times that humans can play a role but downplayed how significant that role might be.

For a statement that is accurate but needs additional clarification, our rating is Mostly True.

PolitiFact's conclusion consists of spin.

The Senate amendment was not simply about "a man-made role in climate change." It stipulated a significant role as well as a worsening effect on severe weather.

When Johnson said humans play a role in climate change he did not express skepticism about whether humans play a role. He expressed skepticism about the extent of that role. They're not the same thing, and skepticism about the latter does not contradict Johnson's recognition that humans play some role in affecting the climate. PolitiFact says Johnson says humans "can" play a role. But that's just more spin. Johnson did not simply say humans "can" play a role. He said humans do play a role, and he said he does not deny humans play a role.

If Johnson says humans play some role in causing climate change, that statement cannot support Feingold's claim that Johnson does not believe humans play any role in climate change.

Johnson's statement cannot reasonably justify the "Mostly True" rating with which PolitiFact Wisconsin gifted Feingold. The statements could reasonably justify "False" or "Mostly False" ratings if PolitiFact's definitions for its ratings meant something.

PolitiFact's continued inability to apply simple logic in the course of its fact checks continues to boggle our minds. At the same time, we're not surprised. This is the type of error that results when left-leaning journalists rate the truth of political statements on a subjective scale.

Promises, promises

What's more useless than PolitiFact's trademarked "Truth-O-Meter"? How about its device for rating presidential promises, the Obameter?

Years ago, we pointed out an absurd rating when PolitiFact gave President Barack Obama a "Promise Kept" rating for staying in office while the nation achieved on its own what Obama had proposed achieving through a "Renewable Portfolio Standard." Obama promised to change the RPS. PolitiFact gave him credit for keeping the promise before it was kept.

In this new case, we will partly defend the president, albeit without doing him any favors.

This "Obameter" item is focused on the $2,500 promise, as PolitiFact separately failed on the promise to sign a health care bill providing universal coverage.

Obama fulfilled this promise in its literal sense. "Up to $2,500 a year" covers everything equal to or below $2,500 per year. If Obama increased costs to families by $5,000 per year, it fulfills his promise to decrease costs to families by up to $2,500 per year.

Absurdly, Obama could only break this promise by saving a typical family more than $2,500 per year.

Obama's promise was a classic political promise because it didn't really mean anything while at the same time sounding like a wonderful promise. The president implied the typical family would save about $2,500 per year under the legislation he promised. The promise paints operations like PolitiFact into a corner. They can either grade the promise on its implied meaning, which PolitiFact did, or else admit Obama's promise effectively promised nothing.

Obama delivered on the empty literal promise, unquestionably.

PolitiFact deserves partial credit for highlighting Obama's failure to achieve the promise he implied. But a fact-checker can help arm the public against misleading campaign rhetoric by explaining the deception of an "up to" clause.

PolitiFact did not do that.