Former PolitiFact expert source Brian Riedl on PolitiFact

 The tweet:

When Politifact was founded in 2007, their leaders visited me in person to recruit me as one of their "go-to" fact-checking sources.

I eventually quit working with them when it became clear that my role was to provide conservative cover for partisan-left fact checks. (1/3) https://t.co/rSgDn3lngh

— Brian Riedl 🧀 (@Brian_Riedl) March 30, 2021

Click the link for the other two tweets in the thread.

Add Riedl's name to a fairly impressive list of similar experiences from people like Michael F. Cannon and Judith Curry.

The unquotable Judith Curry

Judith Curry's Twitter avatar.

A reader tipped us to the fact that climate research expert Judith Curry has posted interview questions she received from PolitiFact's John Kruzel, along with her responses to those questions.

Kruzel solicited Curry's views in the context of fact-checking a statement about carbon dioxide's role in climate change. Kruzel's fact check lists his email interview of Curry in its source list, but the fact check does not quote, paraphrase or summarize Curry's views.

In accordance with its Creative Commons licensing, we present Curry's account of her PolitiFact interview, following the format she used at the blog she hosts, Climate Etc. (we added a bracketed explanation of the IPCC acronym):

On June 20, John Kruzel, the author of the Politifact article, sent me an email:

We’re looking into Energy Secretary Rick Perry’s recent claim that the main cause of climate change is most likely “the ocean waters and this environment that we live in.” We’ve asked the Department of Energy why Perry disagrees with the IPCC  [Intergovernmental Panel on Climate Change] that human activity is the main cause of climate change; we’ve received no response so far.

I’d be grateful if you’d consider the following questions:

Questions from Politifact to JC, and JC’s responses:

Do you consider the IPCC the world’s leading authority on climate change and why?
The IPCC is driven by the interests of policy makers, and the IPCC’s conclusions represent a negotiated consensus.  I don’t regard the IPCC framework to be helpful for promoting free and open inquiry and debate about the science of climate change.
 .
Do you agree with the IPCC that effects of man-made greenhouse gas emissions “are extremely likely to have been the dominant cause of the observed warming since the mid-20th century.”
It is possible that humans have been the dominant cause of the recent warming, but we don’t really know how to separate out human causes from natural variability.  The ‘extremely likely’ confidence level is wholly unjustified in my opinion.
 .
How solid is the science behind the conclusion that human activity is the main cause of climate change?
Not very solid, in my opinion.  Until we have a better understanding of long term oscillations in the ocean and indirect solar effects, we can’t draw definitive conclusions about the causes of recent warming.
 .
What is your response to Perry’s statement?
I don’t have a problem with Perry’s statement.  There is no reason for him to be set up as an arbiter of climate science.  He seems clearly committed to a clean environment and research to developing new energy technologies, which is  his job as Secretary of Energy.

JC question:  So what are we to conclude from PolitiFact’s failure to even mention or consider my responses, after explicitly asking for them?

We suggest it's safe to conclude Kruzel had his mind made up on this fact check before contacting his expert sources. Asking experts if they agree the IPCC is the leading authority on climate change qualifies as a classic leading question, and offers a strong indication that the IPCC's leading role in the story was central before Kruzel contacted Curry. The second question counts as another leading question, set up by the first leading question.

It looks like Kruzel was trying to lead the experts toward giving quotations to back what he had already decided to write.

Kruzel's third and fourth questions are fine. A serious fact check could have worked based on those questions alone, dropping the leading questions and Kruzel's/PolitiFact's confident proclamation regarding the IPCC (bold emphasis added):

The world’s leading authority on climate change, the Intergovernmental Panel on Climate Change, has concluded that human activity is "extremely likely" to be the main driver of warming since the mid 20th century.

While it’s still possible to find dissenters, scientists around the globe generally agree with this conclusion.

Kruzel might have added: "We actually found one such dissenter without even really trying!"  But since PolitiFact does not publish its email interviews (unlike one transparent fact checker we know of), there's no telling whether PolitiFact found more than one such dissenter in its small pool of expert sources.

Seriously, what is the basis in fact for calling the IPCC "the world's leading authority on climate change"? Such designations stem from popular or expert opinions, don't they? Objective reporting makes such distinctions clear. What Kruzel did was not objective reporting.



Correction June 28, 2017: Belatedly added a hyperlink to the PolitiFact fact check that cites Curry without quoting Curry

Fact-checking the wrong way, featuring PolitiFact

Let PolitiFact help show you the right way to fact check by avoiding its mistakes

Fake and skewed news do present a problem for society. Having the best possible information allows us to potentially make the best possible decisions. Bad information hampers good decision-making. And the state of public discourse, including the state of the mainstream media, makes it valuable for the average person to develop fact-checking skills.

We found a December 11, 2016 fact check from PolitiFact that will help us learn better how to interpret claims and make use of expert sources.

The interpretation problem

PolitiFact believed it was fact-checking Republican Reince Priebus' claim that there was no report available saying Russia tried to interfere with the 2016 presidential election:

Was Priebus saying there was no "specific report" saying Russia tried to "muddy" the election? Here's how PolitiFact viewed it:

"Let's clear this up. Do you believe -- does the president-elect believe that Russia was trying to muddy up and get involved in the election in 2016?" Meet the Press host Chuck Todd asked on Dec. 11, 2016.

"No. 1, you don't know it. I don't know it," Priebus said. "There's been no conclusive or specific report to say otherwise."

That’s wrong. There is a specific report.

It was made public on Oct. 7, 2016, in the form of a joint statement from the Department of Homeland Security and the Office of the Director of National Intelligence. At the time, the website WikiLeaks was releasing a steady flow of emails stolen from the Democratic National Committee and top Hillary Clinton adviser John Podesta.

"The U.S. Intelligence Community (USIC) is confident that the Russian Government directed the recent compromises of e-mails from U.S. persons and institutions, including from U.S. political organizations," the statement said. "These thefts and disclosures are intended to interfere with the U.S. election process."

Based on the context of Priebus' appearance on "Meet the Press," we think PolitiFact botched its interpretation. NBC's Chuck Todd went back and forth with Priebus for a number of minutes on the nature of the evidence supporting the charge of Russian interference with the 2016 U.S. presidential election. The main topic was recent news reports suggesting Russia interfered with the U.S. election to help Republican candidate Donald Trump. Todd's question after "Let's clear this up" had little chance of clearing up that point. Priebus would not act unreasonably by interpreting Todd's question to refer to interference intended to help the Republican Party.

But the bigger interpretation problem centers on the word "specific." Given the discussion between Todd and Priebus around the epistemological basis for the Russian hacking angle, including "You don't know it and I don't know it" in the immediate context, both the "conclusive" and the "specific" definitions of the word address the nature of the evidence.

"Conclusive" means incontrovertible, not merely featuring a conclusion. "Specific" means including a specific evidence or evidences, and therefore would refer to a report showing evidences, not merely a particular (second definition as opposed to the first) report.

In short, PolitiFact made a poor effort at interpreting Priebus in the most sensible way. Giving conservatives short shrift in the interpretation department occurs routinely at PolitiFact.

Was the report PolitiFact cited incontrovertible? PolitiFact offered no argument to that effect.

Did the report give a clear and detailed description of Russia's attempt to influence the 2016 election? Again, PolitiFact offered no argument to that effect.

PolitiFact's "fact-checking" in this case amounted to playing games with the definitions of words.

The problem of the non-expert expert

PolitiFact routinely cites experts either without investigating or reporting (or both) their partisan leanings. Our current case gives us an example of that, as well as a case of giving the expert a platform to offer a non-expert opinion:

Yoshiko Herrera, a University of Wisconsin political scientist who focuses on Russia, called that letter, "a pretty strong statement." Herrera said Priebus’ comment represents a "disturbing" denial of facts.

"There has been a specific report, and politicians who wish to comment on the issue should read and comment on that report rather than suggest there is no such report or that no work has been done on the topic," Herrera said.

What relevant expertise does a political scientist focused on Russia bring to the evaluation of statements on issues specific to U.S. security? Even taking for granted that the letter Herrera talks about was objectively "a pretty strong statement," Herrera has no obvious qualification that lends weight to her opinion. An expert on international intelligence issues might lend weight to that opinion by expressing it.

The same goes for PolitiFact's second paragraph quoting Herrera. The opinion in this case gains some weight from Herrera's status as a political scientist (the focus on Russia counts as superfluous), but her implicit opinion that Trump made the error she cautions about does not stem from Herrera's field of expertise.

Note to would-be fact checkers: Approach your interviews with experts seeking comments that rely on their field of expertise. Anything else is fluff, and you may end up embarrassing the experts you cite by relying on their expertise for information that does not reflect their expertise.

Was Herrera offering her neutral expert opinion on Trump's comment? We don't see how her comments rely on her expertise. And reason exists to doubt her neutrality.

Source: FEC. Click image for larger view.

Yoshiko Herrera's FEC record of political giving shows her giving exclusively to Democrats, including a modest string of donations to the campaign of Hillary Rodham Clinton.

Did PolitiFact give its readers that information? No.

The wrap

Interpret comments fairly, and make sure you only quote expert sources when their opinion comes from their area of expertise. Don't ask an expert on political science and Russia for an opinion that requires a different area of expertise.

For the sake of transparency, I advocate making interview material available to readers. Did PolitiFact lead Herrera toward the conclusion a "specific report" exists? Or did Herrera offer that conclusion without any leading? An interview transcript allows readers to answer that question.

PolitiFact had announced that it plans to start making interview materials available as a standard practice. Someday? Somewhere over the rainbow?



Afters

Since the time we started this evaluation of PolitiFact's fact check, U.S. intelligence agencies have weighed in with comments hinting that they possess specific evidence showing a Russian government connection to election-related hacking and information leaks. But even these new reports do not contradict Priebus until the reports include the clear and detailed evidence of Russian meddling--from named and reliable sources.

Jon Feere: "Does Hillary support a border wall? Someone should ask"

Jon Feere of the Center for Immigration Studies was recently cited as an expert by PolitiFact Wisconsin.

Like a number of experts in the past, Feere came away unimpressed with the way PolitiFact made use of his expertise. Feere wrote an article questioning PolitiFact's finding that it is "Half True" Hillary Clinton once favored building a wall on the Mexican border.

We love articles like Feere's that reveal the correspondence between PolitiFact and the experts it cites in its stories. People like Feere give us the type of transparency PolitiFact should provide in the first place.

We recommend reading the whole thing for the detailed analysis Feere offers of PolitiFact's approach to the subject.

Here's a two-paragraph teaser:

Had Trump said, "By the way, Hillary's 700 miles is 300 miles shorter than the 1,000 miles I'm supporting" PolitiFact would have had to give the statement a "True" rating which means it probably would never have been picked up by PolitiFact in the first place.

Bottom line: It's totally true that Clinton once supported a double-wall border fence that never got constructed, and it's totally true that Trump's proposed wall is 300 miles longer.

Also check the tag "Biased interviews of experts" for another example of an expert with this type of PolitiFact experience. More if we get around to tagging them.

True statements ruled "Mostly False"

What happens when PolitiFact finds that a statement is literally true?

That issue was brought up indirectly when Jeff retweeted economist Howard Wall. Wall had tweeted:

.@jamestaranto I was an expert in Politifact once. Sole purpose of the interview was to illicit a few words to show pre-existing conclusion.

— Howard Wall (@GatewayEcon) January 10, 2016

We looked up the story where Wall was quoted as an expert. It was a fact check of Mitt Romney from the 2012 presidential election. The Romney campaign said women had suffered the greatest job losses under Obama, implying Obama's leadership had been bad for women.

PolitiFact ruled the claim "Mostly False."

The Romney campaign pushed back. PolitiFact looked at the issue again and ruled the claim "Mostly False." But at the same time, PolitiFact said "The numbers are accurate but quite misleading."

Don't blame the Romney campaign. It probably operated under the assumption that PolitiFact's definitions for its "Truth-O-Meter" ratings mean something.

Taking PolitiFact's definitions literally, the lowest rating one should receive for a literally true claim is "Mostly True." Once below that level, the definitions start talking about "partially true" statements that give a misleading impression ("Half True") and "The statement contains some element of truth" but ignores facts that could give a different impression ("Mostly False").

What's our point? We've always said PolitiFact's ratings reveal more about PolitiFact than they do about the entities receiving the ratings. It's a scandal that social scientists keep their eyes closed to that. Search PolitiFact's ratings for claims it says are literally true. Note the rating given to the claim. Then take a look at the ideology of the entity making the claim.

There's your evidence of journalistic bias by fact checkers.

This is an important issue. If social scientists aren't looking at it, it suggests they don't care.

Why wouldn't they care?



Jeff Adds: We highlighted a Mark Hemingway critique of PolitiFact's Romney claim back in 2012 that is still worth a read. It would seem little has changed at PolitiFact since then.



Update 0956PST 1/12/2016: Added "Jeff Adds" portion - Jeff