PolitiFact updates its website, makes "Corrections and Updates" even harder to find (Updated: Fixed!)

Over the years we've enjoyed poking fun at PolitiFact's haphazard observance of its policy on corrections. Aside from simply not doing quite a few needed corrections, PolitiFact does things like:

• Correcting stories without a correction notice
• Not tagging stories with "Corrections and Updates" as promised in its statement of principles

We've also needled PolitiFact over the way it hides its supposedly transparent page of corrected or updated claims. Looking up "corrections and updates" along with "PolitiFact" using a search engine would allow readers to easily find the page, but finding that page from PolitiFact's home page was so hilariously complicated that we posted instructions on how to do it.

Now in February 2020 PolitiFact has revamped its website and at long last fixed the problem succeeded in making the problem even worse.

Hopefully the worsening of the situation is only temporary, but PolitiFact's history marinates that hope in thick, gooey skepticism.

Our Feb. 1, 2020 survey of the PolitiFact website makes the "Corrections and Updates" page look like an orphan.

We tried to help. Seriously.

When I (Bryan) heard on Twitter that PolitiFact was updating its website, I tweeted out a reminder for PolitiFact to make its "Corrections and Updates" page more available to readers:

Don't forget to feature a prominent link to your corrections page on your landing page.

Even better if you figure out a way for all corrected stories to appear there (tags for articles!).

— Bryan W. White (@ZebraFactCheck) January 31, 2020

Instead of fixing it, the "Corrections and Updates" page is one of the very few (this is the only other one we found) that did not experience a facelift in keeping with the new look of the website.

For our money, the redesign looks pretty bad on the big screen. And it's not much better on mobile.

But one thing we did like, though perhaps that means it won't last.

What We Liked

In addition to PolitiFact's dodgy behavior on corrections, we've endlessly criticized PolitiFact for publishing sciencey-looking graphs of aggregated "Truth-O-Meter" ratings with no regular disclaimer attached. The ratings are subjective and PolitiFact does not attempt to choose a scientifically representative sample of claims. So the graphs are nonsense in terms of representing a politician's overall reliability.

PolitiFact still isn't attaching any disclaimer, but the new design largely neuters the visual impact of its graphs.

Let's look at Donald Trump's PolitiFact "scorecard" before and after the update.

Before

That has some visual impact. The graph groups the bars closely, emphasizing the visual difference between, say, 5 percent and 35 percent.

After

What a difference! Thirty-four percent looks visually smaller on the new graph than 5 percent looked on the old graph. Sure, there's an attempt to spice it up by adding colors, but the short graph scale and thin lines suck away almost all of the impact.

Do we think PolitiFact did this intentionally so that the graphs would do less to mislead readers? No, unless it's part of an effort to farm out the deception.* But if it stands, it doesn't really matter if it's an accident or a mistake. PolitiFact will probably deceive fewer of its readers as a result.

One Other Positive!

PolitiFact has always offered the total number of each rating for individual politicians. But now it is publishing the totals for PolitiFact as a whole, as well as the states.

That makes doing certain types of research on PolitiFact's numbers easier. Though researchers will still need to realize that the subjectivity of the ratings means they tell researchers about PolitiFact, not so much about the politicians making the claims.

It's a very simple matter now to document how many more ratings Donald Trump has received than did Barack Obama, and in a shorter span of time as candidate/president.

*The downside? Those who are motivated to use PolitiFact "data" to prove Republicans are liars and whatnot will have less work to do in collecting the numbers. People irresponsibly publishing such nonsense may end up misleading more people in spite of the positives we noted.


Afters

Centered text? Seriously?


Updated Feb. 3, 2020 with edits thought already complete: strikethrough and URL linking earlier PFB post about finding PolitiFact's "Corrections and updates" page.

Update Feb 4, 2020: Whether it was the plan all along or whether in response to our cajoling, PolitiFact has added "Corrections and Updates" to its menu, under the heading "About Us."  That fixes one of our major complaints about PolitiFact. Now will PolitiFact add corrected "articles" to its list of corrected stories?

PolitiFact ratings aren't scientific--or are they?

We stand bemused by PolitiFact's attempt to straddle the fence regarding its aggregated "Truth-O-Meter" ratings.

On the one hand, as PolitiFact Editor Angie Drobnic Holan told us a few weeks ago, "It’s important to note that we don’t do a random or scientific sample."

On the other hand, we have this eye-popping claim Holan sent out via email today (also published to the PolitiFact.com website):

Trump remains a unique figure in American politics and at PolitiFact. There’s no one we’ve fact-checked as many times who has been as consistently wrong. Out of 738 claims we’ve fact-checked as of this writing, more than 70% have been rated Mostly False, False or Pants on Fire. After four years of watching Trump campaign and govern, we see little to no increase in his use of true facts and evidence to support his arguments.

Even though it is supposedly important to note that PolitiFact doesn't do a random or scientific sample, Holan makes no mention of that important caveat in her email. Instead, she makes it appear to the reader as though Trump's "Truth-O-Meter" record offers a reasonable basis for judging whether Trump has increased "his use of true facts and evidence to support his arguments."

How would PolitiFact judge whether Trump was using more true facts to support his arguments other than by looking at unscientific fact checker ratings?

PolitiFact has made a pattern of this deception.

The occasional admission of non-random, unscientific story selection is perfunctory. It is a fig leaf.

PolitiFact wants readers to judge politicians based on its aggregated "Truth-O-Meter" ratings.

For some unknown reason, though maybe we can guess, the fact checkers think their collected ratings are pretty much accurate judges of character regardless of their departure from the scientific method. And that's why PolitiFact over its entire history has implicitly and explicitly encouraged readers to rely on its graphs and charts to judge politicians. And has steadfastly resisted the obligation to attach a disclaimer to those charts and graphs making the supposedly important point that the selection process is not random or scientific.

We call it an obligation because we view it as a requirement for journalists to avoid deliberately deceiving their audiences.

PolitiFact deceives its audience daily in this way without any visible repentance.

'Not a lot of reader confusion' VI

PolitiFact editor Angie Drobnic Holan has claimed she does not notice much reader confusion regarding the interpretation of PolitiFact's "report card" charts and graphs.

This series of posts is designed to call shenanigans on that frankly unbelievable claim.

Rem Rieder, a journalist of some repute, showed himself a member of PolitiFact's confused readership with a Nov. 10, 2017 article published at TheStreet.com.

While most politicians are wrong some of the time, the fact-checking website PolitiFact has found that that [sic] Trump's assertions are inaccurate much more frequently than those of other pols.

When we say Rieder showed himself a member of PolitiFact's confused readership, that means we're giving Rieder the benefit of the doubt by assuming he's not simply lying to his readers.

As we have stressed repeatedly here at PolitiFact Bias, PolitiFact's collected "Truth-O-Meter" ratings cannot be assumed to reliably reflect the truth-telling patterns of politicians, pundits or networks. PolitiFact uses non-random methods of choosing stories (selection bias) and uses an admittedly subjective rating system (personal bias).

PolitiFact then reinforces the sovereignty of the left-leaning point of view--most journalists lean left of the American public--by deciding its ratings by a majority vote of its "star chamber" board of editors.

We have called on PolitiFact to attach disclaimers to each of its graphs, charts or stories related to its graphs and charts to keep such material from misleading unfortunate readers like Rieder.

So far, our roughly five years of lobbying have fallen on deaf ears.

"There's not a lot of reader confusion" Update 2

The Houston Chronicle joins those suffering confusion over the significance of PolitiFact's "report cards."

The New York Times breathlessly excreted the "All Politicians Lie. Some Lie More Than Others." headline over an article by PolitiFact's Angie Drobnic Holan, even though the article mentioned nothing about intentional deceit--the commonly understood definition of "lie."

Rolling Stone published a graphic based on PolitiFact's ratings, following the Times' lead by calling false statements "lies" and framing the story as though percentages of false statements as judged by PolitiFact represent something significant.

Now along comes the Chronicle to provide its own endorsement of PolitiFact's junk science:

(PolitiFact) looked at top politicians and 2016 presidential candidates to analyze which ones have told the most false statements since 2007. The site’s editor Angie Drobnic Holan wrote an Op-Ed in the New York Times about the results of their fact-checking campaign. See the politicians who lie the most in the gallery.

Once again we have equivocal language with "lie" standing in for false. And the word "analyze" creep in there somehow, as though totaling the different ratings for each candidate is some type of science. If it's any kind of science it's junk science.

Do we get any disclaimer about PolitiFact's dubious objectivity (and that's being generous) or the obvious selection bias problem?

No. Nothing closer than this:

Politifact only evaluates statements that are clear, precise and are not “obviously true.” The site does not look at opinions or predictions, but statements that the general public would be interested in knowing the truth about.

Chronicle readers who are familiar with science may be able to figure out from that paragraph that PolitiFact's stories are not chosen to make up a representative sample. That means the percentages of "lies" reported in the story have no firm basis in evidence.

PolitiFact editor Angie Drobnic Holan claims there's not much reader confusion about PolitiFact's report cards. If she was right then stories based on the report cards would not draw much reader interest. The report cards draw reader interest precisely because of the way they feed confirmation bias.

PolitiFact, The New York Times, Rolling Stone and the Houston Chronicle all appear perfectly willing to stoke that confirmation bias.

Here's a sampling of the absence of reader confusion from the comments section under the Chronicle story:

"SHOCKER...Conservative Republicans and Religious Right candidates lie more than their Liberal counterparts. Those darn facts are always getting in the way."

"I don't know what is more disturbing--the fact that Republican candidates lie up to 80% of the time, or that the more they lie, the more popular they are within the party."

"The GOP candidates have issued a higher percentages of lies then their Democratic counterparts and the President...shocking. These are only the Republicans we know of. The GOP own the majority of Congress thus probably own the largest share of those lies."

"Lying is what the GOP does best, it would appear."

"The far right nuts obviously have a problem with truth and accuracy. But we've known all this for years."

"There is a clear pattern in these data. Only the feckless liberals feel the need to be accurate. Only the weak form their opinions and policies based on evidence and logic."

"If you ever visit the site and read the analysis, the impact of the statement is weighted to determine how egregious any misinformation is, and also whether there is benefit of doubt."

Facts, evidence, logic.

No reader confusion there. No sirree.

Rolling Stone jumps on PolitiFact's junk science bandwagon

Rolling Stone has joined The New York Times in promoting PolitiFact's unscientific ratings of politicians' truthfulness.

PolitiFact admits it exercises discretion in choosing which fact check stories to pursue. That results in obvious selection bias. So PolitiFact two-facedly admits its work is journalistic and not social science while encouraging readers to draw conclusions from its data with caution.

What kind of "caution"? The data are approximately worthless.

 

Image from Rolling Stone magazine

Rolling Stone indulges in the same equivocation games The New York Times foisted on its readership.

If PolitiFact rates a statement "False," Rolling Stone helpfully translates that into "Lied." That's exactly the kind of word game political ads often play. The liberal press does exactly the same thing while donning a deceitful mask of fairness.

They're hypocrites.

And how about the deck?

And Hillary Clinton and Bernie Sanders are the most truthful, according to PolitiFact's analysis

 "Analysis."

So the stories are selected via discretion (selection bias) and PolitiFact has an impossibly subjective ratings scale, but totaling the ratings of the various candidates is supposed to qualify as "analysis." That's a lie, if we can borrow the way the Times and Rolling Stone employ the term.

And why is the word "bias" completely absent from the story? Is Rolling Stone unaware that journalism is the profession exhibiting the greatest ideological imbalance in the United States?

Maybe these journalists know the data don't mean anything but promote it anyway to get clicks and influence elections.

Maybe these journalists don't have a clue.

Either way, not good.

Angie Drobnic Holan and the Times do some fact-chucking

Does PolitiFact accuse politicians of lying?

In a Dec. 7, 2015 review of a research paper I published over at Zebra Fact Check, I assured readers PolitiFact consistently says it avoids accusing politicians of lying, its "Pants on Fire" rating and "Lie of the Year" awards notwithstanding.

Days later, on Dec. 11, 2015, The New York Times published an op-ed by PolitiFact editor Angie Drobnic Holan that threatened to contradict my claim.

Though the Times' headline proclaimed "All Politicians Lie," the op-ed never really supports its blaring headline. Holan herself does not use the word "lie" in her op-ed and doesn't even refer to the concept of intentional deception--the most commonly understood meaning of "lie."

So, aside from the title of the op-ed, Holan stays consistent with PolitiFact's stance that it does not accuse politicians of intentionally deceiving people.

That's the good news. The bad news is that Holan's op-ed is misleading.

I’ve been fact-checking since 2007, when The Tampa Bay Times founded PolitiFact as a new way to cover elections. We don’t check absolutely everything a candidate says, but focus on what catches our eye as significant, newsworthy or potentially influential. Our ratings are also not intended to be statistically representative but to show trends over time.

Jeff, the other editor at PolitiFact Bias, highlighted Holan's literary pretzel on Twitter. What kind of trends do non-representative statistics show?

They show a non-representative trends, that's what kind. Anyone inclined to say differently should make their case by supporting with representative sampling the trend PolitiFact represents with its graphs and numbers. And after that vet PolitiFact's findings for accuracy and consistency. Without both, the graphs should not be taken seriously.

Even without knowing that journalists tend to lean ideologically left, it's irresponsible and misleading to present claims such as Holan's without real supporting evidence.

Shame on PolitiFact. Shame on Angie Drobnic Holan. Shame on The New York Times.

"Most dishonest" based merely on whether statements are true or false? Isn't that dishonest?

Clarification Dec. 13, 2015: The original caption under our image capture ended with "That's dishonest." We changed that to "Isn't that dishonest?" to change it from an apparent contradiction into an paradoxical charge against The New York Times.

Clarification Dec. 14, 2015:  Fixed some minor grammatical flaws.

Are PolitiFact's "report cards" misleading?

One of our recurrent themes at PolitiFact Bias concerns the misleading nature of PolitiFact's "report cards." PolitiFact admits it does not perform its fact checks on a scientific basis, particularly in that story choices do not occur randomly. Despite that, it's utterly common for PolitiFact to publish a "report card" story encouraging readers to consider a candidate's "report card."

The latest such asks readers to consider the record of just-announced presidential candidate and former Secretary of State Hillary Clinton.

The reader comments from PolitiFact's Facebook page offer us a window into the degree of deception PolitiFact achieves with its "report card" stories.

We'll omit the names to save these individuals unnecessary embarrassment.

"When you compare the overall honesty of Democrats vs. Republicans, it's no wonder that some Republicans believe that fact checking web sites are liberally biased. It's an easier explanation then the reality that Republicans tend to lie more."

"Better than Ted Cruz."

"Comparison: Clinton, true or mostly true = 48%; Rand Paul, true or mostly true = 15%; Ted Cruz, true or mostly true = 8% (from PolitiFact archives)."

"I would love to see a chart comparison against the other candidates. Paul's record would be a joke, his pants have been on fire so much the fire department had to move into a spare room"

"Still not a Hillary fan, but at least she's more honest that the right wing."

"Not a bad record."

Many more after the break! 

"The Heart of PolitiFact"

We find it fundamentally dishonest the way PolitiFact treats its trademark "Truth-O-Meter" as though it's some sort of machine that objectively measures the truth content of political statements.

The opposite's true.

Have a look at PolitiFact's "About PolitiFact" page (bold emphasis added):

The heart of PolitiFact is the Truth-O-Meter, which we use to rate factual claims.

The Truth-O-Meter is based on the concept that – especially in politics - truth is not black and white.

PolitiFact writers and editors spend considerable time researching and deliberating on our rulings ...

We then decide which of our six rulings should apply.

There's no machinery, only the machinations of biased reporters and editors.  The "Truth-O-Meter" is just the vehicle for the label they put on their decision.  The eventual rulings are even worse than the "coin flip" described by John Kroll, mentioned in our previous post.  The "Truth-O-Meter" is akin to the "Wheel of Fortune," given the level of subjectivity inherent in the system.

The heart of PolitiFact?

More PolitiPundit

We've written before about (former PolitiFact editor) Bill Adair's desire to have it both ways with regard to PolitiFact's ratings. When cornered by skeptics, Adair usually defended himself by saying "PolitiFact rates the factual accuracy of specific claims; we do not seek to measure which party tells more falsehoods." However, when preaching to his flock he would proclaim PolitiFact's ratings create "report cards for each candidate that reveal patterns and trends about their truth-telling." and the tallies of those ratings "provide interesting insights into a candidate's overall record for accuracy."

Either the ratings are scientific measurements or they're not, and they're either revealing patterns or they're not. PolitiFact cannot promote the cumulative results of its ratings as indicative of a person's honesty while simultaneously hiding behind a mask of random curiosity.

Apparently new editor Holan has bought into this contradiction with her eyes wide shut. It also appears Holan has convinced herself and her staff that they can don a magical cloak of objectivity when checking pundits as well as they do with politicians. PolitiFact's selection bias is only poised to be more evident when checking pundits than it is with public servants. Pundits, by definition, deal in nuance and opinion.

But the real howler with this PolitiPundit announcement was the apparent lack of self-awareness in this line (emphasis added):

Although PolitiFact has done occasional fact-checks of pundits and talk show hosts, the new venture will mark the first time that staffers have been dedicated to checking media figures.

I bet the best part of being the Unquestionable Arbiter of Facts is you get to decide what words like "occasional" mean. Are Rush Limbaugh's 17 ratings an occasional event? What about Glenn Beck's 23 ratings, or Rachel Maddow's 16? I suppose Sean Hannity's eight ratings or Bill O'Reilly's 10 count as rare?

It's been common practice for PolitiFact to rate pundits and commentators since its inception. The only thing new here is the devotion of additional resources to its fact checking farce.  I'll go out on a limb and predict PolitiPundit will be an even bigger embarrassment than their flagship site.


Bryan adds: 

Some new readers might wonder:  What's the big deal with PolitiFact being a bit imprecise?  "Occasional" covers a good bit of ground, so what's the big deal?

PolitiFact has often downgraded political figures and pundits for rhetorical imprecision.  It's hypocritical.  To mimic PolitiFact's typical judgmental tone:

PolitiFact left a misleading impression by saying it "occasionally" rates pundits.  The facts show otherwise, so the statement tells a partial truth but leaves out important details.  That meets our definition of "Half True."

PolitiFact: We're not social scientists, but you sure can tell plenty about a candidate from our 'candidate report cards'

PFB editor Jeff D. took notice that new PolitiFact editor Angie Drobnic Holan recently affirmed PolitiFact's minimalist admission of its practice of selection bias (bold emphasis added):

PolitiFact's goal is not to end inaccuracy in political discourse, just to point it out for voters' information, Holan says. And the site doesn't cover everything politicians say, either in ads or interviews. "We're journalists. We pick statements based on what we think is most newsworthy,'' Holan says. "We fact check things that make people go 'Hmmm, I wonder if that's true.''

Not at all to our surprise, PolitiFact's announcement of its "PolitiPundit" project, dedicated to fact checking media figures and pundits the PolitiFact way, reiterates PolitiFact's intent to keep right on publishing "report cards" for the people and organizations it fact checks. 

It's the same old mixed message from PolitiFact:  No, it doesn't use basic scientific methods in choosing its stories.  Yes, readers should treat PolitiFact's "report cards" as though PolitiFact uses scientific methods.

"Meet the new boss.  Same as the old boss."
--The Who

PolitiFact's Paradoxical Positions: The Conflicting Claims of Bill Adair

A press release for a study by George Mason University (CMPA) last week garnered wide attention, though, hilariously, for the wrong reasons. In a case of comically misinterpreted data, the supposedly fact-based left took to the Twitterverse and other media outlets to trumpet what they thought the study said about Republican honesty, when in reality the study was attempting to quantify PolitiFact's liberal tilt. (We don't think it was successful, but more on that later.)

While distorted headlines may comfort some liberals in the warm blanket of confirmation bias, a study highlighting PolitiFact's harsher treatment of the GOP doesn't fit with PolitiFact's self-proclaimed non-partisanship. With headlines like "Study: Republicans are “the less credible party” and tweets claiming "Republicans lie 3x more often than Democrats, says @politifact" quickly being repeated, PolitiFact founder Bill Adair felt compelled to issue a response:

Actually, PolitiFact rates the factual accuracy of specific claims; we do not seek to measure which party tells more falsehoods.

The authors of the press release seem to have counted up a small number of our Truth-O-Meter ratings over a few months, and then drew their own conclusions.

We actually agree with Adair on a few points in his response, but there's still problems with it. Mark Hemingway was quick to spot one of them:

Adair's statement is lawyerly and bordering on dishonest. CMPA did not draw their own conclusions—they simply tallied up all of PolitiFact's ratings during a specific time period to get a representative sample. All the CMPA did was present relevant data, they most certainly did not "draw their own conclusions."

Hemingway is right. Apparently all CMPA did was tally up PolitiFact's ratings and report the results. On the one hand, that's a much less dubious version of fact-checking than anything Adair's ever done. But that's also one of the reasons we don't think it's valuable as a tool to measure PolitiFact's bias.

We've written before about why that method is so flawed. Simply adding up the ratings provides data that has too many plausible explanations to reach a conclusion. The most obvious problem is selection bias, which, by the way, Adair openly admits to in his response:

We are journalists, not social scientists. We select statements to fact-check based on our news judgment -- whether a statement is timely, provocative, whether it's been repeated and whether readers would wonder if it is true.

It's good to see Adair finally admit to selection bias, but we've spoken at length about the numerous other problems with this methodology. The bottom line is a tally of PolitiFact's ratings falls well short of proving ideological bias. A lopsided tally may be consistent with a political bias, but in and of itself it isn't unassailable evidence.

So we agree with Adair that a tally of ratings is a poor measure of which party tells more falsehoods. Of course, that means we also disagree with him.

What Adair forgot to mention in his response to the CMPA study is Adair himself uses the exact same method to promote PolitiFact's ratings:

Instead of traditional articles, our Truth-O-Meter fact-checks are a new form that allows you to see a politician’s report card, to see all fact-checks on a subject or see all the Pants on Fire ratings. We can make larger journalistic points through the automatic tallies and summaries of our work.

You see, when PolitiFact tallies up its ratings, it provides "larger journalistic points" about their work. When academics do it in order to highlight PolitiFact's harsher treatment of the GOP, hey, PolitiFact is just checking specific claims, a tally doesn't tell you anything.
Granted, it's possible Adair misspoke that one time. It's also possible that in 2008 he wrote an article giving readers "tips and tricks on how to find what you want on PolitiFact":

•  Check a candidate's report card  — Our candidate pages...provide a helpful overview. Each one includes a running tally of their Truth-O-Meter ratings, the most recent claims and attacks they've made, attacks against them and their Flip-O-Meter ratings.

Helpful overview? OK. Well, at least Adair knows that a tally of PolitiFact's ratings can't reveal patterns and trends about a candidates truth telling. Right?

Collectively, those fact-checks have formed report cards for each candidate that reveal patterns and trends about their truth-telling.

Wait. What?

The PolitiFact report cards represent the candidates' career statistics, like the back of their baseball cards.

But, but, I thought PolitiFact only checked specific claims and tallies don't show insight about an overall record?

The tallies are not scientific, but they provide interesting insights into a candidate's overall record for accuracy.

Adair is pulling a sneaky trick. In one instance he's claiming a tally of PolitiFact's ratings reveals nothing, yet on multiple occasions he's implicitly telling readers the opposite and claims tallying up their ratings is a valuable tool in order to determine a politician's honesty. Adair cannot simultaneously use that method as both a shield and a sword.

Adair's response to the CMPA study doesn't pass the sniff test. His dismissal of CMPA's method contradicts years' worth of his own lauding of that exact practice. When Adair stops pimping PolitiFact's report cards as if they're some kind of objective referendum on a politician's honesty, we'll be happy to give him kudos. Until then, he's little more than a disingenuous hack trying to have it both ways.



Afters:

At the very least I'll give Adair points for hypocritical chutzpah. The very same day he published his response whining that the study seemed "to have counted up a small number of our Truth-O-Meter ratings" and then "drew their own conclusions", PolitiFact published a recap of Michele Bachmann's record with this admonition:

"At this point, Bachmann's record on the Truth-O-Meter skews toward the red [False]." 

If 100 ratings is a "small number" to draw from for your own conclusions about PolitiFact, then how can 59 ratings tell you anything about Bachmann?

Additional Reading:

-Check out Bryan's piece over at Zebra Fact Check, PolitiFact's Artful Dodger, for a more straightforward analysis of both the CMPA press release and Adair's response.

-We always think Hemingway is worth reading and his take on Adair's response to the study is no exception. He has several good points we didn't mention here.

-You can review our empirical evidence highlighting PolitiFact's liberal bias. Our method avoids the flaws we've seen in so many other studies. You can find it here.