Thursday, June 27, 2019

Selection Bias, Magnified

How PolitiFact uses inconsistent application of principles to help Democrats, starring Beto O'Rourke


PolitiFact Bias has repeatedly pointed out how PolitiFact's selection bias problem serves as a trap for its left-leaning journalists (that likely means somewhere between most and all of them). Left-leaning journalists are likely to fact check suspicious claims that look suspicious to left-leaning journalists.

But beyond that left-leaning journalists may suffer the temptation of looking at statements through a left-leaning lens. Fact-checking a Democrat may lead to confirmation bias favoring the Democrat's statement. The journalist may, perhaps unconsciously, emphasize evidence confirming claims coming from liberal sources. Or cutting the fact-finding process short after finding enough to supposedly confirm what the Democrat said.

When Democratic presidential hopeful Beto O'Rourke claimed to have received more votes than any Democrat in the history of Texas, PolitiFact Texas fact-checked the claim and found it "True."

Note that the fact check was written by long-time PolitiFact staffer Louis Jacobson. PolitiFact National employs Jacobson.

It is literally true that O'Rourke received the most votes for a Democrat ever received in the state of Texas. But literal truth is rarely the benchmark for fact checkers. In this case, we immediately noticed a problem with O'Rourke's claim that typically causes fact-checkers to find fault: As the number of voters in Texas grows, the number of raw votes received shrinks in significance. Measuring the percentage of the total vote (48.3 percent for O'Rourke) or the percentage of registered voters (about 25.6 percent) offers a more complete picture of a candidate's electoral strength in a given state.

For comparison, President Jimmy Carter won Texas in 1976 with 2,082,319 votes. Carter's percentage of the vote was 51.1 percent. His percentage of registered voters was 31.2 percent. It follows that Carter's performance in Texas was stronger than O'Rourke's even though Carter received about half as many votes as O'Rourke received.

We pointed out the problem to a PolitiFact Texas employee on Twitter. PolitiFact elected not to update the story to address O'Rourke's potentially misleading point about his electoral strength.

But it's justified resisting the efforts of conservatives to "work the refs," right? Who would think of trying to put the number of votes in context like we did other than right wing zealots?

Try the BBC, for starters. BBC noted that Hillary Clinton received the most presidential votes in history, then promptly tempered that statement of fact with a caveat:
So the proportion of Clinton votes might be more illuminating than simply how many votes she earned.
Indeed. And even PolitiFact Texas devoted more than one paragraph to the context O'Rourke had left out. Yet PolitiFact had the left-leaning sense not to let that missing information interfere with the "True" rating it bestowed on O'Rourke.
Our ruling

O’Rourke said that in 2018 when he ran for senator, "young voter turnout in early voting was up 500%. We won more votes than any Democrat has in the history of the state of Texas."

His assertion about young voter turnout is backed up by an analysis of state election data by the firm TargetSmart. And he’s correct that no Democrat has ever won more raw votes in a Texas statewide election than he has, an accomplishment achieved through a combination of his own electoral success, a pro-Democratic environment in 2018, and Texas’ rapid population growth in recent years.

We rate his statement True.
PolitiFact does not count the missing information significant, even though it was apparently significant enough to mention in the story.

Partial review of PolitiFact's rating system:
MOSTLY TRUE – The statement is accurate but needs clarification or additional information.

HALF TRUE – The statement is partially accurate but leaves out important details or takes things out of context.
If O'Rourke's statement did not need clarification or additional information, such as the growing number of voters in Texas, then why did PolitiFact provide that clarifying information?

These gray area "coin flips" between ratings offer yet another avenue for left-leaning fact-checkers to express their bias.

PolitiFact has never revealed any mechanism in its methodology that would address this weakness.

Thursday, June 13, 2019

Transparency: How to access PolitiFact's page of corrected or updated fact checks (updated x2)

Note: Page now obsolete, see Update 2 at the bottom

It has long amused us here at PolitiFact Bias how difficult PolitiFact makes it for readers to navigate to its page of corrections and updates. There are pretty much three ways to navigate to the page.


Someone could link to it by hotlinking using the page URL.

This is the method PolitiFact uses to make finding the page seem easy-peasy in tweets or other messages. Works great!



The reader could use a search engine to find it

No, not the search function at the PolitiFact website. That will not get you there.

We're talking about a search engine like Google or DuckDuckGo. Search politifact + corrections + and + updates and reaching the page is a snap.


The reader could navigate to the page from PolitiFact's homepage. Maybe. 

This is the amusing part. We've already noted that using the "search" function at the PolitiFact website won't reach its dedicated page of corrected and updated fact checks (other corrections and updates do not yet end up there, unfortunately).

And without a guide such as the one that follows, most people browsing PolitiFact's website would probably never stumble over the page.

How To Do It

Step 1: On the homepage, move the cursor to the top menu bar and hover over "Truth-O-Meter" to trigger the drop-down menu
Step 2: Move the cursor down that menu to "By Subject," click on "By Subject"
Step 3: On the "Subjects" page, move the cursor to the alphabet menu below the main menu, hover over "c," click "c"
Step 4: Move the cursor to the subjects listed under "c," move cursor to hover over "Corrections and Updates," click "Corrections and Updates"

Done! What could be easier?

The key? Knowing that PolitiFact counts "Corrections and Updates" as a category of "statements" defined by PolitiFact as Truth-O-Meter stories. The list of corrections and updates consists only of fact checks. Corrections or updates of explainer articles, promise ratings and flip-flop ratings (etc.) do not end up on PolitiFact's page of corrections and updates.

What you'll find under "c" at PolitiFact.com



Afters


When I (Bryan) designed the Zebra Fact Check website, I put the "Corrections" link on the main menu.



It's not all about criticizing PolitiFact. It's also about showing better and more transparent ways to do fact-checking.

This isn't exactly rocket science. Anybody can figure out that putting an item on the main menu makes it easy to find.

There is reason to suspect that PolitiFact is less than gung-ho about publicizing its corrections and updates.


Update Aug. 5, 2019
: We do have evidence of promise ratings appearing on the list of corrected stories."Flip-O-Meter" stories have subject tags, so we assume those may appear on the list as well.

Update 2, Feb. 2, 2020: PolitiFact updated its website on Feb. 1, 2020, making our instructions obsolete. Details here.