PolitiFact editor: Principles developed "through sheer repetition"

PolitiFact editor Angie Drobnic Holan this week published her ruminations on PolitiFact's first 10 years of fact-checking.

Her section on the development of PolitiFact's principles drew our attention (bold emphasis added):

We also have made big strides in improving methodology, the system we use for researching, writing and editing thousands of fact-checks, more than 13,000 and counting at PolitiFact.com.

Through sheer repetition, we’ve developed principles and checklists for fact-checking. PolitiFact’s Principles of the Truth-O-Meter describes in detail our approach to ensuring fairness and thoroughness. Our checklist includes contacting the person we’re fact-checking, searching fact-check archives, doing Google and database searches, consulting experts and authors, and then asking ourselves one more time what we’re missing.

The line to which we added bold emphasis doesn't really make any sense. One develops principles and checklists by experience and adaptation, not by "sheer repetition." Sheer repetition results in repeating exactly the procedures one started out with.

PolitiFact's definitions for its "Truth-O-Meter" ratings appear on the earliest Internet Archive page we could load: September 21, 2007, featuring the original definition of "Half True" that PolitiFact not-so-smoothly dumped around 2011. So the definitions do not appear to have resulted from "sheer repetition."

The likely truth is that PolitiFact developed an original set of principles based on what probably felt like careful consideration at the time. And as the organization encountered difficulties it tweaked its process.

Does the contemporary process count as "big strides" in improving PolitiFact's methodology?

We're not really seeing it.

When PolitiFact won its 2008 Pulitzer Prize for National Reporting, one of the stories among the 13 submitted was a "Mostly True" rating for Barack Obama's claim that his uncle had helped liberate Auschwitz. Auschwitz was liberated by the Soviet army. Mr. Obama's uncle was not part of the Soviet army. A false claim received a "Mostly True" rating.

This week, PolitiFact California issued a "Mostly True" rating for the claim a National Academy of Sciences study found undocumented immigrants commit fewer crimes than native-born Americans. If PolitiFact had looked at the claim in terms of raw numbers, it would likely prove true. Native-born Americans, after all, substantially outnumber undocumented immigrants. Such a comparison means very little, of course.

PolitiFact California simply overlooked the fact that the study looked at immigrants generally, not undocumented immigrants. We wish we were kidding. We are not kidding:

We started by checking out the 2015 National Academy of Sciences study Villaraigosa cited. It found: "Immigrants are in fact much less likely to commit crime than natives, and the presence of large numbers of immigrants seems to lower crime rates." The study added that "This disparity also holds for young men most likely to be undocumented immigrants: Mexican, Salvadoran, and Guatemalan men.

While the latter half of the paragraph hints at data specific to undocumented immigrants, we should note two important facts. First, measuring crime rates for Guatemalan immigrants in general serves as a poor method for gauging the criminality of undocumented Guatemalan immigrants. The same goes for immigrants from other nations. Second, PolitiFact California presents this information as though it came from the NAS study. In fact, the NAS study was summarizing the findings of a different study.

Neither study reached conclusions specific to undocumented immigrants, for neither used data permitting such conclusions.

Yet PolitiFact California found the following statement "Mostly True" (bold emphasis added):

"But going after the undocumented is not a crime strategy, when you look at the fact that the National Academy of Sciences in, I think it was November of 2015, the undocumented immigrants commit less crimes than the native born. That’s just a fact."

False statement, "Mostly True" rating.

If PolitiFact has learned anything over the past 10 years, it is that it can largely get away with passing incompetent fact-checking and subjective ratings on to its loyal readers.