The clueless guru?

Late last month, we published a limited study on PolitiFact's execution of a simple math problem: calculating percentage error.  Using search parameters that suitably simulate randomness, we found 14 cases where PolitiFact explicitly or implicitly performed a percentage error equation.  PolitiFact used the wrong equation an astounding nine times.  Two of the cases were ambiguous.  Those two we gave the benefit of the doubt.

We tweaked PolitiFact over this failure on June 14 after Neil Brown, editor and vice president of PolitiFact's parent the Tampa Bay Times, called PolitiFact editor Angie Holan a "guru of best practices" in a June 9 tweet.  We said a guru of best practices would do percent error calculations the right way.

On Friday, June 27, 2014, PolitiFact doubled down on its methods in a fact check of President Obama.  President Obama said child care costs more than college tuition in 31 states.  PolitiFact, with veteran staffers Louis Jacobson writing and Holan editing, said the president was cherry picking and eventually gave him a "Mostly True" rating.

PolitiFact's explanation of Obama's cherry-picking caught our attention:

It’s worth noting some clarifying language in the report --"for an infant in center-based care" -- that is absent from Obama’s statement. This is actually the highest-cost example of the four cases the report looked at.

If you look at the cost for a 4-year-old in center-based care -- rather than an infant -- it costs more than in-state college tuition and fees in 19 states. That’s 39 percent fewer states compared with statistics for infant care. (Generally, care for infants is more intensive, so costs tend to go down as children get older.)

The report also looked at costs for home-based care, which is often a less expensive option for parents. For infants, the cost of home-based care is higher than college costs in 14 states. That’s a 55 percent reduction in states compared to Obama’s 31.

And for 4-year-olds, the cost of home-based care is higher than college in 10 states. That’s a 68 percent reduction in states compared to Obama’s 31.

What's the problem?  One could argue there's no right figure here to use as a baseline for a percent error calculation, except the same principle holds true for calculating a percentage change from a baseline.  And in this fact check we've got a charge of cherry-picking.  Cherry-picking creates a favorable impression compared to alternative baselines.  Calculating the exaggeration above the baseline is exactly like calculating the percentage error.

And guess what?  PolitiFact consistently performs the calculation incorrectly in a way that makes Obama look better.

1. For the 4-year-old group, PolitiFact said the cost was higher for child care in 19 states, 39 percent fewer than the figure Obama used:  31.  Do the calculation using 19 as the baseline and the result tells the effect of Obama's cherry-picking.  The real exaggeration Obama achieves is 63 percent.  PolitiFact's method underestimates the exaggeration by 38 percent (24 percentage points).
2. For home-based care of an infant, the result follows the same pattern.  PolitiFact said the difference was a 55 percent reduction.  In truth, Obama's cherry-picking inflated the number of states by 121 percent.  PolitiFact's calculation reduced Obama's exaggeration by about 55 percent.
   
3. For home-based care of 4-year-olds we see the same story again.  PolitiFact called the difference "a 68 percent reduction."  Using the cost of home-based care for 4-year-olds as the baseline, we find Obama's cherry-picking exaggerates the number of states by 210 percent.  PolitiFact reduces Obama's exaggeration in this case by 68 percent.

The group Obama chose to cherry-pick provided by far the largest group of states.  Any averaging with the other figures from PolitiFact's source, Child Care Aware of America, will lower the figure, especially if we also consider the school-age category that PolitiFact fails to mention.  The costs for that group were lower than for infants and 4-year-olds.

Rigged.

The percentage figures PolitiFact provides do nothing to explain the effects of Obama's cherry-picking.  Instead, they arbitrarily tell the relationship in size between two numbers, doing it in a way that ultimately misleads readers.

It's easy to see what happened with Obama's misstatement.  Obama's figure matches exactly the figure Child Care Aware of America published for four-year-olds receiving child-care services at a center.  Except Obama described the figure incorrectly.  An average for all three groups, considering both center-care and home-care, would render Obama's statement literally false.  He'd be just another politician who described a study using the wrong words, except PolitiFact goes easier on some than it does on others.  Obama's statement is literally false (off by no less than 63 percent).  It misleads his audience.  He gets a "Mostly True" from PolitiFact.

If these are its best practices then PolitiFact needs a new guru.