Different Strokes for Different Quotes: What does "voted for tax cuts" really mean?

"What I find is it's hard for me to take critics seriously when they never say we do anything right. Sometimes we can do things right, and you'll never see it on that site."

-PolitiFact Editor Angie Drobnic Holan

Sometimes PolitiFact can do things right.

PolitiFact New York did something right recently that deserves mention because it's the correct way to journomalist:

PolitiFact added the Trump camp "did not get back to us with information supporting his claim, so we can't say for sure what he was talking about in his endorsement."

PolitiFact noted that Trump tweeted about the Tax Cuts and Jobs Act "four other times in May" but acknowledged Trump did not reference that law in the tweet it fact checked.

In our view this is the correct approach.

We think a persuasive argument could be made that Trump inaccurately implied Donovan was a Tax Cut and Jobs Act supporter, but that argument belongs on the editorial page, not in a fact check. PolitiFact examined the claim Trump made without inventing assumptions about what he meant or what he was implying. In this case PolitiFact stuck to the facts.

Notwithstanding our longtime opposition to rating facts on a sliding scale, we think PolitiFact did this one right and we're happy to point it out.


The Other Guy

Readers may wonder "How could a fact checker screw this one up?" Donovan had a documented history of voting for tax cuts, and Trump's claim was not only unambiguous but also easy to check.

How could a serious fact checker get this wrong?

When the Washington Post's unabashed Trump basher/unbiased truthsayer tweeted that "fact checkers sometimes disagree" we were curious. PolitiFact rated Trump's tweet as accurate, while Kessler deemed the exact same tweet false. How can that be?

As it turns out the two fact checkers aren't disagreeing at all.

PolitiFact correctly identified the claim Trump made and ruled based on his actual words. Kessler invented a claim and then gave Trump a false rating for his own fantasy. The fact checkers aren't disagreeing because they're not checking the same claim.

Kessler says Trump's claim that Donovan "voted for tax cuts" is false because "Donovan voted against Trump's tax cut three times." For those of you that aren't experts in journalism or logic, voting against the Tax Cut and Jobs Act does not negate the fact that Donovan has previously voted for other tax cuts.

As far as we can tell, Kessler offered no justification for calling Trump's claim false other than Donovan's opposition to the 2017 tax bill.

Kessler's reasoning here is flatly wrong. And if one wanted to treat Kessler with the same painful pedantry as he applies to Trump in his chart, one could note there's no such thing as "Trump's tax cuts" because only Congress can pass tax bills.

Petty word games aside, this "disagreement" among fact checkers affirms that our fact-divining betters are neither scientific agents of truth nor objective determiners of evidence. When a fact checker can substitute a person's actual words for their own interpretation of what that person meant it counts as commentary, not an adjudication of facts.

Kudos to PolitiFact New York for taking the correct approach. Sometimes PolitiFact can do things right.

(for Glenn Kessler and PolitiFact) How to fact check the job recovery numbers

Originally posted on May 14, 2012 at Sublime Bloviations

A valuable media watchdog watchdog post at the new blog "counterirritant" pointed out a problem with Glenn Kessler's fact check of a claim from Republican presidential candidate Mitt Romney.

Kessler writes the fact checker feature for the Washington Post.  Romney claimed that in a normal recovery from recession the country should be adding something like 500,000 jobs per month.

counterirritant:

Kessler decided that the best way to “check” this was determine how frequently 500,000 jobs were created in a month in the last 65 year.

The post goes on to very effectively criticize Kessler's methodology throughout.

By a funny coincidence (general leftward lean of the mainstream media, maybe?), PolitiFact used very similar reasoning on the same claim:

Is 500,000 jobs created per month normal for a recovery?

The short answer is "no."

We arrived at this conclusion by looking at the net monthly change in jobs all the way back to 1970. Since Romney was referring to total jobs, rather than private-sector jobs only, we used total jobs as our measurement. And since Romney was talking about job creation patterns during a recovery, we looked only at job creation figures for non-recessionary periods, as defined by the National Bureau of Economic Research. Finally, we excluded the current recovery.

The Kessler/PolitiFact method is entirely wrongheaded.

Romney didn't claim that 500,000 jobs created per month was a normal figure during a recovery.  I can imagine the furrowed brows of Kessler, Louis Jacobson and other mainstream media fact checkers.  Aren't they the experts?  What am I talking about?

It's actually pretty simple.

The size of the economy changes.  If country A enters a recession losing 1 million jobs  then it takes two months to regain the lost jobs at a rate of 500,000 per month.  If country B experiences a recession losing 10 million jobs then it takes 20 months to regain them at a rate of 500,000 per month.

Not only does the size of the economy vary, but so does the depth of the recession.  The rate of recovery for lost jobs needs  to account for both factors.  Neither Kessler nor PolitiFact gave any apparent consideration to those critical criteria.  It's like comparing prices between now and the 1950s without adjusting for inflation.

The Romney campaign has made a number of statements like the 500,000 jobs claim, and they probably relate to the following chart or one like it from the Bureau of Labor Statistics:

Clipped from nytimes.com

Romney's claim almost certainly derives from the fact that post-war recoveries usually replace lost jobs much faster than the present recovery. 

So how does one check that claim?  It's not that hard.  Take the bottom point of employment, then count the number of months that it takes to get employment back up to the peak level.  Divide the number of jobs lost by the number of months it took to return to return to the employment peak at the start of the recession.  Do the same for each of the post-war recessions, then average the numbers to obtain the average job recovery time.  After all of that, divide the number of jobs lost from the 2007 recession by the averaged job recovery time.

Why didn't the Washington Post or PolitiFact do anything remotely resembling the fact check I just described?  It could be gross incompetence.  It could be ideological bias.  Or it could be both.


Update 6/7/2012:  Added post title

Mark Hemingway and Glenn Kessler on NPR

Mark Hemingway, who wrote a key critique of modern fact-checking operations back in December, appeared with the Washington Post's fact checker, Glenn Kessler, for a radio interview on NPR.  It's worth either listening to it or reading the transcript, but one particular section deserves special attention:

CONAN: Here's an email from Noreen(ph). I don't understand that the - that since - excuse me. I don't understand the idea that since PolitiFact demonstrates that Republicans lie three times as often as Democrats mean it's biased. Maybe Republicans actually do lie that much more. The idea that you have to have an even number of lies reported for Democrats and Republicans in order to be considered not biased is ridiculous. One side could lie way more than the other. And by trying to make them even, you are distorting fact. Is simple numerical balance an indication of nonpartisanship?

KESSLER: No. I don't look at them that way, and, as I said, I don't really keep track of, you know, how many Democrats or how many Republicans I'm looking at until, you know, at the end of the year, I count it up. My own experience from 30 years covering Washington and international diplomacy and that sort of thing is there's - both Democrats and Republicans will twist the truth as they wish if it somehow will further their aims. I mean, no one is pure as a driven snow here. And I've often joked that if I ever write an autobiography, I'm going to title it "Waiting for People to Lie to Me."

(SOUNDBITE OF LAUGHTER)

CONAN: That's something reporters do a lot. Mark Hemingway?

HEMINGWAY: Why - I think I said when I even brought this up. I mean, you know, I don't think that, you know, that, you know, numerical selection is indicative of, you know, bias per se. I just think that it's highly suspicious. When it's three to one, you know, if it were 60-40, you know, whatever, yeah, sure, you know? But when it's three to one, you start getting things where, you know, you start wondering about, you know, why the selection bias.

Hemingway's December article was quite valuable, but he missed an opportunity to explain an important aspect of Eric Ostermeier's examination of PolitiFact's story selection.

PolitiFact rated about the same number of politicians from each party.  Yet one party received significantly worse "Truth-O-Meter" ratings.   The key inference behind Ostermeier's study was the expectation that a party-blind editorial selection process should be expected to choose the same types of stories for both parties.  The results, then, if Republicans really do lie more, would show approximately the same distribution of ratings but with one party more heavily represented in the total number of stories.  The approximately even number of stories for each group throws the monkey wrench in Noreen's reasoning.

It would have been good if Hemingway had explained that during the broadcast.

As a side note, it's interesting that Kessler likewise ends up writing approximately as many stories about Democrats as about Republicans.  Run the numbers for Kessler as Ostermeier did for PolitiFact and perhaps the tendencies look alike. The obvious reason for focusing on PolitiFact instead of Kessler is PolitiFact's far greater volume of material.



(1/12/12) Jeff adds: There's a flaw that is often overlooked when discussing the "add 'em up" style of interpreting PolitiFact's ratings, and that's the issue of the quality of the fact checks themselves. Assuming PolitiFact actually adheres to an objective formula for avoiding selection bias, and then rates 50 statements from the left and 50 from the right, it still wouldn't disprove an ideological lean.

Take for example the different standards used when PolitiFact rated similar statements from Herman Cain and Barack Obama. Both included the employer's portion of payroll taxes in their respective calculations, but in Cain's case PolitiFact downgraded him for it, while in Obama's case it pushed him higher up the rating scale. And this still doesn't take into account the dishonest tactic of inventing statements out of thin air.

It may be interesting to review the tallies of who gets what ratings and discuss the merits of the numbers. Ultimately though it's the alternating standards that will offer the best evidence of PolitiFact's liberal bias.


(1/19/2012) Jeff adds: An additional flaw with adding up PolitiFact's ratings is the fact that PolitiFact chooses who to give the rating to.

When Obama claimed that preventative health care "saves money", and David Brooks said he's wrong, PolitiFact gave a True to Brooks. This serves the dual purpose of sparing Obama a False on his "report card" that PolitiFact likes to shill so often, while also providing cover in the "we give Republicans True's too!" sense.

When PolitiFact rated the oft-repeated, and false, claim about $16 muffins at DOJ event, PolitiFact could have given the rating to NPR, the New York Times, or even (gasp!) PolitiFact partner ABC News. Instead, they chose to burden Bill O'Reilly with the falsehood, despite the original claim coming from a government report.

It's these types of shenanigans that will always distort a ratings tally.

Update/clarification (1/14/2012):

Added "for a radio interview on NPR" to the first sentence.