Tuesday, April 26, 2011

Ocean State Policy Research Institute: "Still Leaving Rhode Island"

The Ocean State Polity Research Institute recently published a detailed rebuttal of PolitiFact's assessment of OSPRI's earlier report, "Leaving Rhode Island."  The latter report floated the hypothesis that tax policy was responsible for eroding the number of wealthy residents in Rhode Island.

I don't know whether OSPRI definitively rebuts PolitiFact.  The nature of the material suggests that authoritatively refereeing the debate would require more time than I am willing to devote to the project (and might perhaps exceed my ability).

There is one part of the debate, however, that is quite clear-cut.  That portion occurs clearly when the PolitiFact editor responds to an op-ed that summarizes the case made in OSPRI's rebuttal (OSPRI's Bill Felker wrote the op-ed):
PolitiFact editor Tim Murphy replies: The OSPRI report prominently stated that “The most significant driver of out-migration [from Rhode Island] is the estate tax.” Our ruling, online at ProJoPolitiFact.com, was based largely on OSPRI’s own data, along with assessments from other experts. It was given a “false” rating because, in PolitiFact Rhode Island’s view, the report did not prove the case.
PolitiFact Rhode Island relied on PolitiFact's "Burden of Proof" criterion in making its "False" ruling.  As I have repeatedly emphasized, PolitiFact's use of its burden of proof criterion serves as an example of the fallacy of the appeal to ignorance (PolitiFact doesn't know x to be true, therefore x is false).

Illustrating the absurdity of PolitiFact's conclusion is simple.  PolitiFact does not know x is false, or else no fallacious reasoning would be required to make the determination.  Using the same reasoning PolitiFact uses to rate OSPRI's claim "False" we can likewise rate PolitiFact's claim (that OSPRI's claim is false) as "False."

The same claim cannot be false and not-false (~false) at the same time and in the same sense.  That state of affairs is contradictory, and rightly regarded as an impossibility.

In terms of bias, this case concerning OSPRI simply constitutes another anecdote.  PolitiFact's flawed system is not politically biased, just flawed.  But where the flaw is applied unevenly, measurably affecting one ideology more than another, we have evidence of an ideological bias.

Monday, April 25, 2011

Sublime Bloviations: Michele Bachmann and Taxing the Top 1 Percent

PolitiFact Bias editor Bryan White recently exposed a blatant inconsistency in PolitiFact's treatment of their favorite target, Michele Bachmann, and Ohio Democrat Marcia Fudge.

Over at his blog, Sublime Bloviations, Bryan takes us on a trip back to a forgotten time historians refer to as "March":
Remember the good old days when it was easy to tell what type of taxes a politician was talking about? When Marcia Fudge said many large corporations pay no taxes at all, a journalist could simply assume that she was talking about income taxes and do a fact check based on that assumption.

Things are more complicated these days with evil Republicans in office.
More complicated indeed.

It seems Bachmann went on the the Today show and made a statement that PolitiFact just had to check:
"Well, remember, again, already the top 1 percent of income earners pay about 40 percent of all taxes into the federal government. So if you want to talk about fairness, the top 1 percent are paying 40 percent of all of the income."
Seems like a pretty straightforward statement to rate. And any news outfit with objective and consistent standards would be able to determine the validity of Bachmann's comments quite easily. Unfortunately, as Bryan notes, PolitiFact doesn't have that luxury:
In the bygone days of yore, that would be "40 percent of all income taxes into the federal government," or at least the interpretation would amount to the same thing.
Bryan goes on to explain in detail the dual methodology PolitiFact uses for the different fact checks of Bachmann and Fudge, and also brings up yet another example of their rigid as prairie grass recognition of "misspeaking":
We can thus continue to build our picture of the PolitiMath theorem by noting that PolitiFact takes a range of error from 42 to 76 percent as a "False" numbers claim, setting aside for the moment the fact that the most important thing about a numbers claim is the underlying message. If Bachmann's underlying message was that the top 1 percent pay much more than 1 percent of the total tax income for the U.S. then she's out of luck this time. She's likewise out of luck if she misspoke.

PolitiFact:
Bachmann would have been right if she’d said, "the top 1 percent of income earners pay about 40 percent of all income taxes into the federal government." But she didn’t say that -- and even if she had, her decision to focus on income taxes, rather than looking at the whole federal tax picture, would have presented the numbers in such a way that wealthier Americans would look more heavily taxed than they are.
Likewise, Fudge would have been right if she'd said "There are corporations in this nation, some of the biggest corporations in this nation, who do not pay income taxes when they fail to show a profit after claiming their deductions." Oh, wait. PolitiFact rated Fudge "True" for her inaccurate statement and Bachmann "False" for hers. That's some pretty impressive inconsistency.
With PolitiFact we're finding out the only consistent thing is their inconsistency. Bryan does a superb job of exposing that. For readers interested in the comprehensive review of the differences, we recommend you read the entire post.

Correction 4/27/11: Hat Tip to alert reader Ron Binns for pointing out I had goofed by referring to Ohio Congresswoman Marcia Fudge as "Martha". Whenever errors occur our policy is to make corrections transparent and acknowledge them once they're fixed. Fudge's name has been updated. Thanks Ron!-Jeff

Saturday, April 23, 2011

PFB Smackdown: The TPM critique of PolitiFact

Joshua Marshall of Talking Points Memo joins the fray with his own critique of PolitiFact.  Marshall's critique, like that of Jonathan Chait reviewed here earlier, focuses on PolitiFact's grading of a Democratic Party ad attacking Republican policy on Medicare.  And even more specifically, Marshall criticizes the portions of a recent post at Columbia Journalism Review (also reviewed here) that mostly defended PolitiFact's reasoning while questioning the rating system itself.

TPM:
Did Republicans vote to 'end Medicare'?
Err, not really. As already mentioned, Republicans did not, as the ad suggests, vote to end Medicare. Rather, they voted--in the lower house--for a plan that would change Medicare, were it to reach the president's desk and be signed into law. Which it won't. The ad mentions none of this, instead leaving its bold claim hanging like a piñata for PolitiFact's batsmen.
'End Medicare' is the heart of the question. And as I've already repeatedly noted, ending a program that functions in one way (single-payer guaranteed medical insurance regardless of health status) with one that works in a fundamentally different manner (provide limited subsidies for private insurance which would quite possibly not exist for many seniors) and doesn't provide anything like the same service by any definition counts as 'ending' the program regardless of whether you give the latter program the same label. But look at the reasoning in the excerpt. One of the reasons the claim isn't 'true' apparently is because only the House voted for it so far, not the Senate. And the President would still have to sign it. And he probably won't. So since it likely won't become law in this Congress, House Republicans aren't even really voting to do it.

By that standard, Bernie Sanders doesn't really support single-payer because it's never going to become law.
 Let's look at Marshall's reasoning regarding the reasoning from PolitiFact and CJR.

"End Medicare" may be the heart of the question as Marshall says, but "vote to" is a big part of the claim as well--probably the bigger part of the claim since it constitutes the action supposedly taken without which "end Medicare" doesn't happen.

Marshall argues that Medicare under Ryan would be "fundamentally different" from Medicare, but isn't clear that private Medicare insurance is "fundamentally different" than Medicare single-payer insurance, contrary to what Marshall seems to contend.  Both, after all, qualify as insurance.  And suppose we cut Medicare funding under the present system to 10 percent of what Ryan's plan would pay.  Would Marshall argue that Medicare ceases to be Medicare at some point on the money/benefit continuum?  Or did Medicare stop being Medicare the moment it started paying for hip replacements?  Marshall doesn't delve into the principle, leaving his argument akin to a "slippery slope" argument.  Medicare is fundamentally a health insurance program for the elderly.  The name, in fact, is meant to express that idea and when it was coined Medicare was a far more limited program than it is today.  Marshall then extends the argument by suggesting that Medicare under Ryan "doesn't provide anything like the same service by any definition," but that's just hyperbole if he isn't simply lying. Marshall's case ultimately rests on his opinion.

Marshall also attacks the CJR for reasoning the bill will not reach the Senate.  But he apparently misunderstands the situation.  The bill the House passed is non-binding.  So even if the Senate passed it and the president signed it (which never happens with concurrent resolutions), it wouldn't carry the force of law.  Binding bills do that.

Marshall's argument is ridiculous.  I suppose he can try to blame Joel Meares of CJR for not explaining the details, but I consider Meares within his rights to assume a decent level of civics knowledge for his readers.


April 26, 2011:  Changed "Democrat Party" to "Democratic Party" in the opening paragraph.

Friday, April 22, 2011

CJR's Campaign Desk: "Testing the Truth-O-Meter"

Amid the controversy over a recent PolitiFact rating of a Democratic Party Medicare ad, Joel Meares of the "Campaign Desk" at Columbia Journalism Review recycles a handful of my stock criticisms of PolitiFact in an April 21 post:
(M)ore interesting than the details of this mini controversy about a standard-issue political ad are the questions it raises about the PolitiFact method and the use of its Truth-o-meter. The truth, or at least the truthiness, of some matters, seems to lie in degrees that Adair’s innovative six-level gauge may not quite capture.
That sounds a bit familiar ("Unless PolitiFact comes up with a more precise way of grading, the "Truth O Meter" should be tossed on the scrap heap"--June 1, 2008).

Meares:
But PolitiFact’s “Pants on Fire” assessment still feels unduly harsh for some reason. What makes this, for instance, not simply “False”? Or, given that it is a missing phrase away from seeming quite reasonable, “barely true?” Or, if you don’t buy that—that the numbers are wrong and the vote is not binding is pretty damning—wouldn’t misleading be a more accurate description than “outright lie”?
Meares doesn't sufficiently credit PolitiFact for its trickiness.  It doesn't label anything as an "outright lie" unless one counts the annual "Lie of the Year."  The "Pants on Fire" label denotes a "ridiculous" claim while communicating "lie!" based on the "liar, liar, pants on fire" rhyme that all children seem to learn at some point.  As for the difference between "False" and "Pants on Fire," I've regularly pointed out that devising an objective method of determining the property of ridiculousness is probably beyond human capacity.  In addition, PFB highlighted a complementary assessment of PolitiFact from the "Engineering Thinking" blog where Ed Walker made similar points.

Bottom line:  The "Truth-O-Meter" is subjective.  We shouldn't be surprised that it doesn't operate consistently.

Unfortunately, Meares whiffs on his opportunity to slam another PolitiFact problem:
The concise rulings, says Adair, “are a tremendous reader service.” They cut through the political rhetoric to issue concise judgments and when pooled, like on this page breaking down the statements of Michele Bachmann, they can reveal quite a lot about a candidate or organization’s commitment to accuracy. “The tradeoff,” admits Adair, “is that sometimes complexity doesn’t fit neatly into our six ratings.”
There it is.  Adair is still advocating using the collected rulings for individuals as a type of report card as though PolitiFact is immune to selection bias.  Meares ought to subject that idea to some hard questions like those Eric Ostermeier has asked.

Though Meares is probably too easy on PolitiFact and PolitiFact's Bill Adair, he puts a helpful focus on PolitiFact's flawed methods--methods that help amplify newsroom bias in PolitiFact's fact checking.

Thursday, April 21, 2011

Jonathan Chait: "PolitFact Goes Off The Deep End"

Make no mistake about it, Jonathan Chait's a liberal.  So why bother with his story when PolitiFact Bias normally puts its focus on evidences showing PolitiFact's bias against conservatism?

Because even though Chait is late to the party when it comes to criticizing PolitiFact's bias, he has at least something of a point.  Sure, the survey on which Chait probably relied in making his claims about the conservative bias of journalists has its share of flaws.  But Chait's instinct to look for journalists' ideology to affect their work is reasonable on its face.

So what took Chait so long to detect PolitiFact bobbing in the deep end?  Does he want a Pulitzer Prize or something?

Good luck to Chait on that one.  I was torn between simply highlighting Chait's story and doing a "PFB Smackdown" entry on it.  Chait offers no criticisms of this story that can't be made a hundred times over and more for stories that harm conservatives.  It speaks to Chait's ideological bias that he succeeded in containing himself this long.

In the end, Chait's story is short on substance.  His concluding paragraph illustrates:
Wait -- it's a lie to say Republicans voted to end Medicare because the vote hasn't been signed into law? Lawmakers can be held accountable for votes only after those votes are signed into law? This is so ridiculous I can't believe Politifact is arguing it. The whole analysis is hard to interpret as anything other than an expression of the view that criticizing any proposal that reduces spending on Medicare or Social Security is inherently foul play.
Chait's a smart guy. He knows that a non-binding budget resolution doesn't move on through the legislative process and turn into law. Presidents never sign them. So, yes, it's flatly misleading to say that the Republicans voted to end Medicare because the resolution is non-binding while the ad implies the reverse.

The key to stiff-arming the charge of bias--for PolitiFact and Chait alike--is the consistent application of standards.  Chait's story would suggest that PolitiFact has hit some type of new low.  But that's ridiculous in light of examples such as PolitiFact's 2010 "Lie of the Year" selection.

Was the PolitiFact evaluation of the ad biased?  It's plausible in spite of Chait's rather weak argument to that effect.  But in any case the full picture of PolitiFact's bias emerges not from single anecdotes but from a broad range of evidences.

May I suggest an alternate title for Mr. Chait's column: "PolitiFact Gored My Ox!"

Wednesday, April 20, 2011

Michael F. Cannon: "Why I'm Boycotting PolitiFact"

Michael F. Cannon of the CATO Institute has often served PolitiFact as an expert source.  Today he explained why he no longer cooperates with PolitiFact in that capacity:
Reporters at PolitiFact.com have used me as a resource half a dozen times or so when fact-checking something someone said about health care reform. Sometimes we disagree about where the truth lies, but I’ve always been happy to help. That changed recently, and I should let PolitiFact’s reporters know why.
read it all
Cannon's boycott is conditional:   He will lift it after he is satisfied that PolitiFact has addressed problems with its past two "Lie of the Year" selections.

I'll quote one more important passage, though Cannon's column ought to be read completely through:
Some conservatives think PolitiFact is a left-wing outfit. I don’t think that’s true, and I have defended PolitiFact against that charge. I believe that PolitiFact’s reporters are earnestly doing their best to get at the truth. But there’s a tension between that belief and these errors. Whether PolitiFact recognizes and addresses that tension will tell us a lot about PolitiFact.
I share Cannon's assessment.  It is extraordinarily unlikely that PolitiFact intends to do its fact checking any way other than impartially.  But on the other hand it's hard to reconcile that premise with some of the things PolitiFact puts to print and with some of the errors PolitiFact refuses to acknowledge.

I predict that Cannon will not end up lifting his boycott.  PolitiFact is too invested in its model, given its relative success, to risk tampering with it.

But I'd love to be proved wrong.  Just like Cannon wouldn't mind lifting his boycott under the right conditions.

Tuesday, April 19, 2011

Win a Pulitzer--for criticizing Pulitzer Prize-winning PolitiFact

PolitiFact's defenders love to point to the Pulitzer Prize for national reporting PolitiFact took home for a collection of 13 stories published in 2008.  People apparently fail to realize the limited scope of judgment brought to bear by Pulitzer juries.

Fortunately, the Pulitzer folks have made it dead easy to turn that argument into a flaccid pretzel:  James Rago of the Wall Street Journal won the 2010 Pulitzer Prize for Editorial Writing--and part of the collection he submitted contained his column "PolitiFiction," where he criticized PolitiFact's choice for its 2010 "Lie of the Year."

Congratulations to Mr. Rago.

Rago's column on PolitiFact was one of the first stories highlighted here at PolitiFact Bias.

Hat tip to Big Gov Care.

James Wigderson: "We rate it a full flip for Politifact"

Wisconsin's James Wigderson serves up a potent shot at PolitiFact at his Library and Pub.  After noting PF's correction of a "Flip-O-Meter" piece offered no hint of the rationale for changing the rating, Wigderson makes a helpful recommendation:
Perhaps after reversing this Politifact review the newspaper will take the opportunity to look back at some of the other “Politifacts” to test their accuracy. Some of them, written about candidates on both sides of the aisle, were just awful examples of bias. (The Politifacts on Charlie Sykes and Jim Sullivan spring to mind.)
PolitiFact's penchant for stuffing failed stories down the memory hole represents a disturbing trend for an outfit that is already stingy with its transparency.  In the midst of 2010 PolitiFact revised a flawed story but kept the old version archived.  That seems better in terms of transparency.  Was it a change of policy or the same old chaos we're accustomed to seeing from PolitiFact?

Do stop by Wigderson's Library and Pub to savor the entire (brief) post.

Friday, April 15, 2011

Bewz Newz 'n' Vewz: "One Of These Facts Is Not Like The Others"

PolitiFact Bias collaborator Jeff Dyberg illuminates an additional dimension of PolitiFact's selection bias problem through an examination of the treatment given Rep. Dennis Kucinich (D-Ohio).  Kucinich performed some notable gum-flapping over how President Obama operated outside the law with his Libyan intervention.  PolitiFact ignored the accusation and instead dug up Kucinich's month-old comment that "To my knowledge, none of these existing programs appeared on the GAO's list of government programs at high risk of waste, fraud and abuse."

Jeff's summary of the problem:
What this teachable moment has exposed is that PolitiFact's bias isn't always as obvious as simply picking more statements on the left or the right or rating one group more harshly than the other. Their bias is evident in the specific claims and specific people they choose to rate as well as the things they choose to ignore.
This type of bias is one of the toughest to quantify, but examples such as the one Jeff offers nonetheless help build a cumulative case showing PolitiFact's affinity for the political left.

You can read the whole article here.

Big Journalism: "2012 Preview: Who Will Fact-Check the Corrupt MSM’s Cherished PolitiFact?"

Big Journalism's John Nolte forecasts PolitiFact's role in the 2012 election, using a rambling account of the organization's checkered past.  Nolte's dead-on with his assessment that PolitiFact turned popular liberal opinion into "fact" with its "Lie-of-the-Year" selections and provides music to our ears with his call to hold PolitiFact to account:
We are all activists now. Get involved. Call these people out. Spread the word.

At the risk of sounding corny, this is about our country and it is everyone’s job to watch the self-appointed Watchmen.
Hear, hear.

Thursday, April 14, 2011

Dual hat tips

Hat tips to Big Journalism's John Nolte and WTMJ's Charlie Sykes for posting hotlinks to our site.  We also appreciate the Twitter connections.

PolitiFact keeps flubbing up faster than we can write, so watch for updates.  And those of you who are so inclined, join in on the national critique of PolitiFact.  If you blog a critique of PolitiFact, drop us a line about it.

Tuesday, April 12, 2011

Charlie Sykes: "Is PolitiFact's Dave Umhoefer a Liar, Liar?"

Wisconsin radio personality Charlie Sykes of WTMJ in Milwaukee has discovered the hypocrisy inherent in PolitiFact's evaluation system.  Sykes received a "Pants on Fire" rating from PolitiFact Wisconsin for his claim that the Kloppenberg side outspent the Prosser side by a 3:1 margin in the recent Wisconsin supreme court election.

Sykes put his finger right on the problem ("he" is PolitiFact writer Dave Umhoefer, "they" is PolitiFact):
So, in other words: he did NOT verify the Brennan Center numbers. Instead they relied on an overtly left wing source's "estimate." (Try to imagine the JS ever relying on a Koch Bros. financed group's uncorrobrorated estimates to label a liberal a "liar, liar."

They did NOT verify the Kloppenburg numbers.

They did not include possibly big outside spending by Big Labor.

And they decline to offer any other hard data to support their story... the sort of documentation they demand from others and which they use to declare that others are "liars, liars."

Ironic.
(all caps emphasis in the original)
Though evidently quite ready to admit a error if proved, Sykes makes a good case in noting that PolitiFact holds a different standard for its claims of truth compared to those whose statements it rates.

It's called hypocrisy, and it is expressed directly in PolitiFact's statement of standards:
Burden of proof -- People who make factual claims are accountable for their words and should be able to provide evidence to back them up. We will try to verify their statements, but we believe the burden of proof is on the person making the statement.
...except when the person making the statement is a PolitiFact writer.

Sykes' story is especially instructive as he quotes his correspondence with PolitiFact staffers, so please read the whole thing.  My experience has been similar to his.  PolitiFact staffers do not readily acknowledge problems with their process or results.

LifeNews.com: "Politifact Misleads in Bashing Jon Kyl Over Planned Parenthood"

LifeNews.com, an obviously pro-life/anti-abortion (depending on your POV) organization, published a criticism of PolitiFact's review of a recent statement by Sen. Jon Kyl (R-Ariz.) that jibed considerably with one I published very recently:

“Planned Parenthood calculates the numbers by services provided, rather than dollars spent,” the web site said. “By this tally, abortions accounted for just under 3 percent of the procedures Planned Parenthood provided in 2009, which is the most recent year for which the group is reporting statistics. And that would make Kyl’s statement way off. Kyl has vastly overstated the share of abortions.”

Kyl’s office responded to the criticism in a statement saying his remarks were meant to “illustrate that Planned Parenthood, an organization that receives millions in taxpayer dollars, does subsidize abortions.”

But Planned parenthood’s own figures show Kyl was right in the sense that Planned parenthood is primarily an abortion business and, for pregnant women, abortion is the only option they essentially present.
Do remember that quality content ought to trump charges that LifeNews.com is biased on the topic.  Just as low quality content from PolitiFact contributes to the impression that PolitiFact is biased on the topic rather than impartial.