Thursday, March 29, 2012

PolitiFact rolls out new feature, continues to ignore selection bias

Crossposted from Sublime Bloviations


I've belabored the point that PolitiFact pushes its candidate "report cards" without owning up to the selection bias that is overpoweringly likely to skew the grades.

PolitiFact editor Bill Adair has been asked about selection bias.

Bill Adair probably (based on IP address trails) reads commentary here and at PolitiFact Bias [more appropriately "here and at Sublime Bloviations" after crossposting] regarding PolitiFact's problem with selection bias.

Yet minutes ago we get this:
It's spring, which means it's report card time. So we're unveiling a new feature that allows you to compare the PolitiFact report cards for individuals and groups we check.

Our report cards have always been a popular feature and often generate interesting commentary. Now, you'll be able to compare the report cards more easily.
Adair still won't inform his readers that the process leading to the report card grades is rife with selection bias problems.  Yes, Adair at least linked to a New York Times blog ("interesting") that provided the minimum type of disclaimer that PolitiFact should offer.  The other link ("commentary") was the sort of pointless statistical exercise that simply elaborates on the results of PolitiFact's fundamentally flawed process (the former link I gave a positive review, the latter author I've given a less-than-positive review).

Bottom line, PolitiFact continues to publish candidate "report cards" that appear minus critical context.  PolitiFact (allegedly) rules statements missing critical context "Half True."

PolitiFact apparently knows about its selection bias problem and is deliberately downplaying it.  Selection bias is not merely "interesting."  It is critical to an accurate understanding of the meaning of PolitiFact's "report cards."


Updated 3/29/2012:  Added editor's note in third paragraph

Wednesday, March 28, 2012

Hoystory: "Also known as 'Promise Kept'"

When does a promise qualify as being kept?  It depends.

Matthew Hoy of Hoystory highlights the stratospheric standards of the self-appointed Valuator of Vows. This time PolitiFact checks in on John Boehner's promise to fly commercial planes as opposed to military aircraft (a'la Nancy Pelosi).

What titillating travel tidbits did PolitiFact uncover?
[Boehner] spokesman, Michael Steel, told The Hill that Boehner had no intention of reconsidering a decision he previously made to forgo the use of a private jet for transportation to and from his Ohio district.

When we contacted Steel about this pledge, he said Boehner still flies commercial, and the only time he has flown on a military jet was for an overseas congressional delegation.
Promise Kept, right? Hoy fills us in on the promissory particulars:
His spokesman says he’s flying commercial. They’ve gone through his travel records and have uncovered no evidence that he’s using military jets.
...

So, they’ve got zero evidence that he’s not flying commercial. Zero evidence that he has failed to keep his pledge.

And the best they can muster is: “In the works?”
Poor Boehner. If only he had been rated back in the olden days of eight months ago when PolitiFact's standards for Promise Kept weren't so high.

In a Promise Kept rating we reviewed, Obama was given credit for his promise to "...establish a 10 percent federal Renewable Portfolio Standard (RPS) to require that 10 percent of electricity consumed in the U.S. is derived from clean, sustainable energy sources, like solar, wind and geothermal by 2012." To this day, PolitiFact has yet to cite the actual legislation that Obama signed establishing the 10 percent RPS standard. Our best guess is they haven't done so because it doesn't exist. Rather, Obama was given a Promise Kept simply because market conditions caused energy sources to meet the 10 percent figure, not because of any requirement Obama was able to enact.

Boehner honors his commitment, and it's considered In the Works. Obama fails to act, and earns a Promise Kept for something that happened without his action that was only tangentially related to his pledge in the first place. But it don't worry, PolitiFact is still objectively on the case:
We will continue to seek more concrete records of Boehner"s travels and follow this pledge. If readers see him flying commercial -- or on a military jet -- please let us know.
PolitiFact's inability to maintain consistent standards for the statements and people they rate offer a great example of how the editors' and writers' personal bias creeps into their work. PolitiFact is a collection of partisans convincing each other of their own impartiality. They should not be trusted as a news source, let alone as a fact checkers.   

Check out Hoystory for the full article. His snark is always worth the visit. And for more from Hoy on PolitiFact go here. And see our reviews of Hoy's work here.


Bryan adds:

When I read the PolitiFact item on Boehner I had little problem with it as an interim rating--but when I recall that PolitiFact gave President Obama a "Promise Kept" for troop increases that he's now in the process of reversing it does create some puzzlement.  Obama was, after all, supposedly preparing us for "meeting the challenges of the 21st century."  Inconsistency remains a rampant problem at PolitiFact.

Thursday, March 22, 2012

PolitiFact's Bill Adair dismisses charges of bias

USA Today's semi-fawning review of the world of fact checking gives PolitiFact editor Bill Adair another opportunity to address the issue of journalistic bias at PolitiFact:
PolitiFact editor Bill Adair dimisses [sic] the bias charges. "Which charge of political bias? That we're biased in favor of the liberals or biased in favor of the conservatives? Because we've heard both charges in the past three months," Adair says. "When you're a loyal fan of a team, you're going to think the referee is biased against your team."
This is a redux of Adair's self-defense of PolitiFact's "Lie of the Year" for 2011.  To all you biased people in your echo chambers of course (objective) PolitiFact appears biased.  The problem is you.

Funny thing about referees, though.  Research indicates that referees are biased*.

Adair's defense is no defense at all.  In fact, it is an evasion of an important issue--an issue that should be of acute interest to a fact checker.

Do we want to trust our fact checking to a person who can't address the issue of media bias any better than the "we get criticism from both sides" blather?



*It's unlikely I would have thought to reference scientific studies of referee bias prior to reading Tim Groseclose's book "Left Turn."

Tuesday, March 20, 2012

Grading PolitiFact: Obama, Bush and the auto bailout

Crossposted from Sublime Bloviations.


Context matters -- We examine the claim in the full context, the comments made before and after it, the question that prompted it, and the point the person was trying to make.
--Principles of PolitiFact and the Truth-O-Meter

 Apparently context doesn't matter much, depending on the subject.


The issue:
(clipped from PolitiFact.com)

The fact checkers:

Molly Moorhead:  writer, researcher
Martha M. Hamilton:  editor


Analysis:

This fact check serves as an outstanding example of narrowing the story focus to fish a grain of truth out of an overall falsehood.

The incompetence is overpowering.  Note that PolitiFact frames the issue by stipulating that the $13 billion "given" by the Bush administration was gone "By the time Obama took office."   That bit of timing isn't mentioned in the film, so far as I can tell, though I was able to note that it used a Dec. 2, 2008 television news clip to emphasize the immediacy of the crisis faced by President Obama.

The film and PolitiFact omit a number of important facts.  First, GM received another $4 billion loan in February under the agreement worked out with the Bush administration.   Part of the agreement required the two automakers to submit plans for achieving financial stability by February.  The report of the Congressional Oversight Panel details the response from the Obama administration:
On February 15, 2009, President Obama announced the formation of an interagency Presidential Task Force on the Auto Industry (Task Force), that would assume responsibility for reviewing the Chrysler and GM viability plans.
The timing is far more complicated than either the film or PolitiFact lets on, and the loans from Bush were not necessarily "gone" when Obama took office, particularly in the case of the $4 billion received by GM in February, though that amount is not counted in the $13 billion through the magic of cherry picking the facts.

Let's pick up with PolitiFact's telling (bold emphasis added):
On the subject of Detroit, car company CEOs appear onscreen asking for money in Washington, followed by pictures of empty factories and dire news headlines. The movie talks about the financial pressures on the new president and the unpopularity with the public of more bailouts. But Obama, [narrator Tom] Hanks says, acted anyway to help American workers.

"He decided to intervene, but in exchange for help the president would demand action. The Bush administration had given the car companies $13 billion, and the money was now gone," Hanks says.

Then President Bill Clinton appears onscreen to lend his voice.

"He didn’t just give the car companies the money, and he didn’t give the UAW the money," Clinton says. "He said you guys gotta work together and come up, and everybody’s gotta have some skin in the game here. You gotta modernize the automobile industry."
This segment of the film is not about the history of $13 billion out of a total of $17 billion loaned to automakers by the Bush administration.  It is fully intended to build a contrast between the incoming president and his supposedly irresponsible predecessor.  That point is extremely misleading, as we shall continue to observe.

PolitiFact:
Bush authorized initial loans to Chrysler and GM (and their respective financing arms) before leaving office, using money from the Troubled Asset Relief Program. Chrysler initially received $4 billion, and GM got $13.4 billion in bridge loans meant to keep the companies afloat for a little longer.
Apparently the math amounts to $4 billion plus $13.4 billion equals $13 billion.  And that $13 billion was gone by Jan. 20 even though $884 million was loaned to GMAC on Jan. 16.  It lasted only four days by PolitiFact's account.

Of course the excess $4 billion was loaned in February as described above.  You just don't get to learn that from the PolitiFact version of events.

PolitiFact:
Early in 2009 [mid February], Obama convened a task force to study the companies’ viability. Both were required [through the agreement with the Bush administration] to submit plans for getting back to solvency, but both failed, the task force determined. In the meantime, they were running short of money again.
Pardon my editorial counterspin--which shouldn't be necessary for a fact check.  Unfortunately it is necessary.  GM, by the way, received its last Bush loan on Feb. 17, two days after Obama announced his task force.

PolitiFact:
A report from the Congressional Oversight Panel details the chronology of the spending, including an additional $6.36 billion that GM received between March and May 2009.
The $6.36 billion does not include the $4 billion loaned in February under the agreement with the Bush administration.  Nor does it include $8.5 billion sunk into Chrysler by the Obama administration as part of its restructuring.  Neither does it include the $30.1 billion subsequently sunk into GM as part of its eventual restructuring.  Both the latter figures come from the Congressional Oversight Panel's report PolitiFact cited.

PolitiFact interviewed former Obama team member Steve Rattner about the bailout numbers.  PolitiFact presents Rattner as agreeing that the funds from the Bush administration were exhausted "before we really were in the saddle."  Rattner states that the loans from the Bush administration weren't intended to rescue GM and Chrysler but rather to tide them over until the Obama administration could deal with the situation.

PolitiFact does not totally ignore the film's point about Bush:
We also think it’s worth mentioning the implication in the video that the Bush administration did not put enough restrictions on the money. "He decided to intervene, but in exchange for help the president would demand action," narrator Hanks says just before mentioning the Bush loans.
In case PolitiFact isn't the only party who missed it, note that the filmmaker uses the quotation of Bill Clinton to hammer the point all the more.  It was the main point of the segment, and it was untrue.

What's the verdict?

PolitiFact:
The Obama campaign movie says, "the Bush administration had given the car companies $13 billion and the money was now gone."

It's important to note that the $13 billion was provided as loans, not as grants, as the wording might suggest.

Referring to the time Obama took office, January 2009, GM and Chrysler by then had received almost $14 billion in bailout money. News reports also reflect that the money was basically used up. So, that much is correct. But the movie ignores the fact that this was not unexpected. The Bush administration’s loans were always just a temporary lifeline, meant to keep the companies operating so the new president would have time to decide what to do long term.

This is important information left out of the movie’s extensive discussion of the auto bailouts. That the $13 billion was gone when Obama arrived was no surprise. We rate the statement Mostly True.
The film glosses over quite a few facts that PolitiFact fails to note.  The point of the film is the contrast between the president who demands accountability and Bush who simply gives money away to big corporations.  The movie's account of the auto bailout is thorough spin.  Fact checking isolated statements in the fabric of this filmmaker's fiction will never fully reveal the misleading nature of the narrative.

If Obama went against popular sentiment on the bailout then so did Bush.  If Obama demanded accountability then so did Bush, albeit the latter's attempt was hamstrung by the end of his tenure as president.

PolitiFact disgraces itself again by connecting the film's distortion with a "Mostly True" label.


The grades:

Molly Moorhead:  F
Martha M. Hamilton:  F

PolitiFact let the main misleading message of the auto bailout segment slide.  PolitiFact's reporting corrected a fraction of the film's omissions and shades on the truth.  PolitiFact's version is scarcely an improvement on the original.

But President Obama and his campaign might like it.  That's got to count for something.


3/22/12-Added link to original PF article in first paragraph/fixed link to PoP-Jeff

Nothing To See Here: The Obama movie

I wouldn't ordinarily watch a campaign mockumentary, but since PolitiFact tagged it for fact checking I decided to give it a whirl and see what might have looked worthy of a fact check to a conservative.

This claim came from the film's narrator:
Not since the days of Franklin Roosevelt had so much fallen on the shoulders of one president.  And when he faced his country, who looked to him for answers, he would not dwell in blame by dreaming idealism.
I'm not quite sure what "he would not dwell in blame by dreaming idealism" is supposed to mean.  If it's supposed to mean that idealistic dreams per se do not necessarily blame others then the statement is trivially true.  But it comes across sounding like Obama did not blame others for the nation's predicament.

Another from the narrator:
He acted first with the Recovery Act, giving help where it was most urgently needed.  The country had been hemorrhaging jobs.  More than 3.5 million lost in 6 six months before he took office.  Middle-class jobs and economic security were vanishing.  The funding would keep teachers in the classroom, cops on the street and first responders ready.  And for those who were hurting: small business incentives, tax cuts for the middle class and job training.  Building bridges, highways, and infrastructure.  Laying the groundwork for a new economy and restoring the possibility of growth.
The description of the recovery act contains a number of discrete claims.  Do PolitiFact staffers pay attention to critiques of the stimulus?  And is it plausible to believe that the stimulus restored "the possibility of growth" when the chart provided by its architect shows recovery with or without the stimulus bill?

Yet another from the narrator:
But in exchange for help the president [Obama] would demand action.  The Bush administration had given the car companies $13 billion dollars.
Doesn't that suggest that Obama demanded steps from the car companies to ensure their future success while Bush simply gave money with no strings attached?  The surrounding context in the film only magnifies the impression.

More tough choices?  Mr. Narrator:
He had promised to bring a responsible end to the war in Iraq, and bring the troops home.  It was a promise kept.
It amuses me to think of liberals who consider the line above and earnestly wonder "How is that even the least bit questionable?"  Subsequently, the film explains that the end of the Iraq War was part of Obama's plan to focus on the war in Afghanistan.  There's a clue to the tip of the iceberg.

Another:
He restored science to its rightful place. [caption reads:  "Restores Stem Cell Research Funding"]
That paired claim may be worthy of a contest to see who can come up with the highest number of legitimate reasons it's a knee-slapper.

And another:
Cracking down on credit card companies and mortgage lenders, so the American people would never have to bail out Wall Street again.
This documentary is actually like a patchwork of Obama campaign commercials.

And yet another:
They changed the way the world sees us.
Okay, I do think that one's true.  But I'm interested in the metrics.  The visuals suggest that other nations now love us.

There's even more:
And a landmark law, so that a woman who does the same job as a man can get the same pay.  [caption reads:  "Lilly Ledbetter Fair Pay Act Passes"]
That's the purpose of the Lilly Ledbetter Fair Pay Act?  Who knew?

I could add more but the list is already longer than I expected.  PolitiFact only found two (see Addendum below) items worth checking in this beast?  Amazing. 


Addendum:  As I move to publish, I note that PolitiFact evaluated two more claims from the film, including one of the statements from my list.  Look for an analysis of the PolitiFact treatment at Sublime Bloviations.

Nothing To See Here: Arizona firing shots in the GOP War on Women?

My Emily's List pen-pal (well, she does send me email encouraging me to send money) Stephanie Schriock sent me the following very recently:
Republicans aren't getting the message. They're pushing even more outrageous bills, like the one in Arizona that would require women to tell their boss that they need to be on the pill to treat a medical condition if they want it to be covered by their insurance.
It's a national plea for funds, so I can see no reason to leave this one to the thus-far-nonexistent PolitiFact Arizona.

Schriock was nice enough to provide a link to the bill.

Monday, March 19, 2012

Ramesh Ponnuru: "Stimulus helped debt grow, not the economy"

Via the Columbian, Ramesh Ponnuru reminds us that using models to confirm the effects of the stimulus predictably leaves us with results similar to the results of pre-stimulus predictions using models.

Ponnuru mentioned PolitiFact's role in fact checking the effects of the stimulus in the same context:
Media fact-check organizations have no such doubts. Factcheck.org says it’s “just false” to deny that the stimulus has created jobs. It cites the Congressional Budget Office’s estimate that the stimulus had saved or created millions of jobs. But the CBO, as its director has explained, hasn’t really checked the effect of the stimulus. It has merely reported what the results of additional federal spending and tax credits would be if you assume that spending and tax credits are stimulative.

In other words: If you assume that stimulus works, it must have worked. This circularity doesn’t bother PolitiFact, a group that seeks to elevate the tone of political debates but usually lowers it. It says people who deny the effectiveness have their “pants on fire.”
Ponnuru's column foreshadows the difficulty for the GOP in the coming election.  The mainstream media will defend liberal orthodoxy on issues like the effects of the stimulus and health care bills.

Thursday, March 15, 2012

Nothing To See Here: The futility of drilling, per President Obama? (Updated)

From a recent speech by President Barack Obama:
If we drilled every square inch of this country — so we went to your house and we went to the National Mall and we put up those rigs everywhere — we’d still have only 2 percent of the world’s known oil reserves.
The error should be easy to spot for anyone who's looked into the issue of oil exploration.

While that ought to suggest that PolitiFact will rate this statement, I judge it unlikely that PolitiFact will rate it.


Hat tip to Power Line blog.


Update:

The Washington Post's fact checker scores with a March 15 item.  PolitiFact has yet to publish on this one.

Thursday, March 8, 2012

Nothing To See Here: Blunt Amendment allows Employer control over medical decisions?

Via an Emily's List email from Political Director Jonathan Parker:
Remember last week when right-wing Senator Roy Blunt's amendment would have given employers power over medical decisions from birth control to vaccinations?
Hmmm.

Wednesday, March 7, 2012

Nothing To See Here: No such thing as free contraception?

We have another NTSH item concerning Sandra Fluke straight from PolitiFact's new "In Context" feature:
"Forty percent of the female students at Georgetown Law reported to us that they've struggled financially as a result of this policy.

"One told us of how embarrassed and just powerless she felt when she was standing at the pharmacy counter and learned for the first time that contraception was not covered on her insurance, and she had to turn and walk away because she couldn't afford that prescription. Women like her have no choice but to go without contraception."
It seems like at least two of the contraception methods women used--at least according to stats Nancy Pelosi and the Obama administration have eagerly tossed around--were essentially zero-cost methods.

But we'll leave those details to PolitiFact.

Nothing To See Here: The high cost of contraception for law students

This NTSH item comes to us straight from PolitiFact's "In Context" feature.  So this one coming from newly famous Sandra Fluke is a layup for PolitiFact:
"Without insurance coverage, contraception, as you know, can cost a woman over $3,000 during law school."

Monday, March 5, 2012

Nothing To See Here: Taking away contraception and turning back the clock

This invisible unmentionable comes from Timothy B. Carney of the Washington Examiner:
While PolitiFact and the Washington Post Factchecker blog basically ignored the string of “they’re trying to ban contraception !!!11!!!” lies from Democrats for weeks, last Friday, the Post noted Schumer’s boatload of deception. Here’s one of his gems:
This whole debate is an anachronism. Our country progressed beyond the issue of whether or not to allow birth control a long time ago. Yet here we are in 2012, and the Republican Party suddenly wants to turn back the clock and take away contraception from women. Make no mistake: that’s what this debate is about.

This one's pretty far back in the news cycle already, with no sign of interest from PolitiFact.

New feature: Nothing To See Here

To provide some contrast to the selection process PolitiFact uses to choose its subject matter, PFB introduces a new feature:  Nothing To See Here.

Posts in this series will feature claims a non-liberal might find interesting enough to fact check.  While we won't be surprised if PolitiFact checks some or perhaps even many of the issues we highlight, we still anticipate a pattern of ignored stories that puts PolitiFact in the role of serving the political interests of the left.

Saturday, March 3, 2012

Eric Zorn: "True lies: Media umpires confront the challenge of dishonest facts"

Eric Zorn of the Chicago Tribune produced an excellent column criticizing PolitiFact's (and the Washington Post's) approach to fact checking.

Perhaps its biggest flaw comes from the fact that Zorn is so late to the party, like so many other critics from the ranks of mainstream journalists and the political left. 

The pair of quotations near the end of his column cap it off nicely:
"I've never been able to see an academically defensible way to hand out those kinds of ratings," director Brooks Jackson [of FactCheck.org] told me Thursday. "But I secretly admire the ability of (PolitiFact and The Washington Post) to do that. It engages the readers, and it's fun. But we'll leave questions of 'truth' to the theologians and philosophers."

PolitiFact editor Bill Adair told me that "the Truth-O-Meter is the heart of our organization. We'd never consider getting rid of it, but we always encourage people not to look just at the rating, but also at our reasoning."
The "Truth-O-Meter" misleads PolitiFact's audience, just as Zorn and many others have pointed out.  Yet Adair says "We'd never consider getting rid of it."

Would you buy a used car from that guy?

Zorn makes a good number of important points, so please read and digest the entire column.

Thursday, March 1, 2012

Tales of the Unexpected, featuring PolitiFact

 Crossposted from Sublime Bloviations

You have to love PolitiFact's fact-challenged statements about itself.


Now:
We’ve consistently ruled in the past that the economy is too complex to
assign full blame (or credit) for job gains or losses to a president or a
governor.
Then:
Our ruling

Pelosi compared a select time frame in the Obama administration against the entire length of the Bush administration -- a methodology that treats the two presidents unequally. The irony is that if she had used better methodology, she would have had a sounder argument that more private-sector jobs were created under Obama than under the Bush administration. For her general point, we give Pelosi some credit. For her methodological sins -- repeated at least three times -- we give her thumbs down. On balance, we rate her statement Half True.
There's consistency for you.

The NRCC makes a statement that's correct but represents cherry picking and gets a "Barely True."  Nancy Pelosi makes a statement that's also correct, represents cherry picking and gets a "Half True"--with no mention of docking Pelosi for crediting President Obama.  On the contrary, PolitiFact itself recommends an alternative method for giving President Obama credit for his job creation numbers compared to those of his predecessor.



Jeff adds (3/02/12): For the record, there's at least some consistency in these two articles: Both were written by Louis Jacobson, and both were edited by Martha Hamilton.