Would a North Carolina bill permit people to "murder women for using birth control"? Truth-O-Meter unavailable for comment

 Do Democrats ever make false claims?

If we judge by PolitiFact, falsehoods from Democrats have nearly been eliminated. The last "Pants on Fire" rating for a Democrat was from the summer of 2024. So far in 2026, only one claim from a Democrat has received a rating below "Mostly False."

In reality, however, Democrats likely lie about the same rate these days that they did in 2012, when PolitiFact recorded 135 falsehoods, combining the 109 "False" ratings with the 26 "Pants on Fire" ratings.

A big part of that drop in volume stems from the extinction of many of PolitiFact's state franchises, especially PolitiFact Wisconsin. And Meta's decision to end its partnerships with fact-checking organizations surely affects the number of fact checks published. But at the same time, PolitiFact has apparently just lost interest in issuing harsh ratings for Democrats.

One of the very few remaining franchises, PolitiFact North Carolina, provides our example.

A North Carolina Republican sponsored a bill that would declare life begins at conception. PolitiFact Carolina published an article--not a "Truth-O-Meter" fact check--mentioning a number of claims made about the bill.

That's where selection bias comes in.

When a number of Republicans weigh in on a subject, PolitiFact routinely picks the most extreme statement and publishes a harsh Truth-O-Meter rating.

The PolitiFact North Carolina story has an excellent candidate for a false (or worse?) rating:

The article takes up the issue of murdering birth control users in its latter half. PolitiFact itself passes no judgment on the "murder" claim. Instead, the story explains terms and leaves the judgment up to one of its cited experts:

Hamilton said in her video that the bill would allow people to "murder women for using birth control." Tracy Weitz, professor at American University and senior fellow with the Women's Initiative at American Progress, referred to that claim as "extreme" and "hyperbolic."

PolitiFact North Carolina quote Weitz again a few paragraphs later:

"Yes, contraception works to prevent pregnancy as currently defined," Weitz said. "However, some extreme religious perspectives define contraception as abortion using a religious not medical definition." Still, the allegation that the bill would allow people to murder someone for using birth control is "very farfetched," she said. 

"As a scientist, I try to avoid these kind of extreme hyperbolic statements," Weitz said.

Weitz almost makes the claim sound like it's earned a "Pants on Fire" rating.

Jen Hamilton, on whose claim PolitiFact North Carolina focused, holds no office and so would not count as a politician in our tracking of Democrat falsehoods. But Sophia Chitlik would:

"This session, we've seen legislation in the General Assembly that would allow people to murder women if they prevent fertilization," Chitlik said.

There's a potential harsh rating for a Democrat, just short of the PolitiFact North Carolina finish line.

Chitlik has no "Truth-O-Meter" ratings as we write this.

                                                                                                                         

PolitiFact's "Pants on Fire" bias in 2021

As we noted in our post about the "Pants on Fire" research for 2020, we have changed the way we do the research.

PolitiFact revamped its website in 2020, and the update made it next to impossible to reliably identify which of PolitiFact's various franchises were responsible for a fact check. Instead of focusing on PolitiFact National, it makes more sense to lump all of PolitiFact together. But the new approach has a drawback. The new evaluations represent an apples-to-oranges comparison to the old evaluations.

To deal with that problem, we went back and did PolitiFact's entire history since 2007 using the new method.

With the research updated using the new method, we are now able to compare the new research with the old method.

Spoiler: Using the new method, PolitiFact was 2.66 times more likely to rule a claim it viewed as false as "Pants on Fire" from a Republican than for a Democrat. That's PolitiFact's third-highest bias figure of all time, though PolitiFact National, considered separately, has exceeded that figure at least three times.

 

Method Comparison: New vs. Old 

Our new graph shows the old method, running from 2007 through 2019, along with the new method graphed from 2007 through 2021.

The black line represents the old method. The red line represents the new.

The numbers represent the what we term the "PoF bias number," which is an expression of how much more likely it is that PolitiFact will a claim it regards as false a "Pants on Fire" rating for a Republican over a Democrat. So, for 2009 under the old method (black line), the GOP was 3.14 times more likely to have one of its supposedly false statements rated "Pants on Fire."

As our research has documented, PolitiFact has never offered an objective means of determining the ridiculousness of a claim viewed as false. The "Pants on Fire" rating, to all appearance, has to qualify as a subjective judgment. In other words, the rating represents PolitiFact's opinion.

In 2017, under the old method, the bias number dropped to 0.89, showing a bias against Democrats for that year at PolitiFact National. On average over time, of course, Republicans were significantly more likely to have their false claims regarded as "ridiculous" by PolitiFact.

Notably, the new method (red line) shows a moderating effect on PolitiFact's "Pants on Fire" bias from 2008 through 2014. The red line hovers near 1.00 for much of that stretch. After 2015 the red line tends to run higher than the black line, with the notable exception of 2019.

Explaining the Numbers?

We found two correlations that might help explain the patterns we see in the graphs.

PolitiFact changes over time. From 2007 through 2009, PolitiFact National did nearly every rating. Accordingly, the red and black lines track very closely for those years. But in 2010 PolitiFact added several franchises in addition to PolitiFact Florida. Those franchises served to moderate the PoF bias number until 2015, where we measured hardly any bias at all in the application of PolitiFact's harshest rating.

After 2015, a number of franchises cut way back on their contributions to the PolitiFact "database" and a number ceased operations altogether, such as PolitiFact New Jersey and PolitiFact Tennessee. And in 2016 PolitiFact added eight new state franchises (in alphabetical order): Arizona, Colorado, Illinois, Nevada, New York, North Carolina, Vermont and West Virginia.

The Franchise Shift

We made graphs to help illustrate the franchise shift. PolitiFact has had over 20 franchises over its history, so we'll divide the graph into two time segments to aid the visualization.

First, the franchises from 2010 through 2015 (click for larger view):

We see Florida, Texas, Rhode Island and Wisconsin established as consistent contributors. Tennessee lasts one year. Ohio drops after four years. Oregon drops after five and New Jersey after three.

Next, the franchises from 2016 through 2022 (click for larger view):

I omitted minor contributions from PolitiFact Georgia in 2016 (12) and 2017 (2). The orange bar near the top of 2016 is six states combined (hard to make out in the columns after 2016).

Note that the contributions are skinny, except for the one from Wisconsin. But even Wisconsin cut its output compared to the previous graph. We have a correlation suggesting that the participation of different state franchises impacted the bias measure.

But there's another correlation.

Republicans Lie More! Democrats Lie Less!

Liberals like to explain PolitiFact ratings that look bad for Republicans by saying that Republicans lie more. Seriously, they do that. But we found that spikes--especially recent ones--in the "Pants on Fire" bias measure were influenced by PolitiFact's spiking reluctance to give Democrats a "Pants on Fire" rating.

That correlation popped out when we created a graph showing the percentage of false statement given the "Pants on Fire" rating by party. The graph for Republicans stays pretty steady between 20 and 30 percent. The graph for Democrats fluctuates wildly, and the recent spikes in the bias measure correlate with very low percentages of "Pants on Fire" ratings for Democrats.

As is always the case, our findings support the hypothesis that PolitiFact applies its "Pants on Fire" rating subjectively, with Republicans receiving the bulk of the unfair harm. And in this case Republicans receive the bulk of the unfair harm through PolitiFact's avoidance of rating Democrat claims "Pants on Fire."

Do Democrats lie less? We don't really know. We suspect not, given the number of Democrat whoppers PolitiFact allows to escape its notice (such as this recent gem--transcript). We think PolitiFact's bias explains the numbers better than the idea Democrats lie less.

Notes on the PolitiFact franchise numbers: As we noted from the outset, PolitiFact's revamped website made it all but impossible to identify which franchise was responsible for which fact check. So how did we get our numbers?

We mostly ignored tags such as "Texas" or "Wisconsin" and looked for the names of staffers connected to the partnered newsroom. This was a fallible method because the new-look website departs from PolitiFact's old practice of listing any staffers who helped write or research an article. The new site only lists the first one mentioned from the old lists. And it has long been the case that staffers from PolitiFact National would publish fact checks under franchise banners. So our franchise fact check numbers are best taken as estimates.

PolitiFact ratings aren't scientific--or are they?

We stand bemused by PolitiFact's attempt to straddle the fence regarding its aggregated "Truth-O-Meter" ratings.

On the one hand, as PolitiFact Editor Angie Drobnic Holan told us a few weeks ago, "It’s important to note that we don’t do a random or scientific sample."

On the other hand, we have this eye-popping claim Holan sent out via email today (also published to the PolitiFact.com website):

Trump remains a unique figure in American politics and at PolitiFact. There’s no one we’ve fact-checked as many times who has been as consistently wrong. Out of 738 claims we’ve fact-checked as of this writing, more than 70% have been rated Mostly False, False or Pants on Fire. After four years of watching Trump campaign and govern, we see little to no increase in his use of true facts and evidence to support his arguments.

Even though it is supposedly important to note that PolitiFact doesn't do a random or scientific sample, Holan makes no mention of that important caveat in her email. Instead, she makes it appear to the reader as though Trump's "Truth-O-Meter" record offers a reasonable basis for judging whether Trump has increased "his use of true facts and evidence to support his arguments."

How would PolitiFact judge whether Trump was using more true facts to support his arguments other than by looking at unscientific fact checker ratings?

PolitiFact has made a pattern of this deception.

The occasional admission of non-random, unscientific story selection is perfunctory. It is a fig leaf.

PolitiFact wants readers to judge politicians based on its aggregated "Truth-O-Meter" ratings.

For some unknown reason, though maybe we can guess, the fact checkers think their collected ratings are pretty much accurate judges of character regardless of their departure from the scientific method. And that's why PolitiFact over its entire history has implicitly and explicitly encouraged readers to rely on its graphs and charts to judge politicians. And has steadfastly resisted the obligation to attach a disclaimer to those charts and graphs making the supposedly important point that the selection process is not random or scientific.

We call it an obligation because we view it as a requirement for journalists to avoid deliberately deceiving their audiences.

PolitiFact deceives its audience daily in this way without any visible repentance.

PolitiFact editor: "It’s important to note that we don’t do a random or scientific sample"

As we have mentioned before, we love it when PolitiFact's movers and shakers do interviews. It nearly guarantees us good material.

PolitiFact Editor Angie Drobnic Holan appeared on Galley by CJR (Columbia Journalism Review) with Mathew Ingram to talk about fact-checking.

During the interview Ingram asked about PolitiFact's process for choosing which facts to check (bold emphasis added):

MI
One question I've been asking many of our interview guests is how they choose which lies or hoaxes or false reports to fact-check when there are just so many of them? And do you worry about the possibility of amplifying a fake news story by fact-checking it? This is a problem Whitney Phillips and Joan Donovan have warned about in interviews I've done with them about this topic.

ADH
Great questions! We use our news judgement to decide what to fact-check, with the main criteria being that it’s a topic in the news and it’s something that would make a regular person say, "Hmmm, I wonder if that’s true." If it sounds wrong, we’re even more eager to do it. It’s important to note that we don’t do a random or scientific sample.

It's important, Holan says, to note that PolitiFact does not do a random or scientific sample when it chooses the topics for its fact check stories.

We agree wholeheartedly with Holan's statement in bold. And that's an understatment. We've been harping for years on PolitiFact's failure to make its non-scientific foundations clear to its audience. And here Holan apparently agrees with us by saying it's important.

How important is it?

PolitiFact's statement of principles says PolitiFact uses news judgment to pick out stories, and also mentions the "Is that true?" standard Holan mentions in the above interview segment. But what you won't find in PolitiFact's statement of principles is any kind of plain admission that its process is neither random nor scientific.

If it's important to note those things, then why doesn't the statement of principles mention it?

At PolitiFact, it's so important to note that its story selection is neither random nor scientific that no example from three pages of Google hits in the politifact.com domain when searching for "random" AND "scientific" has anything to do with PolitiFact's method for story selection.

And despite commenters on PolitiFact's Facebook page commonly interpreting candidate report cards as representative of all of a politician's statements, Holan insists "There's not a lot of reader confusion" about it.
If there's not a lot of reader confusion about it, why say it's important to note that the story selection isn't random or scientific? People supposedly already know that.

We use the tag "There's Not a Lot of Reader Confusion" on occasional stories pointing out that people do suffer confusion about it because PolitiFact doesn't bother to explain it.

Post a chart of collected "Truth-O-Meter" ratings and there's a good chance somebody in the comments will extrapolate the data to apply to all of a politician's statements.

We say it's inexcusable that PolitiFact posts its charts without making their unscientific basis clear to readers.

They just keep right on doing it, even while admitting it's important that people realize a fact about the charts that PolitiFact rarely bothers to explain.

This Is How Selection Bias Works

Here at PolitiFact Bias we have consistently harped on PolitiFact's vulnerability to selection bias.

Selection bias happens, in short, whenever a data set fails to serve as representative. Scientific studies often simulate random selection to help achieve a representative sample and avoid the pitfall of selection bias.

PolitiFact has no means of avoiding selection bias. It fact checks the issues it wishes to fact check. So PolitiFact's set of fact checks is contaminated by selection bias.

Is PolitiFact's selection bias influenced by its ideological bias?

We don't see why not. And Taylor Swift will help us illustrate the problem.

PolitiFact looked at Swift's claim that Sen. Marsha Blackburn voted against the Violence Against Women Act. That fact check comes packed with the usual PolitiFact nonsense, such as overlooking Blackburn's vote in favor of VAWA in 2012. But this time our focus falls on PolitiFact's decision to look at this Swift claim instead of others.

What other claims did PolitiFact have to choose from? Let's have a look at the relevant part of Swift's statement:

I cannot support Marsha Blackburn. Her voting record in Congress appalls and terrifies me. She voted against equal pay for women. She voted against the Reauthorization of the Violence Against Women Act, which attempts to protect women from domestic violence, stalking, and date rape. She believes businesses have a right to refuse service to gay couples. She also believes they should not have the right to marry. These are not MY Tennessee values.

 Now let's put the different claims in list form:

• Blackburn voted against equal pay for women.
• Blackburn voted against the Reauthorization of the Violence Against Women Act
• Blackburn believes businesses have a right to refuse service to gay couples
• Blackburn also believes they should not have the right to marry

PolitiFact says it checks claims that make it wonder "Is that True?" 

The first statement regarding equal pay for women makes a great candidate for that question. Congress hasn't had to entertain a vote that would oppose equal pay for women (for equal work) for many years. It's been the law of the land since the 1960s. Lilly Ledbetter Fair Pay Act? Don't make me laugh.

The second statement is a great one to check from the Democratic Party point of view, for the Democrats made changes to the VAWA with the likely intent of creating voter appeals based on conservative opposition to those changes.

The third statement concerns belief instead of the voting record, so that makes it potentially more challenging to check. On its face, Swift's claim looks like a gross oversimplification that ignores concerns about constitutional rights of conscience.

The fourth statement, like the third, involves a claim about belief. Also, the fourth statement would likely count as a gross oversimplification. Conservatives opposed to gay marriage tend to oppose same-sex couples asserting every legal advantage that opposite-sex couples enjoy.

PolitiFact chose its best candidate for finding the claim "True" instead of one more likely to garner a "False" rating. It chose the claim most likely to electorally favor Democrats.

Commonly choosing facts to check on that type of basis may damage the election prospects of those unfairly harmed by partisan story selection. People like Sen. Blackburn.

It's a rigged system when employed by neutral and nonpartisan fact checkers who lean left.

And that's how selection bias works.

Beware, lest Trump & PolitiFact turn your liberal talking point into a falsehood!

PolitiFact gave President Donald Trump a "False" rating for claiming the GOP tax bill had effectively repealed the Affordable Care Act.

We figured there was a good chance that defenders of the ACA had made the same claim.

Sure enough, we found an example from the prestigious left-leaning magazine The Atlantic. The Google preview tells the story, as does the story's URL, though the story's title tames things a little: "The GOP's High-Risk Move to Whack Obamacare in Its Tax Bill."

The key "repeal" line came from an expert The Atlantic cited in its story (bold emphasis added):

“Make no mistake, repealing the individual mandate is tantamount to repealing the Affordable Care Act,” said Brad Woodhouse, campaign director for Protect Our Care, an advocacy group supportive of the ACA.

Would Woodhouse receive a "False" rating from PolitiFact if it rated his statement?

Would The Atlantic receive a "False" rating from PolitiFact?

Would PolitiFact even notice the claim if it wasn't coming from a Republican?

Afters (other liars who escaped PolitiFact's notice)

"GOP tax bill is just another way to repeal health care." (Andy Slavitt, USA Today)

"Republican tax bill to include Obamacare repeal" (Christian Science Monitor)

"Republicans undermine their own tax reform bill to repeal Obamacare" (Salon)

"Another Obamacare repeal effort doesn't actually have to be in the tax cuts bill, says the guy heading up popular vote loser Donald Trump's Office of Management and Budget." (Daily Kos)

Current Affairs: "Why PolitiFact’s 'True/False' Percentages Are Meaningless"

We're not sure how we missed this gem from Aug. 8, 2016 in Current Affairs magazine.

The author, Nathan J. Robinson, was prompted by the slew of misleading stories featuring PolitiFact "data" from its "report cards":

Scores of media outlets have used PolitiFact’s numbers to damn Trump. The Washington Post has cited the “amazing fact” of Trump’s lie rate, with bar charts showing the comparative frequency of his falsifications. The Week counted only those things deemed completely “True,” and thus concluded that “only 1 percent of the statements Donald Trump makes are true.” Similar claims have been repeated in the US News, Reason, and The New York Times. 

But all of these numbers are bunk. They’re meaningless. They don’t tell us that lies constitute a certain percentage of Trump’s speech. In fact, they barely tell us anything at all.

Robinson does a great job of hitting nearly every problem with PolitiFact's rating system. The article is somewhat long, but worth the investment in time.

Mark of the Least: PolitiFact Avoids Hillary's Most Damaging Lies

Pictures of last night ended up online, I'm screwed. OH WELL!

Yeah I think we broke the law, always say we're gonna stop...

This Friday night do it all again. 

-Katy Perry

This Tuesday, PolitiBlogger Lauren Carroll did me the favor of making my prediction come true, thus cementing my status as the world's least daring fortune teller.

Carroll's post helped PolitiFact readers sort out the truth of a mystery that CNN, Reuters, NBC, AP, New York Times, Wall Street Journal, and Politico had already confirmed and reported on extensively a week before PolitiFact even touched it. 

Carroll offered up this lame excuse:

We haven’t yet put the issue on the Truth-O-Meter because there were too many unknowns.

Carroll explains the IG report was the smoking gun they finally needed to put Hillary to the scientific rigors of the Truth-O-Meter.

Carroll failed in "sorting out the truth" of what she was sorting out the truth of.

Readers didn't benefit in any way by PolitiFact's delayed press time. Carroll's post didn't include any exclusive or developing details that weren't already reported on by much more popular journalism outlets the week before. Carroll only succeeded in regurgitating a widely known story, included a weak defense for ignoring it earlier, then slapped a gimmicky "Truth-O-Meter" graphic on it.

Carroll continued:

But the inspector general’s report has clarified some of those unknowns and demonstrated that Clinton’s exclusive use of personal email was, in fact, not allowed.

We've known for years Clinton's exclusive use of a personal email account violates State Department policy. Who does Carroll think she's fooling? Here's the source:

First of all, the State Department’s policy as of 2005 (Clinton joined in 2009) is that all day-to-day operations are to be conducted on the official State Department information channel. Clinton never once used this State Department email system.

The quotation comes from Carroll herself. As far back as March, 2013 it was reported that Clinton was exclusively using her personal email account for government business, contrary to State Department policy. PolitiFact is just figuring this out now? The IG report didn't confirm Clinton's email impermissibility so much as it reiterated it.

Carroll's excuse that there wasn't enough information doesn't pass the laugh test.

And what spared Clinton from the dreaded "Pants on Fire" rating? On Twitter, I asked both PolitiFact and Carroll herself what objective criteria they used to determine Clinton's claim was false, but not ridiculously false (That's the only difference between a False and Pants on Fire rating, as Bryan explains here.)

Neither responded so we're left to assume Clinton's repeated, years long, blatant lie wasn't too offensive to the political sensibilities of PolitiFact staff. It's false, they admit, but not ridiculous.

Finally, we have the issue of selection bias. It's arguable that of all the sordid details of Clinton's private email practices, her lie that it was "allowed" is arguably the least politically damaging to her. Breaking a few rules for convenience is hardly something most Americans would become outraged over, especially when so many are dealing with voluminous and complicated email work rules themselves. A partisan may even be able to paint Clinton in a sympathetic light if all she did was use the wrong email.

Rating Clinton's "allowed" lie is almost as helpful as that other time PolitiFact dipped a tepid toe into the Clinton email scandal. In that case they informed readers that Hillary Clinton was not under investigation. Instead, the FBI was investigating her email server, an inanimate object.

Why not check this Clinton whopper?

“There is no classified marked information on those emails, sent or received by me." 

No need for an IG report here. That claim was demonstrably false when she made the claim. (Nor has PolitiFact provided so much as an "In Context" article to explain it's irrelevant if Top Secret information was marked or not.)

Oddly enough PolitiFact editors don't see any news value in determining if Clinton put American lives at risk by failing to protect our most valuable secrets. That's why you won't see it rated on the Truth-O-Meter.

Besides, it seems that Carroll has stayed busy trying to sort out the truth of much more important things.

PolitiFact isn't holding anyone accountable. They're pushing narratives based on their own political inclinations. And politicians that lie will do it all again.





Edits: 
0638 PST 6/3/2016: Fixed various text formatting errors. Deleted duplicate final sentence in antepenultimate paragraph. -Jeff
0840 PST 6/4/2016 While fixing formatting errors the text "Oddly enough PolitiFact editors don't see any news value in determining if" was inadvertently deleted in antepenultimate paragraph. It has been restored. -Jeff  

Are PolitiFact's "report cards" misleading?

One of our recurrent themes at PolitiFact Bias concerns the misleading nature of PolitiFact's "report cards." PolitiFact admits it does not perform its fact checks on a scientific basis, particularly in that story choices do not occur randomly. Despite that, it's utterly common for PolitiFact to publish a "report card" story encouraging readers to consider a candidate's "report card."

The latest such asks readers to consider the record of just-announced presidential candidate and former Secretary of State Hillary Clinton.

The reader comments from PolitiFact's Facebook page offer us a window into the degree of deception PolitiFact achieves with its "report card" stories.

We'll omit the names to save these individuals unnecessary embarrassment.

"When you compare the overall honesty of Democrats vs. Republicans, it's no wonder that some Republicans believe that fact checking web sites are liberally biased. It's an easier explanation then the reality that Republicans tend to lie more."

"Better than Ted Cruz."

"Comparison: Clinton, true or mostly true = 48%; Rand Paul, true or mostly true = 15%; Ted Cruz, true or mostly true = 8% (from PolitiFact archives)."

"I would love to see a chart comparison against the other candidates. Paul's record would be a joke, his pants have been on fire so much the fire department had to move into a spare room"

"Still not a Hillary fan, but at least she's more honest that the right wing."

"Not a bad record."

Many more after the break! 

PunditFact: Our scorecards measure all claims made by pundits on air

Bless your heart, PunditFact.

Some people think PolitiFact's network scorecards can't be taken as a reliable litmus test of network truthfulness. After all, these skeptics say, PunditFact doesn't check every claim by every pundit on the networks.

But today PunditFact broke the news that it does check every claim on the networks (bold emphasis added):

MSNBC and CNN have improved ever so slightly on our TV network scorecards, while Fox News has moved a touch in the opposite direction.

We last looked at our network scorecards, which examine all the claims made by pundits on air, in September. The scorecards measure statements made by a pundit or a host or paid contributor on a particular network. They do not include statements made by elected leaders, declared candidates or party officials.

So that lends a great deal of credibility to PunditFact's scorecards. After all, if the scorecards are examining all the claims then there's no issue with selection bias. Right?

Or not. PunditFact added this later on in the article:

As we have said in the past, be cautious about using the scorecards to draw broad conclusions. We use our news judgment to pick the facts we’re going to check, so we certainly don’t fact-check everything. And we don’t fact-check the five network groups evenly.

Bless your heart, PunditFact.

PolitiFact: We're not social scientists, but you sure can tell plenty about a candidate from our 'candidate report cards'

PFB editor Jeff D. took notice that new PolitiFact editor Angie Drobnic Holan recently affirmed PolitiFact's minimalist admission of its practice of selection bias (bold emphasis added):

PolitiFact's goal is not to end inaccuracy in political discourse, just to point it out for voters' information, Holan says. And the site doesn't cover everything politicians say, either in ads or interviews. "We're journalists. We pick statements based on what we think is most newsworthy,'' Holan says. "We fact check things that make people go 'Hmmm, I wonder if that's true.''

Not at all to our surprise, PolitiFact's announcement of its "PolitiPundit" project, dedicated to fact checking media figures and pundits the PolitiFact way, reiterates PolitiFact's intent to keep right on publishing "report cards" for the people and organizations it fact checks. 

It's the same old mixed message from PolitiFact:  No, it doesn't use basic scientific methods in choosing its stories.  Yes, readers should treat PolitiFact's "report cards" as though PolitiFact uses scientific methods.

"Meet the new boss.  Same as the old boss."
--The Who

Disconnect at PolitiFact Ohio

Nothing's better than getting PolitiFact editors on the record about PolitiFact.  Their statements probably do more than anything else to show that the PolitiFact system encourages bias and the people who created it either don't realize it or couldn't care less.

Statements from editors at PolitiFact's associated newspapers come in a close second.

Ted Diadiun, reader representative for the Cleveland Plain Dealer (host of PolitiFact Ohio), answering a reader's question:

In July you printed a chart with two years of PolitiFact Ohio results. It showed Democrats with 42 ratings of Mostly False, False or Pants on Fire, while the Republicans had a total of 88 in those categories. Doesn't that prove you guys are biased?

Well, it doesn't necessarily prove that. It might prove instead that in the statements our PolitiFact team chose to check out, Republicans tended to be more reckless with the truth than Democrats.

Diadiun apparently doesn't realize that if PolitiFact Ohio chooses more Republican statements to treat harshly then it is a likely sign of institutional selection bias unless PolitiFact Ohio either randomizes its story selection (highly unlikely) or coincidentally chose a representative sample.  How would we ever know that the sample is representative unless somebody runs a controlled study?  Great question.  It's such a good question that it is reasonable to presume that a disparity in the treatment of the respective parties by PolitiFact results from an ideologically influenced selection bias.  That was the point of Eric Ostermeier's study of PolitiFact's 2011 "Truth-O-Meter" results.

Diadiun, continuing his answer to the same question:

Or, it might prove only that there are a lot more Republicans who have been elected to the major offices that provide most of the fodder for fact-checking.

It would prove nothing of the kind.  PolitiFact has one state operation in a state that is firmly controlled by Democrats:  PolitiFact Oregon.  PolitiFact Oregon, despite a state political climate dominated by Democrats, rates the parties about evenly in its bottom three "Truth-O-Meter" categories (Republicans fare slightly worse).

Diadiun:

It is also a fact that Republicans had a few more statements rated True than the Democrats did, but the Truth-O-Meter was indeed a bit tougher overall on Republicans. You can find the report here

Does that show bias? I've said it before, and I'll say it again here: The PolitiFact Truth-O-Meter is an arbitrary rating that has the often impossible job of summing up an arduously reported, complicated and nuanced issue in one or two words.

Diadiun goes on to tell his readers to "ignore" the Truth-O-Meter.

That's quite the recommendation for PolitiFact's signature gimmick.

He's partly right.  PolitiFact ratings cram complicated issues into narrow and ill-defined categories.  The ratings almost inevitably distort whatever truth ends up in the reporting.  So shouldn't we ask why a fact checker steadfastly continues to use a device that distorts the truth?

The answer is pretty plain:  The "Truth-O-Meter" gimmick is about money.  PolitiFact's creators think it helps market the fact checking service.  And doubtless they're right about that.

There is a drawback to selling out accuracy for 30 pieces of silver:  Contrary to Diadiun's half-hearted reassurances, the "Truth-O-Meter" numbers do tell a story of selection and ideological bias.  Readers should not ignore that story. 


Jeff adds:

The hubris on display from Diadiun could fill gallon-sized buckets. Notice that he completely absolves PolitiFact from the role they play in selecting which statements to rate, and immediately implies the "Republicans tended to be more reckless with the truth than Democrats." Incompetence or deceit are the only reasonable explanations for such an empty claim.

For the benefit of our new readers, I'd like to provide an exaggerated example of selection bias: Let's say I'm going to call myself an unbiased fact checker. Let's say I'm going to check 4 statements that interest me (as opposed to a random sample of claims). I'll check Obama's claim that he would close Guantanamo Bay, and his claim that he "didn't raise taxes once." I find he's telling falsehoods on both accounts.

Next, I'll check Rush Limbaugh's statement that he's happy to be back in the studio after a long weekend. I'll also check his claim that he's one of the most popular radio shows in the nation. Of course, these claims are true.

What can we learn from this? According to PolitiFact's metrics, Rush Limbaugh is a bastion of honesty while Barack Obama is a serial liar. I'll even put out "report cards" that "reveal patterns and trends about their truth-telling." I'll admit the "tallies are not scientific, but they provide interesting insights into a candidate's overall record for accuracy." It's unbiased because, as the popular defense goes, I checked both sides. The reality that I checked statements that interested me supposedly has no influence on the credibility of the overall ratings. If you don't like the results, it's because you're biased!

See how that trick works?

It's something to keep in mind the next time you see Obama earning a Promise Kept for getting his daughters a puppy or a True for his claim that the White Sox can still make the playoffs. A cumulative total of PolitiFact's ratings serves the purpose of telling readers about the bias of PolitiFact's editors and what claims are interesting to them. It's a worthless measure of anything else.

Bill Adair still ignoring the poison of selection bias

In an otherwise okay article about adapting journalism to the Internet, PolitiFact editor Bill Adair ended up making his usual pitch for pretending that journalists' selection bias does not affect fact check stories:

At PolitiFact, we’ve created two new forms. Instead of traditional articles, our Truth-O-Meter fact-checks are a new form that allows you to see a politician’s report card, to see all fact-checks on a subject or see all the Pants on Fire ratings. We can make larger journalistic points through the automatic tallies and summaries of our work.

We’ve done the same thing with the Obameter and the other meters we use for tracking campaign promises. The unit of journalism is the promise and then we write updates and rate whether the promise is kept or broken. The promises also get tallied so you can see how the politician is doing.

PolitiFact's "two new forms" of journalism encourage readers to accept a non-scientific process as science.  PolitiFact and Bill Adair continually sell the falsehood that its candidate report cards serve as a useful guide for judging whom to support in an election.

He's not entirely wrong.

PolitiFact's left-leaning bias makes its report cards primarily useful to liberals.

For review, here's how it works.

PolitiFact writes fact check stories its left-of-center staff thinks are relevant and interesting.  Left-of-center staffers find stories favoring their left-of-center views more interesting than others.  This naturally results in more fact checks of conservatives since left of center staffers question conservative ideas more than those jibing with their own.  Should PolitiFact experience concerns that its fact checking looks overly focused on conservative claims, it picks a few claims by liberals to help balance the ledger--claims that were in the first place less likely to warrant a fact check by left-of-center lights.  As a result, to no one's surprise, liberals tend to fare better on the report cards.  Liberals feel no surprise because of confirmation bias.  Conservatives feel no surprise because they've learned to expect biased reporting from journalists.

PolitiFact's defenders like to suggest that conservatives perceive a non-existent bias.  The simplest explanation, some of them say, is that conservatives simply lie more.

That response ignores two absolutely obvious points. 

Number one:  Journalists tend to lean to the political left of the general population.  The difference is not small.

Number two:  The story selection process--indeed, PolitiFact's entire process--might as well be designed to reinforce liberal bias.  There's no check on selection bias, and the three-editor panels PolitiFact uses to assign grades help ensure that the liberal majority in the newsroom decides the ratings by popular vote on any occasions where the voting isn't done only by liberal judges.

PolitiFact's process requires those who understand its workings to suspend disbelief when looking at its report cards and collected numbers.

Balancing act at PolitiFact Ohio?

Since its 2011 "Lie of the Year," the claim from Democrats that Republicans voted to end Medicare, PolitiFact has found itself trying to answer the criticism that it tries to achieve a false balance in its fact checking operations.

Past questions about the dangers of selection bias should have amply prepared PolitiFact editors to answer this sort of question.

Should have.

PolitiFact Ohio editor Robert Higgs of the Cleveland Plain Dealer took a shot at addressing the issue of balance in comments to the Plain Dealer's reader representative, Ted Diadium:

I asked Bob Higgs, the editor who oversees the PolitiFact Ohio operation, if he deliberately tries for balance:

"The belief is that if we apply the same constructive standards to all claims, we'll end up treating all sides fairly," he said. "Some of the state operations (there are 10 in the PolitiFact organization), as well as the national operation, do not tally the rankings at all."

Higgs admits that he does tally up the results by party (which shows them remarkably even), "but only to see after the fact how we've done."

Even if PolitiFact Ohio applies its evaluation standards consistently to all its stories, balanced treatment need not result. In fact, it probably won't result.

It won't result because selection bias will occur without active steps taken to avoid the problem.

Select nine stories likely to make Democrats look bad while selecting one that will likely made Republicans look good will not achieve balance regardless of applying identical standards in the evaluation--not that we at PolitiFact Bias believe PolitiFact applies its standards consistently.

Newsflash for Higgs:  Every writer and editor at PolitiFact likely has a sense of how the stories break down by party.  The "remarkably even" count that results at PolitiFact Ohio helps prove the point.

PolitiFact markets its stats as candidate report cards and the like, but the real value of PolitiFact's numbers comes from the insights we obtain into PolitiFact's behavior--not the behavior of those featured in the stories.


Correction:  Changed the first of two consecutive instances of "the" to a "from" in the concluding paragraph.  Hat tip to Jeff Dyberg for catching the the error.

Does Bill Adair understand selection bias?

My heart fairly skipped a beat minutes ago when I ran across an interview of PolitiFact editor and founder Bill Adair where he was directly asked how PolitiFact avoids selection bias.

National Press Foundation:

How do you avoid selection bias?

There are many things that go into deciding what we are going to choose. We try to be timely, we try to stay on top of the news and we try to have balance so we check people from both parties. That can be challenging though, because if you have eight voices speaking up in a Republican primary and only one Democratic incumbent – naturally you have eight times the number of statements being made on the Republican side than on the Democratic side. We try to check roughly the same number of claims by Democrats as we do for Republicans, but we have to go where the claims are and lately there have been more made by Republicans. In terms of avoiding selection bias, I think the key is to be guided by what serves the reader. Once you get past claims selections, our fact-check process is entirely driven by journalistic and independent assessment.

Shorter Bill Adair:  "We don't avoid selection bias."

The guy just doesn't get it.

Selection bias is simply the non-random selection of subject matter.  Adair has described two different methods of story selection.  First, PolitiFact tries to sniff out suspicious-sounding claims.  Second, PolitiFact tries to pick out roughly equal numbers of claims by Democrats and Republicans.

That's a recipe for producing selection bias, not a method for minimizing it.

 
Jeff adds: Adair's misdirection may placate PolitiFact's sympathetic followers, but his excuse that "naturally you have eight times the number of statements being made on the Republican side than on the Democratic side" because of the current campaigning doesn't pass the laugh test. That may be a plausible reason if PolitiFact limited it's fact checking only to current presidential contenders, but that's not the case.

PolitiFact checks statements by Congressmen, aging pop singers, television personalities, Senate conferences, and chain emails. Are right leaning Congressmen making more statements because of the upcoming election? Are conservative pop stars issuing statements at an 8-1 ratio to their liberal counterparts? No.

And as we've explained before, simply choosing an equal number of statements by opposing parties is an insufficient method to avoid selection bias. Adair knows this, and his evasion on the issue would be embarrassing if it wasn't so insulting.

PolitiFact rolls out new feature, continues to ignore selection bias

Crossposted from Sublime Bloviations


I've belabored the point that PolitiFact pushes its candidate "report cards" without owning up to the selection bias that is overpoweringly likely to skew the grades.

PolitiFact editor Bill Adair has been asked about selection bias.

Bill Adair probably (based on IP address trails) reads commentary here and at PolitiFact Bias [more appropriately "here and at Sublime Bloviations" after crossposting] regarding PolitiFact's problem with selection bias.

Yet minutes ago we get this:

It's spring, which means it's report card time. So we're unveiling a new feature that allows you to compare the PolitiFact report cards for individuals and groups we check.

Our report cards have always been a popular feature and often generate interesting commentary. Now, you'll be able to compare the report cards more easily.

Adair still won't inform his readers that the process leading to the report card grades is rife with selection bias problems.  Yes, Adair at least linked to a New York Times blog ("interesting") that provided the minimum type of disclaimer that PolitiFact should offer.  The other link ("commentary") was the sort of pointless statistical exercise that simply elaborates on the results of PolitiFact's fundamentally flawed process (the former link I gave a positive review, the latter author I've given a less-than-positive review).

Bottom line, PolitiFact continues to publish candidate "report cards" that appear minus critical context.  PolitiFact (allegedly) rules statements missing critical context "Half True."

PolitiFact apparently knows about its selection bias problem and is deliberately downplaying it.  Selection bias is not merely "interesting."  It is critical to an accurate understanding of the meaning of PolitiFact's "report cards."


Updated 3/29/2012:  Added editor's note in third paragraph

Ranting and Rating: Why PolitiFact's Numbers Don't Add Up

Crossposted from Bewz, Newz, 'n' Vewz.

"I've given you a decision to make,
Things to lose, things to take,
Just as she's about ready to cut it up, She says:
  "Wait a minute, honey, I'm gonna add it up."

-Violent Femmes

One of the more common methods of using PolitiFact's findings is to add up total ratings and form a conclusion based on the data. In its simplest form, this is when someone looks at, for example, 100 PolitiFact ratings, 50 from Republicans and 50 from Democrats, then adds up who received more trues and who had more falses, and concludes from that total who is more credible. The reality is that a collection of PolitiFact's ratings provides far more information about the ideological bias of PolitiFact's editors than it does about the people they check.

One of the reasons this flawed method is so popular is that PolitiFact frequently promotes it as part of its shtick. Whether it's the ubiquitous report cards, or the iPhone app with its absurd Truth Index (described as a "Dow Jones Industrial Average of truth"), PolitiFact implicitly tells readers they can simply click a link to find out the credibility of a particular politician. Like most diet pills and get-rich-quick schemes, it's snake oil science and complete junk.

The most obvious flaw with this method is selection bias. There's simply no way for PolitiFact, or anyone for that matter, to check every statement by every politician. This means PolitiFact needs to have some sort of random selection process in order to ensure their sample reflects the wide variety of political statements being made, as well as the politicians making them. Without a random process, editors and writers might investigate statements that pique their own ideological interests. And how does PolitiFact choose its subjects?

"We choose to check things we are curious about. If we look at something and we think that an elected official or talk show host is wrong, then we will fact-check it."

Ruh-roh.

This "things we're curious about" method may explain why Senate hopeful Christine O'Donnell (Rep-RI) garnered four PolitiFact ratings, while the no less comical Senate hopeful Alvin Greene (Dem-SC) received none. 

Officially, PolitiFact only checks claims that are the "most newsworthy and significant." (Unless it's about Obama getting his daughters a puppy. Or baseball). PolitiFact also has a penchant for accepting reader suggestions. Anyone visiting PolitiFact's Facebook page is aware that their fans overwhelmingly reside on the left side of the political spectrum. If PolitiFact asks 50,000 liberals what statements to check, guess what? Statements about Fast and Furious won't be as popular as, say, Sarah Palin.

It's also important to consider the source of the statement being rated. For example, when Barack Obama made the claim that preventative health care is an overall cost saver, and Republican David Brooks wrote a column explaining Obama is wrong; PolitiFact gave a True to Brooks. This spares Obama a demerit in his column* while granting Republicans an easy True. Another example of this source selection is evident in the rating about $16 muffins for a Department of Justice meeting. Despite the claim being made in an official Inspector General report and being repeated by several media outlets, including the New York Times and PolitiFact partners NPR and ABC news, PolitiFact hung a False rating around Bill O'Reilly's neck. PolitiFact refrained from judging the nominally liberal media outlets--and the source of the claim--all while burdening O'Reilly with a negative mark in his file.

One of the most overlooked problems with analyzing a tally of the ratings is the inconsistent application of standards PolitiFact employs in different fact checks. Even if one was to assume PolitiFact used a random selection process and assigned its ratings to the appropriate source, we still have a problem when subjects aren't checked according to the same set of standards. For example, Politifact rated Obama "Half True" when he made a claim about the rates certain taxpayers pay. His claim only earned that rating when PolitiFact considered the amount their employers contributed to the employees' tax burden. Almost simultaneously, they labeled Herman Cain Mostly False in a similar claim specifically because he used the same formula. A cursory analysis of total ratings fails to detect this disparate treatment. When considering such flexible guidelines, the "report cards" don't seem like such a credible evaluation.

Ultimately, the sum of PolitiFact's ratings tells us far more about what interests PolitiFact's editors and readers than it does about the credibility of any individual politician. With so many flaws in their process, and such a minute sample size in a vast ocean of statements, conclusions about a subject's overall honesty should be considered dubious. We recognize that this flawed process will undoubtedly affect liberal politicians as well. However, it's our contention that the personal bias of the editors and writers will harm those on the right side of the aisle more often and more dramatically than those on the left.

Adding up PolitiFact's ratings in an attempt to analyze a person's or party's credibility produces misleading results. Until PolitiFact includes a check for selection bias and establishes and adheres to transparent and objective standards, an analysis based on their cumulative work is ill-founded at best, and grossly inaccurate at worst.




*PolitiFact did eventually give Obama a False after he repeatedly made the claim, but still spared several high profile Democrats for the same statement. Unlike perennial fact check favorites like the jobs created by the stimulus or Obama's birth certificate, PolitiFact seems to think the issue isn't worth revisiting.


Update (4/8/2012)-Shortly after (because?) we published this post PolitiFact did revisit Obama's  "Preventative Care Saves Money" claim, and gave him another False. Fair enough. But there's still plenty of other claims that give Obama a pass while simultaneously earning their "We give good grades to the GOP too!" chits that the original point still stands.-Jeff



(2/1/2012): Corrected Alvin Greene's state and name.

Bill Adair describes selection bias at PolitiFact

We've noted before, along with Eric Ostermeier, that PolitiFact's self-described methods of choosing stories amount to a recipe for selection bias.  The Parker Report recently published an interview with PolitiFact editor Bill Adair that describes more of the same:

How do you decide which reporters take on which claims? How many people check PolitiFact’s facts before publishing? In other words, what is the process? 

Here’s how it works: Every day, our interns look through transcripts, campaign videos, news coverage and interviews for factual claims. The editors review them and choose the claims to fact-check based on whether they are timely, newsworthy and whether people will wonder if the claim is true. That is our biggest criteria for selecting a claim to check — to satisfy people’s curiosity.
 
Most of the time, our reporters choose the claims they want to check. They do the research and write the article, which typically takes a day.

The articles, which include a recommended Truth-O-Meter rating, are then edited by one of our editors and then reviewed by a three-editor panel. The panel makes the final decision on the rating.

Though Adair does not mention it in this interview, PolitiFact also solicits fact check ideas from readers (see image below).

(clipped from http://www.politifact.com/)

If those who send in story suggestions are predominantly liberal or predominantly conservative, this feature will tend to skew story selection toward one political pole or the other.

Likewise, in Adair's description we have two factors that will help introduce partisan bias.  First, if the interns' political biases influence their choices of stories to send to the editors, then the system again will tend to skew story selection toward one political pole or the other.  Second, if the editors use their own sense of curiosity in choosing which statements to rate we have another case where ideology may influence the selection process left or right.

The three-editor panel that chooses the final "Truth-O-Meter" rating will by its nature tend to preserve a majoritarian ideological bias in PolitiFact's ratings.  Suppose the panel has two conservatives and one liberal.  In any rating where ideology serves as an influence, the two conservatives could consistently outvote the liberal.

Interviewer Erik Parker did not ask Adair any particularly tough questions (at least it doesn't appear on the record). Maybe next time.

PolitiFact's Bill Adair skirts the issue of selection bias on C-SPAN (Updated)

For a second there I thought PolitiFact editor Bill Adair was going to do it.  He was going to admit that PolitiFact "report cards" are pretty much worthless because of selection bias.

Not quite.  He dropped a few hints but avoided the confession.  Check it out:

 The almost-confession is drowned out by Adair's language hinting that the main problem with the individual truth ratings comes from small sample size. Good grief. Update 10/17/2024: We removed our original embeds because Blogger no longer supports their format. Fortunately C-SPAN continues to host the full set of user clips we created. The updated format should work: C-SPAN allows users to make custom clips of its videos. Thanks to that feature, we have clipped the segment down to its relevant section.

    

Associated Content: "The Misuse of PolitiFact's Rankings"

Anonymous contributor "Thales" of Yahoo's Associated Content provides us yet another reminder of how PolitiFact's collected ratings are pretty much useless without a check on selection bias:

There's no way they could cover every political statement made by even a couple politicians, let alone dozens of them. But if PolitiFact hasn't given a comprehensive evaluation of all of the statements of these politicians -- or at least equally representative samples from each of them -- then how can they be used to rank these people in terms of veracity or ignorance? They can't.

Thales also notes PolitiFact's tacit approval of the misuse of its rankings:

PolitiFact has said nothing about this use -- I would argue, misuse -- by The Daily Beast of their Truth-O-Meter rankings. Although The Daily Beast is accurately reporting PolitiFact's evaluations, they're doing it in a way that leads to false (or at least questionable) implications. This kind of situation -- accurate statement with questionable implication -- has been variously rated by PolitiFact in the past as being anywhere from "True" to "Half True" (again, they're inconsistent in these rankings). But, to date, PolitiFact and the Truth-O-Meter remain silent on The Daily Beast's "10 Most Ignorant Politicians" list.

As to Thales' first point, PolitiFact would not need to rate every political statement in order to negate selection bias if the selection process were random.  But that process is not random, unless we disbelieve PolitiFact's description.

And to amplify Thales' second point, PolitiFact can hardly scold the Daily Beast for its misuse of the data without undercutting its own practice of directing readers to judge individuals according to their Truth-O-Meter "report card":

(W)e thought it would be timely to look at Beck's record on the Truth-O-Meter. As you can see from the running tally in his PolitiFact file, we've rated 17 statements by the Fox News talk show host. It's fair to say that record skews toward the False end of the Truth-O-Meter.

Yes, and?