The (annotated) Principles of PolitiFact

In early 2011 PolitiFact published "Principles of PolitiFact and the Truth-O-Meter" in response to questions about its story selection process.  After another year of watching PolitiFact bend its principles like so much cooked spaghetti, we at PolitiFact Bias introduce The (annotated) Principles of PolitiFact and the Truth-O-Meter, or TAPoP+tToM.  This feature will continue to grow and change as the PFB team adorns it like a Christmas tree with PolitiFact's gaffes, blunders and inconsistencies.  The original text is in black.  Our annotation occurs in red italics punctuated by the shades of visited and unvisited exit links.   Enjoy.

PolitiFact is a fact-checking website that rates the accuracy of claims by elected officials and others who speak up in American politics.  PolitiFact National is run by editors and reporters from the St. Petersburg Times, an independent newspaper in Florida. The PolitiFact state sites are run by news organizations that have partnered with the Times. The state sites follow the same principles as the national site.

PolitiFact staffers research statements and rate their accuracy on the Truth-O-Meter, from True to False. The most ridiculous falsehoods get the lowest rating, Pants on Fire.

The site has two other features:
  • The Flip-O-Meter, which rates whether an elected official has been consistent on an issue.
  • Promise meters, such as the Obameter and the GOP Pledge-O-Meter, that rate the status of elected officials' campaign promises.
PolitiFact checks claims by elected officials, candidates, leaders of political parties and political activists. We examine officials at all levels of government, from county commissioners to U.S. senators, from city council members to the president.

We also check claims by groups involved in the discourse -- political parties, advocacy groups and political action committees. We check claims by pundits, talk show hosts and columnists. We also examine claims in widely circulated chain e-mails.

Choosing claims to check

Every day, PolitiFact staffers look for statements that can be checked. We comb through speeches, news stories, press releases, campaign brochures, TV ads, Facebook postings and transcripts of TV and radio interviews. Because we can't possibly check all claims, we select the most newsworthy and significant ones.  We also solicit suggestions from our (predominantly liberal) readership, but we decided not to mention that as one of our principles.  That's on a need-to-know basis and you don't need to know.

In deciding which statements to check, we ask ourselves these questions:

  • Is the statement rooted in a fact that is verifiable? We don’t check opinions (except when we make an opinion the most prominent fact check of the year), (twice), and we recognize that in the world of speechmaking and political rhetoric, there is license for hyperbole. (That doesn't mean we won't check hyperbole.) (Repeatedly.)
  • Is the statement leaving a particular impression that may be misleading?
  • Is the statement significant? (Or  cute?, or insignificant?) We avoid minor "gotchas"’ on claims that obviously represent a slip of the tongue.   Exclusions apply (1,2,)
  • Is the statement likely to be passed on and repeated by others?
  • Would a typical person hear or read the statement and wonder: Is that true?
  • Was it a joke? (1)

Transparency and on-the-record sources

PolitiFact relies on on-the-record interviews and publishes a list of sources with every Truth-O-Meter item. When possible, the list includes links to sources that are freely available, although some sources rely on paid subscriptions. The goal is to help readers judge for themselves whether they agree with the ruling.  That said, one of our favorite tricks is to use a very limited quotation that features an ellipsis (indicating material missing from the quotation) and sourcing it to a subscription source. Either just trust us or pay up! Ha ha.

Truth-O-Meter rulings

The goal of the Truth-O-Meter is to reflect the relative accuracy of a statement.  Relativism, after all, is the key to getting at the truth of things.

The meter has six ratings, in decreasing level of truthfulness:

TRUE – The statement is accurate and there’s nothing significant missing.  If we say it's significant then it's significant.  Otherwise it's not significant.

MOSTLY TRUE – The statement is accurate but needs clarification or additional information.

HALF TRUE – The statement is partially accurate but leaves out important details or takes things out of context.  This definition is subject to change without any public announcement at PolitiFact's sole discretion, and PolitiFact reserves the right to use two different and contradictory definitions of "Half True."  Or change the definitions without notice.  We will, however, make a big deal about something like changing the name of the third-to-lowest rating from "Mostly True" to "Mostly False."

BARELY TRUE – The statement contains an element of truth but ignores critical facts that would give a different impression.  We will serve as the judges of what is critical and what is not.  What could be more objective?

FALSE – The statement is not accurate.  Well, sometimes the statement is accurate but the speaker doesn't deserve any credit for it.

PANTS ON FIRE – The statement is not accurate and makes a ridiculous claim.  The point at which a statement is deemed "ridiculous" will fall to the sole discretion of PolitiFact's editors based on subjective impression, since they are the objective ones.

CAVEATS:

  • We might bump up a notch or two if the subject is working from memory (1).


Principles in Truth-O-Meter rulings  

Words matter -- We pay close attention to the specific wording of a claim.  This enables us to parse things in favor of claims to which we are partial and against claims to which we are averse.  For example, if a pol says "pistol" and all we can find are stats for handguns then we'll treat "pistol" like it means "handgun." Is it a precise statement? Does it contain mitigating words or phrases?  Again, we'll judge what counts as "mitigating."  Trust us.  We're a nonpartisan fact checking organization and we won a Pulitzer Prize.
Context matters -- We examine the claim in the full context, the comments made before and after it, the question that prompted it (if the question is inaudible, however, we're free to make assumptions (1)), and the point the person was trying to make.
Burden of proof -- People who make factual claims are accountable for their words and should be able to provide evidence to back them up. We will try to verify their statements, but we believe the burden of proof is on the person making the statement.
Statements can be right and wrong -- We sometimes rate compound statements that contain two or more factual assertions. In these cases, we rate the overall accuracy after looking at the individual pieces.
Timing – Our rulings are sometimes based on when a statement was made and on the information available at that time.  But sometimes not (1).

Process for Truth-O-Meter rulings

A PolitiFact writer researches the claim and writes the Truth-O-Meter article with a recommended ruling. After the article is edited, it is reviewed by a panel of at least three editors that determines the Truth-O-Meter ruling.  If the three editors ever disagree that's only for us to know.  We don't want you that informed.  Majority rule helps us out just in case, God forbid, a non-liberal editor ever joins our team.

The Flip-O-Meter

The Flip-O-Meter rates an official's consistency on an issue. The rating is not making a value judgment about a politician who changes positions on an issue. Indeed, voters often like politicians who are flexible and have the ability to compromise or adapt their positions to the wishes of constituents. Still, accusations of shifting positions are so common in politics that it is valuable to have us provide an analysis of a shift and rate the amount of change. The Flip-O-Meter has three ratings:
  • No Flip – No significant change in position.
  • Half Flip – A partial change in position.
  • Full Flop – A complete change in position.
The writing, editing and rating process for Flip-O-Meter items is the same as the process for Truth-O-Meter items.

Corrections and review

We strive to make our work completely accurate. When we think we make a mistake, we correct it if we feel like it and note it on the original item. If the mistake is so significant that it requires us to change the ruling, we will do soAnd if that's a tautology then it's a tautology.  But sometimes we'll change our ruling even if we won't admit to any mistake.  Go figure!


Readers who see an error should contact the writer or editor.  This is typically a waste of your time since we only change things when we feel like it.  We very probably will never bother to address your concerns (1,2,3,4,5,6).  But putting it on our Principles page at least makes it look like we care. Their names are listed on the right side of every Truth-O-Meter item. Clicking on their names will take you to their bio pages, where you can find their e-mail addresses.

When we find we've made a mistake, we correct the mistake.  LOL.  Sorry.  Even we can't keep a straight face after that one.
  • In the case of a factual error, an editor's note will be added and labeled "CORRECTION" explaining how the article has been changed.  If we don't want to admit factual error we'll skip the "CORRECTION" tag and just say we received new information.  Or we'll just correct the story and without an editor's note at all (1).
  • In the case of clarifications or updates, an editor's note will be added and labeled "UPDATE" explaining how the article has been changed.
  • If the mistake is significant, we will reconvene the three-editor panel. If there is a new ruling, we will rewrite the item and put the correction at the top indicating how it's been changed.
  • Much of the time, though, we'll do changes simply labeled "Editor's note" whether or not we made an error (1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,)Isn't that good enough?
  • Or we might change a ruling without "Editor's note," "Correction" or "Update." (1)
  • Or we might make a factual error and label the notice as an "Update" rather than a "Correction." (1)  And the "Update" may occur in the text of the item instead of in the editor's note (1).  Confused? Good!  We're amazingly accurate and you can trust us.
  • Would you believe that as of Oct. 29, 2011 we've published only (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16*, 17, 18*, 19, 20**, 21, 22, 23, 24, 25, 26) 26 correction notices according to the specified form and only six prior to Jan. 1, 2011?  We're that good.  Really.
  • When we finally get around to publishing a corrections page we'll omit some of the corrections just listed.  Now guess whether we'll publish a correction of that mistake!
* Giving credit for "Correction appended"
**  A correction notice with a change in the rating

We respect that reasonable people can reach different conclusions about a claim.  That's what relativism is all about. If you disagree with a ruling, we encourage you to e-mail the writer or editor with your comments about our ruling. We find these very amusing except when it indicates that our majority audience of liberals is displeased, in which case we'll revise our ruling based solely on our readers' outrage. Of course, we'll cite new information but won't tell you what it is. You can also post comments to our Facebook page or write a letter to the editor. We periodically publish these comments in our Mailbag feature.  No, not all of them, you dolt.


We will not be keeping a record of updates to this page, with the exception of correction notices.  Those will appear below if needed.

3 comments:

  1. Well... I've seen just about enough, so I think I can wrap this up. I came here with the sincere intent of looking for convincing evidence of liberal bias at Politifact. (I don't want to cite Politifact data if it's biased.) but what I've found is so much circular reasoning it's made me dizzy. I suppose that if I had come here already believing that Politifact is biased, based upon its finding that right-wing politicians lie much more often than left-wing, I'd have plenty of fodder to satisfy my confirmation bias. But I didn't. I came here looking for fact-based evidence to disprove the null hypothesis that Politifact is not biased. I can't find it, so my null hypothesis survives... I'll keep looking for a while longer. What I've found instead is so much of the sanctimonious petulance, playground taunts, and small-mindedness I typically find on right-wing blogs, and frankly, I find that very, very tedious.

    So... Just keep on preaching to the choir folks. For my part, I'm close to concluding that Politifact is what it claims to be... And even if Politifact were to have a liberal bias of, say, 2X (which is highly unlikely, based on what I've seen), it's still a dismal indictment against the ravings of right-wing politics... And that leads me to ask: If the far-right is so bloody correct and justified, why should they EVER have to spout the BS that they so often do? Hmmmmmm????

    ReplyDelete
  2. TreeDoctor wrote:

    **what I've found is so much circular reasoning it's made me dizzy.**

    We take great care to avoid fallacious reasoning, so we would be extremely grateful if you would give one or more examples of our circular reasoning.

    **I came here looking for fact-based evidence to disprove the null hypothesis that Politifact is not biased.**

    Why on earth would that be your null hypothesis, given that surveys of journalists (especially Washington journalists) show that they skew well to the left of the average American? We point that out on our "About" page. Was that one of the pages you visited?

    **I can't find it, so my null hypothesis survives...**

    What a scientist! What if you had started with a different null hypothesis, that PolitiFact biased either left or right? Those would have likewise survived your search for truth, right?

    **And that leads me to ask: If the far-right is so bloody correct and justified, why should they EVER have to spout the BS that they so often do?**

    It's a pity you decided not to provide concrete examples in support of any of your complaints. I'll give you one sensational counterexample. Democrats often cite the raw gender wage gap. Why? It is well known that the raw wage gap does not accurately represent a wage gap caused by gender discrimination. There's literally no good reason to cite the raw wage gap when advocating equal pay *except as a means to deceive.*

    But if a Democrat cites the raw wage gap and avoids specifically saying it's for the same work, PolitiFact may give it a "True" or "Mostly True" rating. Even when it occurs plainly in the context of arguing for equal pay for equal work.

    If you think that's us using circular reasoning then maybe you're reasoning in a circle.

    ReplyDelete

Thanks to commenters who refuse to honor various requests from the blog administrators, all comments are now moderated. Pseudonymous commenters who do not choose distinctive pseudonyms will not be published, period. No "Anonymous." No "Unknown." Etc.