PunditFact's editorial page

Oops--are we being redundant? Sorry! Here's a portion of PunditFact's main page from Dec. 28, 2015.

Partial screen capture from PolitiFact.com/PunditFact/ Dec. 18, 2015

That, ladies and gentlemen, is a pictorial editorial.

It's not an unaltered photograph of Donald Trump. It's not a photo at all. It is an artist's rendering of Donald Trump, created to communicate an editorial message to PunditFact's readers. Note how Trump's fingers are crossed as he speaks, a traditional gesture of those who know they are speaking falsely.

Material like this makes PunditFact itself a pundit of sorts.

We've long noted the fact that PolitiFact and its various franchises blur the traditional line between straight news reporting--what some might expect from journalism billed as "fact-checking"--and editorializing. That's why we call PolitiFact "the Fox News of fact-checking" and PolitiFact staffers "liberal bloggers."

Their journalism is not objective. When PolitiFact creator Bill Adair calls fact checking a "new form of journalism" perhaps he has in mind that deliberate blurring of the lines.

Or not. Either way, we don't appreciate it.

Yet more scintillating offsite criticism

Gotta love it.

A member of a message board posted some of our material for discussion--it seems somebody was using Angie Drobnic Holan's chart to draw firm conclusions about candidate truthfulness.

How were our views criticized this time?

Behold:

For the record, the site (...) is run by two bloggers. One of whom lists beer as one of his interests and the other documents the bands that he likes as a credential.

So this time we get not just ad hominem fallacies but incredibly lame ad hominem fallacies. And the earlier post of our material included some of our responses to similar criticism. I guess it truly doesn't help to point out to people when they're reasoning fallaciously.

Dude: It's the movies I like that serve as a credential. Not the bands. Sheesh. What a maroon. j/k

On the positive side, he didn't accuse us of posting anonymously as some have done. So, credit where it's due.

Clearly we need to rush forward with our new page answering the most common criticisms (count ad hominem fallacies as one such) we receive.

PolitiMath from PolitiFact New Hampshire

What's new with PolitiMath?

PolitiFact New Hampshire, lately the Concord Monitor's partnership with PolitiFact, gives us a double dose of PolitiMath with its July 2, 2015 fact check of New Hampshire's chief executive, Governor Maggie Hassan (D).

Hassan was the only Democrat to receive any kind of false rating ("False" or "Pants on Fire") from PolitiFact New Hampshire in 2015. PolitiFact based its ruling on a numerical error by Hassan and added another element of interest for us by characterizing Hassan's error in terms of a fraction.

What type of numerical error earns a "False" from PolitiFact New Hampshire?

PolitiFact summarizes the numbers:

In her state of the state address, Hassan said that "6,000 people have already accessed services for substance misuse" through the state’s Medicaid program.

There is no question that substance abuse in the state is a real and pressing problem, and the statistics show that thousands have sought help as a result of the state’s expanded Medicaid program. But Hassan offered (and later corrected) a number that simply wasn’t accurate. The real total is closer to 2,000 -- about one-third the amount she cited.

We rate her claim False.

Describing Hassan's mistake as a percentage error using PolitiFact's figures, Hassan exaggerated her figure by about 230 percent. PolitiFact gave Hassan no credit for her underlying point.

In our PolitiMath series we found the closest match for this case from PolitiFact Oregon. PolitiFact Oregon said conservative columnist George Will exaggerated a figure--by as much as 225 percent by our calculations. The figure PolitiFact Oregon found was uncertain, however, so Will may have exaggerated considerably less using the range of numbers PolitiFact Oregon provided.

In any case, PolitiFact Oregon ruled Will's claim "False." PolitiFact Oregon gave Will no credit for his underlying argument, just as PolitiFact New Hampshire did with Gov. Hassan.

Percent Error and Partisanship

One of our research projects looks in PolitiFact's fact checks for a common error journalists make. We reasoned that journalists would prove less likely to make such careless errors for the party they prefer. Our study produced only a small set of examples, but the percentage of errors was high and favored Democrats.

PolitiFact New Hampshire's fact check of Gov. Hassan draws some consideration for this error, giving us the second mathematical element of note.

PolitiFact could have expressed Hassan's mistake using a standard percentage error calculation like the one we used. We calculated a 230 percent error. But PolitiFact New Hampshire did not use the correct figure (1,800) as the baseline for calculating error. Instead, the fact checkers used the higher, incorrect figure (6,000) as the baseline for comparison: "about one-third the amount she cited."

Using the number "one-third" frames Hassan's error nearer the low end. "One-third" doesn't sound so bad, numerically. Readers with slightly more sophistication may reason that the "one-third" figure means Hassan was off by two-thirds.

Sometimes using the wrong baseline makes the error look bigger and sometimes it makes the error look smaller. In this case the wrong baseline frames Hassan's mistake as a smaller error. The Democrat Hassan gains the benefit of PolitiFact's framing.

"Supreme conservative whinybaby" strikes back

As ever, we're interested in answering criticism of PolitiFact Bias. But you wouldn't believe the work we have to do to find anything that's actually worth taking to time to answer.

For example, check out this devastating set of criticisms from the "Radio Free Liberal" message board. One member of the board posted a link to our site, saying it would help explain how/why PolitiFact is biased to the left.

Then we get this:

This is an anti-PolitiFact website which means it is biased. Really, you could have done something better than picking the first website appearing on Google. This is almost as funny as the time when you were challenged to backup one of your claims and you posted a link to a white supremacist website and then claimed you didn't know it was a white supremacist website.

An anti-PolitiFact website is automatically biased? Doesn't that beg the question? Plus we're indicted for choosing a website name that puts us at the top of search results!

There's no substance at all to this criticism. Is it really helpful to people to explain when they're assuming their conclusion or committing a genetic fallacy? If that helps, here we go: The criticism assumes that criticism of PolitiFact is biased and therefore invalid. How would one justify that assumption?

How about this:

All he had to do is to go to the sponsor of the web blog to get the back story of how they manipulated their own "politifact" story. It's all in the open, Tell a lie and repeat it often... the dummies will follow.
http://bewznewznvewz.blogspot.com/2012/ ... ts_31.html

I have no idea what this person is trying to say. The link goes to Jeff's cross-post of an article we also have posted at PolitiFact Bias. As far as I can tell, this chap thinks it represents evidence that we are admitting we are making up stuff about PolitiFact. If anybody thinks they see the same thing in that link he does, please drop us a line.

This next commenter apparently likes the guitarist Steve Morse, and therefore can't be all bad:

And the other guy sounds like a supreme conservative whinybaby. Look at these sites of his

https://www.blogger.com/profile/07608604859044029293


Although, he seems to be a Steve Morse fan. I can get with that.

I guess it's a board for liberals to get together and whine about how whiny conservatives are. But let's just assume I'm not just a supreme conservative whinybaby. Let's assume I'm the supreme conservative whinybaby. With that out of the way, can we agree that a criticism that fails to get beyond name-calling amounts to an ad hominem fallacy?

Come on, Radio Free Liberal. Give us something challenging to work with. Assuming that criticism of PolitiFact is invalid thanks to bias is essentially the same error as assuming PolitiFact is invalid because it is biased.

You can do better, right? We await your counterwhine.

Refuting PolitiFact on mismatch theory

Back on Dec. 11, 2015, PolitiFact published a fact check of a statement by Supreme Court Justice Antonin Scalia. Scalia told his courtroom audience of a brief that said most black scientists do not come from top-flight schools for which they may not be academically prepared. Scalia was alluding to "mismatch" theory. As Scalia put it, "They come from lesser schools where they do not feel that they’re being pushed ahead in classes that are too fast for them."

Mismatch theory is the idea that if students aren't academically prepared to go to a higher-level school, the student is better off going to a school better suited to their level of preparation.

PolitiFact rated Scalia's claim "Half True." Scalia, PolitiFact said, was accurate that most black scientists come from lesser schools, but the statistic doesn't stem directly from academic mismatch.

But what really caught our eye about the PolitiFact fact check was this (bold emphasis added):

Regarding the larger point — that black students fare better at "lesser" schools because their academic credentials are better matched to the curriculum — the evidence is mired in controversy. There is some scholarly research that backs up this point, but there is also scholarly research that refutes it.

It's not uncommon to see the word "refutes" misused this way, improperly substituting it for a word like "rebuts." An argument that is refuted is a defeated argument. An argument that receives a rebuttal is contested. PolitiFact's choice of words was poor, biased or both.

We were reminded of PolitiFact's skim-the-surface treatment of Scalia's remarks by a much better treatment by Conor Friedersdorf in The Atlantic. This portion we found striking:

I’m baffled that any journalists are treating it as settled, even as tenured social-scientists at top-tier universities declare that it deserves to be taken seriously. No one, it seems, can yet provide a precise answer to the question, “at what point do disparities in GPA, SAT score, or high-school quality start to matter,” even though everyone surely agrees that they matter at some inflection point.

While PolitiFact stops short of claiming the issue is settled against mismatch theory, its treatment is decidedly more negative than the one at The Atlantic.

And since when is it a proper fact check of Scalia without specifically referencing the brief to which he referred?

PolitiFact points toward this brief as the likely source of Scalia's statement, but the fact check provides no link (whether by error or not). We can't find the document linked by PolitiFact and we don't see a clear connection between Scalia's remarks and the document PolitiFact pointed toward.

Moreover, as Alison Somin points out, the mainstream media narrative in which PolitiFact wallows skipped over the fact that justices routinely ask attorneys questions based on briefs they do not necessarily agree with. PolitiFact glossed over that context.

Critics have not refuted mismatch theory. It's fair game for Scalia to bring up that theory using words from an amicus brief. It's notable that the attorney in question did not respond as pedantically as did the mainstream media.

Matching Scalia's questioning to the right brief--that would be fact-checking. What PolitiFact did--not so much. PolitiFact's "fact check" was an op-ed about mismatch theory slanted against Scalia and mismatch theory.



Edit: Added link to PF story at "Half True" text- Jeff 12/23/2015 1508PST

The unbearable lameness of PolitiFact's 2015 "Lie of the Year"

PolitiFact has chosen statements by Donald Trump as its "Lie of the Year" for 2015, effectively making Donald Trump PolitiFact's liar of the year.

Why do we say it's lame?

PolitiFact's "Lie of the Year" has always been the fact checkers' editorial. As such, it will show a tendency to follow a journalistic narrative.

What's this year's key journalistic narrative? Donald Trump vs. the media.

Out of one side of its mouth PolitiFact denies it charges people with lying. With the other side ...

Two years in a row (arguably three), PolitiFact chose a winner that was not on the list of candidates readers were given when voting for the readers' version of PolitiFact's "Lie of the Year."

Picking a group winner works well for PolitiFact since it's hard to contest the choice. Contest one of the supposedly false claims on the basis of fact and PolitiFact can always point to another on the list and emphasize that one. Justified, see?

It's so predicable that I predicted it shortly after PolitiFact announced its candidates:

I'm going to invoke PolitiFact's history over the last two years of not putting the eventual winner on the list of candidates. My pick is "Statements by Donald Trump" as a parallel to last year's "Statements about Ebola."

It shouldn't be that easy.

Lame.

PolitiFact's Unethical Internet Fakery

What's Fake on the Internet: Unbiased fact checkers

We stumbled across a farewell post for Caitlin Dewey's rumor-debunking "What was Fake” column in the Washington Post. In the column, Dewey notes a change in the Internet hoax business, namely that rumors are often spread intentionally via professional satire sites seeking click traffic, and is calling it quits "because it’s started to feel a little pointless."

While Dewey's column focused more on viral Internet rumors than politics specifically, we were struck by the parallel between her observations and our own regarding PolitiFact. She laments that the bogus stories are so easily debunked that the people who spread the misinformation aren't likely to be swayed by objective evidence. She then highlights why hoax websites have proliferated:

Since early 2014, a series of Internet entrepreneurs have realized that not much drives traffic as effectively as stories that vindicate and/or inflame the biases of their readers. Where many once wrote celebrity death hoaxes or “satires,” they now run entire, successful websites that do nothing but troll convenient minorities or exploit gross stereotypes.

Consider that trend when you see this chart that ran with PolitiFact editor Angie Holan's NYT opinion article:

Image via NYT screengrab

The chart, complete with bar graphs and percentages, frames the content for readers with a form of scientific legitimacy. But discerning anything from the aggregate total of their ratings amounts to pure hokum. The chart doesn't provide solid evidence of anything (with the exception of PolitiFact's selection bias), but it surely serves to "vindicate and/or inflame the biases of their readers."

We've gone into detail explaining why PolitiFact's charts and report cards amount to junk science before, but simply put there's multiple problems:

1) PolitiFact's own definitions of their ratings are largely subjective, and their standards are applied inconsistently between editors, reporters, and individual franchises. This problem is evidenced by having nearly identical claims regarding a 77-cent gender wage gap being rated everywhere from True to Mostly False and everything in between.

2) Concluding anything from a summary of PolitiFact's ratings assumes each individual fact check was performed competently and without error. Further, it assumes PolitiFact only rates claims where actual facts are in dispute as opposed to opinions, predictions, or hyperbolic statements.

3) PolitiFact's selection bias extends beyond what specific claim to rate and into what specific person to attribute a claim to (something we've referred to as attribution bias). For example, an official IG report included an anecdote that the government was paying $16 for breakfast muffins. Bill O'Reilly, ABC and NPR all reported the figure in the report. PolitiFact later gave Bill O'Reilly a Mostly False rating for repeating the official figure. This counts as a Mostly False on his report card while ABC, NPR, and even the IG who originally made the claim are all spared this on their "truthiness" chart.

4) The most obvious problem is selection bias. Even if we assume PolitiFact performed their fact checks competently, applied their standards consistently, and attributed their ratings correctly, the charts and report cards still aren't evidence of anyone's honesty. Even PolitiFact admits this, contrary their constant promotion of their report cards.

To illustrate PolitiFact's flaw consider investigating a hundred claims from President Truman to determine their veracity. Suppose you find Truman made 20 false claims, and you then publish only those 20 False claims on a chart. Is this a scientific evaluation of Harry Truman's honesty? Keep in mind you get to select which claims to investigate and publish. Ultimately such an exercise would say more about you than Harry Truman. The defense that PolitiFact checks both sides falls flat (PolitiFact get's to pick the True claims too, and in any event is an appeal to the middle ground).

We've documented how PolitiFact espouses contradictory positions on how to use their data. PolitiFact warns readers they're "not social scientists," but then engages in a near constant barrage promoting their "report cards," claiming they're useful for showing trends.

Whenever PolitiFact promotes charts like the one posted in the NYT article, the overwhelming response on Facebook and Twitter is to send the chart viral with unfounded claims that the conservative bogeymen are allergic to the truth. How can PolitiFact ethically promote such assertions when they know their report cards offer no objective data about the people they rate?

Instead of telling readers to discount any conclusions from their misleading charts, PolitiFact actively encourages and promotes these unscientific charts. That's not how honest, professional journalists seeking truth behave. On the other hand, it's behavior we might expect from partisan actors trolling for web traffic.

So why are people so intent on spreading PolitiFact's bogus charts based on bad information? Perhaps Dewey uncovered the answer:

[I]nstitutional distrust is so high right now, and cognitive bias so strong always, that the people who fall for hoax news stories are frequently only interested in consuming information that conforms with their views — even when it’s demonstrably fake.

No worries, though. PolitiFact editor Angie Holan assures us "there's not a lot of reader confusion" about how to use their ratings.

We're bummed to see Dewey close down her weekly column, as she arguably did a more sincere job of spreading the truth to readers than PolitiFact ever has. But we're glad she pointed out the reason hoax websites are so popular. We suggest the same motivation is behind PolitiFact publishing their report cards. Is there much difference between hoax sites spreading bogus rumors and PolitiFact trolling for clicks by appealing to liberal confirmation bias with its sham charts and editorials masquerading as fact checks?

Contrary to being an actual journalistic endeavor, PolitiFact is little more than an agenda-driven purveyor of clickbait.

Attack of the offsite critics!

Unlike PolitiFact, PolitiFact Bias regularly engages its critics.

We gained an opportunity recently to have some back-and-forth with some off-site critics. We ran across some traffic to our site coming from a message board. Somebody had linked to this site, and the link had drawn dismissive responses. But the responses were quite vague about the reasons behind their dismissiveness. So I joined the forum to see if I could uncover those reasons.

The first two detailed reasons were based on misunderstandings. One critic thought we were saying that Republicans and Democrats lie equally. We don't say that because we have no idea. Finding out would take a detailed and complex study. We're not aware of any serious attempt at such a thing (we're aware of one unserious attempt).

Another critic thought we were saying Democrats and Republicans lie proportionately to each other. Again, that's not what we say. Finding out the proportions of Democrat and Republican lies would take a detailed study. We don't pretend to have done anything of the kind.

These misunderstandings stem from our attempts to describe what results PolitiFact's ratings ought to give if PolitiFact used the types of editorial discretion it describes, but did so with ideological blindness. PolitiFact's collection of ratings would still suffer from selection bias, so the ratings would not support generalizations about the proportions of lies for either of the two political parties. But the selections would have a "shape" of sorts, and we say that shape should be about proportional for both parties regardless of whether one states more untruths than the other.

What did we learn from engaging our critics on this occasion? We found no legitimate criticism, but found good reason to improve our description of a minor point we make at the site.


Correction Dec. 21, 2015: Fixed some minor typos and added a comma.