Wednesday, October 18, 2017

Fact-checking the president

When accused of focusing its fact checks on conservatives more than liberals, PolitiFact has been known to defend itself by pointing out that it has fact checked Barack Obama more than any other political figure.

We properly ridiculed that claim because it is natural for a national political fact checker to place special importance on the statements of a president. We should only be surprised if the fact checker fails to fact check the president most frequently. And now that President Donald Trump has succeeded President Obama in office, we can do some comparisons that help illustrate the point.

Please note that this comparison does have an apples-to-oranges aspect to it. PolitiFact started out with the aim of fact-checking the election campaign. Therefore, we should allow for PolitiFact to get a slow start on President Obama's first term.

We based the comparisons on the number of fact checks PolitiFact performed on the presidents between their inauguration (two of those for Obama) and Oct. 18. In fact, PolitiFact fact checked Obama more frequently in 2009 than it did when he launched his second term in 2013.



As the graph shows, through Oct. 18 PolitiFact has fact checked Trump more in 2017 than it did Obama in 2009 and 2013 combined.

Trump has an excellent shot at supplanting Obama as the figure most fact checked by PolitiFact within just four years of taking office.

And perhaps we'll never again hear PolitiFact's balance defended on the basis of its fact-checking Obama more often than other political figures.

11 comments:

  1. Like you said, presidents are going to get special attention. Also, it's entirely possible that Trump has simply made more assertions of fact than Obama did, which would naturally lead to more fact-checks. It certainly seems like Trump makes public statements more often, anyway, what with his twitter account.

    Maybe it would be a better metric if you ignored the presidents and presidential candidates, and counted up the fact-checks done on all the other political office-holders, R and D? I'd guess that presidents and candidates tend to make more statements than the average politician, which might lead to a wider variation in frequency of checks on them, but looking instead at all the other office holders should even that out a bit and make any check-frequency bias stand out more.

    Also note that something like 80% of the Politifact audience is liberal, and Politifact checks are at least partially driven by reader requests, so there might be some bias in the input as well.

    ReplyDelete
    Replies
    1. (ordinarily I'll let typos stand, but I'm deleting the earlier version of this post that misspelled "public.")

      Jake wrote:

      **Like you said, presidents are going to get special attention.**

      That's really the entire point, Jake.

      We're criticizing PolitiFact for the spurious argument that fact-checking President Obama frequently serves as some sort of evidence of its impartiality. That's poppycock, and Executive Director Aaron Sharockman of PolitiFact ought to be embarrassed to offer that reasoning in public. Yet you can see it in the Washington Post article straight from his mouth. Review it and see.

      Delete
    2. I was proposing that you look at fact checks on other elected officials because you started the piece with "When accused of focusing its fact checks on conservatives more than liberals".

      However, if you want to compare the rate of checks on Trump vs Obama, specifically, then it seems like you'd need to account for the total number of assertions each one made. In other words, if Obama made 1000 assertions of fact in public in 2009 and 2013, and Politifact only checked 88 of those, and Trump made 500 assertions of fact in, say, 2016 and 2017, but Politifact checked 250 of those (increasing that number because I'm adding a year), then you'd have a higher *rate* of checking on Trump vs Obama.

      I'm just trying to brainstorm ways you could account for the variation in the number of assertions of fact made, since comparing raw numbers won't reveal the truth if there are significant differences in the sample size.

      Delete
    3. Jake wrote:

      **I was proposing that you look at fact checks on other elected officials because you started the piece with "When accused of focusing its fact checks on conservatives more than liberals".**

      That's not one of our accusations. But it's a common one ever since Eric Ostermeier published his evaluation of PolitiFact's story selections some years ago. It's not really a good indication of bias, so it doesn't interest us that much.

      **However, if you want to compare the rate of checks ...**

      That's something we don't really want to do. To repeat our point, Aaron Sharockman's argument is spurious to an embarrassing degree.

      **I'm just trying to brainstorm ways you could account for the variation in the number of assertions of fact made, since comparing raw numbers won't reveal the truth if there are significant differences in the sample size.**

      PolitiFact uses no systematic method of selection. For a few years PolitiFact said it tried to rate an equal number of claims from each party. We've pointed out how using that criterion along with trying to pick interesting and controversial claims leads to a particular type of selection bias.

      It's inarguable that PolitiFact makes its story selections subjectively. Since we already know that (PolitiFact admits it), we don't see much point in studying it.

      Delete
  2. **That's not one of our accusations.**

    It's a bit disingenuous to start your post with an accusation and then disown it in the comments. Would it be fair to conclude that you don't currently agree with those who say Politifact checks conservatives more than liberals?

    Let me try to make my point this way. Trump has made more than twice as many assertions of fact between inauguration and October 18, 2017 than Obama did between inauguration and October 18, 2009. Therefore, Politifact is checking Trump no more often than it did Obama. Can you show that this is wrong?

    ReplyDelete
  3. Jake wrote:

    **It's a bit disingenuous to start your post with an accusation and then disown it in the comments.**

    The accusation we make is that PolitiFact defends itself from the accusation it focuses on Republicans by citing its focus on President Obama. And we're right. It's careless to confuse our accusation with one made by others, and a slander to claim it shows disingenuousness on our part.

    **Would it be fair to conclude that you don't currently agree with those who say Politifact checks conservatives more than liberals?**

    No, it would not be fair. We are agnostic on the topic. If we thought having an opinion on it was important then we would do research to support it. We don't think it's important, for reasons we have already described.

    **Let me try to make my point this way. Trump has made more than twice as many assertions of fact between inauguration and October 18, 2017 than Obama did between inauguration and October 18, 2009. Therefore, Politifact is checking Trump no more often than it did Obama. Can you show that this is wrong?**

    Rather than showing it wrong, we would approach the question by showing that you don't know it's right. Your attempt to make your point is a non-starter. There are considerable barriers to counting up the number of assertions of fact, and we have good reason to doubt you have surmounted them.

    Can you make a reasonable case that we need to show you wrong before you make an attempt to show you're right?

    ReplyDelete
    Replies
    1. The point of your post is that Trump has been fact checked more than Obama over a similar time span. I'm trying to make the point that this comparison is only valid if you control for the total number of assertions of fact each president made during that time period.

      If I were to claim that, since there are 30 sunrises in November, but 31 in December, therefore the sun likes December more, obviously that's wrong because I didn't account for the difference in the number of days in each month. Likewise, if there were 30 sunrises in both months, that would look like equal sun treatment. The correct way to compare would be to say the sun rose on 30 of 30 days in November and 30 of 31 days in December, therefore the sun favors November.

      You've counted the raw number of fact checks against Obama and Trump, and Trump's number is higher, but how does that lead to the conclusion that Politifact is fact-checking Trump more?

      Delete
    2. Bless your heart, Jake.

      **The point of your post is that Trump has been fact checked more than Obama over a similar time span.**

      No, the point of the post is what I have said it is: It shows the Sharockman's argument for PolitiFact's neutrality is bogus. And if you'd apply your brainpower to PolitiFact's methods of self-defense in a way anything like you're attempting here, you'd experience far greater success.

      **I'm trying to make the point that this comparison is only valid if you control for the total number of assertions of fact each president made during that time period.**

      That's rubbish. Let's review the quotation of Sharockman from the Washington Post.

      ****he bristles a bit at the conservative critique ("The basic critique is that fact-checkers cherry-pick statements and facts to create a false impression — usually that conservative candidates are less truthful than the liberal kind."--bww). “People say, ‘Why didn’t you fact-check Hillary Clinton’s claim about coming under fire [as first lady] in Bosnia?’ Well, we did. The person we fact-checked more than anyone else is Barack Obama. . . . The person we fact-check the most is the president. We’re going to hold the president accountable.”****

      Sharockman defends PolitiFact based on the many times it has fact-checked President Obama. Does that defense have any meaning if Sharockman doesn't control for the number of assertions of fact each president made compared to other figures? Sharockman has never given a shred of evidence that he means anything other than the sheer number of times PolitiFact did a fact check. That puts you on the horns of dilemma with your argument against us, Jake. You can't attack our argument but not Sharockman's and remain consistent.

      **You've counted the raw number of fact checks against Obama and Trump, and Trump's number is higher, but how does that lead to the conclusion that Politifact is fact-checking Trump more?**

      Because it obviously does, until you revise your sentence to make it say what you're trying to get it to say. Maybe something like "You've counted the raw number of fact checks against Obama and Trump, and Trump's number is higher, but how does that lead to the conclusion that any individual statement of Trump's has a higher chance of getting checked by PolitiFact than a statement of Obama's?"

      And once you put it that way, we'll remind you that isn't our point. Our point is to show that Sharockman's argument has no validity to it. Period. And if you feel like it you can start pestering Sharockman to prove that Obama's statements got more attention than the statements of others on a per capita basis. And then maybe you'd be consistent.

      Delete
  4. I'm from the South, too, Mr White. I know what "bless your heart" means. Let's keep things civil, please.

    I am not arguing that you should not control for the timespan over which they were rated rather than simply calculate the raw totals. I agree with that criticism of Sharockman's statement. However, even a stopped clock is right twice a day, so this doesn't necessarily mean that Sharockman didn't land on the right answer simply by flipping a coin.

    **No, the point of the post is what I have said it is: It shows the[sic] Sharockman's argument for PolitiFact's neutrality is bogus. And if you'd apply your brainpower...**

    Forgive me, but my applied brainpower notes that you did not stop at saying Sharockman's argument is bogus. You went on to conclude:

    **...through Oct. 18 PolitiFact has fact checked Trump more in 2017 than it did Obama in 2009 and 2013 combined.**

    **Trump has an excellent shot at supplanting Obama as the figure most fact checked by PolitiFact within just four years of taking office.**

    You make the claim "it shows the[sic] Sharockman's argument for PolitiFact's neutrality is bogus." In other words, that Politifact can be shown to be biased based on the comparative raw number of fact-checks of Trump vs Obama when controlling for (and only for) the timespan over which you measure them. My criticism of your method is that you still have not controlled for all factors, specifically the raw number of all assertions of fact from each president during that time period.

    Mr White, you will find that I have side-stepped the bull and left you on the horns of that dilemma of yours. You can not argue both that you should control for timeframe and that you should ignore relative rates. An apples-to-oranges comparison can lead you blindly to a false conclusion; that's why there's an idiom.

    Let's say Luke has a total of ten points in school so far, and Kermit has 100. "Kermit is way ahead of Luke," you conclude. This is essentially Scharockman's defense. It's flawed because you didn't account for the fact that Luke has been in school for one year while Kermit has been in school for eight (as you ably point out).

    So now we look at just the first year of Kermit's record and see he had earned five points at his one-year mark. Compared to Luke's ten points, you now retract your earlier conclusion and instead state, "Over a comparable time span, Luke would be ahead of Kermit." This is analogous to your original article's criticism of Scharockman's defense.

    The flaw in your article's criticism is that you didn't account for the fact that, in their respective first years, Kermit had 5 of 5 possible points in that year, while Luke has 10 of 20 possible points in his first year. Knowing that, you would conclude that Kermit is, in fact, in the lead in this comparison, because he scored 100% in his freshman year while Luke scores 50%.

    ReplyDelete
    Replies
    1. ...
      Please note that I'm also not saying Politifact is fair and Sharockman is correct. Obama 2009 was checked on 53 out of (unknown number!) assertions. Trump 2017 was checked on 107 out of (unknown number!) assertions. I don't know either of those unknowns. However, I do know that until you can give at least a reasonable estimate of those numbers, you are in no more position to draw conclusions about who gets checked the "most" than is Sharockman.

      I have no problem accepting that by "Sharockman's argument has no validity to it" you can be taken to mean "Sharockman's argument does not have a solid mathematical basis."

      However, you make clear your further conclusion that Trump is under more scrutiny from Politifact than was Obama:

      **...through Oct. 18 PolitiFact has fact checked Trump more in 2017 than it did Obama in 2009 and 2013 combined.**

      **Trump has an excellent shot at supplanting Obama as the figure most fact checked by PolitiFact within just four years of taking office.**

      I'm not arguing that you didn't add up the checks that exist and find a higher number for Trump. I'm arguing that using this calculation as the basis of your assertion about Politifact's bias in this instance is insufficient to support your conclusion. Of course, that doesn't necessarily mean that you didn't land on the right answer simply by flipping a coin.

      **And if you feel like it you can start pestering Sharockman...**

      That's funny, he actually sent me to you. =D

      Delete
    2. Jake wrote:

      **I know what "bless your heart" means**

      4c: https://www.merriam-webster.com/dictionary/heart

      **You make the claim "it shows the[sic] Sharockman's argument for PolitiFact's neutrality is bogus." In other words, that Politifact can be shown to be biased based on the comparative raw number of fact-checks of Trump vs Obama when controlling for (and only for) the timespan over which you measure them.**

      That's a non-sequitur. I've already explained to you that we see no value in using these numbers to show bias. Many critics of PolitiFact have taken that route. We do not and never have. The numbers provide evidence of selection bias, and that's it.

      Speaking of civility, stop saying that I have to mean thus-and-such when I explicitly say otherwise. If we thought tallying these types of numbers helped serve to prove bias then we'd be using that approach in our research. We don't do that. End of story.

      **An apples-to-oranges comparison can lead you blindly to a false conclusion; that's why there's an idiom.**

      You've agreed with the conclusion, have you not (Sharockman's argument lacks value)? Any other conclusion is your invention (and a straw man candidate).

      **This is essentially Scharockman's defense. It's flawed because you didn't account for the fact that Luke has been in school for one year while Kermit has been in school for eight**

      It's flawed for all kinds of reasons, timespan representing one of the least of them.

      **The flaw in your article's criticism is that you didn't account for the fact that, in their respective first years, Kermit had 5 of 5 possible points in that year, while Luke has 10 of 20 possible points in his first year.**

      Do you know what a "counterexample" is in argument? We used PolitiFact's numbers as a counterexample to Sharockman's argument. We don't need to account for rates or percentages of total numbers of statements unless that is a feature of Sharockman's argument. There's no evidence of that in Sharockman's argument.

      You're attacking a straw man.

      **you are in no more position to draw conclusions about who gets checked the "most" than is Sharockman.**

      On the contrary, it's perfectly accurate to say PolitiFact has checked X the most by PolitiFact if the number of PolitiFact fact checks for X is greater than for any other figure. The only issue is what the numbers mean apart from that. Like, does that number show that the party to which X belongs receives the same treatment in terms of fairness compared to the other?

      I see that as you continue with your comments you fail to appreciate that point:

      **
      However, you make clear your further conclusion that Trump is under more scrutiny from Politifact than was Obama:

      **...through Oct. 18 PolitiFact has fact checked Trump more in 2017 than it did Obama in 2009 and 2013 combined.**

      **Trump has an excellent shot at supplanting Obama as the figure most fact checked by PolitiFact within just four years of taking office.**
      **

      Those are factual observations, and that's it. They do not support your logical leap that we take those numbers to mean that Trump receives closer scrutiny from PolitiFact than did Obama.

      **I'm not arguing that you didn't add up the checks that exist and find a higher number for Trump. I'm arguing that using this calculation as the basis of your assertion about Politifact's bias in this instance is insufficient to support your conclusion.**

      You mean that conclusion that we do not state and you create for us?

      **Of course, that doesn't necessarily mean that you didn't land on the right answer simply by flipping a coin.**

      Or evaluating parallel fact checks to look for inconsistent standards. That's an option, right?

      **That's funny, he actually sent me to you.**

      Now that *is* funny. You should definitely start pestering him about his silly argument. He really ought to lay that one to rest.

      Delete