Intro: Media Trackers Announces Series on PolitiFact
Part 1: PolitiFact and the Political Parties
Part 2: PolitiFact and Third-Party Organizations
Part 3: PolitiFact and Talk Radio
Part 4: PolitiFact and Governor Scott Walker
Part 5: Conclusion on PolitiFact
We were unimpressed with the start of the series, but by the conclusion Media Trackers reached solid ground.
Concern over the direction of the series started early:
On the whole, PolitiFact can’t be called completely biased towards conservatives or liberals. By Media Trackers count, PolitiFact has devoted nearly equal ink to conservative/Republican statements as to liberal/Democrat.
The remainder of Part 1 built a comparison between PolitiFact Wisconsin's treatment of state Republican Party statements with those of its Democratic Party counterpart. The number of statements involved was very small (11 combined), but suggested that PolitiFact's editorial focus fixed more on the Democratic Party and doled out harsher ratings.
The second installment focused on the treatment of what Media Trackers calls "third party" organizations. That is, political action groups not directly associated with the political parties.
Media Trackers noted a trend opposite that from part one, albeit the two mini-studies share the problem of small sample size. The conclusion of the second part found Media Trackers on top of a live spoor:
(D)oes PolitiFact lead readers to believe that conservative third-party organizations are less likely to tell the truth? How come the organization that spent the most on negative advertising in the recall elections had just one statement reviewed? Why more scrutiny to Pro-Life groups than Pro-Choice? Why were One Wisconsin Now’s statements reviewed four times more than the MacIver Institute? And what about statements on critical stories such as the denial by Citizen Action of Wisconsin of a connection to Wisconsin Jobs Now!? Why did PolitiFact choose not to tackle that statement?In other words, the selection bias problem at PolitiFact is pretty obvious.
No one expects PolitiFact to be the “be all end all” of watchdog journalism. But when they set themselves up as the judge and jury for all political statements in the state, one has to question how they select stories and why certain groups receive far and away more scrutiny than others.
Part three looked at PolitiFact Wisconsin's treatment of local radio personalities and established Media Trackers' modern day record for small sample size. Conservative Charlie Sykes received two ratings while fellow conservative Mark Belling received one. All three ratings were of the "Pants on Fire" variety. Again, it smells like selection bias.
The fourth installment examined PolitiFact's treatment of Republican governor Scott Walker.
Media Trackers forgave PolitiFact for rating a high number of Walker's statements because of his position of power. Time will reveal the reliability of that measure.
The Media Trackers analysis noted that PolitiFact appeared to go a bit hard on Walker:
It seems that PolitiFact’s burden for truth is a bit higher for Governor Walker than it is for others. Given the “lightening rod” status of Walker, it certainly seems a bit disingenuous to call the Governor’s claim that Wisconsin is “broke” a false claim because he could just layoff workers and raise taxes to fix the deficit. And to say that Walker did not campaign on the reforms found in the Budget Repair Bill is also disingenuous given that the Governor spoke on a number of the reforms he sought, even though he did not spell out the eventual changes to collective bargaining.The anecdotes can add up.
Media Trackers seized on the common thread in its conclusion:
As Media Trackers has shown with this series, PolitiFact arbitrarily applies its scrutiny. Statements from the Democratic Party of Wisconsin have been evaluated seven times to the Republican’s two. Conservative Club For Growth have been examined seven times (three during the recall elections) while We Are Wisconsin was examined just once. Pro-Life groups have been scrutinized twice and never a Pro-Choice group.Sample sizes aside, Media Trackers settles on a conclusion well supported by a huge mass of anecdotal material collected by others. The final installment also refers to Eric Ostermeier's study pointing out PolitiFact's selection bias problem (highlighted at PolitiFact Bias here).
Each of these political groups and officials are putting out an equal number of statements on a myriad of issues every day. If PolitiFact intends to claim the mantle of watchdog journalism by “calling balls and strikes” in the name of “public service,” PolitiFact needs more transparency about how they select their stories and a review of why certain groups and individuals receive more scrutiny than others.
Though the Media Trackers conclusion about PolitiFact isn't exactly groundbreaking, the outfit deserves credit for overcoming its initial stumble and doing an independent examination of its local version of PolitiFact with the conclusion supported on those data.
Jeff adds: It's worth mentioning that PolitiFact Wisconsin is by far the most frequent target of accusations of right-wing bias. We've never found anything that sufficiently corroborates those claims and Media Trackers seems to do a capable job of dispelling that myth.