E-mail to Politifact

Readers of this site will likely know I am critical of the rise in "fact checking" as I feel what these "fact checkers" do is often more akin to punditry than fact checking. One such organization, Politifact, did a Q&A session on Twitter today where I happened to see it say:

In response to claims it is liberally biased. I felt this tweet exaggerates the impartiality of Politifact due to seeing a number of articles it published which seemed biased against conservative or toward liberal views. As such, I asked:

The answer I got directed me to e-mail Politifact with any specific concerns. I did so. I initially planned to give a sampling of issues in a number of "fact checks," but in the process of finely parsing one article I selected, I discovered so many problems there was little space. Indeed, there were enough problems with that article I couldn't even discuss them all in my e-mail.

I don't know what to expect from Politifact regarding this e-mail, but since I went through the trouble of writing it, I figured I might as well post it online for people to see. Perhaps it will give some insight as to why I don't hold much respect for "fact checkers." The e-mail discusses only a tiny fraction of the issues I've seen in Politifact's fact checking, and even so, it runs over a 1,000 words. Oh, and yes, I do realize there are a few typos in it. That's what I get for not having an editor. (I count three. How many do you count?)


Hello,

I have noticed a number of errors in the "Fact checks" on your site, and it was suggested to me I should contact you here about them. I will focus primarily on articles about scientific issues due to my personal knowledge and views on science commonly being portrayed as partisan.

To begin, I would like to highlight a relatively trivial example regardin one Rush Limbaugh. A "fact check" on your site lists this as what is being checked:

The presence of gorillas calls into question the concept of evolution.
— Rush Limbaugh on Tuesday, May 31st, 2016 in the Rush Limbaugh Show

http://www.politifact.com/punditfact/statements/2016/jun/03/rush-limbaugh/rush-limbaugh-asks-why-cincinnati-zoo-gorilla-hadn/

A naive person might think that this statement is attributed to Rush Limbaugh means he actually said it. He did not. Unlike most things being checked on your site, this statement is not placed in quotation marks because it is a paraphrase rather than a quotation. What Limbaugh actually said is:

"A lot of people think that all of us used to be gorillas, and they're looking for the missing link out there," Limbaugh said. "The evolution crowd. They think we were originally apes. I've always had a question: If we were the original apes, then how come Harambe is still an ape, and how come he didn't become one of us?"

The paraphrase given for this is inappropriate. While Limbaugh may have meant to use his question to create the rhetorical effect given in that paraphrase, there are many other potential interpretations. For instance, a teacher might ask the same question in order to get students to think about details how of evolution works. A person who knows little about evolution might ask the question because they sincerely want to know the answer.

Limbaugh's statement is rated as "False" because the "fact check" assumes one possible interpretation of the purpose of Limbaugh's question is the only possible one, to the extent it never even attempts to justify that assumption. That is inappropriate.

In an article on global warming:

http://www.politifact.com/truth-o-meter/statements/2015/sep/02/rick-santorum/santorum-un-climate-head-debunked-widely-cited-97-/

There are multiple inaccurate remarks. A notable example is a strange transition the article goes through, demonstrated by these two sentence:

"Santorum is likely referring to Richard Tol, an economist at the University of Sussex who’s been a vocal critic of the 97 percent figure."
"Santorum’s claim confuses several points. First, the critic of the 97 percent he’s referring to isn’t the "head" of the UN’s climate panel, but an economist who has collaborated with but has since left the IPCC."

It is difficult to see how one could argue "Santorum is likely referring" to a person then conclude as fact Santorum is referring to "an economist" of any sort. If one says they are not certain who Santorum is referring to, it is misleading to state as fact he was referring to a specific individual. It is the probable conclusion, but "fact checkers" should not state as fact things which are merely probable.

There are specific errors in the article as well. I won't try to document them all, but I'll list a few. The article says:

Santorum is likely referring to Richard Tol, an economist at the University of Sussex who’s been a vocal critic of the 97 percent figure. He is not, as Santorum claims, the "head" of the IPCC, though he was the convening lead author of a chapter of the IPCC’s fifth report. He has since parted ways with the IPCC.

Richard Tol did not part ways with the IPCC. The article linked to to support that claim does not say he had. What it says is Richard Tol "was withdrawing from the team writing the summary" of the IPCC Working Group 2 Fifth Assessment Report. That Tol withdrew from a team working on one aspect and one Working Group's report does not mean he parted ways with the IPCC as a whole. Indeed, Tol remained a lead author of a chapter of that very report.

The article also says

The study Santorum is attempting to describe is a 2009 survey by Peter Doran, a professor of earth science at Louisiana State University. About 90 percent of around 3,000 surveyed earth scientists said they think climate change is happening, and about 82 percent said human activity is contributing to it.
The 97 percent figure comes from a subsample of climate scientists in Doran’s study, and Santorum correctly describes its small size: 74 out of 77 respondents said they agreed that climate change is man-made.

This contains multiple inaccuracies: 1) The second question did not ask if "human activity is contributing to" global warming, but rather, if "human activity is a significant contributing factor to changing mean global temperatures." It is not appropriate to conflate "contributing to" and "a significant contributing factor. A person could reasonably believe there are contributing factors which are not significant.

2) It is misleading to say "74 out of 77 respondents" said anything. This is true on two counts. First, the reported results of the Doran & Zimmerman study say 75 out of 77, not 74 out of 77.

3) 79 people met the criteria to be included in this subset. All 79 were asked if they believed global mean temperatures had generally risen, generally fallen or remained relatively constant compared to levels prior to 1800. Two of the 79 answered "remained relatively constant." They were then filtered out and not asked the follow-up question. Excluding them in this way without warning people is inappropriate. The actual numbers would change from 96.1% to 93.7%. (This same pattern of filtering out respondents was used for the entire survey, meaning all of its reported results are similarly distorted if one provides no caveat.)

Cook’s study found that among over 4,000 studies that took a position on man-made climate change, 97.1 percent "endorsed the consensus position that humans are causing global phase" and 97.2 of 1,300 authors who responded agreed with the position.

There is no such thing as "global phase." The quotation should read "global warming." It appears several words from the original quotation have been inadvertently removed during the editing process.

Additionally, Cook and his friends did not state anything about "over 4,000 studies." They specifically refrained from examining studies as a whole. Instead, they looked at the abstracts (basically, summaries) for scientific papers. While summaries may in many cases express the same view on the "consensus" as the body of the paper might, the body of papers will say many things not included in their summary. In many cases, that might include a position on the "consensus." It is inappropriate to conflate results for the summaries of papers with results for the papers as a whole.

Finally, it is worth noting the "consensus" position expressed by Cook et al does not match that expressed by Doran & Zimmerman. Cook et al included in its consensus any statement which at least acknowledged the greenhouse effect is real. Many papers were included in its "consensus" simply because they said things akin to, "Methane is a greenhouse gas." Statements like those do nothing to establish human impact is a signficant contribution to global warming. Readers are unlikely to notice this distinction unless it is highlighted.

If you have any questions or wish for further information regarding anything I have said, feel free to ask. I'd be happy to help. I think fact checking is a very important thing. I just can't view what you do as actually being fact checking given this e-mail highlights only a small sample of the many issues I have found with your articles.

I strongly suspect fact checking your "fact checks" COuld be a full-time job. Not that anyone would pay for it.

Best wishes,
Brandon Shollenberger

16 comments

  1. Who fact checks the fact checkers? Facebook is about to use Politifact and, Factcheck.org, snopes, and some others to filter out news stories on their site. I think it will end up like Wikipedia's articles.

  2. " I think fact checking is a very important thing. I just can't view what you do as actually being fact checking..."
    I bet they loved that.

    Brandon, you are an excellent fact checker. Have you ever worked on Wikipedia? I think Mann's and McIntyer's pages could likely use some work last time I looked.

  3. Ron Graf, thanks. I think part of what makes me good at fact checking is I don't shy away from being "pedantic." A lot of people just don't care about details and nuances that are necessary getting things factually correct. I find it enjoyable like I would a puzzle.

    As for Wikipedia, I don't edit it because the social dynamics of Wikipedia are terrible and a number of their rules are nonsensical. Editing Wikipedia requires playing all sorts of "games" because of how disputes work. That said, I have suggested the value a climate wiki could have in the past a number of times, and I have drafted plans on how it could be managed. I've done some testing of it, and I've confirmed the technical aspects of it are manageable. I even have a wiki online right now because I didn't see any reason to delete it after testing the setup process that last time. It only has one article, but it's basically "good to go" except for the lack of content.

    I've hinted at this project a couple times, and the more dissatisfied I've gotten with climate discourse (primarily because all "skeptics" I encounter are acting just the same as "warmists"), the more I've been looking into it. I currently am in the process of outlining the articles I'd want for paleoclimatology. It's looking like a lot of work. Just making the outline is a sizable task given the number of papers, subjects and proxies that need to be considered.

    Given the complexity of that field, I'll probably start producing content by writing pages regarding the "consensus." Some of the articles for that are easy to write (the wiki's one article right now is about the Doran & Zimmerman paper), but trying to figure out what to write for Cook et al 2013 and Cook et al 2016 gives me a headache. There is so much to say about them it's difficult to figure out how to make an encyclopedic entry.

  4. Ron, there are details about it, but editing Wikipedia is impossible for skeptics. William Connolley runs roughshod and blocks people and puts his own spin on things. There was a concerted effort at an edit war, with people using all of Wikipedia's processes for adjudication and going after Connolley's violations of various rules. The result was that Connolley was suspended for about six months, and so were the accusers. The rest of Connolley's team then proceeded to continue as usual. Hard to believe, but some of the pages were improved in tone as a result. For example, I remember having my edit of Richard Lindzen's page removed very quickly. It had something to do with
    "He also emphasized the fact that the summary had not been written by scientists alone. The NAS panel on which Lindzen served says that the summary was the result of dialogue between scientists and policymakers.[c]"
    The original version was an attack on Lindzen, with something like the second sentence serving as a refutation of the first.

  5. Brandon, I think you could provide a valuable service by providing a climate wiki and MikeN's comment gave me an idea. In politically controversial topics wikis (organic encyclopedias) should have one box reserved for the pro-side and another for the counter argument.

    WRT Connolley, it seems as though liberals are only a step below having the zeal of 9/11 highjackers. With Trump and the conservative congress I can't even begin to imagine their pain.

  6. The problem with mmaking a climate wiki would be the amount of work involved. From a policy perspective, a useful and effective climate wiki could be made for ~$500,000. A full and complete one could be made for ~$2 million.

    Obviously, I'm not going to be capable of doing anything on that scale. Just getting a good wiki for paleoclimatology would be a major project for any one individual. It'd be great to have, but it'd mean passing up on doing a number of other things. For instance, I really ought to be writing another eBook.

    As for having pro/con sections, I can't agree with that. In any complex subject, there will be more than two "sides." Forcing things into a pro/con structure creates a polarized system in which "winning" is the focus. I'm all for enabling dissent (and have a standing offer to run a guest post by anyone who disagrees with me), butt that's not how you do it.

    In any event, wikis are supposed to be about sharing facts. There shouldn't be room for differing views in the articles. The reason there are so many problems with Wikipedia is people often don't edit it for the purpose of sharing facts.

    Anyway, the wiki is probably going to be on hold until after Christmas as I have a technical subject I want to start writing up. I'm hoping to have my write-up finished in time for the new year. I'll probably do some preliminary posts about it in the next couple weeks. I might even see if I can use it as part of a new eBook as I feel it is an important enough topic.

    In the meantime, you can see the Wiki as it is now here. That article is pretty rough, but it should kind of show what the idea is. I'm happy to hear any suggestions/submissions for the wiki, but it's not going to be publicly editable so people can't just start directly adding content. It's a bit of a shame as "many hands make light work," but there's no way I'm going to have edit wars become a thing.

  7. I don't see anything wrong with Politifact's critique of Rush Limbaugh. I could argue that 'calls into question' makes it valid where Politifact was arguing as if he used 'refutes'.

  8. MikeN, fact checks are supposed to involve checking facts. While people can ask questions for rhetorical effect and thus effectively make an assertion, there's no indication that's what Rush Limbaugh did. The context of his question might show he was trying to dispute evolution, but it might show he was simply uncertain as to what the mainstream understanding of evolution was. Or it might show he knew the answer to his question and was only asking that to set up the opportunity to answer it in order to inform his readers. Or it might show any number of other things.

    I have no problem with fact checking something a person implies but doesn't outright state. That's fine. You just have to establish what the person's implication was. PolitiFact didn't even try. It didn't attempt to give any context or reasoning which showed that was Limbaugh's intended meaning. It just assumed that was his intended meaning and ran with it.

    If PolitiFact wanted to "fact check" this idea, it should have either taken steps to establish this was Limbaugh's intended meaning or it should have just said something like, "It is uncertain exactly what Limbaugh meant by this question, but it is worth examining what the answer to his question ought to be." That would be fine. That would be responsible.

    What they did was just hack writing.

  9. MikeN, anecdotes are notoriously difficult to fact check. A second-hand story about something somebody says happened in 1973 is not the sort of thing you would normally try to verify. It would be much easier to examine what the article says about hacking. Leaving aside it's silliness about the "UAE electronic army," this paragraph is just dumb:

    We have a great example of such news this week in the Yahoo one billion account hack. Sure, it’s all over the web but it happened in 2013. Are we really supposed to believe that one billion user records were stolen from Yahoo and it took three years for somebody to notice??? The story is that law enforcement officials came across the stolen data, or some of it, and took it to Yahoo for verification. Maybe, but having written these stories for 30 years I think it is much more likely that somebody already knew about the breach and simply chose not to say when it happened. This is not to say that Yahoo CEO Marissa Mayer knew or didn’t know about the breach, just that shit happens and often isn’t reported if jobs are perceived as being on the line.

    Of course nobody is supposed to believe that. Yahoo has known about that hack for years. It just did what lots of companies do when their data is stolen - they tried to hide it. The fact people try to prevent the world from finding out embarassing things about themselves doesn't mean there is "real news you never hear about" like this article claims. People get away with crimes all the time. That we don't hear about them is often because of nothing more than things not becoming news if nobody discovers them.

    For the record, I've known of several breaches at Yahoo that were never officially reported or covered by the mainstream news. The same is true for other companies. Quite often, things like this become "open secrets." It's sort of like how there are hundreds of people who know about a military base's breach only being discovered because the man who broke onto base fell asleep in the base domn (he was homeless and just wandered onto base and into some enlisted man's bed). You're not going to see any news stories about that. Does that mean it is "real news you never hear about"?

    That all said, I can do a bit of checking on the barge anecdote. My understanding is the conversion rate from oil barrel to gallons is 1 to 42. THe United States was producing over nine million barrels of oil a day in 1973. The amount of imported oil fluctuated greatly that decade, so let's just go with 9 million barrels a day as an estimate for daily consumption within the United States (this should account for exported oil). Given that, let's approximate the United States used ~375 million gallons of oil a day. That'd be over 11 billion gallons for 30 days. The anecdote says:

    “During the summer of 1973 I worked on a tow boat on the Mississippi River. Every 10 to 14 days, we’d load our barges on the Gulf Coast and deliver petroleum products to some place in the Midwest. That was the summer of the big gasoline shortages. As we would travel up and down the Mississippi, we’d pass an Exxon tow. It would have eight barges (a double unit) fully loaded, or about 10 million gallons of gasoline. The tow wouldn’t be moving, it would be tied up in a quiet spot on the river. Each trip we find more tows tied up. Shell, Texaco, Exxon, Amoco were all doing it. One day they announced in the news how much gasoline would be used in the USA in a single day. I made some quick calculations and realized we had passed a month’s supply on our last trip.”

    Given the crude estimate above, this person would have had to pass 11 billion gallons of oil in a single trip on the Mississippi River. According to the anecdote, each tow this person passed would hold "about 10 million gallons of gasoline." That means he would have had to pass a thousand tows that had been tied up. That is, he'd have had to pass a thousand tug boats pulling eight thousand barges, all tied up and not moving on the Mississippi River in a single trip. Being generous, that'd be like one barge every two miles or two tows every 15 miles.

    Do you believe that's plausible? I don't. In fact, I'd wager there weren't enough tug boats or barges in operation on the Mississippi River in 1973 for that to have even been possible. I have no doubt there were tows tied up at various points. That's normal. It's particularly common thanks to the lock systems on the Mississippi River where tows can get held up for several hours at what is basically a checkpoint if things are busy. But the idea multiple oil companies had thousands of tug boats just chilling out on the Mississippi River in order to create a fake oil shortage is... not plausible. It certainly isn't something people should believe just because some guy told some other guy that he saw it 45 years ago.

    By the way, if I were being paid for this, I'd dig up information on how many tug boats and barges those companies had in operation in the 1970s. That's not the sort of information I can find with simple Google searches though. You get what you pay for.

  10. Oh, I should mention two caveats about that last comment: 1) Towboats push barges while tugboats pull them. I'm treating them the same for this comment for simplicity. 2) The 10 million gallon per tow figure provided in this anecdote is not something I can confirm. That would require each barge hold ~30,000 barrels. There are certainly modern barges which do so, but I doubt each barge this person might have passed would have been that size. If the figure given for how much oil each tow carried were an over-estimate, the anecdote would become even less believable.

    A couple other notes as well. First, I should mention I sent the e-mail shown in this post to PolitiFact four days ago. I got an automated message confirming they received it. It may mean nothing. I don't know how long it might take to get a response, if I will get one at all. I just wanted to point it out.

    Second, I have a new post ready to publish but am waiting as I've contacted a person to give them a chance to respond to what I said before posting it. There may not be any new posts for a few days as I'm probably going to focus on preparing a couple follow-up posts. I wanted to mention this so people don't misunderstand any silence on my end. It's not that I'm not being active. I just want to be delicate with this next subject because comes close to saying a person's PhD should be revoked because their work was fundamentally invalid.

  11. OK, so I wasn't too far off. I got 4000 tows, using the numbers someone else put up in the comments, which is even more ridiculous.
    I suspect this guy does stuff like this a lot.

  12. Pingback: Go Figure? | Izuru

Leave a Reply

Your email address will not be published. Required fields are marked *