This post isn't going to be exhaustive, and I will likely have much more to say within the next few days, but I wanted to get this out right away. As you probably know, last month I found a CONFIDENTIAL draft version of a new paper by:
John Cook1,2,3, Naomi Oreskes4, Peter T. Doran5, William R. L. Anderegg6,7, Bart Verheggen8, Ed W. Maibach9, J. Stuart Carlton10, Stephan Lewandowsky11,2, Andrew G. Skuce13, Sarah A. Green12, Dana Nuccitelli3, Peter Jacobs9, Mark Richardson14, Bärbel Winkler3, Rob Painting3, Ken Rice15
This is just about everybody publishing on the consensus messaging approach to the global warming debate, where they repeatedly say there is a consensus and expect that to convince people we need to take action on global warming. There is one exception, an exception which is quite notable now that the final version of the paper has been published. That exception is a man named James Powell. Powell claims to have shown there is a 99.9% consensus on global warming, yet he is not a co-author of this paper, and his result doesn't show up anywhere in the study. The complete lack of mention of his results is rather notable since the draft version talked about them quite a bit. In fact, Powell's work is featured heavily in the abstract of the draft version:
This 97% result has been criticised for being both too high (Tol 2015) and too low (Powell 2015). In some cases, Tol assumes that when the cause of global warming is not explicitly stated ("no position"), this represents non-endorsement, while Powell assumes the opposite. Neither assumption is robust: as argued by Powell, Tol’s approach would reject the consensus on well-established theories such as plate tectonics. On the other hand, Cook et al surveyed authors of the studies considered and some full papers rejected the consensus even when their abstracts were classified as "no position", contradicting Powell's assumption.
In fact, Powell's name comes up 43 times in the draft version, yet it only shows up twice in the final version. One might think the authors simply changed the focus of their paper, yet they explicitly state this in their new abstract:
We examine the available studies and conclude that the finding of 97% consensus in published climate research is robust and consistent with other surveys of climate scientists and peer-reviewed studies.
The authors must know this statement is incredibly deceptive. The devoted a great deal of time and effort to discussing the study by Powell, yet after completely excising such discussions from the published version of this paper, they still claim to "examine the available studies." There is no doubt the authors are aware of the Powell study. There is no doubt the Powell study is available. As such, there is no excuse for the authors to ignore it and simultaneously claim to "examine the available studies." They must know that statement is a lie.
Don't believe it is a lie? Rather than quote all the text showing the authors were aware of Powell's study when they wrote the draft version, we can see more direct proof. This is from the draft version's table listing the studies they examined and the results those studies had:
Powell's study with its 99.9% result is right there, plain as day. This is from the same table in the final version:
The draft version's table gives results for studies by Stenhouse et al, Verheggen et al, Pew Research Center, Powell, Carlton et al. The final version gives results for studies by Stenhouse et al, Verheggen et al, Pew Research Center, Powell, Carlton et al. Powell (2015) has simply been disappeared.
But it gets worse. The authors of this paper haven't just excluded Powell 2015's results from all their analysis and excised the significant amount of discussion of those results that was present in their draft version. They've done that while still citing the paper to support their criticisms of a paper by one Richard Tol (which to be fair was terrible):
Powell (2015) shows that applying Tol’s method to the established paradigm of
plate tectonics would lead Tol to reject the scientific consensus in that field because nearly all current papers would be classified as taking ‘no position’.
So while claiming to examine the available studies, the authors intentionally exclude results from one study they knew is available then turn around and cite that study to support their views on a different matter. It's obscene.
I can't think of any word to describe this other than "lying." The authors apparently feel free to claim to "examine the available studies" while simply ignoring some studies. Not only is that wrong and dishonest, it raises the important question: What other results have they ignored? If the authors are willing to lie about the existence of one study, who knows what other studies they might have pretended don't exist?
Or for that matter, who knows what else they might have lied about? How can anyone know they accurately described the studies they did include? We can't. In fact, we have strong reason to believe we shouldn't trust their descriptions of studies. The central study for this paper, created by John Cook and many other co-authors, described its methodology saying:
Each abstract was categorized by two independent, anonymized raters.
One of the authors of that study, Sarah Green, is also a co-author of this paper. She said this when carrying out the study:
But, this is clearly not an independent poll, nor really a statistical exercise. We are just assisting
in the effort to apply defined criteria to the abstracts with the goal of classifying them as objectively as possible. Disagreements arise because neither the criteria nor the abstracts can be 100% precise. We have already gone down the path of trying to reach a consensus through the discussions of particular cases. From the start we would never be able to claim that ratings were done by independent, unbiased, or random people anyhow.
Despite saying she and her colleagues could never describe the ratings as having been "done by independent" raters, she co-authored a paper which explicitly stated the ratings were done by "two indepenent, anonymized raters." No explanation was ever given for that contradiction, but in the Supplementary Material the new paper, the authors say:
Tol (2015) questions what procedures were adopted to prevent communication between raters. Although collusion was technically possible, it was - in practice - virtually impossible. The rating procedure was designed so that each rater was assigned 5 abstracts selected at random from a set of more than 12,000. Consequently, the probability two raters being assigned the same abstract at the same time was infinitesimal making collusion practically impossible.
When I discovered the draft version of this paper, I summarized why this is not a fair or reasonable description. We can see the number of raters responsible for most the work in the study was small with this image John Cook created and published for everyone in his forum to see:
So I said:
There were a total of 24 raters, but given how few people were responsible for the majority of the ratings, collusion would have been perfectly possible. It would have been easy for people to talk to one another about how to game the system. This new paper tries to play that off by saying people wouldn't have been able to collude on individual papers, but nobody said that's what happened.
Whether or not people collude on how to rate specific papers doesn't determine whether or not they colluded in general. It would have been easy for raters to talk to one another and say things like, "Hey, whenever you and I do a rating, let's always add +1 to our score." If they did, their ratings wouldn't be independent of one another even though they never discussed a specific paper.
The authors made no effort to prevent such a thing from being possible for their project. They made no effort to monitor for such happening. The raters all had each others' e-mail addresses meaning they could easily contact one another directly and privately, but the authors want everyone to just assume that wasn't an issue. All while offering false excuses of how any communication between raters was merely to seek clarifications/amendments to the rating process.
The reality is the raters themselves recognized the problems created by talking to one another about how to rate specific items, with one rater saying:
Giving the objective to being as close to double blind in methodology as possible, isn’t in inappropriate to discuss papers on the forum until all the ratings are complete?
John Cook, the head of this project, was well aware of this issue. He ran the forum where people talked to one another about how to rate individual papers. He ran the forum where people acknowledged they had cheated the study's methodology by looking up extra information about the papers they were rating. Rather than speak out against any of it, he actively participated in those forums and consequently encouraged this sort of behavior.
That last part was a reference to how the authors of the Cook et al (2013) study claimed their ratings were based solely on the title and abstracts of papers even though they had discussed in their forums how they looked up the author names and full text of papers they were rating. This new paper now admits they did this, meaning they lied when they claimed to have only used titles and abstracts for their ratings:
During the rating process of C13, raters were presented only with the paper title and abstract to base their rating on. Tol (2015) queries what steps were taken to prevent raters from gathering additional information. While there was no practical way of preventing such an outcome, raters conducted further investigation by perusing the full paper on only a few occasions, usually to clarify ambiguous abstract language
So first the authors lied and claimed they had only used the abstracts and titles of papers for their ratings. Then, after enough people have pointed out the authors had discussed how they looked up author names and full texts of papers in the forum John Cook (the head of the project) ran, they admit they cheated and used titles and abstracts but claim they only did so a few tiems.
There's bound to be a ton more to say about this paper, but for now, I'm signing off to await the media flurry I expect will come and manage to completely ignore how the authors of the paper blatantly lied.