We've been discussing a strange chart from a recent paper published by John Cook of Skeptical Science and many others. The chart ostensibly shows the consensus on global warming increases with a person's expertise. To "prove" this claim, Cook et al assigned "expertise" levels to a variety of "consensus estimates" they took from various papers. You can see the results in the chart below, which I've added lines to to show each category:
As you can see, the "consensus estimates" are all plotted one next to the other, without concern for how the categories are spaced. The result is Category 4 doesn't exist in the chart, and Category 5 covers more than half of the chart. This creates a vastly distorted impression of the results.
But while that is a damning problem in and of itself, there is much more wrong with the chart and corresponding paper. One of the key issues we've been looking at in this series is how Cook et al arbitrarily chose which results from the studies it examined to report and which ones not to. Today I'd like to discuss one of the most severe cases of this. It deals with the paper Verheggent et al (2014).
This is how the new Cook et al (2016) paper reported the results for Verheggen et al (2016):
That is one line. Their chart which has 16 data points on it has only one point devoted to this paper's results even though other papers have as many as three data points. This is peculiar as Verheggen et al report their results in great detail. For instance, here is a chart showing responses as a whole:
And here are the responses broken down by type of person who gave the answer:
And here is some of the same information again, in tabular form:
That is a wealth of information. According to the authors of Verheggen et al, it shows:
Consistent with other research, we found that, as the level of expertise in climate science grew, so too did the level of agreement on anthropogenic causation.
This is exactly what Cook et al (2016) seeks to prove. Why then does Cook et al (2016) not include this data? The only results it includes are the answers given by people who "Published more than 10 climate-related papers (self-reported)." Why is that the one group who has its results reported? Why would a person having published 10 papers mean they get excluded while a person having published 11 papers get included?
It gets even worse when we look at the Expertise category assignment. These values were arbitrarily assigned by Cook et al, and there is no guidelines given for how they were chosen. As our last post showed, climate scientists who've published on climate science recently but published more on other topics were put in Expertise Category 3 for the Stenhouse et al (2014) survey.
Some of those people likely published 10 or more papers on climate science. After all, in five years (the period of time covered), a scientist could easily have written over 20 papers. If 10 were on climate science and 15 on something else, they'd be rated as Expertise Category 3 for Stenhouse et al (2014) yet fall in the Expertise Category 5 for Verheggen et al (2014). It gets even worse when you realize Stenhouse et al (2014) judged people's expertise on papers published in the last five years but Verheggen et al judged their expertise on how many papers they had published in their entire lifetimes.
Cook et al (2016) don't offer any explanation for why they only published results for one subgroup of the Verheggen et al (2014) data set. They offer no explanation for why some data was excluded and other data was not. They claim:
We examine the available studies and conclude that the finding of 97% consensus in published climate research is robust and consistent with other surveys of climate scientists and peer-reviewed studies.
Yet they intentionally exclude a great deal of data from many of "the available studies" they claim to examine. And when they display whatever results they did report, they make no effort to ensure the "expertise" categories they use are consistent or coherent. It appears all they're doing is picking up results that are convenient for them, assigning arbitrary "expertise" values to what they picked out and then displaying the results in a heavily skewed manner which gives far more visual weight to the most favorable results when a fair depiction:
Would look very different.