2012-03-02 10:56:44Question: will TCP be the biggest ever survey of peer-reviewed climate papers?
John Cook

john@skepticalscience...
121.222.175.176

Would be a nice label to pin on the work if it was true.

2012-03-02 11:46:16
dana1981
Dana Nuccitelli
dana1981@yahoo...
71.137.110.129

Hmm good question.  I can't think of a larger one, but I'm not sure how you'd check that.

2012-03-02 17:00:16
Brian Purdue

bnpurdue@bigpond.net...
138.130.140.206

There was the Oreskes study and this one in 2010 but I think they both failed in their marketing strategy.

Both studies had a flaw that deniers were able to attack by concentrating their fire on a small target. Monckton did it with Oreskes. That is, they both had a small number of study authors behind the findings.

What would give TCP findings far more credibility and resistance to attack is if it was released under the banner of more than just SkS but that will require a couple of “impartial” organisations /institutions being give access to the findings for their scrutiny and co-authorship.

Remember this is the closing comment in the article about the 2010 study.

In this long-running battle over scientific credibility and how to measure it, the Anderegg paper analyzes a particularly large database of climate researchers, and therefore goes farther than any previous effort in attaching hard numbers to the discussion.

But it didn’t stop the deniers - did it. TCP must muffle them so the message gets out and is absorbed by the target audience.

2012-03-02 17:43:05
John Cook

john@skepticalscience...
121.222.175.176

What was the main criticism of Anderegg et al? I seem to recall the main criticism was it was a "black list". Eg - Anderegg was about the scientists and questioning the denier's level of expertise (fairly so). The difference between Anderegg and TCP is we are about what the research says, not what the scientists are saying. That is an important distinction - we are not about opinion but about peer-reviewed science.

So TCP is much more along the lines of Oreskes' study rather than Anderegg's. So it's a good idea to look at what criticisms have been made towards Oreskes 2004. For example, the Monckton critique you link to is a good example of what to expect lobbed at TCP. His critiques (in red):

  • Some nitpicky technical details. Oreskes originally wrote 'climate change' when in fact she searched for 'global climate change'. Peiser also nitpicks about papers that didn't have abstracts and discrepancies between the quantities cited. These are minor technical issues but we need to double check and triple check all this to ensure no distracting side-issues.
  • Significantly, Oreskes’ essay does not state how many of the 928 papers explicitly endorsed her very limited definition of “consensus”. This is true although she does flesh it out in a later, longer article. We will be much more transparent, publishing all our results online.
  • Dr. Peiser’s research demonstrated that several of the abstracts confounded Oreskes’ assertion of unanimity by explicitly rejecting or casting doubt upon the notion that human activities are the main drivers of the observed warming over the last 50 years. We have identified some rejection papers and will be transparent in listing them all - but also stress that our survey doesn't mean we've identified ALL rejection papers but just that the number is vanishingly small compared to the total # or # of endorsements.
  • According to Dr. Peiser, fewer than one-third of the papers analyzed by Oreskes either explicitly or implicitly endorsed the “consensus”, contrary to Oreskes’ assertion that the figure was 75%. Actually, I think Oreskes' assumption of 75% was erroneous - she arrived at that by assuming all mitigation and impact papers implicitly endorsed the consensus. We've seen that is not the case. She also assumed all Methods papers were neutral, we've also seen some of them do endorse the consensus. As for what the final quantity of endorsement will be in Oreskes' sample, we'll know that soon enough as we will have rated all the papers that Oreskes rated.
  • 44 abstracts focused on the natural as opposed to anthropogenic causes of climate change, and did not include any direct or indirect link or reference to human actitivies, carbon dioxide or other greenhouse gas emissions, let alone anthropogenic forcing of recent climate change.  We've been pretty careful in our definition of endorsement.

A key element to our paper will be transparency - we will challenge deniers to do the analysis themselves and create an interface that makes it dead easy for them to do it. This will be a big contrast with Oreskes 2004 which was largely opaque.

But your suggestion about involving other institutions is interesting. Do you think if there were a few more institutions attached to the Anderegg or Oreskes study, that it would have made any difference to the denier attacks? I don't see the deniers attacking the small number of authors. I see them attacking the content of the study and don't see them doing it any differently if there was more names or institutions on the papers. Even if we attached the Academy of Science from a dozen countries to our paper, they would still come after it just as hard.

Anticipating the possible attacks is a very useful exercise and we should continue to gnaw on this bone for the next two months while we work on the rating. Also think about our target audience, the undecided majority - we don't do this to convert deniers and we know they're going to attack TCP no matter what. The important thing is to produce a result that has a compelling narrative that captures the public's interest and is robust enough that there are no flaws that might compromise the narrative.

2012-03-02 17:46:08More on Anderegg
John Cook

john@skepticalscience...
121.222.175.176

Note also that Anderegg didn't survey the papers but the scientists who wrote the papers, splitting them into "convinced by the evidence" and "unconvinced". So they weren't a "survey of peer-reviewed climate papers" and hence not comparable to us. So as far as I know, Oreskes 2004 with 928 papers is the largest review I know of.

The only other survey of peer-reviewed climate papers I know of is Ari's ongoing survey. Just curious, Ari, how many have you guys done so far?

2012-03-02 18:58:29
Brian Purdue

bnpurdue@bigpond.net...
138.130.140.206

John - I don’t think the deniers will attack TCP first – they will attack SkS first.

Because SkS is very active in the “debate” their past tactics show they will drag out every mistake SkS has ever made and make up a lot as well (like SkS is funded by Nazis) and say “would you believe what these lying alarmists say about the consensus”.

If SkS does go it alone we will have to counter this drumbeat while convincing the target audience about the consensus. This would be very hard because they are exceptionally good at destroying reputations. Diversifying the authors would make this very difficult for them, and particularly if the other authors were genuinely “impartial”.

I’m sure there are other ideas though.    

2012-03-02 19:29:19
Ari Jokimäki

arijmaki@yahoo...
194.251.119.197

John, my project is on hold for the moment as I'm concentrating to this one, but there were only few hundred rated before I put it on hold. Also, I'm the only one in that project so there's no "guys". :) But I'm hoping that I can interest all of you after this one to concentrate on my project.

There's one thing about Oreskes that I thing went a bit wrong. I think the search phrase she chose ruled out most of the papers rejecting consensus. Those that reject consensus use the term global warming more, I think. Also, in some other thread here I mentioned that the lack of uncertainty analysis in Oreskes' paper was bad.

2012-03-02 19:39:22Second thought
Brian Purdue

bnpurdue@bigpond.net...
138.130.140.206

Maybe co-authors are not required if instead there was an independent audit done by a professional survey company to certify the methodology used. If we have that to show it will hopefully shut the deniers up while the study was gaining public acceptance.

2012-03-02 21:22:11
Riccardo

riccardoreitano@tiscali...
192.84.150.209

John
a second thought on challenging the deniers to rate the papers themselves on your database. One of the criticism we'll surely face is the bias of the people selected to rate the papers. Giving access to the database to have deniers do their ratings will let them dismiss your analisys right away. Think of the "best science blog" or the well known astroturfing organizations or the flood of deniers comments in MSM blogs. Deniers are strongly motivated and in some cases well organized. In a week or so they will "demonstrate" the bias of your analisys. It would be disastrous.

You have to choose the playing ground, which is the scientific litterature. There are rules, competence, quality standars, the filter of a more or less working peer review. The internet is a jungle and fair play is non-existent.

2012-03-02 21:31:30
Ari Jokimäki

arijmaki@yahoo...
194.251.119.197

I agree with Riccardo. Letting deniers rate the papers results in most papers being explicit rejections and they will then parrot that result regardless of how well you debunk their rating effort afterwards.

2012-03-03 03:17:10
dana1981
Dana Nuccitelli
dana1981@yahoo...
64.129.227.4

Making the system transparent and available for anyone to rate the papers themselves will be critical.

Deniers are going to find a way to dismiss our results.  That's fine, it doesn't matter because they're not our target audience.  Our methodology will be consistent, replicated by multiple contributors, and our paper will be peer-reviewed.  Thus we will trump any ridiculous biased denier ratings anyway.

But making the system public will allow the unconvinced and open-minded to see the papers for themselves and confirm that our results are correct.  That will be a very valuable tool.

Remember, the deniers are largely just talking to themselves.  WUWT is the only denier blog with significant traffic, and that's just because it's where all the deniers conglomerate.  Thus what they say is of no significance.

2012-03-03 04:50:00
Riccardo

riccardoreitano@tiscali...
2.33.129.129

If the deniers declared war to science and we think the war can be won scientifically, we're going to loose. Much better to keep the two battlefields separated.
Now, we have to make the analisys scientifically sound in first place, no doubt; this is the easy one. Switching to the PR field we need to avoid the counter attack. Letting the deniers rate the papers en mass means looking for troubles. But I see the point of letting people look at the abstracts and rate the papers themselves.

We may limit the access on a per session basis (using cookies and user IP maybe) to be sure that each one rates as many papers as he likes but avoiding large scale collaborations. People in good faith will be satisfied by one or two hundred papers, the others may well go to hell!  :)

2012-03-03 05:22:14Public ratings .....
BaerbelW

baerbel-for-350@email...
2.200.172.29
....should be somehow kept separate from the results used to create the TCP-paper. This could be achieved with different means: store them only temporarily for whoever is doing this sample rating, mark them with a special flag, set a cut-off date, require a user-ID and login before rating, store the IP-address for identification purpose to name just a few.
2012-03-03 06:07:04
Andy S

skucea@telus...
209.121.15.232

I think that there will be a very large number of papers that we will be conservatively rating as "neutral"that their authors will be rating as supportive of AGW in some degree or other. As long as we are fair to the truly skeptical papers and grade them as such, the deniers won't have a leg to stand on. Even if they find the odd one or two that we missed in the database or misgraded, the statistics are still very likely going to be overwhelmingly in our favour.

If we restrict access to our data and ratings in any way, that will provide an opportunity for the deniers to start the meme that we are hiding stuff from scrutiny. That would include the real identities and backgrounds of all the participants in this exercise.

My guess is that most skeptics will dismiss this study as a waste of time, proving something they have never doubted, the literature is dominantly pro-AGW (like the reliability of the temperature records after the BEST analysis) and that it merely proves the fact that the scientific establishment indulges in group-think on a massive scale. The lunatic fringe will strive to fight on the details but we can always challenge them to do their own study and get it reviewed and published.

There may be some copyright problems in making the abstracts freely available.

2012-03-03 07:44:25
John Cook

john@skepticalscience...
121.222.175.176

One problem with showing our rating of every individual paper is deniers will pick through the whole database and find isolated examples that they can dispute, leading to an endless series of blog posts on the denialosphere saying "TCP said this is an endorsement but we think it isn't... RESULT INVALID". Imagine that meme posted over and over again.

Here's an alternative approach to think about. Rather than post our specific individual ratings, we instead offer the following interactive feature:

  1. Readers can rate a random selection of papers - as many as they like
  2. Then the average rating results are compared to our rating results and the scientist self-ratings. Eg:
    Your average rating 2.5 (neutral)
    TCP average rating 2.7 (neutral)
    Scientists rating their own papers 3.1 (implicit endorsement) 

The way we counter the "SkS and results are biased" meme is by using the scientists' own ratings of their papers. So if a denier does a rating and rates intentionally lower, then says "see, my 'unbiased' rating is lower than SkS proving SkS is biased", the answer to this is their rating is lower than the actual scientists rating their own papers. It's not deniers vs SkS - it's deniers vs the scientists who wrote the actual papers - who knows better what level their paper was written at?

In fact, the scientist rating - if it has a higher endorsement level than ours - is the main counter to the "SkS bias" accusation. It will prove that our result is biased towards underestimating the level of endorsement. That's a good counter narrative and if that is the result we get (which I expect to see but we'll just have to wait for the data), we should get on the front foot with that counter narrative before the deniers have a chance to post their critiques, cut them off at the pass.

Rob H, I was initially skeptical and resistant to your idea of polling scientists for TCP but now I'm fully on board with this idea, all the way!

2012-03-03 08:03:54
dana1981
Dana Nuccitelli
dana1981@yahoo...
64.129.227.4

I like that idea, John.  Nip the potential cherry pick in the bud - the net rating (i.e. percentage of endorsements vs. rejections) is what's ultimately important anyway.

2012-03-03 08:05:33More on this idea
John Cook

john@skepticalscience...
121.222.175.176

Clarifying that the public ratings that readers give to papers is completely separate to our initial ratings that we're doing now. Consider the initial ratings we're doing now is the 'official' published analysis - the public ratings is for communication purposes, as a catchy interactive feature to stimulate interest and a way of nullifying the naysayers.

A key element to the interactive feature will be to say "Your ratings will be compared to the scientists' rating of their own papers". It's actually been shown in the psychological literature that that kind of accountability (comparing participant's results to results from an objective expert panel) decreases participant's bias, motivates them to be more impartial. So will be important to emphasise that.

Also thinking about making it so you need to register an SkS user account in order to rate the papers. I probably won't do this because you need to make things as free and easy as possible as well as anonymous if desired. Instead, perhaps I'll have a "if you want to save your ratings, register a user account" so their ratings are saved and they can even come back and add to them later. Something to think about.

There are so many ways we can approach the interactive feature, this will require a lot of thought as if it's not done right, it can create lots of opportunities for deniers to attack TCP. But if done right, anticipating all avenues of attack, it can be used to nullify a lot of potential attacks.

2012-03-03 15:35:00
Brian Purdue

bnpurdue@bigpond.net...
138.130.140.206

Don’t know if this has been talked about before but the way of guaranteeing TCP is the largest consensus study ever is to make it open-ended.

As new papers are peer-reviewed and published in the future they are added to TCP data base. Every 3 months? Every 6 months? Once a year? This is then published on SkS to keep TCP in the public eye.

This can be made part of the PR fanfare when TCP is first released because it would add to the consensus’s transparency and give the deniers heartburn.

The risk is there may be a bunch of negative papers that deniers would latch onto so time between each update would need to be made long enough to even out any possible distortions. If new data is just added to old data any short-term distortion should not be a worry. The consensus trend is up so that would negate this too. It’s going to be difficulted getting from 97% to 100% though.

2012-03-03 17:36:51
Ari Jokimäki

arijmaki@yahoo...
91.154.109.19

It will be a opportunity for deniers to show that SkS and scientists are biased.

2012-03-03 19:19:50
John Cook

john@skepticalscience...
121.222.175.176

Brian, continuing TCP into the future was always the plan. See Introduction to TCP:

Phase 4: Repeat each year

...in early 2013, we can repeat the process for all papers published in 2012 to show that the consensus is still strengthening. We beat the consensus drum often and regularly and make SkS the home of the perceived strengthening consensus.

My vision is that SkS becomes the home of consensus. More specifically, the home of the strengthening consensus.  It will be an ongoing, regularly updated feature on the website. Maybe we'll create an embeddable widget. At least once per year, we update the analysis with new data. Possibly come up with new ways to analyse the data and republish peer-review papers on a yearly basis. There are many ways to skin this cat.

I'm pretty confident of what we will find each year based on what we've seen over the last 21 years and we will let the data speak for itself. TCP needs to be a message that we beat loudly and regularly. I heard someone say if you are completely bored with your message because you've said it so many times, then maybe, just maybe, people will start getting the message.

2012-03-03 20:56:36
Brian Purdue

bnpurdue@bigpond.net...
138.130.140.206

John, as I said I wasn’t sure but I must have read your post because I have the one straight after it. But you read so much you’re flat out remembering it all.

I still think it’s important to make a big deal of the continual update of TCP when it’s launched for reasons given but with an embeddable widget how can people forget so I for one think it’s a great idea.  

2012-03-06 00:44:08
Sarah
Sarah Green
sarah@inlandsea...
67.142.177.24

Do you know about this one?

http://scienceblogs.com/deltoid/2007/08/classifying_abstracts_on_globa.php

2012-03-06 04:33:04
dana1981
Dana Nuccitelli
dana1981@yahoo...
64.129.227.4

You mean Schulte, Sarah?  Much smaller than ours (just 2004-2007).  I love his conclusion - because 'neutral' is the largest category, there is no consensus!  Somebody never took a basic logic class.

2012-03-06 06:33:17
MarkR
Mark Richardson
m.t.richardson2@gmail...
193.167.135.17

Dana: I think that's a big problem we'll encounter if it's not cleared up. Most papers I've rated so far are 'neutral', and I suspect it's true overall. Deniers will just pick up on this and say it means there's not a consensus. Really stupid and ignorant, but that's the denialsphere for you.

2012-03-06 07:12:23
dana1981
Dana Nuccitelli
dana1981@yahoo...
64.129.227.4

I'm sure we'll say something about it in the paper.  But the denialsphere being stupid isn't really a problem for us.  We're targeting the 'undecided masses' who don't realize there's a consensus.  They wan't be expose to some idiotic WUWT post.  The deniers mostly just talk to themselves.

2012-03-06 11:05:58
John Cook

john@skepticalscience...
121.222.175.176

MarkR raises a good point and this is something we will need to look at carefully. Deniers will try to shift the focus onto the # of neutrals. It helps to look at past critiques of Oreskes 2004 - particularly the Schulte paper. His paper is seriously flawed in that he overestimates the rejection papers by taking papers about uncertainty to mean rejection of AGW - and he also compares Naomi's 1993 to 2003 results to his 2003 to 2004 results to argue a "consensus is crumbling" meme. But it's a flawed comparison as his yardstick is different to Naomi's - it's comparing apples to oranges. Our data will cover both those periods and all the papers they covered so we can answer definitively whether Schulte's conclusion of a crumbling consensus is correct (hint, it's not). 

Here is Schulte's paper - recommended reading for anyone involved in TCP:

http://thedgw.org/definitionsOut/..%5Cdocs%5Cschulte_scientific_consensus_on_climate_change.pdf

From a Debunking Handbook point of view, I would argue that Schulte reinforces the meme he's trying to debunk in his very title :-)

So as to how to respond to the "it's all neutral" response, several thoughts come to mind. Firstly, many papers are neutral because they're not directly addressing AGW. Secondly, we overestimate the # of neutrals because of our restricting our analysis to just the abstract (which hopefully the scientists' self-ratings will confirm). Thirdly, and this is premature because we will know better once we have the final data but probably the result/image we will emphasise is the # of endorsements vs the # of rejections. That is probably going to be the take-home message we hope to communicate to the 'undecided masses'.

While our target audience is the general public, not the denialosphere, it's useful to anticipate all possible avenues of denier attack and either neutralise them or even use those attacks jujitsu style to reinforce our results.

2012-03-06 15:38:56
dana1981
Dana Nuccitelli
dana1981@yahoo...
71.137.110.129

Simple response - neutral doesn't mean anything except the authors didn't clearly indicate a position regarding AGW in the abstract.  The key question is, of the papers that did take a position, what are the percentages?

2012-03-06 17:22:42
Ari Jokimäki

arijmaki@yahoo...
194.251.119.197

Fourthly, scientists are neutral by nature, so it is expected that there are more neutrals than others. (If there is a conspiracy related to AGW, why is there so much neutral science being published?)

2012-03-06 17:42:09
Andy S

skucea@telus...
209.121.15.232
The predominance of neutrals is not a bug but a feature. When looking at the impacts of a temperature increase it doesn't always matter what the reason is. Similarly for some papers on co2 increases. I'm sure that for the majority of impact-neutral authors, the anthropogenic nature of the warming is also taken for granted. similarly, the relative lack of #1 rated papers is also not that big a deal; there are few earth-is-round mentions in recent geography abstracts either.