2010-08-12 09:52:41Proposal for peer-review system (and new system updates)
John Cook

john@skepticalscience...
121.222.17.49

Have been talking to Doug about issues of quality control and I suggested the following peer-review system. Feedback welcome, here is the proposed process:

  1. Someone posts a new rebuttal here on the forum
  2. We all post comments/suggestions
  3. The original author then goes back and updates their rebuttal (I'll add a 'Edit' function to the forum) - then they add a post letting us know they've updated the rebuttal
  4. We all read the new rebuttal and when each author is happy with the final wording, they give it the thumbs up (I'll add a thumbs up feature).
  5. Once a post receives three thumbs up, it's "passed peer-review"
  6. Doug suggests as a final step, I be the one to officially go live with it (I can take or leave this last point but Doug thinks editorial control is still important)

So questions/comments/thoughts/suggestions?

Also, I've majorly revamped the Rebuttal List. It's colour coded (I love using colours). You can now edit and preview your rebuttals. You can claim an unclaimed rebuttal - then if you change your mind, you can unclaim it. Note: none of these rebuttals are live yet, we can only see them from here and only Authors can view them. So you can play around in here without fear of prying eyes.

2010-08-12 11:22:55Agree with Doug
Robert Way

robert_way19@hotmail...
142.162.181.139
I think this idea is great but I am hesitant about it only requiring 3 thumbs up. I think ultimately you will have to keep editorial control if it is a small number of thumbs up.
2010-08-12 11:58:05# of thumbs up
John Cook

john@skepticalscience...
121.222.17.49
If we have enough active authors, I'll beef it up to 5 thumbs up. We'll see how it goes.
2010-08-12 16:25:50Sounds good to me
James Wight

jameswight@southernphone.com...
58.105.164.221
Sounds good to me. I also agree with Doug about editorial control.
2010-08-12 16:47:15More agreement
gpwayne
Graham Wayne
graham@gpwayne...
81.132.177.33

The thumbs-up aspect is a bit odd in my view. I say to deniers that science can't be voted on, and in terms of factual accuracy this must be true of our basic rebuttals. Whether we include/exclude graphs or other materials (I'm basing this on suggestions made regarding my submissions) is pretty arbitrary, and as such requires a final arbiter. I think that's you John - sorry mate, it's a dirty job, right... :)

2010-08-12 18:28:03JC to have editorial control + IMPORTANT style issues.
John Russell

jr@johnrussell...
82.70.63.102

Much as I like democracy, I know that often more gets done when someone has absolute power -- provided their heart's in the right place. In this case that's JC and I agree that he should have final say.

One issue comes out straight away. I think the list of basic rebuttals is developing into rather a mess of styles, dependant on who started the thread. To state the obvious; what I write (being very much a layman) is very different in style from what some of the scientists have written. However no one has said that what I've written is not appropriate for the requirement and vice versa.  IMHO someone -- JC? -- needs to decide which way we should go for the basic rebuttals. If we take topic 7, "it hasn't warmed since 1998", JC started it in one style and then, as much as an experiment as anything else, when he questioned certain aspects of what he'd written I waded in with a completely different approach. So which is the way we want to go? Note that I'm not saying there's a right and wrong way -- it's just that the reader should see a consistent approach.

John originally suggested 260 words and -- I admit, it agreed with my preferred approach -- I've tried to adopt that length where it works for the topic. Some others seem to me to be overly long -- more like the medium rebuttal that are already up on the site.My approach is that the basic rebuttal is just that -- basic; it leaves out many strands of the argument that can and should be covered in the deeper versions of the rebuttals. The main thing is that what it does include should be correct and clear.

Graham's point is that a basic rebuttal should be capable of standing alone when taken out of context. I know where he's coming from, but I do worry about that as it does create a constraint on the content. This is an example of where we need strong editorial guidance. JC? 

Please can we clarify these points? I don't have any axe to grind -- I just want to know which way I go when I start the next rebuttal. 

Best wishes,

JR

2010-08-12 23:22:21Style guide for basic rebuttals
John Cook

john@skepticalscience...
121.222.17.49
This is a really hard question - because I'm not sure there is one magic style. What I think I'll do is start a thread which will be the "Style Guide for Basic Rebuttals". This will lay out a few general principles. I will program in a "sticky" feature in the forum so the Style Guide stays at the top of the forum. Then as we go through this process of refining our rebuttals, getting them ready for public consumption, I think we'll get more of a clearer idea on how best to do this. And I'll edit and refine the Style Guide as we go.
2010-08-13 05:58:04Style guide yes!
doug_bostrom

dbostrom@clearwire...
184.77.83.151

That will save a lot of time and aggravation and as well addresses something that was nagging me regarding maintaining a strong identity for the site.

We're (the world) fortunate that John has contributed what as far as I can tell is a uniquely orderly presentation and calm manner to the world of online climate discussion.  I realize now that some of my through-the-roof reaction to John's making big changes to the site was about fear of losing those benefits.

Thinking of this as a publication and thus being able to draw from the very deep well of wisdom about publication logistics and operations is a great way to go. Many things will not need to be reinvented by trial-and-error.

 

2010-08-13 16:41:19On consistency
gpwayne
Graham Wayne
graham@gpwayne...
81.132.177.33

We're all on the same page here, by the look of it. I am concerned for the reputation of the site - as Doug says, JC (!) has done a remarkable job of keeping things consistent, calm, orderly and sensible - everything that I personally fight for in the face of such intractible irrationality.

As John R suggests, having a single, final authority that make the decisions is the way to get things done. Peer-review is one thing, trying to herd cats (my opinion of most committees) can be terribly frustrating, unproductive and bloody time consuming.

The standard format is important in my view. The opening title, links to the site, the intermediate version etc - every rebuttal should follow a template that the final versions can adopt, once we get the arguments written up in an accepible way. I'm planning to write some more rebuttals, then go through the lot, edit and revise, and standardise.

2010-08-21 05:06:01Comment
Robert Way

robert_way19@hotmail...
24.224.230.22
John,
I'm finding the system a little cumbersome in the sense that it is sometimes annoying that everytime i put in revisions I have to do it in two places. Is there any ways to automatically link the argument in the rebuttal list with the forum post?
2010-08-21 05:47:42
doug_bostrom

dbostrom@clearwire...
184.77.83.151
Robert, try doing your revisions in the same place you drop your first draft, here. Once everybody's satisfied, use the edit window here to select and copy all of the final draft, then paste it into the edit window over on the rebuttal list. This is extra handy because suggestions for revisions are visible in the same browser window as you're using to perform edits.