2012-02-18 21:13:17Specific technical questions about Phase 2b: rating papers
John Cook


Am starting coding the 'rating paper' functionality and have a few specific questions:

  1. One category was 'Opinion' which identifies articles that aren't actually peer-reviewed papers. These will be eliminated from our final analysis as we're only interested in peer-reviewed papers. It's extremely difficult to identify an opinion paper just from the title and abstract. In Phase 1, Jim did a lot of detective work in identifying opinion papers. For example, identifying articles that were posted at regular intervals by the same author in the same journal. Or articles in Scientific American which isn't peer-reviewed. I suggest our approach is we leave off Opinion as a category but people can tag the paper as a possible opinion paper in the text box that goes with each rating, and at the end, we draw upon Jim's identification of possible opinion papers.
  2. I'm planning to only show the title and abstract. So when we rate papers, we don't see the author, year or journal. Any arguments against that approach?
  3. I've put an "Undecided" option in our list of categories so if someone gets stuck, they can file it under "come back to it later". I don't have an Undecided for Endorsement Levels though - do people want one there also?
2012-02-18 22:48:33
Tom Curtis


John, (1) is good as far as it goes.  It is a good method of ensuring raters do not falsely tag as "opinion peices" which are in fact not opinion peices.   However, in the review drawign on Jim's expertise, if it is only applied to articles taged as possible opinion peices we will retain a significant number of opinion peices among our supposedly peer reviewed articles.

I would suggest that ideally:

(a) Jim should prepare a list of probable opinion peices independantly;

(b) Papers tagged as potential opinion peices by reviewers of title and article which are on Jim's list then be treated as opinion pieces;

(c) Papers Jim tags but which are not flagged as opinion peices be given a review of the full article and then tagged as opinion or not based on that review;

(d) Papers flagged as opinion pieces but which are not on Jim's list be likewise reviewed to settle the issue; and

(e)  The sample of papers for extensive review (based on my earlier sugestion) be drawn from the full list of papers including those listed as opinion pieces  by Jim to give data on error rates on the proceedure.


I can understand you considering this approach as too intensive.  If so, I believe it is more important that opinion peices be kept out than that all peer reviewed peices be excluded in that the incorrect elimination of a small number peer reviewed articles will not bias the result.  That being the case:

m) Jim should prepare the list of all papers he considers to be opinion peices, all of which should be dropped from the title plus abstract review process but not from the full paper review process;

n) Any paper flagged by both reviewers of title and abstract as an opinion pieces should be dropped, while if only one reviewer flags the article as an opinion piece, it should go to a tie breaking review as per normal; and

0) Full paper reviews should be drawn from among all papers to provide data on the error rate.

This latter procedure will save work all round by eliminating regular columns from the primary review process.  (I am not sure what you mean exactly by "... at the end, we draw upon Jim's identification of possible opinion papers" so it is of course possible that either of these suggestions is what you already had in mind. 

(2) is good except you must decide what to do with those few papers with no proper abstract.  Do you include the introductory paragraph(s) as the abstract or just show title only (which wouldn't be very informative).  Excluding author, year, and journal is necessary for a double blind approach.  We all have some ideas when Denier friendly editors have had influence at particular journals and including year and journal data could in priniciple bias our rating.  So definitely exclude.

(3) I am inclined to think that if you are undecided about whether the article quantifies the effect, it doesn't; if you are undecided about whether it explicitly accepts (rejects) AGW it doesn't; and that if you are undecided about whether it implicitly accepts (rejects) AGW, it doesn't.  Therefore not undecided category is needed for Endorsement Levels.

2012-02-18 23:31:41
Ari Jokimäki


#2 is a good idea. Regarding Tom's question of papers without abstracts, I would say they don't belong to the sample. Perhaps you need an extra category for no abstract, or are all these papers already filtered for that (i.e. all the papers in the sample already have an abstract)?

2012-02-19 01:15:29Papers without an abstract
John Cook


When we started, we had over 200 papers without abstracts but Jim did most of the legwork in tracking down abstracts and we've got it down to 93. If someone with library access wants to track down the remaining papers, let me know and I'll set you up. Some are simply impossible - one I spent ages trying to track down, turns out it's in French and the paper is not available online. I tried emailing the author online, no response.

So I suggest we try to whittle down the papers without abstracts to as small a figure as possible, then we eliminate the left-over papers out of the sample.

Tom, wow, that's a lot of bullet points :-) Sounds like a good plan except I don't think we should drop any papers, even Jim's opinion pieces, at the title+abstract stage. So:

a) Jim has already compiled a list of opinion papers

b) Hmm, I'll include Opinion as a category option

c) & d) Papers that either Jim or we rate as opinion will be flagged for closer inspection

2012-02-21 17:35:59Can't read the roll-overs properly
Glenn Tamblyn


When I mouse over anything with a pop-up text - abstracts, categories etc - the top of the pop-up text box is missing. I suspect it isn't aligning to the top of the main window but rather something like the bottom of the tab bar Only when I run with just 1 or 2 toolbars does it work properly.

2012-02-21 23:14:29Hmm
John Cook


Glenn, what's your browser and operating system?