2012-01-31 13:26:32Are these guaranteed to be climate science?
scaddenp

p.scadden@gns.cri...
161.65.53.59

Also, I notice looking at some papers that these do not appear to be about climate science.

Eg

An Examination Of Existing Data For The Industrial Manufacture And Use Of Nanocomponents And Their Role In The Life Cycle Impact Of Nanoproducts

 

for DM Meyer. Do these belong?

2012-01-31 13:34:59methodology for rating papers
John Cook

john@skepticalscience...
130.102.158.12

The methodology is to replicate Naomi Oreskes' 2004 study - whatever the Web of Science spits out at us for the searches "global climate change" and "global warming".

Naomi's survey was fairly one dimensional and we do have the ability to dive deeper if we so desire. For example, group the papers by journal area - Ari's idea of seeing how consensus changes as journals get further away from climate science. That would address the "hey, that paper is not strictly climate science" objection.

2012-01-31 17:30:31
Ari Jokimäki

arijmaki@yahoo...
192.100.112.211

Perhaps we should do my project as a follow-up to this one?

2012-01-31 19:00:15
Rob Painting
Rob
paintingskeri@vodafone.co...
118.92.45.196

There's a fair amount of crap that doesn't seem to have much to do with global warming and we are missing heaps of papers that explicitly support the consensus. But...ah well.......

2012-01-31 21:01:39Ari's follow-up
John Cook

john@skepticalscience...
121.215.212.33

Yes, definitely, either as a follow-up or as part of this project, as a way of making our result that much more bullet proof. This is something to discuss once we've got some results to chew on.

2012-01-31 23:24:02
Ari Jokimäki

arijmaki@yahoo...
192.100.112.211

Ok, I'll start writing up some things about the methodology of the journal-project.

2012-02-01 10:53:38Journal info
John Cook

john@skepticalscience...
130.102.158.12

Ari, would you be interested in me spitting out a webpage that lists all the journals that our 12,000 papers come from and how many papers in each - probably ordered by the # of papers so you can see which journals are the biggest contributors?

2012-02-01 18:05:12
Ari Jokimäki

arijmaki@yahoo...
192.100.112.211

Yes, sure.

2012-02-02 12:03:35Journals page now added
John Cook

john@skepticalscience...
130.102.158.12

I've programmed up some quick code to display the top 200 journals in our database:

http://www.skepticalscience.com/tcp.php?Action=Journals

Now if you're interested, I can set it up so you can start categorising the journals as soon as you like - so that while we are collecting emails and then moving onto rating the papers, you can be rating the journals. Then when we've both finished our ratings, we can punch out the endorsement levels across the different types of journals. Very nifty, you can have an answer to the question "how does the level of endorsement of AGW vary across different fields of science?" in a very short time. I know it's not as comprehensive as your idea of surveying all papers from a journal but 12000 papers from over 2000+ journals is a helluva large sample size and will give you all the information you need to thoroughly answer the question.

Also, musing on it, I think this is worthy of a paper all on its own - a nice complement to Doran 2009's survey of scientists across different earth science areas. It's also consistent with my idea of a long-term campaign of regularly beating the consensus drum, looking at it from different angles in a series of analyses over time. Our SkS database will be a very powerful resource.

2012-02-02 17:11:30
Ari Jokimäki

arijmaki@yahoo...
192.100.112.211

I can start categorizing but have we decided the final categories yet?

 

2012-02-02 20:45:28No, nothing on categories yet
John Cook

john@skepticalscience...
121.215.212.33

That's the topic for a new thread, presumably part of that methodology you were working on. No rush on that, we haven't even finalised the consensus methodology yet. But whenever you're ready, start a new thread on the category methodology. I suggest also having another look at Doran 2009 and do a literature search to see if there's been anything done in this general direction in the past, to both ensure no replication and also may give some good ideas on categorisation and other possible meta-data we might want to capture.

2012-02-02 21:52:41
Ari Jokimäki

arijmaki@yahoo...
192.100.112.211

No, I meant that have we decided on the categories and classification methods for this current project already? There's a thread on that but I don't think there's a final decision made yet. But you can forget all this, because I just realized that I misunderstood you. I thought you were asking me to start classifying papers in the journals but you actually asked me to start the preliminary work on the journal project and categorize the journals (by their relevance to climate science, I assume). Yes, I can start doing that. :)

2012-02-03 04:25:11
Ari Jokimäki

arijmaki@yahoo...
91.154.97.114

I think categorizing journals might not be needed, if we introduce another category for paper classification scheme. If we add category "unrelated to climate", we can calculate how large percentage each journal has climate related papers. We then use that percentage to determine how far each journal is from climate science. This is far more accurate method for categorizing journals than having just a few discrete category bins.

2012-02-03 08:20:00Unrelated to climate
John Cook

john@skepticalscience...
121.215.212.33
Im keen to keep the survey consistent with the Oreskes 2004 survey. Currently when we encounter an "Unrelated to climate" paper, we relegate it to "Methods". But adding this new Unrelated category gives us the option of collapsing down to methods while retaining more info.

A few things though. It does add more options, makes it just that little bit more complicated although to me, that's not a deal breaker. Would need to come up with some guidelines on what makes a paper Unrelated. But the main issue with this strategy - it will only yield useful data for those top tier journals that have lots of papers. Most journals have only a few papers. In contrast, grouping the journals by category allows us to lump like journals together into significant numbers.

2012-02-03 17:29:13
Ari Jokimäki

arijmaki@yahoo...
192.100.112.211

With this unrelated category I was thinking more of my project of going through all papers in a single journal so the new category is not that important in this search word project where search words have been used in an effort to filter out only relevant papers. With your current sample it is not possible to determine the percentage of climate related papers as I described above. However, with unrelated category it would be possible to determine how succesful your search words were. If all unrelated would be placed in "methods" then separating them to two categories (methods and unrelated) wouldn't break anything. You would just add up the two categories together, when comparing to Oreskes' work.

The guidelines are needed of course, but they would also be needed when categorizing journals. I think it might be that defining unrelated category would be easier than defining several journal categories, because unrelated vs. related would just have one middleground while several categories would have several middlegrounds between the categories.

Furthermore, think of all the nice graphs we could make if we would have a number that would decscribe the journal relevance. I think it might be even worthwhile to determine the climate index (which I will from now on call the percentage of climate related papers) for each journal in your current sample (well, at least for the journals having most papers). That could be estimated by finding out the number of relevant papers in a single year. That way we wouldn't need to go through full journals at this stage. However, in the future it would be interesting to find out how the climate index of each journal develops through times.