The Berkeley Earth team has created a preliminary merged data set by combining 1.6 billion temperature reports from 10 preexisting data archives (4 daily and 6 monthly). We have performed a series of tests to identify dubious data and merge identical data coming from multiple archives. Our process was to flag dubious data rather than simply eliminate it. Flagged values were generally excluded from further analysis, but their content is preserved for future consideration.
The Berkeley Earth team is currently writing a methods paper to describe the process used. We are also working on a paper on the urban heat island effect, a paper on station quality (using the Watts rankings), and a results paper that will analyze the full set of global surface temperature data. As of July 6 2011, we have not yet run the analysis on 100% of the data. This is in part because we want to have the papers written before we complete the analysis (to help ensure we do not inadvertently introduce a bias), and in part because we have discovered we need greater computing power to do so. We have requested, and been granted, access to use Berkeley Lab's cluster computer for our analysis, and are currently adapting our programs so that they will run on the cluster. We are optimistic that we will be able to make the dataset available to the public, along with our papers, by the end of the summer 2011.
All of our papers will be sumbitted to peer reviewed journals, and made available to the public when they have been submitted. In order to obtain notifications and updates, please sign up here.