A couple weeks ago I suggested it is peculiar for groups like Berkeley Earth (BEST) to make claims about what years were or were not the hottest in the temperature record. Each time they update their results, values for previous months change, sometimes by more than the stated uncertainty in their results. My view was uncertainty levels that can't handle changes between versions of a data set should be taken with a grain of salt.
There's a more interesting issue though. That post showed there are at least four different sets of results given by BEST calculations. In a later comment, I updated that number to seven. There are at least seven different sets of results published by BEST, and BEST hasn't archived a single one of them. BEST hasn't done a single thing to allow BEST to compare one version to another.
Yesterday, I discovered this problem goes even further. It turns out there are currently results from three different sets of calculations published on the BEST website. They're all published alongside one another as though they represent the same thing.
It is weird to publish results from different calculations as though they are all taken from the same calculations. The basic idea of version control is to make it clear what results go with what data and code. Failing that, you should at least make it clear what results go with what results. You shouldn't publish a mishmash of results from diffent calculations as though they're all from the same set of calculations.
BEST was created to address concerns people had raised about the global temperature record. One of those concerns was a lack of meaningful version control. I've long suggested BEST should address that concern. BEST has dismissed that idea. When I raised the issue just last month, BEST spokesperon Steven Mosher said:
If you want to keep a history of changes its simple. just do it.
you dont want to keep a history. I judge your intentions by your behavior. Your behavior says you dont care. I judge your actions, not your words.
Here we have a BEST spokesperson telling me it is my responsibility to keep track of the changes BEST makes. BEST creates tens of thousands of different files. It doesn't tell people when it is going to make changes to them. It doesn't even tell people when those files have changed. It would be an enormous task for a person outside of BEST to make a full archive of BEST's files, much less to keep track of when they change.
But even if that weren't true, why should it be up to people not associated with BEST to do basic version control for BEST? Isn't that something BEST ought to be doing? It turns out, BEST thinks so. Or at least, it did three years ago. BEST prepared a slideshow for a presentation at the American Geophysical Union annual meeting, one of the largest science conferences in the world:
One of the slides in it shows BEST agreed with my view:
Three years ago, BEST said it should "Subject the whole thing to version control so users can see how it changes over time." They thought users should be able to see how their methodology and results change over time. It was even part of their promotional strategy for their work.
Now, having never done anything to deliver on this idea, BEST members tell people if they want any sort of version control, they should do it themselves. It's remarkable how their standards have changed.