My last post had this footnote:
*This claim of independence is incredibly misleading. BEST estimates breakpoints prior to running “the entire Berkeley Average machinery.” It does so by examining every station in its data set, comparing each to the stations around it. This is effectively a form of homogenization (BEST even stores the code for it in a directory named Homogeniety).
That means BEST homogenizes its data prior to performing its jackknife calculations. Whatever series are removed in the jackknife calculations will still have influenced the homogeneity calculations, meaning they are not truly removed from the calculations as a whole.
It’s trivially easy to show homogenizing a data set prior to performing jackknife calculations means those calculations cannot reflect the actual uncertainty in that data set. I’m not going to do so here simply because of how long the post has already gotten. Plus, I really would like to get to work on my eBook again at some point.
It occurs to me I ought to demonstrate this is true rather than just claim it. I tried to show what effect this has on BEST's results by fixing BEST's mistake and rerunning the analysis, but I couldn't because my laptop doesn't have enough memory to handle all the processing. As such, I'll just provide a couple of excerpts from BEST's code to help show what is done.
The main entry point for BEST's code is a file named BerkeleyAverage.m. The bulk of the work is done by a file it calls named BerkeleyAverageCore.m. Here is the code for that call:
% The following code executes the "scalpel" method of breaking records into % multiple pieces at metadata indicators and empirical breaks. [se, sites, start_pos, break_flags, back_map] = scalpelData( se, sites, options ); % This is where the real heavy lifting is actually done if nargout > 1 [results, adj_flags] = BerkeleyAverageCore( se, sites, options ); else results = BerkeleyAverageCore( se, sites, options ); end
As you can see, prior to calling the main file, BEST calls the scalpelData.m file. This file holds the code which determines where BEST believes breakpoints exist. It is only once those breakpoints have been determined that BEST passes the results to the main processing file.
The scalpelData.m file is never called again. When BEST performs its jackknife calculations, it does so with this call:
% The real effort for statistical uncertainty results.statistical_uncertainty = computeStatisticalUncertainty( se, sites, options, results );
The parameters passed to computeStatisticalUncertainty are the same as those which were passed to BerkeleyAverageCore.m plus the results doing so generated. computeStatisticalUncertainty.m doesn't call scalpelData. It just goes through some loops to see what happens if you remove 1/8th of the data used in the main calculations and repeat those calculations.
BEST does not recalculate its breakpoints when doing its jackknife calculations. It does not examine how much variance there is in its data set. BEST only looks at how much variance there is within its homogenized data set. This causes BEST to underestimate the actual uncertainty in its temperature record.
It is incredibly misleading to take the amount of variance in a homogenized data set as the amount of variance in the unhomogenized data set. It's baffling BEST does so.