2011-02-03 09:20:37Global Average Temperature Rise from 1880, based on annual counts of monthly maximum & minimum records
tobyjoyce

tobyjoyce@eircom...
86.46.184.199

In a recent paper, Wergen (2010), two mathematicians studied record-breaking daily temperatures in the European temperature database. In sober and cautious scientific language, they found that five out of seventeen temperature records in 2005 could be attributed to climate change (rather than to chance, presumably).  In other words, they confirmed a warming from European temperature data.

Can the investigation of temperature records be simplified in some way? Here is one possible way.

  • Remove any dependence on actual temperature and make a binary count of record months (1 for a record, 0 for no record) – separately for both maxima and minima.
  • Count over succeeding years and make counts of the number of record months in successive years.

Let n(y) be the number of monthly records (maxima or minima) in a year, and let N(y) be the cumulative sum of n(y) over successive years. We will use n(t) and N(t) when talking about years measured from a particular starting year.

When there is more than one database, we take the average e.g. If n(1960)=2 for the GISS database and n(1960)=1 for the NOAA database, we count it as the average = 1.5. That is because we are interested in the average cumulative number of records and its rate of change.Graphic

Figure 1: Differences in Counts of Average Annual Maximum and Minimum Monthly Records. Nx(t) = cum. average number of maximum monthly records, Nn(t) = cum. average number of minimum monthly records.

Figure 1 shows the cumulative annual differences in counts of average maximum and minimum records in 5 databases (GISS, HADCRUT and NOAA from 1880 to 2010, RSS and UAH from 1979 to 2010).

Comments:

  • There is an “early measurement effect” because the first year’s temperature measurements will all be both maximum and minimum records. Subsequent months will modify the records so that it will take a few years for the annual counts to settle down.
  • We could expect that in a period of warming, maximum records will exceed minima, and vice versa in a period of cooling. Generally, there should be periods of stasis if there is no trend, and perhaps a sine wave indicating long term natural variation.
  • In Figure 1, the early century shows “early measurement effect”, following by perhaps a 20-year period of cooling. After 1920, a mid-century warming commences, and this looks like natural variation (a half-sine wave) up to about 1940.
  • Then a period of stasis ensures (for 12 years) until the excess of maximum over minimum records starts again with an accelerating increase up to 2010.
  • Around 1980, an “early measurement effect” occurs again with UAH and RSS data are added to the counts.
  • The effect of Mount Pinatubo in 1991 is visible, causing a couple of minimum records, and the heavy El Nino of 1998, causing a spate of maximum monthly

Graphic

Figure 2: Cumulative Rise in Annual Average Monthly Records.

The key parameters to be estimated are the functional form of M(t), the expected value of N(t) at year t and its rate of change m(t) =dM(t)/dt. Let AROCOX= Annualised Rate of Occurrence of Maximum Monthly Records = m(t). Usually a power function is used, of the form at^b. The rate of change is the derivative of the function. Clearly, M(t) is increasing linearly if b=1. If b>1, then M(t) has an increasing slope. If b<1, then it has a decreasing slope i.e. the records are becoming rarer and occurring an increasing distance apart. Then, AROCOX  m(t) = abt^(b-1)

Comments:

  • The Maximum Likelihood Estimates of a and b in Figure 1 are 0.08 and 1.71 respectively. The estimate b=1.71 indicates an accelerating trend.
  • The AROCOX (time-dependent) estimate is 0.14t^0.71. It is interesting to look at the average trend in different decades:
    • 2001-2010            1.81r/yr
    • 1991-2000            1.56r/yr
    • 1981-1990            1.27r/yr
    • 1970-1980            0.94r/yr
    • 1960-1970            0.56r/yr
  • Since the incremental increase in temperature for each record reflects the temperature rise, the average temperature rate can be estimated from the temperature data. Let ∆T=Average Temperature Rise over all maxima. Then Temperature Rate = ∆T x AROCOX.
  • Plugging in ∆T=0.011C, the following values are estimated for temperature increase:
    • 2001-2010            0.20C/decade
    • 1991-2000            0.17C/decade
    • 1981-1990            0.14C/decade
    • 1970-1980            0.10C/decade
    • 1960-1970            0.07C/decade
  • Predictions for the next decade (assuming continuance on current conditions):
    • 2020 AROCOX = 2.09r/yr
    • 2020 Rate of Temperature Increase = 0.23C/decade
    • The probability of 2011 not having a record month is 0.11

This basic, and even crude, analysis confirms the model of temperature rise given by mainstream climate science.  That is no surprise. However, it can be expanded. An analysis of residuals for the model used here clearly shows autocorrelation due to natural variation. Variables for these factors can be incorporated in a logistic or Poisson regression model, and a comparison made of natural variation in the earlier and later 20th century.

This analysis outline undermines, yet again, many of the simplistic contrarian analyses e.g. that natural variability is operating, or that the earth is cooling since some time in the period 1998-2002. As Professor Richard Lindzen said: “Temperature is always rising and falling”. However, that would imply an equalization of maximum and minimum monthly records over a long period. Indeed, something like that happened from 1880 to around 1920 with both maximum and minimum records occurring, with a slight preponderance for minima.

 

This analysis shows that, except for isolated cases caused by the Mount Pinatubo eruption and the “early measurement effect” artefact, there has not been a sequence of minimum records in global temperature databases for some time.  The last one ended in 1917, almost one-hundred years ago. The rate of occurrence of minimum records is 0 per year, and the rate for maximum records is consistently outstripping that of minima by almost 2 per year, and rising.

2011-02-03 17:58:16Interesting post but a bit opaque
John Cook

john@skepticalscience...
123.211.149.21
I think it could be explained a little more simply - it's a bit technical. Some specific comments:
  • The lettering on the graphs needs to be bigger - you can't read them. You can also increase your graphs to 570 pixels wide (but no wider)
  • I think it reads better if you introduce Figure 2 before showing it - "here's a graph of..." so people are told what they're looking at
  • In fact, from this point of the article, it gets difficult to read. I would suggest adding some more explanatory text making it more intuitive. It's a pity it's not a rebuttal - this could be the advanced version and we could have more easy versions for the layperson.
2011-02-03 20:51:53Come all ye sceptics .... :)
tobyjoyce

tobyjoyce@eircom...
86.46.184.199

Thanks, John, I will take that on board. I want the logic to be clear and accessible without doing violence to the underlying statistical theme.

Please, anyone who reads the post, drop a few comments here, as negative as you like.

2011-02-06 00:28:43I will re-post this article later today.
tobyjoyce

tobyjoyce@eircom...
86.46.184.199

I will be reposting a revised version of this article later today.

The basic idea is to look at the temperature data as a set of discrete events (much as you might keep a running count of breakdowns to your car) instead of sampling a continuous distribution. For many reasons, simple stochastic processes like this one are more easily handled mathematically than probablility distributions.

Any useful remarks about the re-posted article would be acepted gratefully.

2011-02-06 03:19:13
Ari Jokimäki

arijmaki@yahoo...
91.154.104.236

Another approach is to study the diurnal temperature range:

http://agwobserver.wordpress.com/2010/11/24/papers-on-diurnal-temperature-range/

2011-02-06 03:55:40
tobyjoyce

tobyjoyce@eircom...
86.46.184.199

Ari,

Thanks for that. I hope what I doing leads eventually to a published journal paper (or at leat a few blogposts!).

2011-02-06 06:00:00Global Average Temperature Rise from 1880, based on annual counts of monthly maximum & minimum records: Repost
tobyjoyce

tobyjoyce@eircom...
86.46.184.199

Global temperature data are very “noisy”. There are multiple databases with slightly different observations and different baselines. The usual way to reduce this noise is by “filtering” though a moving average or other smoother to isolate the trend. However, this often destroys much of the richness of the information. Here is an alternative way to de-noise the data while still retaining much of the interesting detail.

A simple count is kept in each database, for each month, marking a 1 if a new temperature record is achieved (compared with the same months in previous years), and a 0 if it not achieved. Then, for each year in the database, there is a running count of the number of new record months in that year. This can be done for both maxima and minima.

To combine the databases, an average is taken. If, for 1960, the GISS database shows 1 new record month,  the NOAA database shows 0, and the HADCRUT database shows 1, it is counted as average = 0.66 for 1960 in the running count of yearly record months. That is because the average cumulative number of new monthly records and its rate of change are of great interest.

We intuitively expect that in a period of warming, new maximum records will exceed minima, and vice versa in a period of cooling. Generally, there should be periods of stasis if there is no trend, and perhaps a sine wave indicating long term natural variation.

Graphic

Figure 1: Cumulative differences in Counts of Average Annual Maximum and Minimum Monthly Records.

Figure 1 shows the cumulative annual differences in counts of average new maximum and minimum records in 5 databases (GISS, HADCRUT and NOAA from 1880 to 2010, RSS and UAH from 1979 to 2010).

 

Comments:
  • There is an “early measurement effect” because all the first year’s monthly temperature measurements will all be both maximum and minimum records. Subsequent months will modify the records so that it will take a few years for the annual counts to settle down.
  • In Figure 1, the early decades show “early measurement effect”, following by perhaps a 20-year period of cooling. After 1920, a mid-century warming commences, and this looks like natural variation (a half-sine wave) up to about 1940.
  • Then a period of stasis ensures (for 12 years) until the excess of maximum over minimum records starts again with an accelerating increase up to 2010.
  • Around 1980, an “early measurement effect” occurs again with UAH and RSS data are added to the counts.
  • The cooling effect of Mount Pinatubo in 1991 is visible, causing a couple of new minimum records, and the heavy El Nino of 1998, causing a spate of new maximum monthly records.
  • Figure 1 resembles charts of the temperature anomaly – but it has a more “natural” origin than subtracting the temperature observation from a chosen baseline.

 Figure 2 is a chart of the cumulative rise in new annual maximum monthly records from 1956 onwards. Note is a non-linear, increasing trend – for each 10 year division, more records are occurring.

Graphic

Figure 2: Cumulative Rise in Annual Average Monthly Records since 1956.

Comments:

  • It is possible to fit a function to the curve and use the model to predict the rate of occurrence of future new records.
  • The rates are estimated from the fitted function, for different decades, in new maximum monthly records (r) per year:
    • 1960-1970            0.56r/yr
    • 1970-1980            0.94r/yr
    • 1981-1990            1.27r/yr
    • 1991-2000            1.56r/yr
    • 2001-2010            1.81r/yr
  • To understand the previous table better, in the decade 1960-1970, new maximum monthly records occurred on average about once every 21 months (=12 x 1/0.56). In the decade 2001-2010, they occurred on average every 7 months (=12x1/1.81).
  • Since the incremental increase in temperature for each new record reflects the temperature rise, the average temperature rate can be estimated from the temperature data. Let ∆T=Average Temperature Rise over all maxima. Then Temperature Rate = ∆T x Rate of Occurrence of Records.
  • Plugging in ∆T=0.011C, the following values are estimated for temperature increase in degrees C per decade:
    • 1960-1970            0.07C/decade
    • 1970-1980            0.10C/decade
    • 1981-1990            0.14C/decade
    • 1991-2000            0.17C/decade
    • 2001-2010            0.20C/decade
  • Predictions for the next decade (assuming continuance of current conditions):
    • 2020 Rate = 2.09r/yr
    • 2020 Rate of Temperature Increase = 0.23C/decade
    • The probability of 2011 not having a new record month is 0.11

This basic, and even crude, analysis confirms the model of temperature rise given by mainstream climate science.  That is no surprise. However, it can be expanded. A deeper analysis of the model used here clearly shows effects due to natural variation. Variables for these factors can be incorporated in a more complex regression model, and a comparison made between natural variation in the earlier and later 20th century.

 

This analysis undermines, yet again, many of the simplistic contrarian models e.g. that natural variability is driving warming, or that the earth is cooling since the period 1998-2002. As Professor Richard Lindzen said: “Temperature is always rising and falling”. However, that implies an equalization of maximum and minimum monthly records over a long period. The data show that, except for isolated cases caused by the Mount Pinatubo eruption and the “early measurement effect” artefact, there has not been a long sequence of minimum monthly records in global temperature databases for some time.  The last one ended in 1917, almost one-hundred years ago. The rate of occurrence of minimum records is 0 per year, and the rate for maximum records is consistently outstripping that of minima by almost 2 per year, and rising.

 

2011-02-06 07:44:04Much better
John Cook

john@skepticalscience...
123.211.149.21
Thanks so much for simplifying the explanation, this is much better. A few tiny things:

In the 2nd paragraph, don't you subtract minimum records and add maximums? It reads like min and max both get added.

For figure 2, is it possible to put the year in the x axis rather than year since 1956?

And for fig 2, can I suggest making the min time series blue so there's more colour contrast between min and max (plus I just love lots of colour in graphs).

2011-02-06 09:42:47
Andy S

skucea@telus...
66.183.174.250

This is interesting, thanks.

But let me react as a climate "skeptic" would, which is to say, commenting before I've properly thought this through.

It would be possible to set more monthly minimum records in a year even if the overall annual average trend is up. For example, you could have a couple of exceptionaly cold months in a year where none of the other months broke maximum records but they generally tended to have temperatures a little above the averages for those months. This is unlikely, of course, to happen year after year but you should perhaps hedge your fourth paragraph to say something like We intuitively expect that in a period of warming, new maximum records will exceed minima in most years.

Some of the "natural cycle" proponents might argue that this study says nothing about cycles longer than the 140-year period that you have looked at. 

Rather than just look at look at whether the monthly temperature is yes/no a record, have you looked at the magnitude of the records (ie the record minus the previous record in degrees K)? My guess would be that this would show a big peak during the "early measurement effect" periods, which would rapidly and then more gradually diminish. This might help in identifying the periods when the "early measurement effect" was a problem. You could even test this on a purely  random dataset, as you could your current approach, come to think of it.

Similar to what  John said, the caption for your figure1 confused me and it took me a while to figure out why the cumulative records could be negative or have a negative slope.

2011-02-06 20:39:28
tobyjoyce

tobyjoyce@eircom...
86.46.184.199

Andy S,

Thanks. Your points:

(1) Good input. I will make the changes. However, in a stochastic process, "near misses" are part of the stochasticity. A count of road accidents in a particular area would not include "near misses" either, as they would be unreported. In theory, an identical count could be made for new 2nd places, new 3rd places etc. but lower places would be infrequent. That comes under "future work".

(2) Yes, I was thinking about ENSO, PDO etc., all the old favourites. I will add a rider to that effect.

(3) I want to preserve the simple counting process method because the mathematics is more tractable, and well understood from survival analysis. However, I have already labelled that "future work".

Regards,

Toby

2011-02-07 09:54:01
Riccardo

riccardoreitano@tiscali...
93.147.82.133
Although some readers will miss the details of the analisys the message is clear. A nice way to look at the temperature record, good job.

"there has not been a long sequence of minimum monthly records in global temperature databases for some time."
This sentence is a bit generic and obscure. Maybe instead of "a long sequence of minimum monthly records" you may say that the number of record minimum does not exceed the number of record maximum, quoting fig. 1.
2011-02-07 13:36:43Random efforts
Andy S

skucea@telus...
66.183.174.250

Toby:

I cobbled together a fit-for-purpose Excel spreadsheet (translation, it's a mess)  that took an approach similar to yours to analyze a random temperature series. I ran two cases, one with a warming trend superimposed and one with no trend. I looked not just at the number of records but also the magnitude ( for each year, I just added together the magnitudes of the excess of the new records over the ones that they beat from the same months the previous year. This is all really pretty mickey mouse since monthly temperatures in reality  are not purely random (ie you can get successive global warm months during an El Nino). Also, I set the temps to vary just +/- 1 degree and the random generator in Excel is a box distribution, not gaussian. The trend is 0.3 degrees per century. You could just double the y axes to get a more realistic 0.6k/yr and a year-to year variation of +/-2k.

 

 

 

The results basically show that the "early measurement effect" does last for about 20 years as you thought and that the relative size of the records makes the early measurement records really stand out.

I'll email you the spreadsheet if you are interested.

Added: I'm struggling with uploading the graphs. I'll try again later . Fixed

2011-02-07 19:14:09
Riccardo

riccardoreitano@tiscali...
192.84.150.209
The length of the "early measurement effect" strongly depends on the interannual variability relative to the trend, the larger the variability the longer the effect lasts.
2011-02-08 04:37:37
tobyjoyce

tobyjoyce@eircom...
86.46.184.199

Andy S & Ricardo,

Thanks for the inputs. Here is the chart of the monthly average records (uncumulated, presented as in Andy's simulation.

Graphic

I stuck a minus sign in front of the minimum records to make them show up better. The LOESS smoother for the minimum records looks like a damped sinusoidal, while the smoother for maxima looks like a sinusoidal with its last lobe stretching.

Andy, you are welcome to send me that file.

2011-02-08 16:08:40
Andy S

skucea@telus...
66.183.174.250

Toby, email me at agskuce@gmail.com since I don't have your address.


John, I note that you have a list of the moderators' email addresses. Would it be possible to do the same for the authors? Sometimes I spot something that may be of interest to a particular author, for example, there's a new post at RealClimate where Lindzen is mentioned in respect to the thermal inertia of the ocean and I thought that Dana might like to be notified of it (although in this case he may well have spotted it already).

 

2011-02-08 19:17:44
Ari Jokimäki

arijmaki@yahoo...
192.100.112.210

This is an interesting method. Some comments:

I somewhat disagree with the claims in the first paragraph. First, I don't think that filtering destroys the richness of the information. On the contrary, I think it reveals the richness from the noise. Second, I'm not yet convinced that this is a good way to "de-noise the data while still retaining much of the interesting detail". Look at this 5 year running mean of GISS surface temperature analysis:

http://www.woodfortrees.org/plot/gistemp/mean:60

You can see there that quite a lot of big things are happening between 1940 and 1970 at the same time when your method shows a flat line (in your figure 1). What leaves to be demonstrated is that your method indeed shows the temperature evolution reliably.

Here's one point that argues that your method might not be a reliable temperature indicator: consider a temperature record that has huge warming spike in the early phase and huge cooling spike right after that. After them, the record shows quite large fluctuations, but the fluctuations don't reach the maximums and minimums of the early spikes. If I have understood your method correctly, it produces the spikes in the beginning, but after that shows a flat line. It seems to me that in a longer record (where old time records are harder to break) your method tends to show less an less variability and only shows exceptional periods. This effect probably produces the flat line after 1940.

At any case, I think your method needs more testing.

"Around 1980, an “early measurement effect” occurs again with UAH and RSS data are added to the counts."

What's the point in adding the RSS and UAH, if they cause error to the result?

"Figure 2 is a chart of the cumulative rise in new annual maximum monthly records from 1956 onwards. Note is a non-linear, increasing trend – for each 10 year division, more records are occurring."

Why select 1956 specifically?

From figure 2 it is easy to argue that the trend might as well be linear but starts, say, at 1970.

2011-02-08 20:24:42
Riccardo

riccardoreitano@tiscali...
192.84.150.209
Ari
the method works fine when the variability is (roughly) normally distributed and with constant variance. This is reasonably true for the temperature record.
The reason why the inclusion of the satellite datasets produce an early measurement effect is because they show a larger variance than the surface record. I agree with you, probably they should be left out of the analysis, although the effect is barely visible.
2011-02-09 05:04:11
tobyjoyce

tobyjoyce@eircom...
86.46.184.199

Ari,

At best, let's say, it is a viable alternative method to smoothers. Instead of "a lot going on between 1940 and 1970", maybe that is just noise, which this method removes. Figure 1 is comparable to this record of the temperature.:

Graphic

You could compare the counting method to a simple count of breakdowns in a car, or crashes of a software system. The first thing you want to know is - is this system getting better or worse with time? This is pretty much a facsimile of the meithod used to determine that. Admittedly, it does not do very well when there is no trend, but that is not necessarily a disadvantage. In my business, which is industrial statistics, that reduces the probability of a false alarm, or false positive.

As for the "large spikes" problem, I am thinking of generalising the method to any rank (a 2nd or 3rd or arbitrary placing ...), and see what can be deduced from a "current set" of rankings. Not only is the number of maxima increasing, so so have high rankings increased in general. Only 9 months in the decade 2001-2010 were outside Top 10 ranking. In the previous decade it was 28.

I considered leaving out the UAH and RSS results but decided to show "warts and all". If there is a strong opinion that it would be better to leave them out, I will take that advice.

1956 was the year temperature records started again .. first time since mid-1940s or so. Again, some "change point" statistics could be introduced to verify a change, but that would complicate an analysis I want to keep reasonably simple. You could start at an arbitrary time, and the last decade results are close to what you would get with a linear estimate.

2011-02-09 05:53:13
tobyjoyce

tobyjoyce@eircom...
86.46.184.199

Ari

I should have mentioned this paper by Wergen & Krug http://arxiv.org/PS_cache/arxiv/pdf/1005/1005.3145v2.pdf

The look at daily European records for 2005.

Daily records are correlated so that is a problem they have to handle. I should have mentioned autocorrelation specifically. If the increments in your counting process are such that the counts in disjoint intervals are independent, then the mathematics gets simpler, and you have what is called a Poisson Process. That is basically what you have here, so that autocorrelation as a source of noise is ruled out.

Thanks for you input .. I would value your further opinion.

2011-02-10 17:26:43
Ari Jokimäki

arijmaki@yahoo...
192.100.112.211

tobyjoyce: "At best, let's say, it is a viable alternative method to smoothers."

The methodological flaw I identified seems to be real, so I don't think this method can be used to describe temperature. It is a good method to illustrate the records in the series, though. If you want your method to show temperature, you should reset the records so that you wouldn't count all time records but you would count records in last 30 years, for example. That way the method would not get stuck forever by huge records early in the series.

tobyjoyce: "Instead of "a lot going on between 1940 and 1970", maybe that is just noise, which this method removes."

Between those years, your method is stuck between the minimums made around 1910 and maximums made around 1940. That shows as a flat line. Now, we know why your method shows a flat line there and based on that we just can't say if there's noise or real thing between those years. However, your method does show the slight cooling episode after 1991 but doesn't show anything during the far larger cooling episode between 1940 and 1950. These both events exist in the same series so in order to remove that 1940's event your method should be able to detect that the 1940's event is noise but 1990's event is not even if 1940's event is far larger. I think there's no way of telling which one of these events are noise from looking at the series alone, which is what your method does.

tobyjoyce: "If there is a strong opinion that it would be better to leave them out, I will take that advice."

Another possibility is to develop the method so that it can handle the addition of more data later in the series without distortion.

2011-02-10 19:13:41
Riccardo

riccardoreitano@tiscali...
192.84.150.209
The method is not flawed, it smooths the time serie with a strength dependent on the variability and its statistical distribution. The difference with other smoothing techniques is that you can not decide how much you want to smooth and that it cannot be used if the variability is ill distributed.
2011-02-10 23:29:04
tobyjoyce

tobyjoyce@eircom...
86.44.235.15

Ari,

It is a fundamental misunderstanding to say that the counting method is "trying to reproduce the temperature record". I will adjust the wording to make sure no other reader thinks that. The method is not recommended to reproduce the average global temperature in a particular year - I am not even sure where the notion of reproduction came from. The method does do a good job of reproducing the temperature rate of change over a numbers of years, and I think I have shown that.

Yes, it misses the "slight surface temperature decrease" in mid-century, something like -0.01C/decade over 20 years or so, clearly not enough to generate new minimum monthly records within that time period. But if that rate of change continued long enough, it would inevitably be picked up. See http://www.skepticalscience.com/global-cooling-mid-20th-century-advanced.htm . I do not see that as a great fault. It does seem to pick up a +0.07C/decade in 1960-70, so a minimum rate of change it can pick up is probably about +/- 0.05C/decade over at least 10 years.

Look upon it is an alternative method of measuring climate change - it looks at a different signal, one that happens not to suffer from autocorrelation noise.

2011-02-11 03:20:27
Ari Jokimäki

arijmaki@yahoo...
91.154.104.236

By using "reproduction" I may have used badly chosen word, but the idea arises from your first paragraph, where you offer this as an alternative to normal filtering techniques. This suggests to me that you are trying to offer this as a method to show global temperature.

But even as a method to measure climate change this method fails if there has been much larger climate changes in the past. It seems to me that this method only measures unprecedented climate changes well (which might be very well application for the method - detecting unprecedented climate changes). Take an example: If you would use this method to a temperature record covering last 20,000 years, your method would not show for example Little Ice Age or Medieval Warm period or perhaps even the climate change of last decades, but would show a flat line there because cold records would have been set during that last glacial maximum and warm records would have been set during the holocene climate optimum.

I'm not suggesting that this is a bad method, but it just seems to me that one should be very careful when considering where this method applies and where not.

2011-02-11 05:36:32
tobyjoyce

tobyjoyce@eircom...
86.44.235.15

Graphic

Hi, Ari, here is the (Cum Average Max Records - Cum Average Min Records) compared wtih a LOESS smoother and a 5-year running mean. I think the maxmin plot (temporary name, until I think of a better one) does better overall than the LOESS smoother, maybe lacks the granularity of the 5-yr plot. For example, the maxmin curve does not catch the troughs very well, especially 1945 to 1955.

However, I question the "reality" of that trough a bit. It comes as a decline from a very warm period from Oct-1943 to Mar-1945. The anomalies before and after that period are not very different. I know it is usually ascribed to sulphate aerosols. The stability of the temperature prevented any records. On the other had, towards the end of 1950s some substantial maximum monthly records were set.

I think I will use this chart to illustrate similarities/ differences & weaknesses/ strengths - thanks for bring this to my attention.

Note: I think the rates I gave in the last post are off - I mixed up monthly and annual rates in my calculations.

2011-03-04 06:02:26
Shoyemore
Toby Joyce
tobyjoyce@eircom...
86.41.147.143

I will be posting a final draft of this later under by new login name "Shoyemore".

Had to change as a defence against trolls.

Toby

2011-03-04 23:26:18
Shoyemore
Toby Joyce
tobyjoyce@eircom...
86.41.147.143

Maximum and minimum monthly records in global temperature databases

Global temperature data are very “noisy”. There are multiple databases with slightly different observations and different baselines. The usual way to reduce this noise is by “filtering” though a moving average or other smoother to isolate the trend. Here is an alternative way, which looks at a different signal, one which is much less noisy.

A simple count is kept in each database, for each month, marking a 1 if a new temperature record is achieved (compared with the same months in previous years), and a 0 if it not achieved. Then, for each year in the database, there is a running count of the number of new record months in that year. This can be done for both maxima and minima.

To combine the databases, an average is taken. If, for 1960, the GISS database shows 1 new record month,  the NOAA database shows 0, and the HADCRUT database shows 1, it is counted as average = 0.66 for 1960 in the running count of yearly record months. That is because the average cumulative number of new monthly records and its rate of change are of great interest.

We intuitively expect that in a period of warming, the number of new maximum records will exceed minima, and vice versa in a period of cooling. Generally, there should be periods of stasis if there is no trend, and perhaps a sine wave indicating long term natural variation. Figure 1 shows the cumulative annual differences in counts of average new maximum and minimum records in 3 databases (GISS, HADCRUT and NOAA from 1880 to 2010).

 Chart

Figure 1: Cumulative differences in Counts of Average Annual Maximum and Minimum Monthly Records.

Comments:

  • There is an “early measurement effect” because all the first year’s monthly temperature measurements will all be both maximum and minimum records. Subsequent months will modify the records so that it will take a few years for the annual counts to settle down. Since the effect influences both maximum and minimum records, Figure 1 is on the average free of this effect.
  • In Figure 1, the early decades show perhaps a 20-year period of cooling. After 1920, a mid-century warming commences, and this looks like natural variation (a half-sine wave) up to about 1940.
  • Then a period of stasis ensues (for 12 years) until the excess of maximum over minimum records starts again with an accelerating increase up to 2010.
  • Figure 1 resembles charts of the temperature anomaly – but it has a different origin than subtracting the temperature observation from a chosen baseline. It is more “granular” than (for example) a LOESS smoother. However, it misses mid-century cooling, which did not generate any cold monthly records.

 Chart

Figure 2 is a chart of the cumulative rise in new annual maximum monthly records from 1956 onwards. Note is a non-linear, increasing trend – for each 10 year division, more records are occurring.

 

Figure 2: Cumulative Change in Annual Average Maximum Monthly Records since 1956.

Comments:

  • It is possible to fit a function to the curve and use the model to predict the rate of occurrence of future new records.
  • The rates are estimated from the fitted function, for different decades, in new maximum monthly records (r) per year:
    • 1960-1970            0.56r/yr
    • 1970-1980            0.94r/yr
    • 1981-1990            1.27r/yr
    • 1991-2000            1.56r/yr
    • 2001-2010            1.81r/yr
  • To understand the previous table better, in the decade 1960-1970, new maximum monthly records occurred on average about once every 21 months (=12 x 1/0.56). In the decade 2001-2010, they occurred on average every 7 months (=12x1/1.81).
  • Since the incremental increase in temperature for each new record reflects the temperature rise, the average temperature rate can be estimated from the temperature data. Let ∆T=Average Temperature Rise over all maxima. Then Temperature Rate = ∆T x Rate of Occurrence of Records.
  • Plugging in ∆T=0.011C, the following values are estimated for temperature increase in degrees C per decade:
    • 1960-1970            0.07C/decade
    • 1970-1980            0.10C/decade
    • 1981-1990            0.14C/decade
    • 1991-2000            0.17C/decade
    • 2001-2010            0.20C/decade
  • Predictions for the next decade (assuming continuance of current conditions):
    • 2020 Rate = 2.33r/yr
    • 2020 Rate of Temperature Increase = 0.26C/decade
    • The probability of 2011 not having a new record month is 0.09

This basic, and even crude, analysis confirms the model of temperature rise given by mainstream climate science.  That is no surprise. However, it can be expanded. A deeper analysis of the model used here clearly shows effects due to natural variation. Variables for these factors can be incorporated in a more complex regression model, and a comparison made between natural variation in the earlier and later 20th century. It is hoped that this is the first in a series of posts on this topic.

This analysis undermines, yet again, many of the simplistic contrarian models e.g. that natural variability is driving warming, or that the earth has been cooling in the period 1998-2002. As Professor Richard Lindzen said: “Temperature is always rising and falling”. However, that implies an equalization of maximum and minimum monthly records over a long period. The numbers of minimum monthly records in these global temperature databases has not even been close to numbers of monthly maxima for some time.  The last such sequence in these databases ended in 1917, almost one-hundred years ago. The current rate of occurrence of minimum records is 0 per year, and the rate for maximum records is consistently outstripping that of minima by almost 2 per year, and rising.

Further Reading:

How often can we expect a record event? <a href=”http://www.intres.com/articles/cr2003/25/c025p003.pdf”>Benestad(2003)</a>

Record-breaking temperatures reveal a warming climate<a href=”http://arxiv.org/PS_cache/arxiv/pdf/1005/1005.3145v2.pdf”>Wergen(2010)</a>

Detection Probability of Trends in Rare Events: Theory and Application toHeavy Precipitation in the Alpine Region<a href=”http://dubaobien.vn/dhkhtn/stores/files/0907_Tailieu_CuaLo/TLTK/Detection%20Probability%20of%20Trends%20in%20Rare%20Events%20-%20Theory%20and%20Application%20to%20Heavy%20Precipitation%20in%20the%20Alpine%20Region.pdf”>Frei(2001)</a>

 

 

 

2011-03-05 00:51:51
nealjking

nealjking@gmail...
91.33.98.95

Please explain to me:

Who is the intended audience?

What are they supposed to get from this?

What % will get past Figure 1?

 

2011-03-05 02:14:40
Shoyemore
Toby Joyce
tobyjoyce@eircom...
86.41.147.143

Nealjking,

(1) Intended audience: people like me, who are interested in the quantitative measurement of climate change, such as those who follow blogs like Open Mind, or who might like to peek at the Further Reading.

(2) An alternative to plotting temperature records - tidier, less noisy, a methodology that may be useful in the future . "What use is a new born baby?" as Ben Franklin used to say.

(3) See (1). Not sure how many of us anoraks are out there.

2011-03-05 04:42:40
nealjking

nealjking@gmail...
91.33.98.95

Shoyemore,

I think what you have written will not be understood by the great majority of readers of SkS.

I also do not think that the basic thrust of SkS is introducing alternative data-analysis methodologies. If you can boil this down to some specific insight or conclusion, that could be comprehensible. I believe this is not.

2011-03-05 05:42:25
Shoyemore
Toby Joyce
tobyjoyce@eircom...
86.41.147.143

nealjking,

I diagree. Why should readers of SkS expect the same material be presented in the same way all of the time? Sounds pretty boring to me. That is not reason I visit this site. I would prefer to be confronted with new perspectives, even on familiar matters. For example, there is this post on the correlation of Temperature with CO2 I thought was particularly interesting and useful: http://www.skepticalscience.com/Graphs-from-the-Zombie-Wars.html

Personally, I cannot figure out how people can read through yet another exposition of the Hockey Stick controversy and what Mann wrote in an e-mail to Briffa many years ago ... my eyes glaze over at the very mention. But maybe that is just me. :) No doubt you lap up posts like that, so there is always individual difference and individual opinion.

Thanks for the insight, though. There is always room to improve, and I will try to make adjustments. Ultimately, I will leave it up to the John to decide if the post merits publication. If not, it won't be the first or last time I will have been turned down for publication :)

2011-03-05 07:46:02Comment
Robert Way

robert_way19@hotmail...
134.153.163.105

Im personally in favor of this post being published but I would suggest that the language be turned more into a narrative for user friendliness. Though I'm probably not one to talk given most of my posts haha..

2011-03-05 08:15:56
Shoyemore
Toby Joyce
tobyjoyce@eircom...
86.41.147.143

Thanks, Robert .... I do read YOUR posts! More avidly from now on. :)

2011-03-05 09:01:13
Rob Painting
Rob
paintingskeri@vodafone.co...
118.93.202.208

Toby, I'm with Neal on this. There will be only a small percentage of readers that will understand this. Sure it might attract the trolls and generate comments, but it will pass completely over the head of most people. If you could make it somehow more accessible that would be the ticket.  

2011-03-05 19:28:15
Shoyemore
Toby Joyce
tobyjoyce@eircom...
86.41.147.143

Rod,

Ok, but I am puzzled ... a lot of the posts on SkS are quite technical, with some fine points of physics, chemistry and biology discussed. I find a lot of them quite tough going, and some of them just defeat me as I cannot get to grips with them in the time available. However, I would never advise the poster not to bother for that reason. The posts are there as a resource if needed.

I think the post as written is accessible to the average numerate reader. Ok, I can dress it up a bit better, but there is an irreducible minimum technical kernel to every post, and there is not much I can do about that.

2011-03-05 19:41:27
Rob Painting
Rob
paintingskeri@vodafone.co...
118.92.94.236

Tony, I agree many posts here could be made more comprehensible, and not everything can simplified, but shucks yours is really hard going. What is the point of that when we are trying to convince people they need to get off their arses and do something about global warming?. 

2011-03-05 21:56:46
nealjking

nealjking@gmail...
91.33.103.53

There's a quote I often use in this context, from Lord Rutherford (pioneer nuclear physicist): "if you cannot explain, to a barmaid, what you are doing, then you don't understand it."

What is the explanation you would give on this paper to a barmaid? And after she's understood it, why should someone not directly involved in temperature time-series analysis care?

To the extent that I have time to comment on proposed posts, I try to ensure that some reasonable effort is made to address these points. Even if I do not have time to study all the papers in detail, I generally have some idea of why it could be worthwhile for someone to read them at SkS. SkS is a blog for scientific journalism, not an alternative climate-science journal. There has to be a pay-off for the citizen reader, or we are wasting his/her time.

2011-03-05 22:29:01Taking both sides
John Cook

john@skepticalscience...
124.186.229.6
Toby's right in that there'd are many quite advanced, opaque blog posts on SkS.

On the other hand, I think we need to be making more effort toward layperson friendly posts. Not every post but as much as possible. Before you start any post, think about whether it could be written for Treehugger or the Guardian and if there's a possibility, aim for those audiences.

In this case, Toy, I think some fairly plain English description of your methodology immediately after the 1st paragraph would help, before you leap into the methodology. Also, perhaps some explanation before Figure 2 to lead the reader into the pic, explain what they're looking at before they look at it.

2011-03-07 06:38:57
Shoyemore
Toby Joyce
tobyjoyce@eircom...
86.41.147.143

Ok, I hear yiz! :) I'll give in another go.

2011-03-13 02:19:31Maximum and minimum monthly records in global temperature databases
Shoyemore
Toby Joyce
tobyjoyce@eircom...
86.41.147.143

 The worldwide success of books like The Guinness Book of Records is an example of human fascination with record-breaking – the smallest, fastest, farthest, the first etc. The frequency and size of records can tell us something about the underlying process. For example, the world mile record was broken on average once every 3 years between 1913 and 1999, by about 1 second per record. The shift in dominance of middle distance running from the Anglophone countries to Africa has meant the record has only been broken once since 1993.

This post describes a method of recording and graphically presenting successive annual counts of record-breaking months by temperature, e.g. the warmest or the coldest since records were kept, over more than one database. The rate of appearance of record-breaking (warmest or coldest) months intuitively provides a signal of climate change complementary to the usual temperature data. See the “Further Reading” section at the end of the post.

Such data of maximum or minimum records are quite useful as they might, for example, provide evidence of a warming (or cooling) climate, when the temperature data is apparently static over short periods. As we will see, such is what has occurred in the 2000s.

Steps to follow:

(1)    Download monthly climate data into a spreadsheet, either the raw data or the temperature anomaly. For easier manipulation, re-arrange the data with successive years in rows underneath each other, and the months in 12 columns, from January to December.

(2)    Create a second matrix of rank numbers. In Excel, the RANK function will return the ranking of each monthly temperature datum since the first datum was recorded i.e. the top month in the column. Consult the Excel Help to tell you how to use RANK to find the minimum records, which you can do in a separate worksheet. The IF function can be used to set all ranks, other than the one of interest, to 0. Figure 1 shows the result for the first four years, using GISS data for an example.

(3)    In a further column to the right, simply add the number of record months in each year.

(4)    If using more than one database, an average is taken. If, for 1960, the GISS database shows 1 new record month,  the NOAA database shows 0, and the HADCRUT database shows 1, it is counted as average = 0.66 for 1960, and entered into a score of average yearly record months, which you can keep in another column.

(5)    You now have two columns, each of the average maximum and minimum records in each year. You can use two further columns to create running totals of each, and a further column to find the difference between the two running totals.

 Chart

Figure 1: Conversion of GISS temperature anomaly into a binary indicator of maximum monthly records for first four years.

We intuitively expect that, in a period of warming, there should be more maximum monthly records than minimum, and vice versa in a period of cooling. If we assume that the frequency and duration of warming and cooling periods even out in the long run (natural variation), the running totals of maximum and minimum records should be approximately equal. The differences obtained by subtracting one running total from the other should centre on zero like a sine wave. Figure 2 shows the annual differences in cumulative sums of average new maximum and minimum records in 3 databases (GISS, HADCRUT and NOAA from 1880 to 2010).

 Chart

Figure 2: Annual differences in cumulative sums of Average Annual Maximum and Minimum Monthly Records. As an example, in 1911 there was an excess of 30 minimum monthly records over maximum, counting since 1880.

Comments:

  • There is an “early measurement effect” because all the first year’s monthly temperature measurements will all be both maximum and minimum records. Subsequent months will modify the records so that it will take a few years for the annual counts to settle down. Since the effect influences both maximum and minimum records, Figure 2 is, on the average, free of this effect.
  • In Figure 2, the early decades show perhaps a 20-year period of cooling. After 1920, a mid-century warming commences, and this looks like natural variation (a half-sine wave) up to about 1940.
  • Then a period of stasis ensues (for 12 years) until the excess of maximum over minimum records starts again with an accelerating increase up to 2010.
  • Figure 2 resembles charts of the temperature anomaly – but it has a different origin than subtracting the temperature observation from a chosen baseline. It is more “granular” than (for example) a LOESS smoother. However, it misses mid-century cooling, which did not generate any cold monthly records.
  • It is difficult to reconcile Figure 2 with the expectation of a long term average of 0, if the record months are occurring randomly and in equal proportions. The mathematics to prove this is a bit tougher, so we will not go into that level of detail.

 Figure 3 is a chart of the running total of new annual maximum monthly records, starting with the 1955 value set to 0. Note is a non-linear, increasing trend – for each 10 year division, more records are occurring.

chart

Figure 3: Cumulative Change in Annual Average Maximum Monthly Records since 1956. The 1955 value is set = 0.

Comments:

  • It is possible to fit a function to the curve and use the model to predict the rate of occurrence of future new records. The mathematics of the curve fitting will not be described.
  • The rates are estimated from the fitted function, for different decades, in new maximum monthly records (r) per year:
    • 1960-1970            0.56r/yr
    • 1970-1980            0.94r/yr
    • 1981-1990            1.27r/yr
    • 1991-2000            1.56r/yr
    • 2001-2010            1.81r/yr
  • To understand the previous table better, in the decade 1960-1970, new maximum monthly records occurred on average about once every 21 months (=12 x 1/0.56). In the decade 2001-2010, they occurred on average every 7 months (=12x1/1.81).
  • Since the incremental increase in temperature for each new record reflects the temperature rise, the average temperature rate can be estimated from the temperature data. Let ∆T=Average Temperature Rise over all maxima. Then Temperature Rate = ∆T x Rate of Occurrence of Records.
  • Plugging in ∆T=0.011C (estimated from the temperature record), the following values are estimated for temperature increase in degrees C per decade:
    • 1960-1970            0.07C/decade
    • 1970-1980            0.10C/decade
    • 1981-1990            0.14C/decade
    • 1991-2000            0.17C/decade
    • 2001-2010            0.20C/decade
  • Predictions for the next decade (assuming continuance of current conditions):
    • 2020 Rate = 2.33r/yr
    • 2020 Rate of Temperature Increase = 0.26C/decade
    • The probability of 2011 not having a new record month is 0.09

This basic, and even crude, analysis confirms the model of temperature rise given by mainstream climate science.  That is no surprise. However, it can be expanded to incorporate natural variation (factors like ENSO and volcanic eruptions) using methods like logistic regression, which is more robust than ordinary least squares. The advantage of this method is that the mathematics of a noisy temperature process has been replaced by the mathematics of a simple stochastic process. Stochastic processes are well understood and used in many situations like monitoring time between crashes of a computer system (in software reliability engineering) or time between events (in health survival analysis).

This analysis undermines, yet again, many of the simplistic contrarian models e.g. that natural variability is driving warming, or that the earth has been cooling in the period 1998-2002. As Professor Richard Lindzen said: “Temperature is always rising and falling”. However, that implies an equalization of maximum and minimum monthly records over a long period. The numbers of minimum monthly records in these global temperature databases has not even been close to numbers of monthly maxima for some time.  The last such sequence in these databases ended in 1917, almost one-hundred years ago. The current rate of occurrence of minimum records is 0 per year, and the rate for maximum records is consistently outstripping that of minima by almost 2 per year, and rising.

Further Reading:

How often can we expect a record event? <a href=”http://www.intres.com/articles/cr2003/25/c025p003.pdf”>Benestad(2003)</a>

Record-breaking temperatures reveal a warming climate<a href=”http://arxiv.org/PS_cache/arxiv/pdf/1005/1005.3145v2.pdf”>Wergen(2010)</a>

Detection Probability of Trends in Rare Events: Theory and Application to Heavy Precipitation in the Alpine Region<a href=”http://dubaobien.vn/dhkhtn/stores/files/0907_Tailieu_CuaLo/TLTK/Detection%20Probability%20of%20Trends%20in%20Rare%20Events%20-%20Theory%20and%20Application%20to%20Heavy%20Precipitation%20in%20the%20Alpine%20Region.pdf”>Frei(2001)</a>

 

 

 

2011-03-13 07:18:37
Rob Painting
Rob
paintingskeri@vodafone.co...
118.93.202.209

Much better. Funny how a few tweaks here and there make such a difference. The reader knows where you're headed from the get go. Interesting how the warm records were still climbing during the cooler period (1940's to 1960's). We now have 30 years of the MSU satellite record. Have you had a look at that?.

And yes, your post is no more complex than a few others. That's for putting up with the criticism. Take this!. 

2011-03-13 07:32:52
Shoyemore
Toby Joyce
tobyjoyce@eircom...
86.41.147.143

Rob,

Including the satellite data actually gets in a few minimum monthly records in the aftermath of Mount Pinatubo, but it would require a bit more explanation of Fig 3. I deemed it better to focus on the essentials.

And thanks. Constructive criticism is what it is all about .. however much we might hate to hear it sometimes. :)

2011-03-14 09:20:16Thumb from me
John Cook

john@skepticalscience...
124.187.101.78
Like the new intro - eases the reader into the subject like getting into a hot bath. One thing - you need to turn the further reading links into active links - your code is showing. Are you okay to get this into blog post (via Author Admin) or do you need help with that?
2011-03-15 06:03:40
Shoyemore
Toby Joyce
tobyjoyce@eircom...
86.41.147.143

John,

Yes, thanks, I'd appreciate some help. I'll e-mail you.

2011-03-15 10:25:23Published
John Cook

john@skepticalscience...
124.187.101.78

Just went live with the blog post