Mann's Screw Up #1 - "Excelgate"

I want to begin this series of posts with something simple and silly - an issue that isn't really important, but does show how ridiculous Mann's actions can be. What better way than something with the incredibly stupid name, "Excelgate" (name coined by Mann's defender Boris)?

Michael Mann argued in his book (Chapter 8, note #45) the early criticisms raised by Steve McIntyre and Ross McKitrick about the original hockey stick:

were false, resulting from their misunderstanding of the format of a spreadsheet version of the dataset they had specifically requested from my associate, Scott Rutherford. None of the problems they cited were present in the raw, publicly available version of our dataset, which was available at that time at

This claim is similar to what he told the Penn State Inquiry Mann now claims shows he was innocent of the charges leveled against him:

The issue of an “incorrect version” of the data came about because Dr. McIntyre had requested the data (which were already available on the FTP site) in spreadsheet format, and Dr. Rutherford, early on, had unintentionally sent an incorrectly formatted spreadsheet.

You'll note in Mann's book, he claimed his critics misunderstood the format of a file, yet Mann told the Penn State Inquiry the file in question was "incorrectly formatted." It's difficult to imagine why he'd give contradictory stories. It's even worse when one looks into what those stories say.

Interestingly, a colleague and sometimes co-author of Mann's, Tim Osborn, e-mailed Michael Mann (and several others) saying:

The mention of ftp sites and excel files is contradicted by their email record on their website, which shows no mention of excel files (they say an ASCII file was sent) and also no record that they knew the ftp address.

However, that was written years before the Penn State Inquiry and Michael Mann's book existed. A more fair comparison would be between that e-mail and what Michael Mann said around the same time:

The spreadsheet file they used was a complete distortion of the actual Mann et. al. proxy data set, and was essentially useless, particularly in the earlier centuries. The authors had access to the full data, which has been available on a public ftp site for nearly two years. When they noticed, as described in their paper, some signs of problems with the Excel spreadsheet version of the data, one might think that they would have bothered to check the data available on our public ftp site.

In contemporary times, Mann said the "spreadsheet file they used was... essentially useless." In his answers to the Penn State Inquiry he said the file was incorrectly formatted. Both of these contradict his book which blames his critics for misunderstanding a file. Obviously, if a file is "essentially useless," we cannot the fault of people viewing it.

While the inconsistent stories are problematic, what is far more problematic is Michael Mann told a blatant untruth. He claimed, more than once, Steve McIntyre and Ross McKitrick requested the data "in spreadsheet format." As his co-author observes, that contradicts the publicly available correspondence (contained in the last link). That correspondence clearly shows no spreadsheet was ever requested nor sent. In reality, McIntyre requested an FTP site for the data. He got this response from Scott Rutherford:

The proxies aren't actually all in one ftp site (at least not to my knowledge). I can get them together if you give me a few days. Do you want the raw 300+ proxies or the 112 that were used in the MBH98 reconstruction?

This seemingly contradicts Mann's repeated claim the data was already available in an FTP site. It certainly shows if there was an FTP site, it is not McIntyre or McKitrick's fault for being unaware of it. They specifically asked for it and weren't given it. Moreover, Mann's own co-author wasn't even aware of it!

McIntyre e-mailed Mann in November 2003, saying:

you are reported as stating that we requested an Excel file and that you instead directed us to an FTP site for the MBH98 data. You are also reported as saying that despite having pointed us to the FTP site, you and your colleague took trouble to prepare an Excel spreadsheet, but inadvertently introduced some collation errors at that time. In fact, as you no doubt recall, we did not request an Excel spreadsheet, but specifically asked for an FTP location, which you were unable or unwilling to provide. Nor was an Excel spreadsheet ever supplied to us; instead we were given a text file, pcproxy. txt. Nor was this file created in April 2003. After we learned on October 29, 2003 that the pertinent data was reported to be located on your FTP site (and that we were being faulted for not getting it from there), we examined this site and found it contains the exact same file (pcproxy.txt) as the one we received, bearing a date of creation of August 8, 2002. On October 29, 2003, your FTP site also contained the file pcproxy.mat, a Matlab file, the header to which read: MATLAB 5.0 MAT-file, Platform: SOL2, Created on: Thu Aug 8 10:18:19 2002. Both files contain identical data to the file pcproxy.txt emailed to one of us (McIntyre) in April 2003, including all collation errors, fills and other problems identified in MM. It is therefore clear that the file pcproxy.txt as sent to us was not prepared in April 2003 in response to our requests, nor was it prepared as an Excel spreadsheet, but in fact it was prepared many months earlier with Matlab. It is also clear that, had we gone to your FTP site earlier, we would simply have found the same data collation as we received from Scott Rutherford.

In addition to what I've discussed above, this e-mail shows the file McIntyre received was prepared prior to his request, thus it couldn't have been created in response to him requesting it. Moreover, the FTP site where Mann says the data was available contained the exact same material as the supposedly requested file.

McIntyre did not request a spreadsheet. No file was prepared in response to a request from him. The FTP site Mann says McIntyre should have used contained the exact same file McIntyre did use. He told all this to Mann, and Mann responded by forwarding the e-mail to other people (as seen here), leading to this comment by Mann's sometimes co-author Tim Osborn:

I do wish Mike had not rushed around sending out preliminary and incorrect early responses - the waters are really muddied now.

Finally, Mann's claim McIntyre and McKitrick's results were due to their use of a particular file are false on their face. McIntyre and McKitrick closely examined the data errors in the file they received, comparing them in detail to the data provided by original sources listed by Mann et al (no quotes provided due to the length of this post).

Put simply, everything Michael Mann made about Excelgate was false. Not only was it all false, almost all of it was said without any basis. Mann was directly told the things he said were false, at least one of his sometimes co-authors indicated to him they were false, and Mann even expressed his own confusion about the various issues:

Take a look at this. You need to explain to us (don’t email this guy anything!) the various versions of the data. I’m really confused, and we need to know the precise history of when the individual MBH98 records were posted, and when the various matlab format files were posted, and in response to what requests, and these latest changes that were made on Oct 29, 2003??
we really now need to know exactly when the data were made available. They claim that the matrix versions of the data files were posted on the ftp site before their request for the data. I’m really confused by this.

Yet despite all that, Michael Mann has, for years, repeated his false claims about an Excel spreadsheet requested by Steve McIntyre and Ross McKitrick being the reason they got different results. He even told this fabricated story to one of the "investigations" which cleared him, and they made no effort to check his story.

That means when Mann now expects people to take that "investigation" as proof of his innocence, he's basically asking them to believe his story which has no connection to reality. In reality, people should take this as proof that "investigation" was not an investigation in any meaningful sense, and its conclusions are untrustworthy, at best.

Or to put it even more simply, Mann has lied about this issue over and over, for years, despite all evidence showing his stories are untrue and those stories often contradicting each other.

And Mann's defenders are cool with that.


  1. I've edited this post to fix a mistake regarding Tim Osborn. Tim Osborn was not a co-author of the original hockey stick papers (Ray Bradley was, instead). He was a co-author of a different paper at the same time, and he was a co-author of another temperature reconstruction Michael Mann published in 2005. He was also a founding member of and frequent posted at Real Climate along with Mann.

    Sorry for the mistake!

  2. Mann also repeated his claims to a Nature editor (ie. that all the data was always available at the FTP site):

    Note that this was on 20 November 2003 - well after McIntyre had pointed out to Mann in an email on 11 November 2003 that his similar contentions were incorrect.

    In particular, Mann told Nature that: "The have been intentionally misleading about the availability of our proxy data. The data
    have all been available on our public ftp site since July 2002"

    This was after McIntyre had told him: " In fact, as you no doubt recall, we did not request an Excel spreadsheet, but specifically asked for an FTP location, which you were unable or unwilling to provide."

    However, even Mann's July 2002 date is suspicious, given McIntyre's 2005 contention that this archive was: "made public in November 2003"

  3. McIntyre himself gives a neat summary of the situation:
    "In April 2003, I requested the ftp location for the data used in MBH98. Mann first said that he had forgotten where the data was. Subsequently Scott Rutherford identified as the location of the data. The data set called here (presently archived at "") was downloaded in April 2003 from Mann's ftp site from the url "". Correspondence regarding his file is located at Problems with this data set were noticed during the summer and in September 2003, confirmation was sought from Mann that this was the data used in MBH98. Mann said that he was too busy to reply to this question and that von Storch and Zorita had been able to replicate his results. After the publication of MM03, Mann said that this data set was the "wrong" data set. In early November 2003, this data set was deleted from Mann's FTP site. Mann said that the correct data was located in, a directory to which no prior reference had ever been made at Mann's website; considerable controversy ensued, which has been described elsewhere."

  4. Finally, after the Climategate emails came out, McIntyre was able to put some more meat on these particular bones:
    "The Mann et al 1998-99 reconstruction had “steps” (grandiosely called “experiments” by Mann), but the results of the individual steps were never archived, only the splice of 11 steps. For statistical analysis, one needs to have the residuals, which we requested in 2003. Mann refused. At this point in 2003, contrary to North’s allegations, Mann hadn’t provided any source code, any computer programs nor any interpretations of computer programs. All that he had provided us was a URL on his FTP site for the proxy data supposedly used in MBH – a data set that he later claimed was the “wrong” data set and which he deleted in Nov 2003 and a new version of the MBH proxy data was made public. At the point that the residuals were refused, Mann hadn’t spent more than a few minutes responding to requests from us.

    The MBH residual series crop up in the Climategate Letters. In July 2003, Mann had sent CRU the very data that he later refused to provide to us. (BTW Osborn also asked Mann for source code and interpretation of matters that he did not understand.) Mann made it very clear to Osborn that the residual series were provided in total confidence, that they were “dirty laundry” which he did not want to “fall into” the wrong hands."

  5. Yawn. You are aware that it is possible to export a spreadsheet to a 'comma-separated values' (csv) file, which is an ASCII text file, and that such files are sometimes called spreadsheet format files? Indeed, by default Windows will associate csv files with the Excel spreadsheet application if its installed. Bring up what looks like a silly and very old misunderstanding 'in court', and you will look ridiculous, IMO.

  6. Phil Clarke, while anyone is welcome to comment here, it is unlikely you will accomplish anything if you comment in the manner you have above. Nothing you said in your comment addresses anything written in this post. That makes your comment nothing but hand-waving mockery.

    If you want to viewed as anything other than a biased individual trolling to defend Michael Mann, I suggest you try reading and responding to what I've written.

  7. @Phil Clarke - Surely you're playing The Onion, aren't you? An ASCII text file is a spreadsheet format file, only in the same way that "01001100010011110100110000100001" is the same as "LOL!"

    In any event, the file format nonsense is a complete red-herring. Here's what actually happened:

    * McIntyre asked for the proxy data used in MBH98
    * Rutherford provided him with a file of data
    * After analysing it and prior to publishing MM03, McIntyre wrote to Mann requesting confirmation of various things
    * Mann said McIntyre had already been provided with the data and he (Mann) was too busy to respond further
    * MM03 was highly critical of MBH98
    * After some furious activity behind the scenes, documented in the Climategate emails, Mann came to the conclusion that Rutherford had provided McIntyre with the wrong data. [Note that this remains debateable.]
    * Someone deletes the data that Rutherford had provided to McIntyre at the FTP site where the data was originally retrieved.
    * Mann says, "Oh, you used the wrong data, at the wrong site. Here - this is the correct FTP site. You'll find the data there. It's been there for over a year."
    * Mann had never disclosed this site location publically. Not as part of the original MBH98 article, not in MBH99, not in the SI's and not when asked for the information by McIntyre.
    * However, Mann did provide the data at this site to his close colleagues, as the Climategate emails confirm.
    * Despite never making this public, Mann maintained the fiction that this data was always there, at this "public" site from July 2002, even though nobody outside of his email group knew of its existence prior to November 2003 (when McIntyre was probing).
    * After McIntyre pointed all of this out to Mann, an editor from Nature queried Mann about some of the allegations in MM03. Mann brazenly repeated that his data was available at the "public" site and that M&M were "intentionally misleading" the community.
    * Mann didn't point out that he left Rutherford to provide the data; refused to confirm (prior to MM03) that it was the correct data; refused (prior to MM03) to enter into any more correspondence about his data; told his friends where the actual data was located on a public FTP, but refused to do so for McIntyre; subsequently claimed that it was all publically available.
    * When McIntyre finally was told about this very non-public "public" FTP site and looked at the data there, he discovered that the results which were reported in MBH98 were most certainly NOT a simple plotting of the data collected, but the result of many steps, including: excluding certain proxies; weighting some proxies more than others; performing principle components analysis in many different places; and so on.
    * McIntyre asked Mann to disclose what he actually did to come up with his analysis (ie. the famous "Hockeystick" chart).
    * Mann refused, claiming - in bald affront to centuries of scientific endeavour - that his processing steps were confidential.
    * McIntyre pointed out that replication is a fundamental element of science.
    * Mann said that he'd given this to a select few of his mates and that they'd reproduced it, and were happy with it. [See the "dirty laundry" segment, above.]
    * McIntyre complained to Science, Mann's universities and others that making a bold scientific claim, then only releasing your data and methods to a small and confidential group for confirmation, was outrageous and patently against all of the fundamental principles of science.
    * To their great and eternal shame, many said to McIntyre, "Too bad. We like Mann's conclusions and we won't let you look at how he came to them."

    History shows that Mann's Hockeystick has been conclusively debunked, apart from a few hyper-green hippies in the lunatic stone-age fringe. Mann is a man who, having fallen off the Empire State building, is now approaching the ground but still in denial about how inevitable, cataclysmic and final things are about to get. "So far, so good", says Dr Mann.

  8. Anto, that's a pretty good summary, but I think you got a couple details. Two stood out to me: 1) I think you misunderstood what the "dirty laundry" was, 2) You say McIntyre requested the data, but he actually asked where the data could be found. The latter is particularly relevant as he specifically asked for an FTP address.

    If there was an FTP address he should have used, as claimed by Michael Mann, why wasn't he given it when he explicitly asked for it?

  9. The last sentence of the 11th paragraph has a type-o and needs a rewrite:

    "Obviously, if a file is “essentially useless,” we cannot the fault of people viewing it."

    It would make more sense if changed slightly to: "we cannot fault the people viewing it," or "it cannot be the fault of the people viewing it."

  10. Brandon - yes, McIntyre did actually ask where it could be found, as you say.

    Regarding the " dirty laundry", yes you're right. This was a separate request for the residuals, made after MM03 came out. McIntyre was brick-batted on that one (probably not surprisingly, given the ordinance they'd just delivered amidships to MBH).

  11. Brandon, there's a CG2 email on October 19, 2003 (1566) that gave additional evidence against Mann's claim that the MBH98 had been "publicly available" all along. Mann wrote to Jones (and this is his first mention of me):

    This guy "McIntyre" appears to be yet another shill for industry--he appears to be the one who forwarded the the scurrilous "climateskeptic" criticisms of the recent Bradley et al Science paper.

    Here is an email I sent him a few weeks ago in response to an inquiry. It appears, by the way, that he has been trying to break into our machine ("multiproxy"). Obviously, this character is looking for any little thing he can get ahold of.

  12. The other damning thing about excelgate is the claim by Mann that it was use of incorrect data that led to M&M being unable to reproduce his results. We know this is false because M&M did not use the data as indicated. But more importantly we also now know that the main reason for the replication failure was Mann's eccentric implementation of principal components. Mann should have known this in 2003. He certainly implicitly admitted this in the correspondance with Nature which I think was in 2004. And all the arguments about the PC4 are effectively about this issue. The question is: why did Mann not tell M&M this in 2003, when they were trying to find out why they could not replicate the results? Was it because it was a mistake, as M&M originally thought, in which case Mann is just incompetent. Or did he just not realise the consequence of short centering - again indicating incompetence. Or did he know and keep quiet, which looks more like fraud

  13. Incorrectly formatted vs misunderstanding the format are not necessarily contradictory. If I sent you an incorrectly formatted spreadsheet, you will have trouble understanding it.

    It is possible that a spreadsheet can produce a text file, though unlikely to be .txt unless they override the default.

  14. HughMcdonough, I don't believe that's true. There is a difference between not understanding and misunderstanding. If I start speaking sentences made up of nothing but a random string of words, you won't understand me. That wouldn't mean you misunderstood me.

    As for the possibility of a spreadsheet being in a text file, that's certainly true. It doesn't excuse anything though. It hardly matters what format the file sent to Steve McIntyre was in. Mann claims McIntyre asked for the data in a spreadsheet format, and that's why there were problems with the data he used. That has no basis in reality.

  15. Here's another example of Mann's duplicitous behaviour here. First, in an email to Steve McIntyre in April 2003:

    Date: Wed, 09 Apr 2003 05:58:29 -0400
    5225.txt: To: Steve McIntyre
    5225.txt- From: "Michael E. Mann"
    5225.txt- Subject: Re: Proxies in MBH
    5225.txt- Cc: Scott Rutherford
    5225.txt: Dear Mr. McIntyre,
    5225.txt- These data are available on an anonymous ftp site we have set up. I've forgotten the
    5225.txt- exact location, but I've asked my Colleague Dr. Scott Rutherford if he can provide you
    5225.txt- best regards,
    5225.txt- Mike Mann

    The, in October 2003 after MM03 had come out and he was trying to explain to various colleagues his take on the paper, he said the following:

    And so, the authors results are wrong/meaningless/useless. The mistake made insures,
    especially, that the estimates during the 15th and 16th centuries are entirely spurious.
    So whose fault is this? Well, the full, raw ascii proxy data set has been available on
    our anonymous ftp site [1]
    and the authors were informed of this in email correspondence.

    In fact, this was not the case. Rutherford referred a different site, with different data, to McIntyre.

    Steve, The proxies aren't actually all in one ftp site (at least not to my knowledge). I can get them together if you give me a few days. Do you want the raw 300+ proxies or the 112 that were used in the MBH98 reconstruction? Scott,d.dGI

    Incidentally, the link above is to a ppt which Steve gave, which is pertinent to the present issues.

  16. Just to show how two-faced Mann was, here's what he said to some other climate scientists in October 2003:

    uncovered the error in what they did. They didn't use the proxy data available on our
    public ftp site, which I had pointed them too

    So, within a brief period of time, it's gone from an "anonymous" ftp site, to a "public" ftp. He has also magically "pointed them to" this site, when he actually never did - instead telling McIntyre back in April that he couldn't remember its address.

  17. A couple of weeks later, and Mann is trying to blame Rutherford for the snafu:

    Take a look at this. You need to explain to us (don't email this guy anything!) the
    various versions of the data. I'm really confused, and we need to know the precise history
    of when the individual MBH98 records were posted, and when the various matlab format files
    were posted, and in response to what requests, and these latest changes that were made on
    Oct 29, 2003??
    Obviously, we don't need to provide these guys with *anything* and we needn't respond to
    any of their emails--the raw data are available on the ftp sites, and have been for some
    time. But we really now need to know exactly when the data were made available. They claim
    that the matrix versions of the data files were posted on the ftp site before their request
    for the data. I'm really confused by this.
    You need to draft a clear explanation of all of this, so we can provide this to people. Can
    you draft an explanation of what was posted when for our internal purposes, and then we can
    decide what information to send on...

  18. By this time, Mann was really irritating his colleagues. This is from Tim Osburn [not "Cook", as incorrectly stated above]:

    Keith and Phil,

    you will have seen Stephen McIntyre's request to us. We need to talk about
    it, though my initial feeling is that we should turn it down (with
    carefully worded/explained reason) as another interrim stage and prefer to
    make our input at the peer-review stage.

    In the meantime, here is an email (copied below) to Mike Mann from
    McIntyre, requesting data and programs (and making other criticisms). I do
    wish Mike had not rushed around sending out preliminary and incorrect early
    responses - the waters are really muddied now. He would have done better
    to have taken things slowly and worked out a final response before
    publicising this stuff. Excel files, other files being created early or
    now deleted is really confusing things!

    [NB. None of my posts above on this thread are really related to Steyn's lawsuit regarding Mann. However, they are amusing and provide context to the situation, IMO.]

  19. Ironically, two years later in 2005, here are the lesson which Phil Jones and Michael Mann had learned from this affair:

    I presume congratulations are in order - so congrats etc !
    Just sent loads of station data to Scott. Make sure he documents everything better
    this time ! And don't leave stuff lying around on ftp sites - you never know who is
    trawling them. The two MMs have been after the CRU station data for years.
    If they ever hear there is a Freedom of Information Act now in the UK, I think I'll
    delete the file rather than send to anyone.

    And Mann's response?

    Thanks Phil,
    Yes, we've learned out lesson about FTP. We're going to be very careful in the future
    what gets put there. Scott really screwed up big time when he established that directory
    so that Tim could access the data.

  20. "Scott really screwed up big time when he established that directory so that Tim could access the data."

    At present, Mann has never explained why this was a "big time" "screw up". I don't know precisely what Mann had in mind.

  21. "...everything Michael Mann made...". I suggest either "...every statement MM made..." or "... everything MM said...".

  22. "Scott Rutherford was one of Michael Mann’s co-authors on the original hockey stick.". Predating MBH98?

  23. omnologos, not inherently, but there are many reasons it could be. For example, if people call Mann dishonest because that's how they view Mann's character, they'll certainly be allowed to try to justify their view if Mann sues them for defamation.

    Coldish, that was supposed to be said, not made. Thanks for catching that. I'll fix it. And I swear, I have no idea why I can't remember Malcolm Huges was the third author. I never forget Ray Bradley or Michael Mann, but for some reason I keep thinking Tim Osborn or Scott Rutherford was the third (they both were co-authors with Michael Mann on later temperature reconstructions he published).

  24. What is intersting is that a judge declared in a rulling that Mann has been cleared by the various investigations.
    Will the judge now allow evidence that calls that ruling into doubt?This case may already be 'settled justice'.
    The illness that has allowed AGW to fester into the nasty money hog it has become is not easily dealt with.

  25. “Scott really screwed up big time when he established that directory so that Tim could access the data.”

    Having raised two boys, I've developed a sixth sense for fibs. There's always something about the construct of a fib that gives it away. Had Mann said, "Scott really screwed up big time when he established that directory" I would accept it as just Mann trying to throw Scott under the bus. But its the second bit that raises my curiosity: “ that Tim could access the data.” That seems like a strange thing to append to the end of a sentence like this. It would be a bit like one of my son's saying "I noticed that we were out of milk, when I got home at 11:35 last night." My radar goes off and I think "hmm, I wonder if he is trying to hide the fact that he actually got home after 12 (curfew) last night"?

    Mann feels the need to explain why they established the directory. It could be that its just Mann's extraordinarily thin skin leading him to provide an explanation. But it seems odd that Rutherford would have created the directory and then forgotten that it existed. So I'm suspicious of Mann's claim that Rutherford set up the directory for Osborne. I think there are real questions about the provenance of that directory and who set it up and for what purpose. Do we know for a fact when this directory was created and when the file in question was added to the directory? Could the file have been added by Mann after the kerfuffle broke out?

  26. I think that a directory was probably set up for Osborn in a protected part of Mann's FTP site, inaccessible to anyone unless that they had exact URLs for files and subdirectories. I trawled Mann's FTP site at the time looking for stuff: Mann accused me of trying to "break into" his FTP site in a Climategate email.

    After publication of MM2003, it is my surmise that Mann either moved the directory or changed the protection and gave out a URL that had never previously been cited or listed on his webpage. He did so very quickly and the directories probably contained material that he would didn't want to make public e.g. the CENSORED directories and/or the code for the principal components calculations showing the short censoring.

    Moving a directory from private to public areas on Mann's FTP site would not change its date: I observed this in connection with directories used in the Corrigendum.

  27. Brandon:
    After years of preparing financial analysis for others, actually I learned it from the first analysis I submitted, Don't add in decisions (false, lie, etc.) or the reader focuses on the decision and not the detail.

    Instead clearly lay out the particulars. e.g. Rather than call something false, point out that neither language nor meaning matches.

    Also try to refrain from long explanations or extraneous language, keep to the details!

    Terrific series of devastating analysis!

  28. "anonymous ftp" and "public ftp" are synonymous. it means anyone can log in using the username "anonymous." if you are so easily confused on simple technical issues, it seems unlikely you've understood the broader situation.

  29. If you're going to criticize someone, you should at least say who you're criticizing. I have no idea who you're talking to. I never said a word about anonymous FTP sites. The word anonymous has barely been used on this page.

    And not that it really matters, but you're wrong anonymous and public are synonymous in this case. While there is a lot of overlap, you can have a private, anonymous FTP site.

Leave a Reply

Your email address will not be published. Required fields are marked *