Double Counting

So I owe you guys a post on how Richard Tol can't do math. Last Sunday, I told you to stay tuned for a post showing it on Thursday. It's now the Monday after. I promise I haven't forgotten. I've just been distracted by some strange problems involving unauthorized attempts to access my bank account.

I hope you'll forgive me for prioritizing sorting out issues of accessing my money over writing blog posts. In the meantime, however, I'll write about a simpler issue related to Tol's inability to do simple analyses to hold you over. It involves a strange problem I discovered when researching his recent use of data upside down.

As you may have read in an article I had published at DeSmog Blog, Richard Tol has had a history of using results from authors who said global warming would be harmful as though they said global warming would be beneficial. While this isn't the only type of error that has consistently cropped up in his work, it is one of the strangest and most disturbing. Tol's most famous conclusion is moderate global warming will be beneficial, and the only way he has found any work other than his own which supports this conclusion has been to invert people's conclusions.

What makes this even more disturbing is article my article was published and attention was drawn to Tol's error, Tol secretly changed his paper to hide the error. There's more to that story which I'll cover in the post I promised you, but I think secretly changing one's work to hide the fact you've made a mistake is a big deal on its own.

It's also something which made me revisit the paper he got those results from. That paper, which he labels "Nordhaus and Yang 1996," is given with the reference:

Nordhaus, W. D., & Yang, Z. 1996. RICE: A Regional Dynamic General Equilibrium Model of Optimal Climate-Change Policy. American Economic Review, 86(4): 741-765.

I knew there was something about this paper which bugged me. Months back, I had spent quite a bit of time checking a bunch of the papers Tol referenced to see if he had used them properly or not. When I did, I found something weird about this one. I even made some notes about it. I don't know where they are though. You see, nobody seemed to care about all the problems in Tol's work or how he managed to sneak it into the latest IPCC report without any external review. Since they didn't, I gave up on researching and writing about it, and when I did, I put my research notes away somewhere and forgot about them.

So given the renewed interest in Tol's work, specifically in this paper, I decided to revisit this paper. I found the copy I had stored on my harddrive, opened it up, and I immediately spotted the problem. Let's see if you see it too:

8_17_Nordhaus_Yang-Table-1

Don't feel bad if you don't. That table is showing the input parameters for a computer model Nordhaus and Yang had created some 20 years ago. The column titled "Climate damage intercept":

equals the reduction in annual net output from an increase of 2.5°C in global mean temperature.

In other words, that column says how much the economy of the listed areas would be damaged by the planet warming by 2.5°C. If you want to know how much the world, as a whole, would be damaged, you would average them together, weighted by some value each area shares (population, GDP, etc.). There's no "right" way to average them, and different papers use different ways. Another issue is different papers use different data sets since the population and GDP of countries in 1996 wasn't the same as they are now.

But those are complexities we don't need to worry about for today. They make comparing results from one paper to another paper difficult, but all that matters for today is what this table says at the very top:

Table 2--Major Input Parameters for the RICE Model

The key word is "Input." Nordhaus and Yang 1996 is a paper about the RICE model, but these damage values aren't output to that model. They're values Nordhaus and Yang plugged into the model. As they explain:

8_17_Nordhaus_Yang-2

So all they really did was use Nordhaus 1994 to get that column of input values for their model. The reference for that Nordhaus 1994 is a book:

8_17_Nordhaus_Yang-3

Which is also referenced in Tol's other papers:

Nordhaus, W. D. 1994b. Managing the Global Commons: The Economics of Climate Change. Cambridge: The MIT Press.

And included in his tables:

table_1_v1

And charts. Which means Tol uses this book as a reference for an estimate for how much damage global warming will do to the economy. Then he turns around and takes a paper which uses that book as a reference for how much damage global warming will do to the economy of various parts of the world and averages those results together, using the result as a separate result.

How does that make any sense? How many times can Nordhaus 1994b be included in Tol's data set? I'd get it if he were just showing the two to highlight any differences there might be due to details in how they were aggregated or whatnot (remember, differences can arise from that), but Tol performed mathematical regressions on his data set. He showed confidence intervals:

figure_1_v2

How can you possibly hope to calculate any sort of uncertainty if you're allowing results to be double counted?

7 comments

  1. Brandon:
    There is no double counting, afaik, as Nordhaus (and others) revises his estimates every so often.

    There is an issue with independence, which is discussed in detail in my recent bootstraps paper in Computational Economics.

  2. Richard Tol, Nordhaus and Yang 1996 provide absolutely no reference for their listed damage values other than Nordhaus 1994. The only way to claim there was no double counting is to say there was no source for them. So if we assume they were some mysterious update, unconnected to Nordhaus 1994, that means you used the numbers without making any effort to verify their validity or source. In that case, your work is based on numbers that could well be pulled out of thin air.

    So tell us, where did the numbers come from? You're telling us they didn't come from Nordhaus 1994. If that's true, where did they come from?

  3. Brandon
    The fact that the numbers in Nordhaus (1994) and Nordhaus and Yang (1996) are different is suggestive of the assertion that they are not the same.

  4. Richard Tol, I didn't dispute your claim the numbers are not taken from Nordhaus 1994 so I have no idea why you commented solely to say they are not. Since you insist on arguing the point though, you aggregated the numbers for Nordhaus and Yang (1996) multiple times, getting three different results in the process. That the numbers you provided for these papers are different tells us nothing. If you want to tell us other numbers, numbers you didn't provide, are different, that might tell us something, but... you should probably provide those numbers then. That would let everyone see you are right about the numbers being different without having to go out and buy a book for $30+.

    Either way though, you need to provide a source for the numbers given in Nordhaus and Yang 1996. If they aren't taken from Nordhaus 1994, then where are they taken from? You don't get to use numbers if they don't have a source. Either they're from Nordhaus 1994 in which case they're being double counted, or they're not from Nordhaus 1994 in which case we have no source for them. Either way, it's a problem you need to address.

    So tell us Richard, where did those numbers come from?

  5. Rivhard Tol, that's not how it works. You don't get to just say, "Those guys used some data so I get to use it too." Every person who uses data is responsible for that data.

  6. Brandon, thanks for your analysis of Tol's work. I was unaware of it until your recent comments over at Lucia's.

    In addition to the serious problems you've linked to, when a man refuses to answer direct questions about his own work, and instead does a fast two-step, his credibility goes in the tank for me.

    I admit I'm late coming to this conclusion, but I've never paid any attention to Tol. My impression of him from afar was that he was seeking publicity rather than knowledge, so I hadn't read any of his stuff.

    Well analyzed,

    w.

Leave a Reply

Your email address will not be published. Required fields are marked *