I've previously documented changes in the Intergovernmental Panel on Climate Change (IPCC) Fifth Assessment Report (AR5) which were made absent any external review. This happened despite the IPCC principles stating:
Three principles governing the review should be borne in mind. First, the best possible scientific and technical advice should be included so that the IPCC Reports represent the latest scientific, technical and socio-economic findings and are as comprehensive as possible. Secondly, a wide circulation process, ensuring representation of independent experts (i.e. experts not involved in the preparation of that particular chapter) from developing and developed countries and countries with economies in transition should aim to involve as many experts as possible in the IPCC process. Thirdly, the review process should be objective, open and transparent.
Working Group/TFI Co-chairs should arrange a comprehensive review of reports in each review phase, seeking to ensure complete coverage of all content.
It is difficult to see how a review process can be "open and transparent" if significant changes are made absent any review. It is difficult to see how a review process can "ensure complete coverage of all content" when entire sections are (re)written absent any review.
I'm not going to dwell on that today though. The changes I discussed before were between various drafts of the IPCC report. Today, I'm going to discuss changes made between the "Final Draft" and "final version" of the IPCC WGII report. The Final Draft was released to the public on March 31st with this word of caution:
The Final Draft Report has to be read in conjunction with the document entitled “Climate Change 2014: Impacts, Adaptation, and Vulnerability. Working Group II Contribution to the IPCC 5th Assessment Report — Changes to the Underlying Scientific/Technical Assessment” to ensure consistency with the approved Summary for Policymakers (IPCC-XXXVIII/DOC.3) presented to the Panel at its 38th Session. This document lists the changes necessary to ensure consistency between the full Report and the Summary for Policymakers, which was approved line-by-line by Working Group II and accepted by the Panel at the above-mentioned Sessions. A listing of substantive edits additionally indicates corrections of errors for the Final Draft Report.
But has been widely treated as being the "IPCC Report." The actual final version was only published two days ago, on October 15th. It has changes as indicated would be made in that disclaimer. The list of "substantive edits" is available here. An errata listing several minor changes is available here. A list of changes necessary for consistency with the Summary for Policymakers (SPM) is available here. That these documents are made publicly available suggests a genuine interest in transparency. That suggestions is completely undermined, however, by the multitude of changes the documents don't mention.
Consider, for instance, Figure 10-1. Here is the previous version (left) and the new version (right):
That it was redrawn to look better is fine. The problem is the data represented in the two figures are not the same. The diamond (representing an estimate published in the last five years) between 2 and 2.5 degrees has shifted almost to 3 degrees. Two points previously at 2.5 degrees have shifted left to 2.2 degrees. A new diamond was added at 3 degrees. A circle at three degrees and about 5%, which previously stood out, has vanished. The diamond at the far right side has shifted even farther right, going from just under five to just to under 5.5 while also dropping quite a bit. The diamond at the bottom has fallen as well, being lower by nearly one full point.
Not a single one of these changes was disclosed. We can verify them, however, by examining the tables provided for both figures. I've previously displayed the table for the first figure:
Here is the new version of the table (found in the IPCC Supplementary Material):
Every difference I highlighted in the figures can be confirmed in these tables. More differences can be found as well. For instance, we can see the estimate from Nordhaus 1994a was changed from -4.8 (-30.0 to 0.0) to -1.9 (median), -3.6 (mean) [-21 to 0.0]. This change is neither disclosed nor explained. Also unexplained is why only both the median and mean values are shown yet only the median value is displayed in the graph.
A partial explanation is available if we examine Nordhaus 1994a. It turns out the original value was taken from the wrong figure in the paper. The earlier value was the mean "likelihood of a high-consequence event" instead of the "impact on global output." The change fixes an obvious error. What isn't obvious, however, is why the table previously listed only the mean value while the new one lists both the mean and median. Also unexplained is why now only the median value is plotted while previously the mean value plotted.
Another difference can be found in the entry for Hope 2006. This entry has some history to it as the creator of the table previously published its estimate as +0.9 instead of -0.9. That error wasn't present in either of these versions of the table though. Instead, a different error was published. The range given for the paper was originally listed as -0.2 to 2.7, a range which excludes the -0.9 estimate listed. This error was one of the few listed in any of the IPCC documents. Its list of substantive edits says:
In cell intersection Hope (2006a) row and Impact (% GDP) column, change ‘-0.2 to 2.7’ to ‘-2.7 to 0.2’
I'd have no problem with this change as it is clearly an error and the error is disclosed, save for one thing. The range given in the final version is not -2.7 to 0.2 as specified. Instead, it's somehow changed to -2.7 to 0.0. This is despite the fact both entries have a footnote which says, "Results aggregated by Tol (2013)."
How can there be two different ranges given when both are supposedly taken from the same paper? I don't know. No explanation is given. When we look at the latest version of that paper, a version published to correct and update the table now used in the IPCC report, we see:
The range for Hope 2006 is listed as -2.7 to 0.2. The new version of the IPCC table does not match the published values of the paper it cites. It's interesting to examine the Nordhaus 1994a entry for this table as well. Unlike the new IPCC table, it only lists the mean value, -3.6, which was not plotted. The median value the IPCC report plots, -1.9, is not listed.
Another issue is the two Mendelsohn entries previously had the footnote saying their "Results [were] aggregated by (Tol 2013)." That footnote has inexplicably vanished. It is difficult to explain this change, along with the fact their degree value changed from 2.5 to 2.2. We can try to figure out what happened by examining the papers as we did with Nordhaus 1994a, but that's complicated by fact the reference section shows:
Only one Mendelsohn 2000 paper is listed. Why are there two entries but only one paper listed? It's hard to say. Examining the paper the IPCC report cites doesn't help. It doesn't provide either estimate listed. It doesn't even examine a scenario with 2.5 or 2.2 degrees of warming. It examines a different scenario:
Country-specific results indicate that the 2°C global-mean warming projected for 2060 will result in net market benefits for most OECD countries and net market damages for most non-OECD countries.
All hope is not lost, however. There is a different, uncited, Mendelsohn 2000 paper titled "Comparing impacts across climate models." This paper says it examines a scenario with a climate sensitivity of 2.5C and a predicted change of 2.21C:
Third, we specify a global temperature sensitivity of 2.5C. The model predicts a global-average temperature of 2.21C by 2100.
It seems plausible the original version mixed up these two values. This is especially true since the paper says:
The market impacts predicted in this analysis do not exceed 0.1% of global GDP and are likely to be smaller.
This does not provide the two values listed in the IPCC table, but it shows the paper's results are in line with what that table shows. Also, the paper does use two different methodologies which could perhaps explain how two different answers were obtained.
It does not, however, explain why the two estimates would be cited as two different papers. Nor does it explain why one estimate would be described as generated via "Enumeration" while the other is generated via a "Statistical" method. Nor does it explain why one is listed as using "Agriculture, forestry, sea level rise, energy demand, water resources" data while the other is listed as using only "Agriculture, forestry energy demand data." And of course, it doesn't explain why values from it would be listed while it isn't cited, yet values from another paper which was cited aren't listed.
But perhaps the most troubling aspect is we still don't have an explanation for where the numbers listed came from. The table published in the IPCC report inexplicably dropped the note the results were aggregated by Tol 2013. That's true even though the same note has accompanied this table time after time in the past. For instance, it is noted in the same table of Tol 2012. Interestingly, that table has an entry for only one Mendelsohn 2000 paper, and its description matches the paper I suggested the values came from. Also interesting is the fact Tol 2012 doesn't actually cite any Mendelsohn 2000 paper.
In any event, the issue of where the numbers come from goes back to the Hope 2006 entry whose given range inexplicably changed. You see, Richard Tol had never published the calculations which went into his "aggregations," even after being asked to repeatedly. That means nobody knew how he generated the "estimates" from those papers. They weren't taken directly from the papers he cited, but rather, were calculated in some undisclosed manner. Tol's reaction to people trying to check his work was to say things like:
Finally, Professor Abraham repeats Mr Ward's lament about lack of transparency. In fact, all data are in the public domain. It does take a bit of multiplication, addition and division to go from disaggregate data to aggregate data, but that should not be beyond someone with a bachelor's degree in geology.
Which is obviously not how a scientist should behave. It's also strange to hear him insist the process is so easy when he produces inconsistent results with it. You see, Richard Tol is the person responsible for this section of the IPCC report, the one heavily relying upon his own work. And you see, the IPCC reacted to pressure about Tol's refusal to say how he got his results by forcing him to upload the calculations. There are a number of issues with those calculations, including the fact several of them don't produce the results Tol mocked people for being unable to replicate, but the most important aspect at the moment is they don't include any calculations for the Mendelsohn papers. Because the note saying their results were aggregated by Tol 2013 was dropped, Tol managed to avoid having to upload the calculations for them.
Does it matter? I don't know. What I know is the lack of explanation for any of this is quite troubling. Similarly, the extreme estimate at the bottom of the figure changed because the Maddison and Rehdanz entry in the tables changed. Originally it was -11.5%, but that inexplicably changed to -12.4%. Again, both of these results were attributed to the aggregation done in Tol 2013, the aggregation Richard Tol refused to document. Now that we have documentation, it doesn't match his previous answers.
Whatever. I'll move onto another example. The table shows two diamonds which changed locations belong to the Roson and van der Mensbrugghe 2012 paper. The previously listed results were -1.8 for 2.3C and -4.6 for 4.9C. The paper says:
According to our preliminary estimates, at the global level, the most serious consequence from climate change will be changes to labor productivity that would induce 84% of the global damage in 2050 (-1.8% of global GDP) and 76% in 2100 (-4.6% of global GDP).
It would appear the IPCC report changed 1.8 to 2.1 because 1.8 is 84% of 2.1. Similarly, it would appear 4.6 was changed to 6.1 because 4.6 is 76% of 6.1. This is wrong. The values in parentheses are the total damage, not the damage caused solely by changes to labor productivity. This can be easily confirmed by examining another paper by the same authors. It says:
At the global level when all of the climate change impacts are included, global output is some 1.8 percent less than in the standard baseline in the year 2050, i.e. the rise in mean temperature of 2.3º C induces an overall economic decline of 1.8 percent
Pushing the simulation out to 2100 leads to an overall loss in 2100 of 4.6 percent, with the labor productivity impact still dominating the overall losses.
It appears the undisclosed changes to the percent values for this paper were made to fix a problem which didn't actually exist. I'm not sure why the temperature values were changed though. The original temperature values had been 2.3 and 4.9. These were changed to 2.9 and 5.4. This is strange as a figure in that paper shows those values are not correct:
In fact, the paper explicitly states the one value:
The global average temperature is found to change in the 21st century by 4.87 °C, reduced to 4.79°C when climate impact feedbacks are taken into account
While the other paper I mentioned lists the other value in its Table 2 (as 2.34). That the values were changed to be something completely different when its clear what values the authors of the paper used is inexplicable.
Also inexplicable is why the figure shows five diamonds when there are only four new entries. The text of the report makes this perfectly clear. The legend for the figure says diamonds are for estimates published after the last IPCC report (AR4). The text says:
Since AR4, four new estimates of the global aggregate impact on human welfare of moderate climate change were published (Maddison and Rehdanz, 2011; Bosello et al., 2012; Roson and van der Mensbrugghe, 2012), including two estimates for warming greater than 3°C. Estimates agree on the size of the impact (small relative to economic growth), and 17 of the 20 impact estimates shown in Figure 10-1 are negative. Losses accelerate with greater warming, and estimates diverge. The new estimates have slightly widened the uncertainty about the economic impacts of climate.
If there are "four new estimates," there should be four diamonds. Instead, there are five. By eyeballing the figure and looking at the table, I'd say the Nordhaus 2008 estimate (which inexplicably changed from -2.5 to -2.6) is being listed as "published after IPCC AR4," but I have no idea why that would be the case. What I also don't know is why the text for this paragraph has changed. Previously, it said:
Since AR4, four new estimates of the global aggregate impact on human welfare of moderate climate change were published (Bosello et al., 2012; Maddison and Rehdanz, 2011; Roson and van der Mensbrugghe, 2012), including two estimates for warming greater than 3°C. Estimates agree on the size of the impact (small relative to economic growth) but disagree on the sign (Figure 10-1). Climate change may be beneficial for moderate climate change but turn negative for greater warming. Impacts worsen for larger warming, and estimates diverge. The new estimates have slightly widened the uncertainty about the economic impacts of climate.
Notice how the old version implied the "four new estimates" disagree on the sign of the impact. The new version improves this by making it clear it is discussing all 20 estimates, not just the four new ones. Additionally, it is good the new version removes the claim climate change "may be beneficial for moderate climate change but turn negative for greater warming" as there was only one data point which suggested there might be any sort of (net) meaningful benefits from global warming. These change are improvements. That just does not excuse the IPCC's failure to disclose them.
I don't claim this discussion has been exhaustive. There are likely other examples of undisclosed changes to be found. There is certainly more to be said about this one section. For now though, I'm content with what I've shown here. The Final Draft of the IPCC WGII Report has been out for nearly seven months. It has been used by the public and media as "the IPCC Report." And yet, we now see substantial changes were made to it before the official final version was published, changes made without any disclosure, explanation or transparency. To add to the strangeness, we see such changes were made to a section, figure and table which were never reviewed in the first place as they were added to the IPCC report only after the last round of reviews.
One could easily believe the apparent need for inexplicable and undisclosed changes wouldn't be necessary if the IPCC would bother to have "a comprehensive review of [its] reports" where "experts not involved in the preparation of" material actually reviewed what went into the report.
But that might require a review process which is "objective, open and transparent."