Richard Tol's Peculiar Argument, Revisited

Richard Tol posted a comment yesterday which was beyond amazing:

Brandon: I’m not gonna go over this again. I thought for a bit that Izuru was someone else. Anyway, the onus is on you to show that a random reordering of a random draw should show a pattern.

This, of course, was in reference to the post I made about his criticisms of the Cook et al paper. Basically, he criticized that paper by saying he found patterns in its data, and that indicates a problem. This is based on the idea a "random reordering of a random draw should" not show a pattern. He freely admits a non-random reordering, that is a sorted draw, can be expected to show a pattern:

That means his criticism rests entirely upon the idea the data he looked at was non-sorted, an idea he himself has shown is false by saying:

On homoskedasticity, the Web of Science presents data in an order that is independent of its contents, namely the date of publication. Cook then randomized the order again, but presents data in the original order.

It turns out his description wasn't accurate. The data is actually sorted by publication year (not date) and then alphabetical order. Still, this shows Tol was perfectly aware the data was sorted. One would think he'd thus be aware we should be able to expect patterns in it. Instead, he insists we shouldn't.

How does he do it? I don't know. I may have a bit of a hint though. I expressed my disbelief about this on Twitter, and during one exchange I got this response:

The person who made it, Shub Niggurath, claimed sorting papers by year published then alphabetical order is performing a random reordering. It seems that must be Richard Tol's view as well. He must somehow feel sorting data is randomly reordering it.

I'd say that's pretty amazing.


  1. Just the sorting issue then...ok.

    Sorting by year or alphabet is sorted no doubt, but it is random with respect to the sequence in which the abstracts were rated. To perform analysis of classifier drift and fatigue, the required piece of data are the timestamps of the individual volunteers. Cook has refused to part with this data.

  2. Exactly. Richard Tol was looking at a sorted data set and drawing conclusions about a randomized form of it. His analysis could never do what he claimed it did. The tests were completely misguided, and the conclusions didn't follow from his results. That's why I kept saying it was obvious nonsense.

    There may be any number of legitimate criticisms one could raise against the Cook et al paper. There may be any number of analyses worth doing. That doesn't change the fact Richard Tol's analysis was nonsensical. It also doesn't change the fact hundreds (thousands?) of people accepted it anyway, even promoting it in media articles. It also doesn't change the fact those people are, ostensibly, skeptics.

    That's why I said maybe my problem is with the human race. The ability to be taken seriously, no matter how wrong one may be, seems to exist on all sides.

  3. No, we don't agree data sorted by year is random. I've never said that. Data sorted by year is certainly not random.

    The only way it is random is in relation to the order the raters rated the data in. The reason it is random in relation to that is because the orders rated the data in a random order. Anything in relation to a random order will be random.

    For the record, declaring someone agrees with you when they've never said they do is a terrible way to respond. It's rude, presumptive, and almost always false.

  4. The data is randomly ordered in relation to the random ordering. That is very different than being randomly ordered.

    Were the data itself randomly ordered, we'd expect not to find patterns related to its order. In that case, the fact Richard Tol found patterns related to the data's order would indicate a serious problem.

    The distinction between being randomly ordered and randomly ordered in relation to the raters' order is vital to the point of this post.

  5. I didn't say I explained the point in this post as I didn't explain it in this post. I didn't think people would need me to tell them the difference between "random data" and "data compared to random data." It seemed one of those things everyone would just get.

    As for where you've made a different point, quite frankly, I have no idea what point you are making. You don't seem to be trying very hard to make it. Your comments are mostly just a couple sentences with no clear meaning, forcing me to try to guess what you might be trying to get at. I'm responding to what you say as best I can, but I'm no mind-reader.

    If you want people to understand your thoughts, you ought to try laying them out.

  6. Good. If you carefully read my tweets and comments, you'll realize that you made up the part you are criticizing me for, and we otherwise agree with each other.

    The real question is, does sorting by year behave like a random assortment (which it is, w.r.t the original order in which abstracts were rated), or, does it display legitimate time trends (which are interpreted as evidence for classifier drift by Tol) and therefore non-random. Am I getting this correct? This is a/the point you have not made above.

    You pointed out in Lucia's blog, if I remember correctly, that numerically, there are more '3's and '4's as you move from 1991 toward 2011. A rolling skewness will necessarily show this, and would/could be legitimate, but appear as evidence for drift under Tol's analysis which would then be wrong. Am I correct in saying this?

    In other words, abstracts sorted by time would display trends that are real and present in the literature as it changed over the years, and therefore cannot be considered random. This is your point, right.

  7. That's not the only point, but yes, it is definitely one of them. It's an especially relevant point as the authors of Cook et al highlighted the change in ratings over time. That is, they published information which would make us expect to see a drift in the data, but somehow Richard Tol took that drift as a sign of a problem.

    The other primary point is data within each year was sorted by alphabetical order. That is certainly not random. A humorous demonstration of this point is a series of three papers that had the exact same name. All three were grouped together within the sorted data. Clearly, we should expect to see patterns if papers with the similar or identical names get grouped together.