A $100,000 Scam

No, this is not a duplicate post. There's no error. My last post was titled, "A $100,000 Scam?" because it was asking the question, is Douglas Keenan's $100,000 contest a scam? Today's post doesn't have the question mark because I'm answering that question with a resounding YES.

Douglas Keenan is a lying, law-breaking cheat who is scamming people out of money.

That might sound a bit harsh, but it's actually rather tame given the situation. You can read about the circumstantial issues I discussed in the last post if you want, but none of them really matter anymore. You see, Keenan just broke the law in order to ensure he wouldn't have to pay out the $100,000. Anything else just seems irrelevant compared to that.

To explain what I'm talking about, Douglas Keenan recently updated the contest page with a couple notes. Amongst them was this text:

The Contest was announced on 18 November 2015. Shortly afterward, a few people pointed out to me that the PRNG I had used might not be good enough. In particular, it might be possible for researchers to win the Contest by exploiting weaknesses in the PRNG. I have been persuaded that the risk might be greater than I had previously realized.

The purpose of the Contest is to test researchers' claimed capability to statistically analyze climatic data. If someone were to win the Contest by exploiting a PRNG weakness, that would not conform with the purpose of the Contest. Ergo, I regenerated the 1000 series using a stronger PRNG, together with some related changes.

The 1000 regenerated series were posted online four days after the Contest was announced—on 22 November 2015. Each person who submitted an entry before then has been invited to submit a new entry with no fee. Everyone who plans to enter the Contest should ensure that they have the regenerated series.

That weaknesses in some random number generator he used might allow people to solve his challenge in an unattended way is a problem that might reasonably need to be addressed. Keenan claims this problem was brought to his attention by a number of people, such as this person. That person had said:

The Encryption Key

If Doug is going to lose money, this is where he is vulnerable. The key encoded answer:
...This is 500 characters long and consists of characters a-zA-Z0-9/+ Thus in total 64, so, this is a 64bit number. The first thing I noticed was the unusually high number of of pairs:

99…cc…ll…ooIl…AAl449xrr…99…UU…uu…ZZXcAgg…77…CC…

There are 15 in 500 characters, but there should be only 500/64 = 7.8. Therefore there are twice as many repeating characters as should occur by chance. This means that the 64 bit number we see is not random! This reminds me of the way the German enigma machine was broken (where characters were repeated). Thus it appears that whatever way Doug Keenan has chosen to encode the text, it is not random and thus (unlike the first challenge), it may be broken. Also, unlike the first challenge, the “code” must make sense to a human, so we will be able to check if the decoding is valid because the answer must make sense. This is in sharp contrast to the first part of the test, where there is no way to validate whether the method we chose is valid except by paying this $10 and asking.

This is a discussion of Keenan's choice of encryption method. Keenan had released an encrypted answer file, and if that file could be decrypted by someone, they could "cheat" to win the challenge. That's a reasonable concern and generating a new set of test series with a new answer file using a better encryption might be reasonable.

But here's the thing. Changing how you encrypt your answer file should never change the nature of the test series created for the challenge. The exact details of the 1,000 series would be different since a new set would have to be created again, but that set should have all the same properties as the old one. Any analysis which worked on the old data set should also work on the new one.

That's not the case. You see, Keenan made "related changes" which he doesn't disclose. Those changes significantly alter the statistical properties of the test series. This is clear to even the briefest of examinations. For instance, one thing many people thought to do for this challenge was to plot a histogram of the trends of the series in Keenan's data set. This is one a user named Magma posted over at the blogger Anders's place:

11_22-Magma-1

Now, I'm using his graph because I didn't think to save a copy of the original data file so I can't create one of my own. I have some of the results from tests I ran, but unfortunately, I can't show a version I prefer. When I was working on this, I took the absolute value of all the trends and effectively "folded" the graph over. It made things simpler, and it shouldn't make a difference since there is no difference between positive and negative trends for this challenge.

That's a non-issue though. I'm really only bringing it up because I want to emphasize the fact Keenan removed the old data set and rendered it unavailable. He didn't just move it somewhere and post a link. He didn't leave an archived copy. He just got rid of it. That's not right. It is, however, necessary for him to be able to try to hide the fact the new data set has a significantly different distribution than the old one. Here is Magma's graph for the new data set:

11_22-Magma-2

As you can see, the distribution has been significantly flattened. Before, there were three clear peaks, each representing a different subset of the data set. Now, two of the peaks aren't visible. If you didn't know to look for them, you might not even guess they were there.

Of course, Keenan's challenge requires people identify which series belong to the subsets corresponding to those peaks (with a 90% accuracy rate). Before, that might have been possible. Now? It's probably not. By flattening out the distribution of the trends in his test series, Keenan has made his challenge significantly more difficult. And he only did this after people paid him money to try their hand at the easier version.

That's horribly wrong. It's completely dishonest. You don't pose a challenge, saying you'll pay $100,000 to anyone who can complete it, then wait around and watch how people try to solve it so you can use that information to change the challenge to ensure nobody can solve it. You don't take people's money then tell them, "Sorry, the easy challenge you signed up for is gone now; you have to do this much harder one if you want the prize money." And you certainly don't say, "No refunds."

But that's exactly what Keenan did. Keenan waited days, letting people work on his challenge and discuss it publicly. He let people submit answers to his challenge, paying him money for the opportunity. He then turned around and changed his challenge to make it much more difficult. And he didn't even offer to give people a refund if they signed up for the original challenge; he just said they could get a free entry to the new one.

That's not just wrong; that's illegal. When a person pays money to sign up for a contest, they form a legal contract with the entity running the contest. If the entity running the contest changes the contest in a way which substantially alters it, such as by making it far more difficult as Keenan has done, they have breached the contract. That means they have no legal right to the entry fees any longer. Saying people can get a free entry into the new contest without offering them a refund is committing theft.

And it gets worse. Because Keenan made undisclosed changes to his data set, nobody can possibly know if their earlier attempts at this challenge worked. That means, for all anyone knows, they might have actually completed the challenge already. Somebody out there may well have already earned the $100,000. Keenan could have easily gotten an entry, saw it was a winner and freaked out, realized he was in big trouble and decided to change the data set to avoid having to pay out the $100,000.

So yeah, Douglas Keenan is a lying, law-breaking cheat who is scamming people out of money. Anyone who has promoted this "challenge" should be embarrassed and should promptly and publicly speak out about his dishonesty.

18 comments

  1. I didn't say it when i commented on Bishop Hill but at the time I also thought the RNG seed that created the 1,000 series was an attack vector, and now he's changed the data series after it was pointed out.

    Other have focused on answers.txt which is understandable given the small size of the encrypted data.

    Looking at his new 1,000 series and one thing immediately jumps out, every single series starts with -0.23. Unless he fed that in as his starting position (why?) his "stronger PRNG, together with some related changes" always comes up with -0.23 first.

  2. I have his old set if you want a copy.

    FWIW Every series originally started with -0.229.

    FWIW2 I think you're probably jumping to conclusions about the change in distribution of trend, seeing patterns that you want to see. Overlay the two and it doesn't look at all obvious.

  3. I spotted the non-randomness in the encrypted answer. Thanks for the above plots but they are not exactly convincing - when I overlay the original "curves" onto the new data, it still looks like a "fit".

    But worse, the (hand drawn) curves are just rubbish. When I estimate the size at 50% maximum. The central peak is 20% larger than the side ones, but all that should be different is +/- 1C/century gradient. Also the peaks are around 0.95 and 1.05 - when they would be +/- 1

    However, what intrigues me most is the method he's used for generating the random numbers, because producing a 1/f type noise signal is not easy - but producing one that is entirely random on a deterministic computer is hellish.

    For more info here: http://uburns.com
    (A little idea springs to mind - perhaps I could add a trend!)

  4. Mike Haseler (Scottish Sceptic) could have followed the link provided by Brandon Shollenberger and read that the comment containing the histogram states "...where I’ve drawn some simple curves to approximate a Gaussian distribution for the positive, negative and untrended cases"

    and could have seen the follow-up post an hour later showing calculated fits to the overlapping peaks

    https://andthentheresphysics.wordpress.com/2015/11/20/a-stupid-100000-bet/#comment-67327

    Too much work to go to the source?

  5. Thanks Paul, I didn't save the initial series to compare it with and didn't realize it also had the same -.229 starting data point. My guess would be it's a fingerprint of him trying to trick up the RNG in some way and failing, or just incompetence.

    The idea that you put up a math prize and then change it when everyone points out it can be solved by different means is ridiculous as Brandon has pointed out. If anyone solves the new rules the "wrong" way then Doug can just change it again.

    It might have taken a few days but I think we can say the original challenge was solvable the "wrong" way given a year of current computing power.

  6. One thing I totally forgot to mention (which was also mentioned in a comment at Anders's blog) is the new file reports fewer digits. You can see that just from this discussion - the old series started with -0.229 but the new series start with -0.23. That changes the signal to noise ratio, which will impact people's ability to solve to the challenge.

    Another thing to mention is the change in these series extends beyond the change in distribution of trends I highlighted in this post. I happen to have the out of some tests I ran on the original data set, and they are notably different than the results I get with the current data set. I just didn't see much value in talking about them since all anyone would have is my word that the results were what I said they were. The original file being overwritten gets in the way of things.

  7. @redc

    The starting figure is to make the generated series look like recorded temperatures. I don't see anything sinister there. Plot them and have a look.

    @Brandon

    The distribution of trends hasn't changed in any significant way.

  8. Paul, while I agree the starting point is a non-issue (save in that it makes the change in rounding obvious as -0.229 is different from -0.23), I have to disagree with you about the distribution of trends. In fact, I have to say I have no idea how you can conclude it hasn't changed. I actually prefer to use a numerical approach for checking for differences between datasets, but I'll stick with histogram charts because they're visually compelling. First, here are two histogram charts I've made, one for the original data set and one for the new:

    Original
    New

    You'll note, they're on the exact same scale, so they are directly comparable. An earlier version I sent to a couple people in e-mail had the x-axis on the same scale but didn't pin the y-axis, so these are a bit different from the first version I made, hence the URLs for these ending in 2. Not really an issue for readers, but I wanted to throw that out there in case one of the people I spoke to in e-mail came across this comment.

    Anyway, I get comparing one graph to another isn't always the easiest thing to do, so I whipped together a quick overlay of the two graphs. You can see it here. It shows the trends of series in the original data set in a greenish color, ones for the new data set in a pinkish color, and ones where they overlap in a purple color. I think the differences are obvious. The distribution has flattened out by a noticeable amount.

    Still, we can demonstrate this even more clearly. There is no practical difference between a positive and negative trend for this challenge. As such, we can treat them as the same. Doing so effectively "folds" the histogram charts in half, putting the data in a simpler form. This is accomplished by taking the absolute value of the calculated linear trends. When we do that, this is the result.

    I think it is beyond obvious there have been significant changes to the distribution of linear trends in the data sets. Can you point to anything in these graphs, or anything else, to suggest otherwise? Other than just saying my charts aren't pretty, I don't see what criticism there could be of what I've said here.

  9. It would be suspicious if the distribution of trends weren't different, the question is whether it is surprisingly different. And I don't see any evidence or argument for that. A simple variation in the proportion of trended series would manifest as a change in the distribution of trends, and a change in that proportion doesn't change the problem of finding them. I think.

    IIUC the issue with the number generator was in the answers rather than the series themselves, I don't know why Keenan reduced the significance of the values, HADCRUT4 is three decimal places and 1880 is given as 0.229.

    I don't think this puzzle is solvable looking at the distribution of linear trend.

    Paul

  10. Paul, first you said the distribution hadn't changed in any significant way. I showed it did. I showed the change was clearly non-random. I don't what sort of change you would find "surprising," but simply dismissing obvious changes as not surprising you is unhelpful. If obvious, non-random changes that make the contest more difficult to solve are not surprising to you, what would be?

    Incidentally, a change in the proportion of series with a trended added/not added could not have caused this as you suggest. There there were distinct peaks associated with the trended and untreded series. A change in the number of these series would change the relative sizes of these peaks. It would not do what we observe here - decrease the size of both peaks. That could only be done by flattening out the entire distribution, which could only be done by changing how the series were created.

  11. That last comment is true whether the peaks are both reduced or if one is reduced and the others are held constant. I point that out because I think the smaller peaks were only held constant, not shrunk. That's what I get for commenting when I'm still groggy in the morning.

  12. I don't think you did show a significant change in distribution. You'd need a lot of runs of Keenan's program to see what was expected. And I guess we can do that next autumn.

    If fewer series were artificially trended then we'd expect a transfer of series from +/-0.01 to around the centre, which is what I think we see.

    Anyway I think looking at the linear trend across all the series is a hiding to nothing for two reasons, it's not in the spirit of the puzzle, which is really about evaluating whether it is possible to look at a series and state whether it has an imposed trend and practically I don't see how you can separate the series on either of your plots into the two classes with any confidence other than for the outliers.

  13. Paul, if you don't agree that constitutes a significant shift, what would? The charts I posted showed a systematic shift of ~100 series between the two data sets, in the exact direction one would expect if Keenan were trying to make the contest more difficult to win. You apparently feel that could be due simply to chance. How many series would have to shift, and by how much, before you'd agree it wasn't a coincidence?

    What if I posted charts showing the same thing for the autocorrelation in the series? And partial autocorrelation? I have them on my hard drive. I can post them when I get home if you'd like. The changes in them aren't remotely random in appearance either. Do you think that's coincidence?

    As for looking at the distribution of linear trends, that is perfectly in line with the spirit of the contest. The contest is about identifying which series have "true" linear trends. Looking at which series actually have linear trends, and how large those trends might be, is an obvious part of the process to do that.

  14. Pingback: car crash russia
  15. Pingback: hip hop mixtapes

Leave a Reply

Your email address will not be published. Required fields are marked *