Hey guys, as you may have picked up on from my last couple posts, I was fairly sick this week. I'm not completely over it, but I have had the energy to do more than lie around all day doing nothing. Naturally, one of my top priorities has been playing Rock, Paper Scissors (RPS).
I'm not going to re-visit the history leading up to today's post. You can read the last post I wrote on this subject here. The short version is it seems no matter what I do, I keep beating a computer opponent that makes random choices. This shouldn't be possible. The odds of winning, losing or tying in RPS should be 1/3 when one opponent picks options at random.
Today's post is about an update to my methodology and the results it leads to. I've played 10,000 matches after the update, and I have won 3,454 of those matches. That gives me a win rate of 34.54%, a result that is "statistically significant" at the 99% level.
There are three key details to the methodology I am using. First, I am doing my tests in batches of 100. That is, I will start a run and play RPS exactly 100 times and record the results of that run in a data file for that run. For completeness, I also store the outcome of each individual match.
Second, I generate the computer's selections prior to starting my run. What this means is at the start of each run, there is a list of 100 choices the computer will cycle through. This means any action or guess of mine cannot influence what comes later.
Third, I do not generate the random choices myself. This is the major update to my testing methodology. In previous posts and comments, I've discussed potential concerns that the two random number generators (RNGs) I've used might somehow have patterns in their output a human could pick up on in order to win at RPS. This seemed highly unlikely to me, but it was the best idea I had. To address it, I've started using a feature provided by the site www.random.org. You can see it in action by clicking this link:
Clicking that link will provide you a list of a hundred selections ranging from 1 to three (inclusive). The site using physical properties of the atmosphere to help it generate its RNG values. There is no way there should be any discernible pattern in its output. It should be impossible to beat a computer opponent using it to play RPS. I've done so anyway:
This chart provides a simple examination of my results. It shows the number of wins I've had minus the number of losses in each set. These results are summed together as the series progresses, giving a cumulative (or net) number of wins I've gotten. Since the outcome of this game should be completely random, we would expect this chart to show a line fluctuating about zero. It doesn't.
As a quick aside, I want to point out the last twenty sets in this chart were done earlier today. I hadn't played RPS in a week, and I'm still sick. It is interesting to see this timing coincides with my results flattening out. It may be a coincidence. I just wanted to put it out there so people could consider it for themselves. It could be similar to how under my last methodology (which handled the RNG on my computer), I got these results:
The first five thousand or so matches under that methodology showed no trend. Over time, it seemed my level of "skill" improved. One could choose to interpret the flat periods in these two charts as indicating periods where something was throwing off my ability to play RPS well. I'm not going to guess at that right now. Instead, I'd like to try using a slightly more in-depth analysis. Consider:
This is a histogram showing the outcomes of the 100 runs I've done using www.random.org for my RNG. In theory, the distribution of wins, losses and ties should be equal (save for random variance). That's because in a random selection, you are just as likely to get 33 wins as you are 33 losses.
That is not what we see in these histograms. In these histogramsk, there is a clear positive skew for winning. it is similar to what we saw in the histograms for my last methodology:
That shows results for 185 runs. That the number of wins is skewed toward the right even after 100/185 runs is remarkable. It is not something which should happen if the RNG is truly random. I don't have an explanation for this. It is simply not plausible that I can subconsciously pick up on patterns in three different RNG systems and beat them like I have.
At this point, the only explanation I can come up with is the program I'm using has a bug in it. As such, I've uploaded the code I've used for the recent tests and the data files for my last 100 runs here. I would like it if someone could check my code and verify there is no major bug that could be causing these results. For instance, please verify that the list used for the computer's choices is updated every run. Obviously, if the same set of choices were used over and over, a human could pick up on that.
The code is in Python, and it is somewhat haphazardly put together. You should be able to follow it, but if you have any questions, feel free to ask. I believe you will need to install the requests package for Python to make it work (and if you don't have it installed, Python as well). Just remember to make the directory to store your data files.
Assuming there is no major bug in my code, I'm not sure what to do. I have data from over 40,000 RPS matches I've played (across three different RNG systems). Across all of it, my win rate has consistently been higher than it should be. What should I do about that? Can I really ignore what appears to be a flagrant violation of the laws of probability and go on with my life like nothing has happened?
I don't know. One suggestion I've received from several people is to record, and perhaps even broadcast live, my test runs. This would provide some manner of evidence that I am not cheating. Would that be worthwhile? Would anyone watch? Would anyone care?
I don't know. What I do know is I'm at my wit's end. All I want is for the randomness around me to be random.