A couple months ago I contacted a scientist asking to examine the data used in three papers which made up the bulk of her PhD dissertation. The initial response contained this:
Thank you very much for your email and interest in our publications.
We follow ethical guidelines from the American Psychological Association, and we are happy to share our data to other competent researchers. Would you please indicate your background and outline how you plan to use the data?
Which struck me as odd as I have no idea how one would determine which people are "competent researchers." I was pessimistic about this response as it seemed like this might be used as an excuse for not sharing data with me, but fortunately, the issue of whether or not I am a "competent" researcher never came up again.
After examining the data for these three papers, I came to the conclusion the papers were fundamentally flawed in a way which invalidated their analysis and conclusions. I informed the author of this thesis of my concerns and tried to give her time to examine the issue privately. I believe several months was long enough so now I'd like to discuss the matter in public. Hopefully, this will demonstrate I am in fact competent.
The scientist in question is named Kirsti Jylhä. You can find her PhD dissertation here. There is a great deal to write about this dissertation, and I won't cover all of it in this post. I plan to write out the problems with this dissertation more fully in the future, but I'd like to use today's post as both a starting point for discussion and an introduction to the subject.
To accomplish this, I'd like to look at the first of two studies included in the first paper Jylha uses in her dissertation. The paper explains its basic concept:
For instance, many individuals still deny climate change, or at least particular aspects of it. An important question concerns where this denial comes from. Previous research suggests that some ideology variables (e.g., McCright & Dunlap, 2011) and exposure to climate related information (e.g., Greitemeyer, 2013) are related to climate change denial. When it comes to ideology variables, research has found social dominance orientation (SDO), right-wing authoritarianism (RWA) and left–right political orientation to be related to denial (e.g., McCright & Dunlap, 2011; Milfont & Duckitt, 2010; Milfont, Richter, Sibley, Wilson, & Fischer, 2013; Whitmarsh, 2011).
Basically, the authors want to try to figure out why people "deny" climate change. In order to do so:
One-hundred-thirty-five (aged between 18 and 61 years, M = 25 .8, SD = 7.5, 68% women) participants were recruited by announcements on notice boards and on a website aimed for recruiting research participants.
This study is built upon asking 135 people to answer a series of questions and looking at those answers for patterns. The issue I want to focus is this claim regarding that study:
Further analyses revealed significant zero-order correlations between all three ideology variables and climate change denial (see Table 1). This outcome is in line with predictions and previous research outlined above.
You can see the correlation scores referred to in this table:
While the authors list a number of these scores as being statistically significant, the M Column in the table is worrying. M stands for mean (average). The authors tell us the possible range of results for each variable:
Climate change denial was measured by a sixteen-item scale (five reverse coded) that was developed by the authors (item example: Climate warming is natural and not due to human influence). The scale was constructed to capture different forms of denial, such as denial of human effect and denial of seriousness (see e.g., McCright & Dunlap, 2011). Right-wing authoritarianism was measured by a scale consisting of 15 items (see Zakrisson, 2005). Social dominance orientation was measured by a 16-item scale (SDO6; see, Pratto et al., 1994). Participants responded to all items on a Likert-like scale ranging from 1 (do not agree at all) to 5 (agree fully). Finally, political left–right orientation was assessed by a single item where participants positioned themselves on a scale ranging from 1 (far to the left) to 7 (far to the right).
On a scale of one to five, used for the first three items on the list, the central value is 3. The mean value in the survey responses was 1.95 for "Climate change denial," 2.16 for "Right-wing authoritarianism" and 1.86 for "Social dominance orientation." The central value for the fourth item, "Left-right political orientation," was 3.5 while the mean survey response was only 3.14.
This tells us the data the authors collected is skewed. That is an undesirable trait for any data set. When collecting data, you want to have a balanced distribution. Still, that doesn't necessarily mean the results are wrong. To examine that possibility, I first loaded the data and attempted to replicate the authors' correlation scores:
Climate_change_denial Social_dominance_orientation Political_orientation Climate_change_denial Social_dominance_orientation 0.53 Political_orientation 0.35 0.30 Right_wing_authoritarianism 0.33 0.51 0.12
As you can see, my results match those of the authors (the values I don't show in my table are Cronbach's alphas, which are a different measure we won't look at today). The authors also report:
We then conducted a stepwise regression analysis entering climate change denial as the dependent and the ideology variables as independent variables. The results showed that SDO was the strongest predictor of denial (b = .46, p < .001, R2 = .28). Also, left–right political orientation made a significant contribution in predicting denial (b = .21, p = .007, R2 = .04). The effect of RWA was not significant (b = .09, p = .28). The model accounted for a total of 32% of the variance in denial.
While I get this when I perform the same regression:
Residuals: Min 1Q Median 3Q Max -1.13218 -0.33581 -0.03661 0.33352 1.65996 Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) 0.54759 0.22601 2.423 0.01676 * SDO 0.44488 0.09334 4.766 4.92e-06 *** RWA 0.11835 0.10914 1.084 0.28021 L-R 0.10147 0.03668 2.766 0.00649 ** --- Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 Residual standard error: 0.5612 on 131 degrees of freedom Multiple R-squared: 0.322, Adjusted R-squared: 0.3065 F-statistic: 20.74 on 3 and 131 DF, p-value: 4.671e-11
These results match up to a significant extent. If we assume the value 0.00649 may was incorrectly rounded to 0.007, the p values match perfectly. The coefficient values (b) don't match up though, and I am uncertain as to why. It seems peculiar I could match the authors statistical significance (p) values while not matching the calculated coefficients. Even so, the difference is small enough I won't worry about it today.
Given the authors and I get similar results on calculations, the next thing to examine is if we reach similar conclusions. The authors conclude:
. These results imply that all three ideological constructs are related to climate change denial. However, only SDO and, to a minor degree, political orientation had unique effects on denial. Thus, SDO seems to be the strongest single predictor of denial while the contribution of political orientation, although significant, could be considered marginal.
This is where we part ways. I argue this conclusion is erroneous and unsupportable by the data. To see why, here is a graph showing the climate change denial item and social dominance orientation item:
While respondents could only pick integer values from one to five, we can see far more values in-between. You'll remember the authors asked 10+ questions for each of these items. What we see in this graph is the averages of them. I had asked the Jylha for the data used in her papers, but instead of getting a raw or complete data set, I was only given a processed, intermediary version of the data set. While this is unfortunate, we can still see an important detail. To make it clearer, I will add a small "jitter" value in which each point is shifted slightly.so each point will be visible:
Now let's try overlaying a grid showing where the central value on each scale is:
This should make the problem quite clear. There is almost no data from anyone who "denies" climate changte. There is almost no data from anyone who scores highly on the social dominance orientation scale. In fact, if we round the averages the Jylha gave me to the nearest whole number, there are only two people who fall in the "denial" side of the climate chnage belief scale. There are only two people who fall in the "dominane" side of:
Social dominance orientation (SDO), one's degree of preference for inequality among social groups,
That's two out of 135 responses. The authors' conclusion we can predict a person will "deny" climate change based upon their preference for inequality between social groups is based on a data set that has exactly two people who share either trait. Here is what the relationship the authors claim to find looks like:
The large empty swath of space on the right is where all the people who "deny" climate change are. With basically no data for those people at all, the authors claim to be able to find a relationship for them. It's nonsense. It's purely an artifact of the inappropriate methodology the authors used. To demonstrate, here is what the authors actually find for their relationship:
As you can see, the authors found a relationship between people who accept climate change and oppose social inequality. That's what their data shows. It doesn't show anything about people who "deny" climate change" or favor social inequality. For all we know, if the authors actually surveyed people who "deny" climate change, they might have gotten a data set like this:
Which would just tell us people who pick values closer to the middle for one question are more likely to pick values closer to the middle for other questions. That would hardly be a surprising result, and it would completely invalidate the idea people's views on social inequality predicts their views on climate change.
Am I saying that would have been the result these authors would get if they bothered to actually survey people who "deny" climate change? Of course not. I don't know what the data the authors didn't collect would show. Neither do they though. It is completely unjustifiable for the authors to draw a line through a data set then just assume they can extrapolate that line out as far as they want in order to draw conclusions about people they have little to no data for.
I'm going to stop this post here. In a future post, I'll discuss how we can mathematically prove the issue demonstrated with these simple graphs. I'll also examine how this issue affects other results from this study and other studies. In doing so, I'll demonstrate the authors have taken the fact people who accept climate change tend to oppose social inequality as proving people who "deny" climate change must favor social inequality. It is that simple.