Establishing Competency

A couple months ago I contacted a scientist asking to examine the data used in three papers which made up the bulk of her PhD dissertation. The initial response contained this:

Thank you very much for your email and interest in our publications.

We follow ethical guidelines from the American Psychological Association, and we are happy to share our data to other competent researchers. Would you please indicate your background and outline how you plan to use the data?

Which struck me as odd as I have no idea how one would determine which people are "competent researchers." I was pessimistic about this response as it seemed like this might be used as an excuse for not sharing data with me, but fortunately, the issue of whether or not I am a "competent" researcher never came up again.

After examining the data for these three papers, I came to the conclusion the papers were fundamentally flawed in a way which invalidated their analysis and conclusions. I informed the author of this thesis of my concerns and tried to give her time to examine the issue privately. I believe several months was long enough so now I'd like to discuss the matter in public. Hopefully, this will demonstrate I am in fact competent.

The scientist in question is named Kirsti Jylhä. You can find her PhD dissertation here. There is a great deal to write about this dissertation, and I won't cover all of it in this post. I plan to write out the problems with this dissertation more fully in the future, but I'd like to use today's post as both a starting point for discussion and an introduction to the subject.

To accomplish this, I'd like to look at the first of two studies included in the first paper Jylha uses in her dissertation. The paper explains its basic concept:

For instance, many individuals still deny climate change, or at least particular aspects of it. An important question concerns where this denial comes from. Previous research suggests that some ideology variables (e.g., McCright & Dunlap, 2011) and exposure to climate related information (e.g., Greitemeyer, 2013) are related to climate change denial. When it comes to ideology variables, research has found social dominance orientation (SDO), right-wing authoritarianism (RWA) and left–right political orientation to be related to denial (e.g., McCright & Dunlap, 2011; Milfont & Duckitt, 2010; Milfont, Richter, Sibley, Wilson, & Fischer, 2013; Whitmarsh, 2011).

Basically, the authors want to try to figure out why people "deny" climate change. In order to do so:

One-hundred-thirty-five (aged between 18 and 61 years, M = 25 .8, SD = 7.5, 68% women) participants were recruited by announcements on notice boards and on a website aimed for recruiting research participants.

This study is built upon asking 135 people to answer a series of questions and looking at those answers for patterns. The issue I want to focus is this claim regarding that study:

Further analyses revealed significant zero-order correlations between all three ideology variables and climate change denial (see Table 1). This outcome is in line with predictions and previous research outlined above.

You can see the correlation scores referred to in this table:

12_20_cor_table

While the authors list a number of these scores as being statistically significant, the M Column in the table is worrying. M stands for mean (average). The authors tell us the possible range of results for each variable:

Climate change denial was measured by a sixteen-item scale (five reverse coded) that was developed by the authors (item example: Climate warming is natural and not due to human influence). The scale was constructed to capture different forms of denial, such as denial of human effect and denial of seriousness (see e.g., McCright & Dunlap, 2011). Right-wing authoritarianism was measured by a scale consisting of 15 items (see Zakrisson, 2005). Social dominance orientation was measured by a 16-item scale (SDO6; see, Pratto et al., 1994). Participants responded to all items on a Likert-like scale ranging from 1 (do not agree at all) to 5 (agree fully). Finally, political left–right orientation was assessed by a single item where participants positioned themselves on a scale ranging from 1 (far to the left) to 7 (far to the right).

On a scale of one to five, used for the first three items on the list, the central value is 3. The mean value in the survey responses was 1.95 for "Climate change denial," 2.16 for "Right-wing authoritarianism" and 1.86 for "Social dominance orientation." The central value for the fourth item, "Left-right political orientation," was 3.5 while the mean survey response was only 3.14.

This tells us the data the authors collected is skewed. That is an undesirable trait for any data set. When collecting data, you want to have a balanced distribution. Still, that doesn't necessarily mean the results are wrong. To examine that possibility, I first loaded the data and attempted to replicate the authors' correlation scores:

                             Climate_change_denial Social_dominance_orientation Political_orientation
Climate_change_denial                              
Social_dominance_orientation                  0.53                        
Political_orientation                         0.35                         0.30                  
Right_wing_authoritarianism                   0.33                         0.51                  0.12

As you can see, my results match those of the authors (the values I don't show in my table are Cronbach's alphas, which are a different measure we won't look at today). The authors also report:

We then conducted a stepwise regression analysis entering climate change denial as the dependent and the ideology variables as independent variables. The results showed that SDO was the strongest predictor of denial (b = .46, p < .001, R2 = .28). Also, left–right political orientation made a significant contribution in predicting denial (b = .21, p = .007, R2 = .04). The effect of RWA was not significant (b = .09, p = .28). The model accounted for a total of 32% of the variance in denial.

While I get this when I perform the same regression:

Residuals:
     Min       1Q   Median       3Q      Max 
-1.13218 -0.33581 -0.03661  0.33352  1.65996 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept)  0.54759    0.22601   2.423  0.01676 *  
SDO	     0.44488    0.09334   4.766  4.92e-06 ***
RWA	     0.11835    0.10914   1.084  0.28021    
L-R	     0.10147    0.03668   2.766  0.00649 ** 
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 0.5612 on 131 degrees of freedom
Multiple R-squared:  0.322,     Adjusted R-squared:  0.3065 
F-statistic: 20.74 on 3 and 131 DF,  p-value: 4.671e-11

These results match up to a significant extent. If we assume the value 0.00649 may was incorrectly rounded to 0.007, the p values match perfectly. The coefficient values (b) don't match up though, and I am uncertain as to why. It seems peculiar I could match the authors statistical significance (p) values while not matching the calculated coefficients. Even so, the difference is small enough I won't worry about it today.

Given the authors and I get similar results on calculations, the next thing to examine is if we reach similar conclusions. The authors conclude:

. These results imply that all three ideological constructs are related to climate change denial. However, only SDO and, to a minor degree, political orientation had unique effects on denial. Thus, SDO seems to be the strongest single predictor of denial while the contribution of political orientation, although significant, could be considered marginal.

This is where we part ways. I argue this conclusion is erroneous and unsupportable by the data. To see why, here is a graph showing the climate change denial item and social dominance orientation item:

12_20_fig1

While respondents could only pick integer values from one to five, we can see far more values in-between. You'll remember the authors asked 10+ questions for each of these items. What we see in this graph is the averages of them. I had asked the Jylha for the data used in her papers, but instead of getting a raw or complete data set, I was only given a processed, intermediary version of the data set. While this is unfortunate, we can still see an important detail. To make it clearer, I will add a small "jitter" value in which each point is shifted slightly.so each point will be visible:

12_20_fig2

Now let's try overlaying a grid showing where the central value on each scale is:

12_20_fig3

This should make the problem quite clear. There is almost no data from anyone who "denies" climate changte. There is almost no data from anyone who scores highly on the social dominance orientation scale. In fact, if we round the averages the Jylha gave me to the nearest whole number, there are only two people who fall in the "denial" side of the climate chnage belief scale. There are only two people who fall in the "dominane" side of:

Social dominance orientation (SDO), one's degree of preference for inequality among social groups,

That's two out of 135 responses. The authors' conclusion we can predict a person will "deny" climate change based upon their preference for inequality between social groups is based on a data set that has exactly two people who share either trait. Here is what the relationship the authors claim to find looks like:

12_20_fig4

The large empty swath of space on the right is where all the people who "deny" climate change are. With basically no data for those people at all, the authors claim to be able to find a relationship for them. It's nonsense. It's purely an artifact of the inappropriate methodology the authors used. To demonstrate, here is what the authors actually find for their relationship:

12_20_fig5

As you can see, the authors found a relationship between people who accept climate change and oppose social inequality. That's what their data shows. It doesn't show anything about people who "deny" climate change" or favor social inequality. For all we know, if the authors actually surveyed people who "deny" climate change, they might have gotten a data set like this:

12_20_fig6

Which would just tell us people who pick values closer to the middle for one question are more likely to pick values closer to the middle for other questions. That would hardly be a surprising result, and it would completely invalidate the idea people's views on social inequality predicts their views on climate change.

Am I saying that would have been the result these authors would get if they bothered to actually survey people who "deny" climate change? Of course not. I don't know what the data the authors didn't collect would show. Neither do they though. It is completely unjustifiable for the authors to draw a line through a data set then just assume they can extrapolate that line out as far as they want in order to draw conclusions about people they have little to no data for.

I'm going to stop this post here. In a future post, I'll discuss how we can mathematically prove the issue demonstrated with these simple graphs. I'll also examine how this issue affects other results from this study and other studies. In doing so, I'll demonstrate the authors have taken the fact people who accept climate change tend to oppose social inequality as proving people who "deny" climate change must favor social inequality. It is that simple.

16 comments

  1. Even before deconstructing the data and statistical method, this research must be considered flawed and dubious based on the methods of selecting the population and the use of arbitrary scales for evaluating psychological states.

    As for sharing data, in some cases (although not this one) there are ethical issues with privacy and confidentiality. I work with data from surveys of students and every three years must be recertified (meaning that I pass a non-trivial course about the ethical concerns of research with human subjects - see https://www.citiprogram.org/) to satisfy my Institutional Review Board. This is a good practice. Everyone tends to cut corners and excuse lapses in judgement when they have work to do. This kind of certification at least confronts researchers with the need to be thoughtful and careful. There have been horrendous abuses in the past and diligence is a way to minimize abuse in the future. Research ethics in general, whether or not with human subjects, is a neglected area of training for new scientists as we can see by how the data are tortured.

    In this present case asking for some indication of competency looks more like keeping it in a circle of "friends" or perhaps to monitor how it's used and to build a network of contacts. There can be multiple motivations.

  2. Gary:

    Even before deconstructing the data and statistical method, this research must be considered flawed and dubious based on the methods of selecting the population and the use of arbitrary scales for evaluating psychological states.

    While I can perhaps sympathize with this view, it is not one accepted by scientists who work in these fields. They hold approaches like these as being normal. For the moment, I don't intend to worry too much about it as I think the basic misuse of statistics/logic papers like these are predicated upon are more interesting and undeniable in my view.

    As for sharing data, in some cases (although not this one) there are ethical issues with privacy and confidentiality.

    Yup. That's why I only sought anonymized data with no personal information. Dealing with John Cook, Stephan Lewandowsky and others has given me quite a bit of insight into how researchers can use/misuse confidentiality concerns to avoid releasing data.

    In this present case asking for some indication of competency looks more like keeping it in a circle of "friends" or perhaps to monitor how it's used and to build a network of contacts. There can be multiple motivations.

    Being cynical, I would say they intended to not release their data to me but changed their mind when I began making a public fuss about it. I have no way of knowing that's true though. As long as there aren't any future problems with data access, I'm content to not worry much about it.

    Though I guess they still didn't provide the actual data. The intermediary data set (averaging the results of related questions) is useful for a number of things, but there are many tests that can't be performed on it. Having the full, underlying data would let us investigate things much more effectively.

  3. You ought to consider submitting some sort of paper on this spurious extrapolation. Maybe you could find a way to quantify it with a computed value -- perhaps a "Lew factor" or a "Shollenvalue".

  4. While I can perhaps sympathize with this view, it is not one accepted by scientists who work in these fields. They hold approaches like these as being normal.
    And they're wrong to do so as Matt Briggs points out in his book https://www.amazon.com/Uncertainty-Soul-Modeling-Probability-Statistics/dp/3319397559/ref=sr_1_1?ie=UTF8&qid=1466337349&sr=8-1&keywords=Uncertainty++The+Soul+of+Modeling%2C+Probability+%26+Statistics. However, it's another conversation and not meant to impose an obligation on you, of course. My point is that if the foundation is unstable, then the superstructure fails no matter how well built because it depends on what lies beneath.

  5. Canman:

    You ought to consider submitting some sort of paper on this spurious extrapolation. Maybe you could find a way to quantify it with a computed value -- perhaps a "Lew factor" or a "Shollenvalue".

    I have no idea how I would write or get such a paper published. I've considered it, but I couldn't think of a way. Maybe someone with experience in getting papers published could figure out a way. That said, it is quite easy to calculate the contribution of each point (or pair-value) to the correlation coefficient. That lets us quantify and examine the nature of the fit. I'll be discussing that in the next post about this.

    What makes things difficult is authors like to use multi-variate regresesions where they perform model fits against many variables at once. That increases the complexity of any analysis like that shown in this post, especially since there's no good way to show data for four sets of variable on one chart.

    But I'll be getting into things like that in future posts. Right now I'm wanting to just give people an easy introduction. Once I finish with this series of posts, perhaps I can figure out some other way to publish them. If nothing else, it might work as part of an eBook.

  6. Gary:

    And they're wrong to do so as Matt Briggs points out in his book https://www.amazon.com/Uncertainty-Soul-Modeling-Probability-Statistics/dp/3319397559/ref=sr_1_1?ie=UTF8&qid=1466337349&sr=8-1&keywords=Uncertainty++The+Soul+of+Modeling%2C+Probability+%26+Statistics. However, it's another conversation and not meant to impose an obligation on you, of course. My point is that if the foundation is unstable, then the superstructure fails no matter how well built because it depends on what lies beneath.

    You might be right. I honestly don't know. I'd want to spend some time researching and thinking about it before forming much of an opinion. It's definitely something I'd like to give thought to, but there is only so much time in a day. Besides, I figure if people can't agree on simple points like, "You need data for a group to draw conclusions about that group," you probably won't get far discussing more complex/abstract ideas

    Who knows? Maybe progress on the topics I bring up in today's post could eventually lead to progress on the issues you bring up.

  7. Brandon,

    It looks to me like the path to getting published is to find an insider coauthor. Steve McIntyre found Ross McKitrick. Willis Eschenbach found Craig Loehle. For you I'd suggest Joe Duarte or Jonathan Haidt.

  8. Your middle two charts were unnecessary. I grasped all in one quadrant right away. What I didn't realize was that I was looking not at the opposite quadrant of what I assumed. Didn't this same thing happen with Lewandowsky. I remember I tried to ask questions on Yahoo about it, and no one got the point(particularly since I used loaded language like global warmers support genocide).

  9. Canman, there might be some truth to that (though I don't know who Jonathan Haidt is). It wouldn't be enough on its own though. A major problem is just figuring out a way to present the problem/results in a way that people will read. That's especially tricky since many journals would likely not want to publish the paper as it'd highlight embarrassing errors in other work published in their journal. It's something to look into though.

    MikeN, I thought I was going overboard with the charts, but I figured it was better to over-explain than risk confusion. Hopefully the extra charts don't bog things down too much. As for Lewandowsky, this is exactly the same problem that existed in his work. That's actually why I came across this dissertation. I've been looking at cases where people made the same error Lewandowsky made. There are quite a few. This one just seemed the best one to discuss, particularly given the severity of it. That someone could get a PhD based largely upon this obvious fallacy is troubling.

  10. You got to love that our universities are inspiring the study of "wrong views" in order to cluster and identify a "problemed population."

    Bless her.

  11. I think you have the beginnings of a case study that would go into journal of applied statistics.

    As much as you might want to avoid a co author ( or not ) I would suggest getting someone.. say Briggs or RomanM

    Both can probably help you with your style.

    and dont take that as a slam on your style.. academic style is just a different thing..

  12. I have no problem with the idea of needing a co-author to get a paper written/published. Few people who haven't published a paper already will publish a paper on their own. If you have little experience with journals, it can be difficult to know what they might be "looking for."

    For instance, one big question is how in-depth to make any article on this topic. There are a surprising number of papers which rely on this same general mistake, and there are a number of different ways it manifests. How many things to discuss and how in-depth to cover them in a journal submission is not an easy thing to figure out. Plus, figuring out what terminology to use for your audience can be difficult.

  13. Brandon, isn't this the same phenomenon that we discussed a few years ago in connection with Wood and Lewandowsky where there was a population of 0 in the common quadrant:

    For example, whereas coherence is a hallmark of most scientific theories, the simultaneous belief in mutually contradictory theories—e.g., that Princess Diana was murdered but faked her own death—is a notable aspect of conspiracist ideation [30].

    See these posts and our discussion in comments:
    https://climateaudit.org/2013/11/07/more-false-claims-from-lewandowsky/
    https://climateaudit.org/2013/11/13/another-absurd-lewandowsky-correlation/

  14. Indeed. I've discussed this general issue quite a few times in the past, mostly in regard to Lewandowsky's work but also in regard to Wood's. Interestingly, the second post I wrote on this site was about my communication with Wood, and how he seemed to be intentionally missing the point.

    While those have been the focus of my discussions since they are the ones most known within the blogosphere, I have spent a significant amount of time identifying at least 17 other papers whose conclusions rely largely, if not entirely, upon abusing correlation scores in this same manner. I chose to highlight the dissertation discussed in this post (and several follow-up posts) because I find it incredibly troubling a person has been awarded a PhD largely upon misusing simple statistical tests in a way which produces bogus results.

    I wish people would have focused on Lewandowsky's methodology back when his Moon Landing paper was first published. Because discussions barely touched on that, many people never understood why Lewandowsky and Wood could get the results they got. Far more ink was spilled on the "fake" Lewandowsy data than the fact Lewandowsky's methodology was completely inappropriate for his data set. I know issues like "fake" data are easier for people to get, but it's not like this issue is particularly complicated.

    Simple correlation tests assume one's data set has particular characteristics. If the data set does not have those characteristics, the meaning of the results of any correlation calculations is altered/destroyed. The result is bizarre conclusions which cannot possibly be correct. For instance, one can take the fact Group A answered "No" to a question as demonstrating Group B would answer "Yes" to the same question, without having any data for Group B.

    I don't think many people will have a problem spotting the logical error in one saying, "We know conservatives say murder is bad. Liberals are the opposite of conservatives, therefore liberals will say murder is good."

Leave a Reply

Your email address will not be published. Required fields are marked *