2012-03-07 11:18:32A question of bias


I don't know if this has been raised elsewhwere, but here goes.


Am I biased?

I would like to see percentages showing individual ratings against whatever turns out to be the norm.  I am 100% convinced that the climate is changing globally and that we humans are the primary cause.  With that in mind, and the need to be objective and impartial, I am trying to give the opposing hypothesis the benefit of the doubt.  That is, if I am undecided between two ratings, I will choose the rating which favours my opponent.  (This does not happen often.)

Despite trying to be fair, am I unconsciously biased towards my own view?  I would very much like to know.

2012-03-07 11:30:46Testing bias
John Cook


Once all papers have been rated twice, I'm going to do an analysis calculating the average of all SkS raters - will be interesting to see who is the most "warmist". Not sure what that will tell us but hey, it's an opportunity for an interesting graph so why not?

My strategy for attempting to remove bias is to firstly adhere to the guidelines, of course, and also adopt the "if in doubt, rate neutral" rule. But it also helps to take the mindset that our goal is not to shoehorn as many papers into endorsement categories - we don't need to pad our numbers. Our methodology by it's very structure will overestimate the # of neutrals. This is actually going to be a useful "bias" as it means our result will underestimate the level of consensus. And if, as I predict, the scientists' self-ratings show higher consensus than our ratings, that will confirm that our results underestimate the consensus. That neutralises the main criticism of deniers - that our results are biased because of our warmist tendencies. The opposite will be true and we will have shown it quantitatively.

Hell, if we're interested enough, I'll even run some code comparing each individual rater's ratings versus scientist ratings to see how each individual compares to the "definitive results". Plenty of ways to pick over this data which is one reason why this is a cool project.

2012-03-07 11:48:09


John: just to clarify, my 'benefit of the doubt' method seems to me to produce more 'neutral' results.


I was thinking ahead about the paper and about pseudo-skeptic attacks.  Maybe something on these lines:


We approached the problem of real or perceived bias in this way.  Our statistical methods show ... error bars ...

As to the possiblitiy of individual observer bias, our analysis of the ratings shows ... error bars ...


This is a good opportunity to publish lots of pretty graphs.  :-)