Wednesday, December 2, 2009

How to Keep Scientists From Becoming Politicians

From Willis Eschenbach's chronicle of his FOI request to CRU, we find the following email from Ben Santer to Tom Wigley et al. buried close to the bottom of this voluminous exchange:
My personal opinion is that both FOI requests (1) and (2) are intrusive and unreasonable. Steven McIntyre provides absolutely no scientific justification or explanation for such requests. I believe that McIntyre is pursuing a calculated strategy to divert my attention and focus away from research. As the recent experiences of Mike Mann and Phil Jones have shown, this request is the thin edge of wedge. It will be followed by further requests for computer programs, additional material and explanations, etc., etc.

Quite frankly, Tom, having spent nearly 10 months of my life addressing the serious scientific flaws in the Douglass et al. IJoC paper, I am unwilling to waste more of my time fulfilling the intrusive and frivolous requests of Steven McIntyre. The supreme irony is that Mr. McIntyre has focused his attention on our IJoC paper rather than the Douglass et al. IJoC paper which we criticized. As you know, Douglass et al. relied on a seriously flawed statistical test, and reached incorrect conclusions on the basis of that flawed test.

I believe that our community should no longer tolerate the behavior of Mr. McIntyre and his cronies. McIntyre has no interest in improving our scientific understanding of the nature and causes of climate change. He has no interest in rational scientific discourse. He deals in the currency of threats and intimidation. We should be able to conduct our scientific research without constant fear of an “audit” by Steven McIntyre; without having to weigh every word we write in every email we send to our scientific colleagues.

In my opinion, Steven McIntyre is the self-appointed Joe McCarthy of climate science. I am unwilling to submit to this McCarthy-style investigation of my scientific research. As you know, I have refused to send McIntyre the “derived” model data he requests, since all of the primary model data necessary to replicate our results are freely available to him. I will continue to refuse such data requests in the future. Nor will I provide McIntyre with computer programs, email correspondence, etc. I feel very strongly about these issues. We should not be coerced by the scientific equivalent of a playground bully.
Well, yeah, there's a certain amount of whining here, but there's also an extremely good point: Do we really want a particular interest group to be able to "paper" a science project into submission? There has to be a point at which repeated FOI requests can be ajudged so intrusive, or so trivial, that the scientist has the right to ignore them and get back to work.

We're used to these kinds of tactics in the political arena, and we tolerate them as a necessary evil. However, scientists need the ability to concentrate on their research without continuously having to fight off frivolous legal actions.

Of course, the big problem here is that CRU was undeniably keeping sources, data, and reduction methodology much more opaque than was proper. They could have staved off a huge amount of FOI paperwork simply by doing a reasonable job of archiving and publishing their data. They were slobs, and then they got up on their high horse when some of their opponents accused them of being slobs.

Seems like we need some modification to the peer-review process. There's no reason in this day and age that a team can't submit a package of raw data, reductions, and code along with their paper. If the peers think that this information isn't sufficiently transparent, it should be grounds for sending the paper back to the drawing board. Of course, this doesn't solve the problem of picking a representative set of peers.

2 comments:

Unknown said...

With all due respect, I think it was a little worse than them being slobs.

They made adjustments to the raw data. That is understandable. The nature of data coming from upstream sources is such that when you get it, it generally has problems. This can be because you care about things the data provider doesn't much care about, it can be because the provider has some constraint (either they're getting it somewhere else that is sub-optimal, or they're using technology that has limitations, or there are economic considerations at play). Point is: making adjustments to data is fair play.

Making opaque adjustments is rarely fair play. Remember, these guys were using mostly public money to do these projects, and they weren't being open about how they were making their subjective adjustments, and they were then using their conclusions to argue for huge amounts of societal resources to be put into their preferred projects. And they were arguing that their conclusions were based on data and scientific methods that weren't questionable.

What the purloined emails and documents show is that they were *intentionally* opaque, as opposed to obtusely opaque. This is an important distinction because intent drives the discussion to motive. The conclusion here seems to be: papering over sloppiness. But if that was really their driving motive, if they judged themselves to be sloppy, then how are we to judge them? And crucially, if they judged themselves to be sloppy, how are we to judge their conclusions? And of all the possible motives, isn't sloppiness by far the most kind?

TheRadicalModerate said...

I'm inclined to give them the benefit of the doubt on their motives. (The essence of Radical Moderation is to question motives less and ideas more.) But I think that the reason they became so obtuse/opaque was because they were slobs. In short, they had lousy source-code control.

I've worked on the occasional project were we suddenly realized that we no longer had the ability to construct an old version of the product. It's not exactly something you rush out and tell your customers about. You hunker down and hope that nobody asks you for a bug fix. CRU did pretty much the same thing, although I agree that their techniques for hunkering down were pretty despicable.

Please note that I am not trying to apologize for the CRU--they deserve to be the main course at the upcoming banquet featuring scientist flambee. However, it's clearly the case that their political detractors are being every bit as craven as they were, and many are tossing around equally lousy science.

There are two somewhat conflicting goals here:

1) Make sure that scientists can work largely unfettered by political and legal attacks. Otherwise, no science gets done.

2) Make sure that policy-relevant science is completely transparent and that any consensus opinion is based on a robust, open peer-review process, rather than some form of groupthink.

The best balance between these two is going to require peer review reform and, coincidentally, recognition that science based on significant software engineering has to be transparent about the engineering and not just the software.

Meanwhile, I'd rather give CRU an Incomplete instead of an F. They need to clean up their mess with all the rest of us looking over their shoulders. Then we'll see where we are.