One of the more interesting talks for us in the Paris conference mentioned previously here was James Porter talking about UKCP09. It turns out there has been a social sciences project (“Project ICAD”) part of which involved looking at the UKCP09 project (and they are based in Leeds University, perhaps an additional reason for a visit there some time?). We were in Japan over the entirety of the interval in which UKCP09 took place, and only had limited contact with the relevant parties, but perhaps know enough about the issues for our perspective to have some relevance. The speaker had spent some time embedded with the Hadley Centre and had talked to a lot of people involved in the production and review of the UKCP09 project.
A significant part of Porter’s talk looked at the question of how the probabilistic predictions were made, and in particular the UKCP choice of basing their probabilities primarily on their ensemble of HadCM3 simulations with different parameter values (perturbed parameter ensemble or PPE), rather than basing their results on the CMIP3 ensemble of different models (multi-model ensemble, MME). I was surprised to see this presented as such a major decision, as my recollection is that most of the critics at the time were really complaining about the willingness of UKMO to generate probabilistic predictions at all since (the critics argued argued) there was not really a sound basis for assigning numerical values by any method.
The main UKCP09 proposal was (according to their web page) funded in 2004 and at that time, it seemed quite widely accepted that PPEs were a better foundation for probabilistic prediction than the MME. In fact this era was very much the heyday of PPEs, with climateprediction.net, the Hadley Centre’s QUMP group and our own rather smaller ensemble research activity all making rapid progress. The UKCP09 approach was externally reviewed back in 2008/9. The full review doesn’t seem to be available (anyone know where it is?) but I don’t see any evidence in either the summary or response that the question of MME vs PPE was seriously raised by anyone even at that later time.
I believe (though I could be wrong and would welcome references) that we were actually the first to argue the contrary. The roots of our argument can be found in this Yokohata et al paper (which although published in 2010 was submitted back in 2008), which pointed to substantial inconsistencies between two PPEs based on our two different GCMs (MIROC and HadCM3). However it was actually our our series of papers on ensemble analysis starting in 2010 (eg here, here, here and here) that most clearly argued not only that PPEs had serious problems, but also that the MME was much better than previously believed. So while I’m encouraged to see that this question is now high on the agenda, I really don’t think it was on the table at the outset of UKCP09 and it doesn’t really seem fair to use it as evidence of insularity or reflexive dismissal of outside ideas, which seemed to be the speaker’s point. Given the work they had already done by 2010 or so, the UKCP09 researchers actually made quite substantial and constructive efforts to account for the (by then) emerging failings of the PPE approach by effectively adding on the MME’s uncertainty to their results. While this may satisfy neither the resolutely anti-Bayesian nor the most purist pro-Bayesian, in my view it certainly improved the credibility of their results.
Some of the interviewees gave excuses for their apparent reticence to air their doubts openly at the time. According to Porter, some of them said they were scared of being labelled sceptics! What a feeble excuse. Perhaps more plausible, is the additional argument that the incestuous and cliquey nature of climate science in the UK made it a bit of a career risk in terms of future funding. But in any case, I certainly recall some people making their criticisms very plain. In particular, Lenny Smith argued eloquently about the risks of assigning probabilities where there was not really a sound basis for them. If the next model generates different results (which is entirely plausible) then someone is going to end up looking rather silly.
So I’m not going to stick the knife into the Hadley Centre for proposing in 2004 to base their probabilistic predictions on a PPE methodology that they had already started to work on. You would have had to be unusually prescient to anticipate our research by several years, although I’m encouraged to see it is now obviously high on peoples’ minds. On the other hand, the Hadley Centre’s apparent continuing preference for PPEs is hard to defend, now that they have a chance to regroup. To that extent, perhaps this Project ICAD analysis contains a truth that is deeper than the actual story they purported to tell 🙂