Better adjusted global temperatures for ENSO, Solar and volcanoes

This is a follow-up to this earlier post, which please see for details. I had got into some difficulty there with using the R function nlm() to estimate both the regression parameters and the delay coefficients for each of the exogenous variables Vol, Sol and ENSO. The solar variable, which interacts most weakly, was apt to be assigned zero or negative delay, which created constant or exponentially rising secular processes, which were used by the fit.

I could avoid this by constraining that parameter. But I think it is better to do as others have done and use a common delay for all three. There is reasonable physical justification for that, and it reduces overfitting.

The result is a much more stable trend pattern across the time intervals and data sets. The trends since 1997 are now mostly between 0.65 °C/century and 1.325. This might still be seen as a slowdown, but surely a minor one. Oddly the only exception is the case studied by SteveF, Hadcrut 4 with linear from 1950. The trend I got there was 0.117°C/cen, I think quite similar to his, as was the decay coefficient at 0.026 (cf his 0.031).

I'll show below the revised table and images.

May GISS Temp up by 0.05°C

GISS LOTI went from 0.51°C in April to 0.56°C in May, almost exactly matching the rise shown by TempLS. Satellite measures went down slightly.

Adjusting global temperatures for ENSO, Solar and volcanoes

This post follows a flurry of activity in the spirit of the paper of Foster and Rahmstorf (2011). There multiple regression was used to remove from various datasets the effects of what could be seen as exogenous variables - the ENSO osciaaltion, solar flux and volcanic eruption aerosols. The result was a much more regular temperature rise, with most of the recent "slowdown" gone. In other words, the exogenous variables appeared to be responsible for the slowdown.

The method was a multiple regression in which the exogenous variables were lagged.
Update - I have a new post with a common lag parameter which seems to work better. 

I blogged about this at the time, and did a display of the trends with significance, showing the great improvement that came with removing the exogenous effects.

Troy Masters took this up in a series of posts, in communication with KevinC of SkS. An improvement was the use of exponential smoothing to achieve the lag effect. Troy found that his version still left some "slowdown" in the recent decade.

A few days ago, SteveF used similar methods on Hadcrut 4, over a longer period, back to 1950. He found a more substantial slowdown than Troy, since 1997, although the trend was still positive.

Going back to 1950 is controversial. Tamino stopped at 1979 because he felt that the linear trend which was used to fit the endogenous part could not be justified going further back. I thought so to, in comments at SteveF's post, and noted the "dip in the middle" in the detrended curve. Tamino wrote a recent more emphatic post on this.

In this post, I have done a similar analysis, but trying quadratic as well as linear, and using the intervals 1979-2011 as well as 1950-2011. But I've added some features. I've used the R non-linear optimiser nlm() to optimise the lags, which are individual to each variable. And instead of detrending, I've just included the trend in a multiple regression.

Update There is a problem pointed out by SreveF that the solar component is sometimes shown with a secular trend. I have tracked down the reason - it happens because nlm() sometimes finds an optimum with a negative exponential trend coefficient. That means that in the recurrence, instead of decaying, errors grow, especially the effect rather arbitrary starting point. This potentially affects all variables, adding a growing exponential component. . I'm working on a remedy.

I have now got reasonable results by constraining the solar delay coefficient to be not less than 0.03 - SteveF's value. That keeps it away from the problem areas. I have posted new images and table.

Significant trends

I see again a fuss at WUWT from Lord Monckton about "no significant warming for seventeen years and four months". I note wryly that the recent Keenan kerfuffle was about the Met Office answering a question about significant rise by citing the exact same statistic - whether with a AR(1) model the linear trend could be distinguished from zero. People wanted Dr Slingo sacked etc. But here we're back as usual - WUWT is  citing that very same statistic.

But it is indeed a fairly pointless statistic (the Met Office produced it on the insistence of a contrarian Lord). Statistical significance is important when you are trying to deduce some proposition from data. You need to know if your deduction could have arisen by chance.

But that's not the case here. We believe temperatures will rise because we've burnt a huge amount of carbon and boosted air CO2 by over 40%. And we look to temperatures and see a rise. Whether noise could have caused it is not the point; if you have a theory that predicts a rise and you see a rise, that's the best you can expect from the theory.

"Significant rise" relates to the wrong null hypothesis. You can only disprove a null, and a failure to disprove that trend is zero is not a very interesting result. It could just mean a not very powerful test. The logical question is - OK we expected a rise and we see a rise - is it the right amount? That is, can we reject the null that there is a trend of the expected magnitude?

That's the proposition that Lucia keeps testing, and though I argue there about whether what she tests is the actual AGW prediction, it is a test that makes sense.

Anyway. I'm sure that we'll hear more about no significant warming for x years, so I thought I would try to say something about the future course of x. It doesn't have a lot of degrees of freedom. And of course, it's as much affected by the ups and downs of temperatures in the '90s as those of today.

I'm basing this on the comprehensive trend plots I started a year or so ago.

TempLS global temp up 0.06°C in May

The TempLS monthly anomaly for May 2013 was 0.480°C, compared with 0.418° in April.

A climate blog reader

Google Reader is kaput at the end of June. I had been lazily eyeing alternatives, but I had also been looking into RSS systems, and it seemed that I could fairly easily write my own. It's a bit like re-inventing, but there are advantages. I used Google Reader a lot, though its limitations were painful. Improved searching is one aspiration. But if you read the feeds yourself, you can accumulate as much back data as you like.

Anyway, I found along the way that I could fairly easily compile an updated searchable list of comments on the main blogs that I was reading. My first attempt is below the jump. So far, I just have a few days data on the main Wordpress blogs. There are a lot of idiosyncracies, so I'll gradually extend it. When it has stabilized, I'll promote it to a page.

Draggable graphs

In the climate plotter a large amount of climate data could be plotted on an adjustable scale. There were bars on which you could click to move and expand the graph vertically and horizontally. Curves with different units could be moved independently. This is all based on the HTML5 canvas.

The bars were a bit clunky. I've been experimenting with mouse dragging. I had thought it would be slow, but it isn't. I'm planning to use it routinely in plotting, and for the climate plotter, and to post the code that enables it. Entering different data is easy.

Here's an example which I'll include in the monthly data tracking. It looks like just the last three years of monthly index data, similar to a graph that is currently shown. But it's backed by data back to 1850, which you can see by dragging back, and shrinking the scale if you want.