Monday, March 2, 2015

Derivatives, filters, odds and ends

I've been writing about how a "sliding" trend may function as a estimate of derivative (and what might be better) here, here and here. There has been discussion, particularly commenter Greg. This post just catches up on some things that arose.

Saturday, February 21, 2015

Regression as derivative

In two recent posts here and here, I looked at a moving OLS trend calculation as a numerical derivative for a time series. I was mainly interested in improving the noise performance, leading to an acceleration operator.

Along the way I claimed that you could get essentially the same results by either smoothing and differentiating the smooth, or differencing and smoothing the differences. In this post, I'd like to develop that, because I think it is a good way of seeing the derivative functionality.

This has some relevance in the light of a recent paper of Marotske et al, discussed here. M used "sliding" regressions in this way, and Carrick linked to my earlier posts.

Wednesday, February 18, 2015

Google Maps and GHCN adjustments

Google Maps and GHCN adjustments

A fortnight ago I posted a Google Maps gadget for viewing GHCN stations colored according to the effect on them of GHCN adjustments. I've been doing some improvements, and rewriting the code in the process. This simplifies the logic, and I'm hoping to produce a generic application to operate on any supplied data.

For the moment, the main improvement is that it displays a count of whatever is colored on the screen. So you can quickly show how many have been adjusted up, or down, with selection criteria specified. The other improvement is that the popup data includes a link to the GHCN display page, giving extensive history and graphs of observations and adjustments.

I have also updated the data to Jan 2015.

The plot is below. And below that, some details about the usage logic. The field Trend_Adj is the trend difference over whole of life made by adjustment, in °C/cen. It is set to NaN for stations with less than 360 months of adjusted data in total (maybe with gaps).

Monday, February 16, 2015

January GISS up from 0.72 to 0.75°C

GISS showed a small rise. TempLS mesh dropped slightly from 0.66C to 0.64C. TempLS grid also dropped, from 0.65C to 0.63C. Based on this, I would expect a small drop in NOAA and HADCRUT. But maybe not. Both satellite indices rose - RSS significantly, from 0.28C to 0.37C. Since the recent warming seems to be SST driven, this lag makes sense. Maps are below. TempLS is continually reported here.

Slightly O/T, but there has been a recent spike in February, according to the NCEP/NCAR index. It has now pulled back a bit.

Thursday, February 12, 2015

Adjusting in the finance world

There has been much talk recently about homogeneity adjustments. Some in the mainstream madia, and none that made much sense. It's one of the most extraordinary scandals of our time. Maybe even criminal:

"Is history malleable? Can temperature data of the past be molded to fit a purpose? It certainly seems to be the case here, where the temperature for July 1936 reported ... changes with the moment," Watts told

"In the business and trading world, people go to jail for such manipulations of data."

So I thought I'd see what does go on in the trading world. I originally commented on this at Paul Homewood's site. I looked up the chart for BHP's share price on our national exchange, ASX. Scrolling down, I read:

"Adjustments - The charts are adjusted to smooth out the effect of bonus issues, rights issues, special dividends, share splits, consolidations, capital reductions, or to link historical values that represent the company's primary equity security. The chart also assumes that all company issued options and convertible securities are converted into ordinary shares."

In other words, not the historic prices at all. And ASX won't show you a chart of the "raw data". One complaint about GHCN adjustments is that they are constantly changing the past. But see what happens here. As with climate, present adjusted values are held equal to present market price. So what happens when BHP issues such a dividend? Its price drops by about the amount of the dividend. ASX adjusts all past prices down, to "smooth" the drop.

Wednesday, February 11, 2015

Surface temperature global average is robust to noise

There has been a lot of argument about GHCN adjustments. Fiddly stuff like whether a certain step introduced in Reykjavik in 1963 was justified. None of this has any significance for a global average. Steve Mosher of BEST is constantly reminding that there is actually a huge amount of data in the average, and there is no point in focussing on the very local scale.

That is a point I was emphasising in this post on homogenisation. The average is little affected by unbiased noise (cancellation), but sensitive to bias. So it makes sense to identify and eliminate bias, even if it increases noise. Homogenise.

There is another common naysayer claim, that averages are not more accurate than the original readings, as in this WUWT thread. I don't know what they think drug companies etc go to great expense to get a large set of responses to average.

Anyway, in this post I'll give a dramatic demonstration of the noise suppression of averaging. I'll take the usual post-1900 monthly GHCN and ERSST data and let TempLS calculate an annual average. Then I'll add Gaussian noise to every single monthly average. Big noise - amplitude (sd) 1 °C. That's a world in which thermometers can barely be read to the nearest degree. In fact much worse, since monthly averaging already reduces noise. Then I'll recompute and show the differences in various ways. I'll do 10 instances.

As you'll guess, the difference is small. The standard deviation of fluctuations about the unperturbed is about 0.006 °C. The effect on trend is even smaller. The unperturbed trend 1900-2014 is 0.7073 °C/Century and the range is about 0.705 to 0.710.

I should emphasise that this perturbation operates on what might be taken to be measurement error, and does not emulate that quoted by NASA, NOAA etc, which is dominated by spatial sampling error. It also assumes white (ie independent) noise; dependence will increase the effect. But from a very low base.

Monday, February 9, 2015

Breakdown of effects of GHCN adjustments.

In a previous post, I showed how running TempLS with unadjusted or adjusted GHCN station data made a small difference to the final global average. That is useful to know, because some say that AGW is a creature of GHCN adjustments. This is certainly not true.

That analysis can be extended. I can trace the contribution to this small adjustment effect from various categories of stations - by continent, rural status, or airport status. And as before, it shows the difference by year, or effect on trend (to present time) by year.

A useful addition to the present post will be some trends calculated for land only in a NOAA paper by Lawrimore et al, 2011, Table 4 (h/t Victor Venema):

v3 Uncorr v3 Corr

Note the change of units (in the original). I got an approx land only difference of 0.17°C/Cen for 1900-2014 by dividing the global value by the land area fraction. That was pretty rough; the value in this post is more accurate, and agrees well.

In this post, I'll start with the usual active plot - see here for guidance. I've added a new "Norm" button. This allows you to toggle to a mode where the average adjustment for each category is calculated. The initial mode gives the contribution of that category to the global average. I think that is most useful, as it is area weighted. The category average can show high average adjustments for relatively rare stations. I'm using TempLS mesh exclusively; the grid style allows too much mixing between categories.

As usual with the active graph, you can select the set to show, move and zoom the graph, and show the numerical data. Then I'll show some ordinary graphs comparing the categories.