Update - I have recently implemented an automatic data updating scheme, described here. It runs every week.

Update - I have added new data sets, described here. I have also added a table of sources at the end.

I did some more programming to allow a fuller set of summary statistics when you click on the main plot. So I thought it was a good time to gather the various facilities in a single plot, since they use the same dataset.

Here is a brief summary of the things you can do:

- You can choose from many datasets (right buttons)

- You can choose time periods - this shows the same data on an enlarged scale for shorter recent periods

- You can select the variable to display - just trend, trend with significance marked, or the upper or lower confidence limit (CI), or the t-statistic, which is the ratio of the calculated trend to its standard error.

- With each dataset selection, a graph of the time series appears top right. There are two balls, blue and red, indicating the ends of the selected trend period (showing the trend). You can click on the red and blue bars to change this selection; there are also nudgers at each corner of the plot.

- The triangle plot has the start of the trend period on the y-axis, and the end on the left. You can click on any point to show details. These appear next to the plot on the right, and the time series plot will update to show the trend.

- At the bottom right is a url which incorporates the current state. You can copy it to use as a link; it will bring up the post with the selections as you had them when you copied.

You'll notice that the colors are mainly rainbow, with some added gray/browns for significant levels. These include trends 0 and 1.7 °C/century - the latter represents a kind of recent average useful as a visual marker. On t-statistics I've colored 1.96 and -1.96, as the 95% significance levels.

There are 255 rainbow color levels - the legend shows a selection of representative values. The scales are non-linear; using an inverse tan mapping so that all possible values are within the color range.

The CI plots are useful for looking up whether a chosen level has been significantly exceeded (or under-run). People are often looking to see if some forecast level has been significantly deviated from, usually on the low side. To check, say, where the trend has been significantly less than 2°C/century, look at the lower CI plot and the corresponding color level. You'll notice that this plot has the zero level in a gray color; that corresponds to the border of significance in the "trend with significance" style plot.

The nudgers in the time series plot move the balls my a small amount, varying depending on how far from the center you click. Smallest jump is 1 pixel, and then by 2x steps to 16. The movements are:

- Top left (blue) blue ball only

- Top rightt (purple) both in parallel

- Bottom left (gold) both in opposite direction

- Bottom right (red) red ball only

A note on performance - there are 264 images here, typically 50 Kb They are downloaded when first requested, so the page loads quite quickly, but if you request a lot of pictures, you may notice a slowdown, since they are held in memory.

I'll try to keep this version updated as new data comes in, and add new datasets as appropriate.

Nick: Is the confidence interval a 95% confidence interval in the trend viewer output?

ReplyDeleteFrank

Frank,

DeleteYes, 95%, with Ar(1) Quenouille correction.

Let's compare the warming trend for UAH6.0 for the first half of the record, the last half of the record and the full record.

ReplyDelete1/79-10/16: 0.883 K/century (95% ci of +0.411 to +1.256 K/century). "statistically significant"

1/79-1/98: 0.283 K/century (95% ci of -0.695 to +1.263 K/century). "statistically insignificant"

1/98-10/16: 0.611 K/century (95% ci of -0.803 to +2.024 K/century). "statistically insignificant"

Fairly strange, isn't it.

The absence of statistically significant warming is not evidence of no warming.

Frank

Frank,

DeleteAs I commented at WUWT, I think those are firures for UAH MT. But yes, you certainly can have quite high trends that are not "significant". For UAH LT 1/79 to 1/98, the trend is 0.982C/cen, and still not significant. In fact, you can have trends that are equal GCM projections that are not significant. That obviously does not denigrate GCMs.

I have just rewritten the Javascript here. The original was from my early days of JS programming, and was messing up the format somewhat with Firefox. Please The new version should have the same functionality - please let me know of anything wrong.

ReplyDeleteNick: Could there be a problem with your routine that calculates the 95% ci for trends? Karl (2015) reports a trend of 0.075 +/- 0.075 K/decade (90% ci including autocorrelation) for the period 1998-2012. Your 95% ci is 0.074 +/- 0/137 K/decade, which I believe translates into a 90% ci of 0.112 K/decade.

ReplyDeleteNOAAsst

Temperature Anomaly trend

Jan 1998 to Dec 2012

Rate: 0.736°C/Century;

CI from -0.636 to 2.109;

t-statistic 1.052;

Range 0.413°C to 0.523°C

Plot data

Trend

The SI for Karl15 is below. It contains dozens of trends with confidence intervals and a full explanation of how they were calculated. (I can't be sure whether your NOAAsst is equivalent to ocean in Karl15 (which presumably is ERSST4). Nor can I be sure whether Karl's 1998-2012 period includes 1/1998 to 12/2012, as I used or some other period. However, the discrepancy appears to be too big to be caused by minor problems like this.

http://science.sciencemag.org/content/sci/suppl/2015/06/03/science.aaa5632.DC1/Karl-SM.pdf

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.730.1459&rep=rep1&type=pdf

Sorry to create work for you.

Frank

Frank,

DeleteI don't know. He's using basically the same formula, on annual rather than monthly data. That might make some difference, but not this much, I think. I'll check.

Frank,

DeleteI've looked into it. One thing is that it monthly vs annual does make a big difference. The reason is that SST has very high monthly autocorrelation (.937), so the Quenouille approx isn't working very well.

However, I've duplicated Karl's arithmetic, and the only way I can make it agree is if 90% is interpreted as I would call 80% - in R terms, they are using qnorm(0.9)=1.28. I don't know if that is really what the IPCC means, but it may be. For my 95% I use qnorm(0.975)=1.96.

I have to say that I also checked the Trendviewer with monthly and got 1.46 spread rather than 1.37. Not a huge discrepancy, but I'll check more.

Delete"Nick StokesFebruary 17, 2017 at 4:52 PMFrank,

I've looked into it. One thing is that it monthly vs annual does make a big difference. The reason is that SST has very high monthly autocorrelation (.937), so the Quenouille approx isn't working very well."

A high pair-correlation value is a good thing if you are looking at detecting an underlying deterministic process. Of course, statisticians who want to see a random process tend to frown at a high pair-correlation value. The more naive they are, the more they tend to get mad at the data instead of paying heed and figuring out what physics is causing the high value.

Thanks for checking this out Nick. Somehow I expected a large team of professional climate scientists have gotten these details right. However, once the conflict is apparent, it should be obvious where the problem lies. You seem to be telling me that Karl15 calculated a 10-90% confidence interval, which is being described as a 90% ci??? And they used annual averages rather than monthly data.

ReplyDeleteSince the difference in trend between ERSST3 and ERSST4 for 1998-2012 were well within the uncertainty in these trends, I'm forced to wonder why Karl15 was published in Science. Perhaps they worked hard to find some way to present the data that makes the results seem more robust. Trends for a decade or two have confidence intervals too wide to be very meaningful.

If you can't tell, I copied the data straight from the screen and pasted it into a comment. I chose SST data because it isn't reprocessed all the time with breakpoint correction algorithms.

Frank

Frank,

DeleteWell, they said they were using annual averages. On my calcs, they do seem to be calculating a 10-90% CI. That doesn't seem right to me; I thought 5-95 is reasonable, since people tend to focus on whether trend lies above zero, which is a one-sided test. It may be an error, or it may be that that is what the IPCC really means by 90%. Or my calc might be wrong.

The "plot" is:

ReplyDelete"AccessDeniedAccess DeniedA301DCCD7A0B9081kJWkjcftRUCGW8pN6hsZQbxIHw8o/MK/4+nbfozWnQpFIPJvt7/GNKn3yWAonOSLQGLL/DRjZ2w="

Thanks, Should be fixed now

DeleteNick