Wednesday, March 22, 2017

Global average, integration and webgl.

Another post empowered by the new WebGL system. I've made some additions to it which I'll describe below.

I have written a lot about averaging global temperatures. Sometimes I write as a sampling problem, and sometimes from the point of view of integration.

A brief recap - averaging global temperature at a point in time requires estimating temperatures everywhere based on a sample (what has been measured). You have to estimate everywhere, even if data is sparse. If you try to omit that region, you'll either end up with a worse estimate, or you'll have to specify the subset of the world to which your average applies.

The actual averaging is done by numerical integration, which generally divides the world into sub-regions and estimates those based on local information. The global result always amounts to a weighted average of the station readings for that period (month). It isn't always expressed so, but I find it useful to formulate it so, both conceptually and practically. The weights should represent area.

In TempLS I have used four different methods. In this post I'll display with WebGL, for one month, the weights that each uses. The idea is to see how well each does represent area, and how well they agree with each other. I have added some capabilities to the WebGL system, which I will describe.

I should emphasise that the averaging process is statistical. Errors tend to cancel out, both within the spatial average and when combining averages over time, when calculating trends or just drawing meaningful graphs. So there is no need to focus on local errors as such; the important thing is whether a bias might accumulate. Accurate integration is the best defence against bias.

The methods I have used are:
  • Grid cell averaging (eg 5x5 deg). This is where everyone starts. Each cell is estimated as an average of the datapoints within it, and weighted by cell area. The problem is cells that have no data. My TempLS grid method follows HADCRUT in simply leaving these out. The problem is that the remaining areas are effectively infilled with the average of the points measured, which is often inappropriate. I continue to use it because it has often very closely tracked NOAA and HADCRUT. But the problem with empty cells is serious, and is what Cowtan and Way sought to repair.
  • My preferred method now is based on irregular triangulation, and standard finite element integration. Each triangle is estimated by the average of its nodes. There are no empty areas.
  • I have also sought to repair the grid method by estimating the empty cells based on neighboring cells. This can get a bit complicated, but works well.
  • An effective and elegant method is based on spherical harmonics. The nodes are fitted with a set of harmonics, based on least squares regression. Then in integrating this approximation, all except the first go to zero. The integral is just the coefficient of the constant.


The methods are compared numerically in this post. Here I will just display the weights for comparison in WebGL.

The display is below, and I'll discuss below that. The gadget and its operation are described here. You'll see there are new checkboxes with the various WebGL objects beside. You can make them disappear. There is a new facility that I'll describe in a later post that creates new objects suffixed _P and _L for the edge nodes and lines. You can toggle these, and also Grid, which is a 5x5° mesh matching what my grid methods use. It should be switched off when looking at the other methods, since it is not then relevant.



The plot is for Jan 2014, since I can then use production weights that may not be calculated on the same day of the month (recent months might have varying data).I have graphed both the weights (adding to 100) and log weights, since they cover a wide range. I have shown the mesh method first. The weights are much as you might expect - high (red) over Africa, poles and Brazil, and low (blue) over US and Europe, where density is high. The mesh does the best job of reflecting local density.

The plain grid is next, and its faults are obvious. If you look at the Arctic, it is blue, not red. Station weights are a fraction of the cell area only, and can't exceed that. The fraction depends on how many stations in each cell. So a cell representing a big area in Africa can have no more weight than a tropical ocean cell, and an Arctic station is limited to much less, because cells are thin there. You'll see odd patterns in the regular ocean mesh. That is from the interaction of the 4x4 SST grid and the 5x5 integration grid. Mostly there is one node per cell, but where there are two, the weight halves, and four, which is possible, brings the weight down further.

The cell infilling method gets much closer to what is expected in Africa and Brazil. It still somewhat underweights the Arctic, probably through incomplete infilling. There is an an apparent artefact near 0°E ocean, which may reflect a problem with the progoram infilling across this point.

The spherical harmonics weights are interesting. SH gives very similar results to mesh (and infill), butit assigns weights with a much broader brush, by regional density rather than local. The good result suggests that the scale that it is not resolving makes little net contribution to the integral.

I now plot all integration results in the new latest data table; you have to click the TempLS button.







6 comments:

  1. Funnier than all get out is a "Dr. O" commenting over on Clive Best's blog who is trying to debunk Nick's approach for estimating surface GAT. Dr. O is using all these big words but can't figure out that the problem is NOT one of solving partial differential equations. The apparently immense effort he put in to this debunking is completely misguided. Will we ever hear a "never mind" from Dr.O?



    ReplyDelete
  2. From: DrO

    Ohhh ... dear me, did those big words give you an ouchi? Why don't you tell us which words bothered you, and maybe we can make your boo-boo better.

    I have no idea who you are, and unless your Clive Best or Mr Nick Stokes masking your identity, I did not see you participate in any of the discussions in questions, or anywhere else for that matter.

    There is no (provable) mathematics, science, fact, reason, or even courtesy in your comments.

    Normally, I would not respond to such nihilism. However, you are a coward. You did not post your comments in the discussion or any place to present your thoughts to me directly ... only here anonymously, and without having the courage to let know you had done so.

    By the way, if you are going whinge about mathematics and precision, you might first learn how to spell a three-letter word correctly. My "handle" on the blog in question is "DrO", not "Dr. O". I have never used "Dr. O" anywhere ... perhaps you should learn how to copy/paste before shooting your mouth off.

    I doubt it would have made any difference to your commentary, given your emotional state, but I have now responded to the colossal mathematical blunders and dishonesty in Mr Nick Stokes reply on that post.

    GAT is utter nonsense. It does not measure anything meaningful, and it cannot be used for anything sensible, even you had "perfect" interpolation and averaging. If you do not understand heat balances and aperiodic systems etc, you have no place in this conversation.

    Your commentary smacks entirely of "tribal narcissism" and the "politics of outrage".

    It would not surprise me if you or one of your ilk will delete my response here, as part of your "intolerant censorship" of anyone who dares to question your position, or worse, prove you wrong ... but I will keep a record, just for posterity.

    It would also not surprise me that you will want the last word, go ahead, I wont respond to you.

    ... here is one last "big word" to describe you, hope it doesn't give you another boo-boo: vexatious.

    ReplyDelete
    Replies
    1. Dr.O thinks the mesh averaging is a PDE problem, and gets steamed like Trump when someone calls him on it.

      Delete
    2. WHUT (whose identity most people here know) has a valid point, never answered. DrO's diatribes are heavy on talk of PDE's. Boundary methods, Stokes Theorem, Green's theorem. "and since these are (highly) non-linear PDE’s" etc. Yet when I point out that Clive is simply integrating, no PDE's in sight:
      "To begin with, this is categorically wrong/dishonest."

      To which I asked a simple question - what are these PDE's. No answer.

      Delete
    3. Indeed, there is a clear distinction to be made between science poseurs and those that know what they are talking about. For example, I have always thought that Clive Best knows his stuff, even though his agenda is a bit suspect. This DrO guy though is clearly flinging the BS and possesses no mathematical intuition, contrary to what he asserts.

      Delete
  3. "...unless your [sic] Clive Best or Mr [sic] Nick Stokes..."

    Substantive issues with science aside, it's difficult to lend credibility to the blather of someone who struggles with the your/you're dichotomy, or who refuses to acknowledge Nick's scientific/mathematical/statistical credentials.

    ReplyDelete