Re: [netcdf-java] [thredds] Help with GRIB encoding - what is "reference time of data" ??

Hi Benno:


Grib1 has center (5), subcenter (26), and table version no (4), all single bytes. ( if process id (6) is used for table lookup, that may be considered "non-standard").

because the parameter tables got full so quickly, all sorts of practices developed. Undocumented AFAICT, though consistent typically within a center. Its also safe to say that local tables are not published in any consistent or authoritative way.

Grib2 is better in many ways, and generally more consistent, but oh yeah, you just have to know that center X uses variant Y except when it doesnt. WMO is publishing their GRIB2 tables now in XML, which I consider programmatically useful, since it can be read by software without human intervention (once you learn the schema).

In terms of backwards compatibility, as Eiza says, it is almost but not always, and really only the writer of the GRIB records knows for sure what table was used.

In short, not good for general GRIB readers, and I have been advocating that GRIB not be used for archival of data until these fundamentals are fixed.


On 12/2/2013 9:45 AM, Benno Blumenthal wrote:
Thank you very much for your answers.  At first I thought this was very
different from grib1 -- but now I think not-so-much.  GRIB1 had center,
subcenter, process, and ParameterTableVersionNumber, all of which got
used (sometimes) to identify the table (and the table could get
overriddened anyway by a local copy), and it looks like identifying the
local table in GRIB2 could very well be a similar process.  GRIB2 has
the advantage that at least all the WMO standard codes come from WMO
tables instead of being copies into the local tables.

On Fri, Nov 29, 2013 at 5:38 AM, TOYODA Eizi <toyoda.eizi@xxxxxxxxx
<mailto:toyoda.eizi@xxxxxxxxx>> wrote:

    Hi Benno,
    1) I had similar question before.  I'm sorry I can't remember the
    conclusion for GRIB, but for BUFR, there are several element
    descriptors whose definition changed over time.  WMO is trying to
    maintain upper compatibility, but sometimes something happened
    unfortunately.  I guess it is safer to assume GRIB is in similar status.
    2)  Again I'm sorry the table posted at following site is the only
    one maintained.

    Collecting local tables is I think the next step.
    3) You'll need the code of the originating centre (and probably
    sub-centre) to identify the local table.  By the way I think NCEP
    and ECMWF are using different short names for GRIB parameters.  So I
    don't think it is easy for WMO to establish its standard name.
    Sorry about unconfortable answers....

        ----- Original Message -----
        *From:* Benno Blumenthal <mailto:benno@xxxxxxxxxxxxxxxx>
        *To:* TOYODA Eizi <mailto:toyoda@xxxxxxxxxxxxxx>
        *Cc:* John Caron <mailto:caron@xxxxxxxxxxxxxxxx> ; THREDDS
        community <mailto:thredds@xxxxxxxxxxxxxxxx> ; Java NetCDF
        *Sent:* Thursday, November 28, 2013 2:33 AM
        *Subject:* Re: [thredds] Help with GRIB encoding - what is
        "reference time of data" ??

        I have been looking at grib2, and was hoping I could get some
        help on some lower-level metadata questions.

        1) wmo code tables have version numbers which are also in the
        data records -- is it sufficient to use a code table that has
        version number greater than or equal to the number in the
        record, or do we have to match the version number in the record?
          WMO is on version 12, my test data of interest is version 1, I
        would much rather have a single latest-and-greatest that keep
        all twelve-and-counting versions.

        2) there are xml versions of the wmo code tables, which can be
        converted into something more programmatically useful.  Is there
        a standard set of converted tables, i.e. XML with clean
        discipline, category, version coding?    Can the same be said of
        local tables, e.g. NCEP for starters?

        3) my understanding so far is that to identify a scientific
        parameter (a.k.a standard_name) in grib we need wmo table
        version or local table version, data source (if local table),
        discipline, category, parameter number, and where (and if) we
        find this info depends on the PDS template number.
        NCEP maps this to short names, though WMO does not. And you are
        presumably mapping this to standard_names as well.   Are these
        mappings available, i.e. in XML format?

        On Wed, Nov 27, 2013 at 5:05 AM, TOYODA Eizi
        <toyoda@xxxxxxxxxxxxxx <mailto:toyoda@xxxxxxxxxxxxxx>> wrote:

            Hi John,

            Yes JMA still issues GRIB edition 1 for:
            - bulletins sent to GTS since long time ago, and
            - the reanalysis project JRA-25 and JRA-55 (coming).
            In the reanalysis there are some products like monthly
            average, for which
            multiple model runs are used in a single message.  In that
            case my
            understanding is that the reference time is set to the
            beginning of the time

            Situations are similar in GRIB2.  I'm hoping the ECMWF
            reanalysis will start
            using GRIB2 sometime.  I'm afraid I don't know actual data
            using "Verifying
            time of forecast" (CT1.2:2).

            The reason why GRIB distinguishes "Analysis" time (CT1.2:0)
            and "Start of
            forecast" (CT1.2:1) might be rather historic nowadays.  It
            has a root in the
            difference between data, i.e. analysis and initialized
            analysis (sometimes
            called data for ft=0).

            Analysis is a best estimate of the state of the atmosphere,
            made from
            observation and past numerical forecast.  In the past NWP
            centers used
            methods called nudging or optimal interpolation (the latter
            is same as what
            geographers call kriging).  The bad thing is the result may
            inbalance between wind and pressure, which causes unnatural
            gravity wave in
            the forecast.  So people had to remove such inbalance after
            analysis.  That
            is the "initialization" heavily mentioned in GRIB edition 1.

            Recent NWP systems use variational assimilation, which has
            less such
            problem.  So the analysis simply becomes the initial field
            without much
            changes.  I don't think somebody still issues "initialized
            separated from "(uninitialized) analysis".  In GRIB edition
            2, the concept
            remains in a footnote in the Code table 1.4, but usually the
            analysis and
            forecast are put in a single GRIB message for Forecast
            products (CT1.4:1).
            In short, people no longer cares.


            ----- Original Message ----- From: "John Caron"
            <caron@xxxxxxxxxxxxxxxx <mailto:caron@xxxxxxxxxxxxxxxx>>
            To: "Eizi TOYODA" <toyoda@xxxxxxxxxxxxxx
            Cc: "THREDDS community" <thredds@xxxxxxxxxxxxxxxx
            <mailto:thredds@xxxxxxxxxxxxxxxx>>; "Java NetCDF"
            Sent: Wednesday, November 27, 2013 12:09 AM
            Subject: Re: [thredds] Help with GRIB encoding - what is
            "reference time of data" ??

                Hi Eizi:

                Thanks for clarifying that terminology, I see that my
                understanding has been a bit fuzzy. I think I do indeed
                mean "Start of forecast", not the actual time the model
                was run.

                Does your agency still use GRIB-1? If so, do you ever
                code anything other than the "Start of forecast" in the
                reference time ?

                If you use GRIB-2, do you set "Significance of reference
                time" equal to 1 = "Start of forecast" ?

                Finally, what in your opinion is the meaning of the
                other "Significance of reference time" codes, esp how
                does "Analysis" differ from "Start of forecast" ?


                On 11/26/2013 2:23 AM, Eizi TOYODA wrote:

                    Hi John,

                    I'm not sure what do you mean by "run time".

                    If you want the date/time at which the forecast
                    model started, I don't
                    think GRIB contains it.

                    If you mean the initial time of forecast model, that
                    is "Start of
                    forecast" and it is really common to use this as
                    reference time.

                    Eizi TOYODA: Japan Meteorological Agency
                    Associate member of WMO/CBS/OPAG-ISS/IPET-DRMM

                    Best Regards,
                    Eiji (aka Eizi) TOYODA

                    On Sat, Nov 23, 2013 at 3:03 AM, John Caron
                    <caron@xxxxxxxxxxxxxxxx <mailto:caron@xxxxxxxxxxxxxxxx>
                    <mailto:caron@xxxxxxxxxxxxxxxx>__>> wrote:

                         The question is whether the CDM can assume that
                    GRIB "Reference time
                         of data" is the "run time" of a forecast model.

                         In GRIB-1 docs, in the PDS there is:

                         "Reference time of data – date and time of
                    start of averaging or
                         accumulation period"

                         In GRIB-2 in Identification section, there is:

                         12 Significance of reference time (see Code
                    table 1.2)

                         Reference time of data:

                         13–14 Year (4 digits)
                         15 Month
                         16 Day
                         17 Hour
                    18 Minute
                         19 Second

                         And Code table 1.2 has the following:

                         Code Table Code table 1.2 - Significance of
                    reference time (1.2)
                      0: Analysis
                              1: Start of forecast
                              2: Verifying time of forecast
                              3: Observation time
                             -1: Reserved
                       -1: Reserved for local use
                            255: Missing

                         None of this obviously refers to "run time",
                    although I suspect
                         that's how many centers use it. However, it
                    appears that when you
                         want to define a time interval, say "average of
                    the temperature,
                    starting 12 hours and ending 24 hours from
                    reference, you may use
                         the reference time to define the start of that
                    interval. In which
                         case, its not the runtime. Im hoping thats not
                    the case, that
                         reference time is the same as the run time for
                    forecast models.

                         So if you know how to interpret these for any
                    or all datasets,
                         please send me a note, or post to this group.
                    Please pass this
                    question on to anyone who might be willing to



                    thredds mailing list
                    For list information or to unsubscribe,  visit:

            thredds mailing list
            thredds@xxxxxxxxxxxxxxxx <mailto:thredds@xxxxxxxxxxxxxxxx>
            For list information or to unsubscribe,  visit:

        Dr. M. Benno Blumenthal benno@xxxxxxxxxxxxxxxx
        International Research Institute for climate and society
        The Earth Institute at Columbia University
        Lamont Campus, Palisades NY 10964-8000   (845) 680-4450

Dr. M. Benno Blumenthal benno@xxxxxxxxxxxxxxxx
International Research Institute for climate and society
The Earth Institute at Columbia University
Lamont Campus, Palisades NY 10964-8000   (845) 680-4450

  • 2013 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the netcdf-java archives: