[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Happy Friday!



Hi Jack:

CFSR has known problems, which we surmise happened when the original GRIB1 files were translated into GRIB2. We are looking for ways to fix these problems on the server, assuming that they arent going to fix them at the source. But it would be certainly better if they were rewritten correctly.

John

PS: I seemed to have lost this file, and its gone on the the ftp site. can you send me another link?

On 10/11/2012 11:45 AM, Jack Settelmaier wrote:
Sorry for not responding sooner helpful gents, as I see in this thread I
was asked a question.

In looking at the below PDS info, I see no valid reason why they would
have this set as they do:
Grib2ProductDefinitionSection
  Interval     = (2012-08-27T00:00:00.000Z,2012-09-27T00:00:00.000Z)

The file is/was a climate model run that ran on
2012-09-27T00:00:00.000Z, with forecast conditions that go out MONTHS in
advance. I see no good reason for them to be referencing
2012-08-27T00:00:00.000Z--that looks like a flat out error.

It seems to me they need to encode that setting as you surmised it
should be:
(2013-05-01T00:00:00.000Z, 2013-06-01T00:00:00.000Z)

So, should I attempt to find the "creator" of this GRIB to get this
corrected, rather than you making changes to your system which would
work if they encoded it properly?

On 9/28/2012 2:19 PM, John Caron wrote:
Hi all:

1. The CFSR dataset is known to have grib encoding issues, so its
possible we are seeing some of that and/or bugs in the software. in
particular, the translation of grib1 to grib2 had bugs in it. ive been
unable to get authoritative info on how to fix this, but i do have
machinery to do so if we can figure out what to do.

2. The 4.3 software is completely different from 4.2. We are only
going to fix 4.3 grib problems. 4.2 doesnt handle time interval
coordinates correctly.

3. In this file:

ftp://ftp.ncep.noaa.gov/pub/data/nccf/com/cfs/prod/cfs/cfs.20120927/00/monthly_grib_01/flxf.01.2012092700.201305.avrg.grib.grb2

I am seeing records like the one I have dumped below.

statistic type is "Average"

reference time is 2012-09-27T00:00:00.000Z

the "end of overall time interval" is  == 2012/09/27T0:0:0

forecast time is 8 months.

we are interpreting the time interval coordinate as:

 (0.000000, 1.000000) == (2012-08-27T00:00:00.000Z,
2012-09-27T00:00:00.000Z)

which seems wrong. ill have to look at this closer.

4. Note that this is an interval coordinate, meaning the "valid
time"is a time interval. From this comment:

"The forecasts from that model run are for the mean conditions over
the months of May 2013"

Im guessing the correct interval should be:

 (0.000000, 1.000000) == (2013-05-01T00:00:00.000Z,
2013-06-01T00:00:00.000Z)

so:
  - we do have a bug
  - the dates are encoded wrong
  - have to figure out how to fix systematically. if theres a way to
derive the correct answer from the info in the PDS below, can you see
it? It will be unfortunate if we have to rely on the filename.



---------

File=0 E:/data/work/ansari/flxf.01.2012092700.201305.avrg.grib.grb2
Header=""

Grib2IndicatorSection
 Discipline = (0) Meteorological products
 Length     = 64370

Grib2IdentificationSection
 Center        = (7) US National Weather Service, National Centres for
Environmental Prediction (NCEP)
 SubCenter     = (0) null
 Master Table  = 2
 Local Table   = 1
 RefTimeSignif = 1 (Start of forecast)
 RefTime       = 2012-09-27T00:00:00.000Z
 RefTime Fields = 2012-9-27 0:0:0
 ProductionStatus      = 0 (Operational products)
 TypeOfProcessedData   = 1 (Forecast products)

Grib2GridDefinitionSection hash=-1824839391 crc=4229097007
 Length             = 72
 Source  (3.0)      = 0 (Specified in Code table 3.1)
 Npts               = 72960
 Template (3.1)     = 40

(3.40) Grid definition template 3.40 - Gaussian latitude/longitude
1: GDS length == 72
5: Section == 3
  6:                                             Source of Grid
Definition (see code table 3.0) == 0 (table 3.0: Specified in Code
table 3.1)
7: Number of data points == 72960
 11:                                             Number of octects for
optional list of numbers == 0
 12: Interpretation of list of numbers == 0 (table 3.11: There is no
appended list)
 13: Grid Definition Template Number == 40
 15: Shape of the Earth == 6 (table 3.2: Earth assumed spherical with
radius of 6 371 229.0 m)
 16: Scale factor of radius of spherical Earth == 0
 17: Scaled value of radius of spherical Earth == 0
 21:                                        Scale factor of major axis
of oblate spheroid Earth == 0
 22:                                        Scaled value of major axis
of oblate spheroid Earth == 0
 26:                                        Scale factor of minor axis
of oblate spheroid Earth == 0
 27:                                        Scaled value of minor axis
of oblate spheroid Earth == 0
 31: Ni - number of points along a parallel == 384
 35: Nj - number of points along a meridian == 190
 39: Basic angle of the initial production domain == 0
 43: Subdivisions of basic angle used to define extreme longitudes and
latitudes, and direction increments == 0
 47: La1 - latitude of first grid point == 89277000
 51: Lo1 - longitude of first grid point == 0
 55: Resolution and component flags == 48
 56: La2 - latitude of last grid point == -89277000
 60: Lo2 - longitude of last grid point == 359062000
 64: Di - i direction increment == 938000
 68:                                     N - number of parallels
between a pole and the Equator == 95
 72: Scanning mode == 0
 73:                                  List of number of points along
each meridian or parallel. == -9999

Grib2ProductDefinitionSection
 Interval     = (2012-08-27T00:00:00.000Z,2012-09-27T00:00:00.000Z)

(4.8) Product definition template 4.8 - average, accumulation and/or
extreme values or other statistically-processed values at a horizontal
level or in a horizontal layer in a continuous or non-continuous time
interval
1: PDS length == 58
5: Section == 4
  6: Number of coordinates values after Template == 0
8: Product Definition Template Number == 8
 10: Parameter category == 2
 11: Parameter number == 17
 12: Type of generating process == 2 (table 4.3: Forecast)
 13:                   Background generating process identifier
(defined by originating centre) == 0 (table ProcessId: Table ProcessId
code 0 not found)
 14:         Analysis or forecast generating process identifier
(defined by originating centre) == 98 (table ProcessId: Table
ProcessId code 98 not found)
 15: Hours after reference time of data cutoff == 0
 17: Minutes after reference time of data cutoff == 0
 18: Indicator of unit of time range == 3 (table 4.4: Month)
 19: Forecast time in units defined by octet 18 == 8
 23: Type of first fixed surface == 1 (table 4.5: Ground or water
surface)
 24: Scale factor of first fixed surface == 0
 25: Scaled value of first fixed surface == 0
 29: Type of second fixed surface == 255 (table 4.5: Missing)
 30: Scale factor of second fixed surface == 0
 31: Scaled value of second fixed surface == 0
 35: Year - time of end of overall time interval == 2012
 37: Month - time of end of overall time interval == 9
 38: Day - time of end of overall time interval == 27
 39: Hour - time of end of overall time interval == 0
 40: Minute - time of end of overall time interval == 0
 41: Second - time of end of overall time interval == 0
 42: n - number of time range specifications describing the time
intervals used to calculate the statistically-processed field == 1
 43:                                 Total number of data values
missing in statistical process == 0
 47: Statistical process used to calculate the processed field from
the field at each time increment during the time range == 0 (table
4.10: Average)
 48:        Type of time increment between successive fields used in
the statistical processing == 2 (table 4.11: Successive times
processed have same start time of forecast, forecast time is incremented)
 49:         Indicator of unit of time for time range over which
statistical processing is done == 3 (table 4.4: Month)
 50: Length of the time range over which statistical processing is
done, in units defined by the previous octet == 1
 54:             Indicator of unit of time for the increment between
the successive fields used == 255 (table 4.4: Missing)
 55:           Time increment between successive fields, in units
defined by the previous octet == 0
 59:                                      As octets 47 to 58, next
innermost step of processing == -9999
 71: Additional time range specifications, included in accordance with
the value of n. Contents as octets 47 to 58, repeated as necessary ==
-9999

Grib2SectionDataRepresentation
  Template           = 40 (Grid point data - JPEG 2000 code stream
format)
  NPoints            = 72960

Grib2SectionData
  Starting Pos       = 196
  Data Length        = 64170


On 9/28/2012 11:45 AM, Steve Ansari wrote:
Hi Jack,

Thanks for bringing this up.  It appears that the GRIB decoders in
the NetCDF for Java library are not detecting the time dimension
correctly.  20120927 should be the runtime and 201305 should be the
time, but the runtime date is showing up as the time dimension and
the valid time of the model is not present.

I'm cc-ing John Caron and Robb at Unidata who are the GRIB masters at
Unidata.

I'm using NCJ 4.2 in the Weather and Climate Toolkit, but I tested
several files with NCJ 4.3 and got the same results.
ftp://ftp.ncep.noaa.gov/pub/data/nccf/com/cfs/prod/cfs/cfs.20120927/00/monthly_grib_01

John, Robb, thanks for your help!

Thanks!
Steve




On Fri, Sep 28, 2012 at 1:03 PM, Jack Settelmaier
<address@hidden <mailto:address@hidden>> wrote:

    I'm poking around with displaying output from the climate (CFS)
    model, and it's forecasts of mean monthly conditions.

    So, in the below files, one gets the CFS model output from the
    Sep 26, 2012 18Z model run.  The forecasts from that model run
    are for the mean conditions over the months of May and June,
    2013, respectively.
    
ftp://ftp.ncep.noaa.gov/pub/data/nccf/com/cfs/prod/cfs/cfs.20120927/00/monthly_grib_01/flxf.01.2012092700.201305.avrg.grib.grb2
    
ftp://ftp.ncep.noaa.gov/pub/data/nccf/com/cfs/prod/cfs/cfs.20120927/00/monthly_grib_01/flxf.01.2012092700.201306.avrg.grib.grb2

    When I Save a single KMZ after viewing each dataset, your
    software apparently assigns a "time stamp" based on first date in
    the filename, or from pulling the model initial time, NOT the
    valid time as I'd like.   So what I end up with in these two files
    http://www.srh.noaa.gov/rtimages/srh/stsd/Jack/CFS_500HAnom4.kmz
    http://www.srh.noaa.gov/rtimages/srh/stsd/Jack/CFS_500HAnom5.kmz

    are two images that have the same "time" in Google, even though
    their valid time is a month apart.  Maybe if I made a loop over
    the 2 grib files it would have inserted a "time step" that would
    allow for some sort of animation in Google, but I'm still not
    sure it would help show me the valid time.

    This is pretty specific to the CFS files and their nuanced
    forecasts and such, but is there something that could be done to
    better the result?