[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [LDM #ZGC-437176]: Dealing with unreadable GRIB files

Hi all:

The short answer is that version 4.3 makes everything ok. The two files look identical, because duplicates are ignored. 4.2 is broken and we cannot support it anymore for GRIB files.

As to why the ldm is getting duplicates, likely the pqact is the problem.

John

On 2/23/2013 6:46 AM, Signell, Richard wrote: 
John,
Welcome back!
Wow, right from a great vacation into GRIB2 issues?
Could there be anything worse?

1. This GRIB2 downloaded from the Unidata TDS can be accessed through OPeNDAP:
<http://geoport.whoi.edu/thredds/catalog/usgs/data0/ldm/data/pub/native/grid/NCEP/catalog.html?dataset=usgs/data0/ldm/data/pub/native/grid/NCEP/GFS_Global_0p5deg_20130222_0000.grib2>

  2. This GRIB2 from our LDM can NOT be accessed through OPeNDAP:
<http://geoport.whoi.edu/thredds/catalog/usgs/data0/ldm/data/pub/native/grid/NCEP/catalog.html?dataset=usgs/data0/ldm/data/pub/native/grid/NCEP/ldm_GFS_Global_0p5deg_20130222_0000.grib2>

If I look at the first file with "wgrib2", eveything looks good:

$ wgrib2 ./tds/GFS_Global_0p5deg_20130222_0000.grib2 | grep 'UGRD:10 m'

221.1:40897348:d=2013022200:UGRD:10 m above ground:anl:
511.1:93268362:d=2013022200:UGRD:10 m above ground:3 hour fcst:
891.1:163581923:d=2013022200:UGRD:10 m above ground:6 hour fcst:
1210.1:221082087:d=2013022200:UGRD:10 m above ground:9 hour fcst:
1529.1:279345269:d=2013022200:UGRD:10 m above ground:12 hour fcst:
...
19886.1:3633573232:d=2013022200:UGRD:10 m above ground:174 hour fcst:
20222.1:3695640054:d=2013022200:UGRD:10 m above ground:189 hour fcst:
20669.1:3775293846:d=2013022200:UGRD:10 m above ground:192 hour fcst:

But the 2nd one retrieved by LDM from the IDD doesn't proceed nicely
from 0 to 192 hours -- there are a bunch of duplicated records
inserted throughout the file, causing the time stamp to not proceed
monotonically, and resulting in a much bigger file.  This one can't be
read via the opendap link on TDS (and therefore produces a zero length
grib index file).
  on 4.2.9

$ wgrib2 ./ldm/GFS_Global_0p5deg_20130222_0000.grib2 | grep 'UGRD:10 m '

202.1:36706943:d=2013022200:UGRD:10 m above ground:anl:
511.1:93268362:d=2013022200:UGRD:10 m above ground:3 hour fcst:
891.1:163581923:d=2013022200:UGRD:10 m above ground:6 hour fcst:
1210.1:221082087:d=2013022200:UGRD:10 m above ground:9 hour fcst:
1515.1:276755666:d=2013022200:UGRD:10 m above ground:12 hour fcst:
...
  9779.1:1793637231:d=2013022200:UGRD:10 m above ground:90 hour fcst:
10059.1:1844502447:d=2013022200:UGRD:10 m above ground:93 hour fcst:
10470.1:1920009884:d=2013022200:UGRD:10 m above ground:96 hour fcst:
10686.1:1959836180:d=2013022200:UGRD:10 m above ground:anl:                 <==
10995.1:2016397599:d=2013022200:UGRD:10 m above ground:3 hour fcst:
11527.1:2114541141:d=2013022200:UGRD:10 m above ground:6 hour fcst:
11801.1:2163994314:d=2013022200:UGRD:10 m above ground:99 hour fcst:
12013.1:2202639475:d=2013022200:UGRD:10 m above ground:9 hour fcst:    <==
12332.1:2260902657:d=2013022200:UGRD:10 m above ground:12 hour fcst:
...
34575.1:6322545271:d=2013022200:UGRD:10 m above ground:180 hour fcst:
34951.1:6390316958:d=2013022200:UGRD:10 m above ground:183 hour fcst:
35240.1:6443194232:d=2013022200:UGRD:10 m above ground:186 hour fcst:
35447.1:6479653290:d=2013022200:UGRD:10 m above ground:174 hour fcst: <==
35780.1:6541465044:d=2013022200:UGRD:10 m above ground:189 hour fcst:
36227.1:6621118836:d=2013022200:UGRD:10 m above ground:192 hour fcst:

Thanks for looking into this,
Rich


On Fri, Feb 22, 2013 at 7:14 PM, John Caron <address@hidden> wrote: 
Hi Rich, can you make the zero length  problem file available to us and/or
the 6.2 vs 3.8 Gb GRIB file?

PS, cant tell who the original questioner is.

John


On 2/22/2013 4:49 PM, Unidata IDD Support wrote: 

Rich,

I'm afraid I know very little about GRIB2 files. So I've forwarded this
reply to some people who know much more than I.

Guys,

The LDM grib file is 6.2GB, while if I download the grib2 file from the
TDS:
wget
http://motherlode.ucar.edu:8080/thredds/fileServer/fmrc/NCEP/GFS/Global_0p5deg/files/GFS_Global_0p5deg_20130222_0000.grib2

it is only 3.8GB.

What the heck?

I guess I need "wgrib" or something to check these out?

-Rich


On Fri, Feb 22, 2013 at 6:31 PM, Unidata IDD Support
<address@hidden>  wrote:


Rich,

In the past 9 months we have had 22 files NAM files and 33 GFS files
that the TDS was unable to read (produces 0 length GRIB index files).

When 0 length GRIB index files get created, the TDS times out on
these
datasets, and our workflow is broken.


Understandable, that is not OK!

This is an error rate of about 10%. 

Well, closer to 2% (4 runs per day), unless you are only requesting
one synoptic time per day, but even 2% is far greater than one would expect,
we see generally 99.98 reliability..if not better.


TCP unreliability is much, much less than 2%. It's far more likely that
there's something wrong with the decoder or the GRIB file.

Do you have something that can check the validity of a GRIB file?

Regards,
Steve Emmerson

Ticket Details
===================
Ticket ID: ZGC-437176
Department: Support IDD
Priority: Normal
Status: Closed



--
Dr. Richard P. Signell   (508) 457-2229
USGS, 384 Woods Hole Rd.
Woods Hole, MA 02543-1598



Regards,
Steve Emmerson

Ticket Details
===================
Ticket ID: ZGC-437176
Department: Support IDD
Priority: Normal
Status: Closed






NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.