[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

20060626: trouble converting GRIB2 products with cnvgrib (cont.)



>From: Rorik Peterson <address@hidden>
>Organization: UAF
>Keywords:  200606262015.k5QKFfSZ017377 LDM GRIB2 CONDUIT status message

Hi Rorik,

>       thanks for the quick response.

No worries.

>What you say makes sense, so I checked 
>my pqact.conf.  Here's what I have, which I tried to copy straight from 
>something I found on the web, perhaps unidata's site, but I'm not sure
>
>CONDUIT 
>MT.gfs_CY.(..)/RD.(........)/PT.grid_DF.gr([12])/(fh.....)_tl.press_gr.(......
> )
>         FILE    /home/ldm/data/gfs/GFS_Global_\5_\2_\100.grib\3

OK.

>Could this be capturing the status file because I'm missing a carrot '^' 
>at the beginning of 'MT.gfs', and the status file matches the above as 
>well?

Yes, exactly.  The regular expression you are using will match if
the pattern is found anywhere in the product ID.

You can use the LDM facility 'notifyme' to list product headers and
come up with a regular expression that does exactly what you want.  For
instance:

notifyme -vxl- -f CONDUIT -o 10000 -p "^.status"

Use of more complicated patterns like the one you have above may force
you to escape characters that get interpreted by your shell (i.e.,
be aware that you may have to escape things like parentheses, etc.).

There is no GFS data flowing in CONDUIT at the moment, but you can
familiarize yourself with the process using other fields included
in the datastream.

'notifyme' is a _very_ handy utility :-)

Cheers,

Tom

Tom Yoksas wrote:
> Hi Rorik,
> 
> re:
> 
>>GRIB2/cnvgrib gurus and other savy problem solvers,
>>
>>I'm trying to convert GRIB2 to GRIB1 files on a 64-bit Athlon linux 
>>machine using 'cnvgrib'.  I've tried 2 different types of files and 
>>succeeded with NAM Alaska grid 242, and failed with 0.5-degree GFS.  I 
>>get them both via the LDM/IDD.  With the GFS files, I actually succeed 
>>converting the first 270 or so messages; I've tried several different 
>>GFS files, and it usually stops around here.  There should be several 
>>thousand records according to 'degrib'.  I used a debugger to see if I 
>>could find out anything, although I don't know much about GRIB2.  This 
>>is what I see happening:
>>The program is generating an index of all records in the GRIB2 file.  It 
>>does this by looking for characters 'GRIB...[12]' at the beginning of 
>>each record, reads forward to where the next record should be, and looks 
>>again.  After about 270 or so records, the next record starts with 
>>'/afs/.nwstg.nws.noaa.gov/ftp/SL.us008001/ST.opnt/MT.gfs-CY' instead of 
>>'GRIB...[12]'.
>>This can't be a coincedence, being part of the product name I get from 
>>the IDD.
> 
> 
> This seems to indicate that you are filing the ASCII status file
> sent in the CONDUIT stream along with the binary GRIB2 messages into
> a single output file.  You can learn more about the catalog file
> in the Unidata web page on CONDUIT data:
> 
> Unidata HomePage
> http://www.unidata.ucar.edu/
> 
>   Data
>   http://www.unidata.ucar.edu/data/
> 
>     CONDUIT
>     http://www.unidata.ucar.edu/data/
> 
>       What is CONDUIT?
>       http://www.unidata.ucar.edu/data/conduit/index.html
> 
>         CONDUIT
>         http://www.unidata.ucar.edu/data/conduit/ldm_idd/index.html
> 
> Here is the relevant comment in the last URL:
> 
>   Since the data is injected as individual products, a status file
>   consisting of the injection time, original file size and data file
>   name is inserted at the end of the product sequence, with the name:
>   .status.ncep_file_name end_sequence  The status file will contain a
>   catalog of each individual LDM product identifier within the original
>   data file, as well as the insertion status code returned
>   by the LDM.
> 
> Please check your ~ldm/etc/pqact.conf action(s) for CONDUIT data
> and make sure that you process the status file differently than
> the GRIB/GRIB2 messages.
> 
> 
>>I don't know if this is suppose to be appearing within the 
>>GRIB2 file or not? (seems like it shouldn't, a waste of space, but I 
>>don't know the GRIB2 spec that well.)
> 
> 
> It isn't.
> 
> 
>>At this point, the program stops 
>>generating an index, and only processes these 270 or so records.
> 
> 
> This is consistent with my theory that the status file is being written
> in the single output file.
> 
> 
>>Being curious, I crudely scanned the entire 1.2GB file and matched for 
>>the 13 character '/afs/.nwstg.n' pattern, and it only begins to appear 
>>at the approximate point in the GRIB2 file that record 270 is, and 
>>occurs several thousand times after that.  Again, not a coincedence, I 
>>figure (although I haven't done the math for a 13 byte sequence in 1.2GB).
>>
>>At this point, I'm stuck.  Is the problem with 'cnvgrib', the GRIB2 
>>files, my LDM, or perhaps something else?
> 
> 
> The problem is in the ~ldm/etc/pqact.conf action you are using to FILE
> the GRIB/GRIB2 products.
> 
> 
>>I reproduced this problem with a GRIB2 file I downloaded straight from 
>>Unidata's motherlode http portal, suspecting problems with my LDM.
> 
> 
> This must have been one of the status files.

--
NOTE: All email exchanges with Unidata User Support are recorded in the
Unidata inquiry tracking system and then made publicly available
through the web.  If you do not want to have your interactions made
available in this way, you must let us know in each email you send to us.

>From address@hidden  Tue Jun 27 19:04:54 2006

>thanks, that did it.  You guys rock.

>rorik