[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[CONDUIT #SCC-464262]: CONDUIT GRIBs

Kevin,

I have posted the fh.0102 as a single file for you to download:
http://www.unidata.ucar.edu/staff/chiz/fh.0102_gfs003

The file size of the individual grib messages cat'd together is:
27074120

This is the exact size as is reported in the .status file you will find in the 
CONDUIT data stream:
/afs/.nwstg.nws.noaa.gov/ftp/SL.us008001/ST.opnl/MT.gfs_CY.12/RD.20060524/PT.grid_DF.gr1/fh.0102_tl.press_gr.on
edeg complete (27074120 bytes) at Wed May 24 16:15:33 2006
Inserted 27074120 of 27074120

As for your other comments:

1) Aside from the .status file in the data stream, you will not find the string 
"afs" in the CONDUIT
data. Assuming you aren't accidentally passing that status file to your 
decoder! Otherwise,
if you are using something like a PIPE action to an xcd_run shell script, then 
that script could be the
source of some log messages into your stream.

2) The control characters 01 0d 0d 0a equate to: SOH CR CR NL
Those are characters that you would find in the HRS data stream at the beginning
of a FOS product. The CONDUIT data does not have these characters, which signals
that you are looking at something else in addition to just CONDUIT data. If
that is the case, and you have a separate invocation to xcd_run fpr HRS and 
CONDUIT,
I was wondering if they could be stepping on each other when inserting into
your circular buffer.

3) The xcd_run script which is in use here by Tom Yoksas is just piping the HRS 
data
stream. He does not pipe the CONDUIT data, so I don't know how that would look- 
but
2 ingebin processes should be correct (the parent and child). What I
was wondering is that if your machine fell behind, if a second invocation of 
xcd_run
could be getting kicked off (if you are logging your process ids, you might look
for overlapping ids).

Note: I will be out of town through next Wednesday. Tom Yoksas is currently out 
of the office as well.
Tom is the knowledgeable person regarding XCD configuration should you have 
questions to that end.



Steve Chiswell
Unidata User Support



> Another thing
> In our GRID 3 files, there are some other characters between the 7777
> and GRIB which I have not seen before which recur in the files
> 2f 61 66 73  (in text '/afs')
> and a sequence of control characters
> 01 0d 0d 0a
> So, the three things I am seeing in our GRID 3 files is
> 7777/afsGRIB
> 7777<01 0d 0d 0a>GRIB
> 7777GRIBGRIB
> with 7777GRIBGRIB by far being the most common
> Kevin
> 
> Unidata CONDUIT Support wrote:
> 
> >>Hi,
> >>I know I have talked about this before, but in the CONDUIT grids, is it
> >>possible to have "GRIBGRIB" in the feed.
> >>So, I have a sequence of characters at the end a GRIB message and the
> >>beginning of the next one as 7777GRIBGRIB
> >>Since GRIB messages are between GRIB and 7777, this is causing problems
> >>in some of our decoding.
> >>Thanks!
> >>Kevin Baggett
> >>UW-SSEC
> >>
> >>
> >>
> >>
> >
> >Kevin,
> >
> >It would be possible to have the string GRIB within a product, but in 
> >general,
> >the beginning of the GRIB product should start with GRIB followed by the IDS 
> >block,
> >so you should not see GRIBGRIB at the beginning of a product. Any sequence
> >of 7777GRIBGRIB could theoretically occur within the GRIB as random 
> >characters.
> >
> >The program which finds the GRIB products from within the NWS ftp server 
> >files
> >is already doing a QC on the GRIB...size...7777 for each product, and is
> >reporting the sum of bytes inserted into the LDM queue as compared to the
> >NWS ftp server file being inserted in the .status file in the data stream.
> >The only time these differ would be if a duplicate product were encountered.
> >Only valid grib products within the NWS file will be able to be placed into 
> >the LDM queue.
> >
> >I store each individual GRIB product from the GFS003 stream at:
> >http://motherlode.ucar.edu/cgi-bin/ldm/conduit_reception.csh
> >If you have a time and or product in question, we can look at the complete 
> >product.
> >I grepped for "GRIBGRIB" for the 12Z GFS today, and didn't find any 
> >occurences.
> >
> >A possible problem is that your output from pqact is stepping on itself. 
> >Have you verified that you
> >don't have 2 XCD occurences? If the pqact process piping data out to XCD 
> >fills up, then you could
> >be generating a second instance of the pipe which I'm assuming is writing to 
> >your curcular buffer.
> >That would explain why a different installation you have does not have 
> >problems if the load, hardware,
> >or other variables differ.
> >
> >Steve Chiswell
> >Unidata User Support
> >
> >Ticket Details
> >===================
> >Ticket ID: SCC-464262
> >Department: Support CONDUIT
> >Priority: Normal
> >Status: Closed
> >
> >
> >
> 
> 


Ticket Details
===================
Ticket ID: SCC-464262
Department: Support CONDUIT
Priority: Normal
Status: Closed