[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[TIGGE #YJF-169907]: NCEP parameters.

Doug,

The format of the CONDUIT LDM patterns is shown here:
http://www.unidata.ucar.edu/data/conduit/ldm_idd/index.html

The sequence number of each grib product within the original data file is the
6 digit number at the far right following the second "!" character.
This will match in the status file as well.

On motherlode, I am filing each grib product such as:
CONDUIT ^(data/nccf/.*) !(.*)! (.*)
        FILE    -close  data/pub/native/conduit/\1/\3

for a single file, that would be
        FILE     data/pub/native/conduit/\1

and the status files such as:
CONDUIT ^.status\.(.*) [0-9][0-9][0-9][0-9][0-9][0-9]
        FILE    -close  data/pub/native/conduit/status/\1

To just log each sequence number, you could do something as simple as:
CONDUIT ^(data/nccf/.*) !(.*)! (.*)
        EXEC     util/conduit_sequence_log \1 \3

where util/conduit_sequence_log could be a script such as:
#!/bin/csh -f

if ( $#argv != 2 ) then
   echo "Incorrect number or arguments" | \
        logger -t "$0 [$$]" -p local0.notice
endif

set LOGFIL="data/conduit/log/$1"

if ( ! -e $LOGFIL:h ) mkdir -p $LOGFIL:h
echo $2 >>! $LOGFIL
exit 0
-------------------------------------------------------------------------


Steve Chiswell
Unidata User Support

> How do I log the sequence numbers?
> 
> On Wed, 04 Oct 2006 10:41:51 -0600
> "Unidata IDD TIGGE Support"
> <address@hidden> wrote:
> >> Hello,
> >>
> >> Is there an indicator that is sent at the completion of
> >>a conduit
> >> send of NCEP's ensemble data?
> >
> > Doug,
> >
> > Each data set that is inserted into CONDUIT has an
> >accompanying status file
> > that provides the inventory of products that should be
> >expected. The
> > status file sequence number will be the last in the data
> >set. However,
> > the LDM does not guarantee what order you will receive
> >those products, in particular
> > beacuse redundant request lines and delivery paths are
> >possible. So the
> > best way to determine that all GRIB pieces ofthe
> >original data file have been received
> > is to log the sequence numbers received for each
> >forecast file that is sent.
> >
> > On motherlode, I actually file each product to a
> >separate file so that users could
> > obtain a single product if needed- which makes an easy
> >check for completeness
> > which I provide through a web cgi:
> > http://motherlode.ucar.edu/cgi-bin/ldm/conduit_reception_new.csh
> >
> > The cgi is a little slow for interactive web beacuse its
> >checking all files on the fly (and there are a lot of
> >GEFS files), but definitely tollerable for a process to
> >check reception on reception of the status file.
> > Green on the file name shows that all sequence numbers
> >have been received. You can click through
> > to the status and individual member download.
> >
> >
> >> This would be helpful in the initiation of further
> >>processing.  Also,
> >> would it be possible to send
> >> UREL and VREL in separate grib messages of NCEP ensemble
> >>data?
> >
> > The data on CONDUIT is just what NCEP is producing. No
> >alteration to the GRIB
> > products are made by the LDM. The GRIB2 standard does
> >provide for multiple parameters
> > to be included within a single grib product, and NCEP is
> >using that for the
> > U and V wind components (not just the ensembles but GFS
> >files are that way too).
> >
> >
> >>
> >> Thanks
> >> Doug
> >>
> >>
> >
> >
> > Steve Chiswell
> > Unidata User Support
> >
> > Ticket Details
> > ===================
> > Ticket ID: YJF-169907
> > Department: Support CONDUIT
> > Priority: Normal
> > Status: Closed
> >
> 
> 


Ticket Details
===================
Ticket ID: YJF-169907
Department: Support CONDUIT
Priority: Normal
Status: Closed