[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

19990511: Conduit AVN data feed

William,

I verified that you are receiving this mornings AVN data.
The request line that you are using should be fine...it
uses the same regular expression syntax as sed/awk etc, and
"^.*gblav.*" says:
^ starting with
.* anything up to  and including gblav 


The AVN data is sent 2x per day. Right now at 15:50Z the 12Z
run is starting to be posting. The reason you didn't see anything last night
at 22Z is that there was no AVN data within the previous hour to download.
The queue here holds data for 1 hour.

I checked the data arriving at your site with the command:
notifyme -v -l - -o 3600 -h mahi.mhpcc.edu

The notifyme program tells me about products arriving from the past
-o seconds (3600 seconds = 1 hour). You can use that command
locally or remotely to see products that arrived. Locally on the machine with
the ldm.pq file, you can use the "ldmadmin watch" command to see procucts
as the arrive.

All of the NMC2 data stream products will have the same naming comvention,
which is the file name of the product on NCEPs computer followed by an
identifier enclosed in "!" characters which identify the individual
grib product. We distribute the products separated as the individual pieces
from the file rather than the 20Mb aggregate files that NCEP posts
for several reasons- the most important is so that you can key on
certain fields in the pqact processing, and/or request only certain
fields of the data to save on bandwith. It is also easier to send
smaller packets than singular large files.

So, now that you have data arriving, you want to do something with it.
As you mentioned, the pqexpire program will periodically clear
data out of the ldm.pq queue if it is older than 1 hour.

The pqact program uses the pqact.conf configuration file to 
decide what to do with data as it arrives. You can FILE the
data to disk, as well as PIPE it to standard input of programs,
or use the EXEC action to run a program when a certain piece
of data arrives.

I see that Robb has also sent you a reply as I am typing.
The pqact.conf entries to FILE the avn data into forecast
files that resemble the aggregate files that are posted on NCEPs
ftp server are at:
http://www.unidata.ucar.edu/staff/chiz/ncep_idd/gblav_files.html

Each AVN run will take ~500-600mb of space to file on disk,
so you will want to scour old data that you save.
The AVN runs post output through F084 (84 hour forecasts).
We can adjust your request pattern to narrow down the amount
of data being transmitted if you don't need data that far out.

I'll let you digest this message (and Robb's) and you can let
me know how you want to proceed to restrict data, of set up
processing etc from pqact.conf.

One bit of advice for the pqact.conf file is that TAB characters
are important, eg:
NMC2 <tab> avn/avn.(......)/gblav.T(..)Z.PGrbF(.*) !.*!
     <tab> FILE <tab> data/avn/\1\2_PGrbF\3.grib

The NMC2 string is the data feed identifier. The next field
is the pattern to match using regular expression syntax.
The fields enclosed in parentheses allow you to use the \1, \2 etc
escape sequences to refer to that part of the product name in later
output file names. The second line is refered to as the "action"
to take when a pattern in the first line is matched. In this case
FILE is the action, and the file location/name is given as the
argument. By default, data is appended using the FILE tag, so
all data matching the pattern will be filed into an accumulating file.

I know its a lot to digest- we're here to help.

Steve CHiswell
Unidata User SUpport




>From: William Smith <address@hidden>
>Organization: .
>Keywords: 199905112343.RAA23440

>Steve,
>
>
>
>> In your ~ldm/etc/ldmd.conf file, you should create request entry:
>> request                NMC2    ".*" flip.unidata.ucar.edu
>
>This generated more traffic than I realized would occur.  Data
>started pouring in like crazy.  Sometimes there were 4000+
>products in the queue.  I discovered I could see what was there
>with the pqcat routine.  There didn't seem to be any problem
>with overflow so the pqexpire must have been working ok.
>
>I saw lots of records for ETA and RUC2, but nothing for AVN.  
>It probably wasn't time, it was about 22:00 GMT.
>
>
>
>> if you want  just the AVN grid #003 data set, then use the line:
>> request      NMC2    "gblav" flip.unidata.ucar.edu
>
>I stopped ldm, commented out the ".*" request, and activated the above
>line the restarted ldm.  It hung up and I found this error in the log file:
>
>May 10 22:20:19 5Q:mahi rpc.ldmd[36693]: Starting Up (built: Nov  5 1998 15:17
> :15)
>May 10 22:20:19 3Q:mahi rpc.ldmd[36693]: /u/ldm/etc/ldmd.conf: line 23: Syntax
>  error
>
>Line 23 is the "gblav" line.  I changed it to "^.*gblav.*" and restarted
>ldm.  Of course I don't know what I'm doing, but at least ldm started.
>As far as I can tell I didn't get any AVN data in.  If you want I can
>send you part of the log where there are RECLASS and skip message from
>flip and Recycled messages from pqexpire.  At the end of the queue it
>now says:
>
>May 11 18:19:04 pqcat: Number of products 0
>
>Where it had 3496 products when I stopped ldm with the ".*" request.
>
>
>
>QUESTIONS:
>
>1.  What do I need to specify to get the AVN data?
>
>2.  Should I use a FILE command to move the product to a file?  
>
>3.  Will the AVN be formated like the ETA products with one product for
>    each level and variable?
>
>4.  Should I activate the dostats program in a crontab?
>
>5.  If I use the FILE should I also use the scour in a crontab to clean
>    out the file area?
>
>
>Sorry for all the questions.  I know this is "old hat" for you, but
>it's really new for me.  I've been reading the LDM5 Site Managers
>Guide and the on-line workshop to figure out how to do this.  Your
>email has been a big help.  Thanks.
>
>
>
>
>
>William
>
>_____________________________________________________________________________
>William Smith                             address@hidden
>Maui High Performance Computing Center    (808) 879-5077  
>550 Lipoa Parkway                         (808) 879-5018 (fax)
>Kihei, HI  96753                          WWW: http://www.mhpcc.edu
>_____________________________________________________________________________
>