[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[IDD #WFA-619667]: Request for CONDUIT feed



Hi Martha,

re:
> We have examined the output of the notifyme generated over a 24 hour
> period.

Very good.

> We wish to collect only one CONDUIT product for now:
> 
> Global GEFS 1p0deg Ensemble data. Ie., all the gefs.* data in the log
> file.

OK.

> The examples you gave below for the edits to the ldmd.conf file to
> collect and process CONDUIT are for the entire feed, and we don't understand
> how that would translate into selecting only one product for ingesting.
> Could you give us a more detailed example?

This is a situation where 'notifyme' is once again your best friend.  Here
is what to do:

1) by reviewing the log file of CONDUIT data products, figure out when the
   products of interest are likely to still be in the LDM queue on the
   upstream machine

2) again, by reviewing the log file for CONDUIT data products, review the 
product
   IDs for the products you want to request.  Here are some examples from a
   log file of CONDUIT data that I collected some time ago:

Oct 24 18:22:49 notifyme[24123] INFO:    52582 20081024164116.711 CONDUIT 004  
data/nccf/com/gens/prod/gefs.20081024/12/pgrb2a/gep12.t12z.pgrb2aanl 
!grib2/ncep/SPEC62MRF/#000/200810241200F000/HGHT/250 Pa PRES! 000004
Oct 24 18:22:49 notifyme[24123] INFO:    21404 20081024164116.711 CONDUIT 005  
data/nccf/com/gens/prod/gefs.20081024/12/pgrb2a/gep12.t12z.pgrb2aanl 
!grib2/ncep/SPEC62MRF/#000/200810241200F000/TMPK/250 Pa PRES! 000005
Oct 24 18:22:49 notifyme[24123] INFO:    31950 20081024164116.719 CONDUIT 014  
data/nccf/com/gens/prod/gefs.20081024/12/pgrb2a/gep12.t12z.pgrb2aanl 
!grib2/ncep/SPEC62MRF/#000/200810241200F000/RELH/700 Pa PRES! 000014

   Notice that all of the GEFS products have the string 'gefs' included in the
   Product IDs...

3) fashion an extended regular expression that matches just the products you
   want to get.  In the GEFS case, this is very easy -- simply use 'gefs'.

4) run 'notifyme' using the extended regular expression you just fashioned
   as the value of the '-p' (pattern) flag:

   notifyme -vl- -f CONDUIT -h idd.cise-nsf.gov -o 10000 -p gefs

If the GEFS products are in the LDM queue on the upstream machine 
(idd.cise-nsf.gov)
you should get a listing of all products whose Product IDs match the very simple
extended regular expression 'gefs'.  If you need to fashion a more complex 
regular
expression, you will likely need to enclose it in quotes in the 'notifyme' 
invocation
command line.  For instance, if you were interested in all products whose 
Product IDs
contain either 'gefs' or 'sref', you would use:

notifyme -vl- -f CONDUIT -h idd.unidata.ucar.edu -o 10000 -p '(gefs|sref)'

Once you have the extended regular expression pattern that requests just the 
products
you are interested in, use it in your ~ldm/etc/ldmd.conf REQUEST line:

request  CONDUIT   "(gefs|sref)" idd.cise-nsf.gov

When you request a small subset of a datastream, it is unlikely that you would
need to split your data request into pieces, so the example I gave in a previous
email will not be important.

Again, 'notifyme' is an _incredibly_ useful utility for:

- determining if an upstream machine is configured to allow you to request data
- determining what the upstream machine is receiving
- determining what you are receiving
- testing extended regular expressions that you will use in ldmd.conf REQUEST
  (or ALLOW) lines and in pqact.conf actions

> We have already updated the pattern action file for THREDDS handling of
> the Global GEFS 1p0deg Ensemble data using the example pqact.thredds you sent
> us.

Very good.

> And about ldm.pq, the results from the pqcheck command is 10,546 and the
> file size is 389 MB.  In the event that we do need to increase this product
> queue's size, how do we go about it (the RAM is 4GB).

This is _way_ too small when one is receiving all NOAAPort data and any
sizable amount of other high volume datastreams like CONDUIT.  Here is
how you resize the LDM queue:

<as 'ldm'>
- edit ~ldm/etc/ldmadmin-pl.conf and:

change:

$pq_size = "400M";

to:

$pq_size = "2G";

- stop the LDM; delete the current LDM queue; make a new queue using the new 
size you
  just set in ~ldm/etc/ldmadmin-pl.conf; restart the LDM:

ldmadmin stop
ldmadmin delqueue
ldmadmin mkqueu
ldmadmin start

> Also, I assume that using xcdscour with the location of the concatenated
> conduit and grib data generated is the way to go in cleaning up?

Actually, no.  If you are processing the CONDUIT data for THREDDS use and
not McIDAS use, then the portion of xcdscour that scours the MySQL database
is either irrelevant or not a good idea.  Instead, you can use the LDM
'scour' utility run from cron.  To use 'scour' (if you aren't already doing
so), you would add an entry like:

0 21 * * * bin/ldmadmin scour

NB: adjust the hour that the invocation runs to be about 1 hour after 0 UTC
at your site.

Next, edit the 'scour' configuration file, ~ldm/etc/scour.conf.  You would
add a line in scour.conf that specifies the directory(ies) in which the
concatenated GRIB/GRIB2 files are kept and specify the number of days of
data you want to keep.  Please take a look at ~ldm/etc/scour.conf to get
a better idea of what I am referring to.

> I see what you mean about THREDDS not updating but I assume that refreshing
> the browser or updating IDV while accessing our THREDDS service will
> take are of that, or could you tell us how MOTHERLODE handles that
> issue?

I don't understand your reference to THREDDS not updating... I don't recall
saying that.  I have been commenting about the need to install a new
THREDDS update when it is released.

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: WFA-619667
Department: Support IDD
Priority: Normal
Status: Closed