[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

20020225: realtime FTPs to motherlode.ucar.edu (cont.)

>From: "Kevin Polston" <address@hidden>
>Organization: NOAA
>Keywords: 200202012111.g11LBCx14907 LDM installation

Kevin,

>Well for the most part everything has been running pretty good. And for
>the most part the data has been very timely....so thats good too.

Sounds promising.

>I tried making the changes you suggested on the satellite directory
>structure....and I did not have great results. Maybe I didn't give it
>enough of a chance but here's what happened. I copiued the changes you
>sent me in the e-mail to the pqact.conf file. They worked but instead
>of getting a EAST-CONUS IR image (for example)....I was getting an IR
>image from Puerto Rico.  Also, there didn't seem to be a way to go and
>change to different images.  For instance ...in NMAP2 when I clicked on
>8km....there was no subdirectory called "WV" to choose from. Or when I
>clicked on satellite there was no "region" to choose from. I have to
>admit I don't understand the patterns very well for the satellite on
>the pqact.conf file...and I haven't had much time to look into them.
>I've got this class I have to do for the next couple of days so that
>will keep me occupied until then.

The pqact.conf patterns are regular expressions that match portions
of the product headers as they are received in the IDD.  To get a
better feel for them, do the following:

1) use notifyme to list out NIMAGE products that you are receiving:

notifyme -vxl- -f NIMAGE -o 3600

Here is an example line from a notifyme run here:

Feb 26 01:39:20 notifyme[24426]: 131dc48221ecbdcbaffb115c5d01e3af    58396 
20020226011151.838  NIMAGE 115344  sat/ch1/GOES-8/12.0/20020226 
0101/PR-REGIONAL/4km/ TIGP03 KNES 260101

Now, the product header starts after the NIMAGE feed type indicator and product
size, 115344.  Stripping away the beginning "stuff" leaves the actual product
header:

sat/ch1/GOES-8/12.0/20020226 0101/PR-REGIONAL/4km/ TIGP03 KNES 260101

The pqact.conf entry I recommended last time was:

# NOAAPORT GINI Images
NIMAGE  ^sat/ch[0-9]/(.*)/(.*)/([12][0-9])([0-9][0-9])([01][0-9])([0-3][0-9]) 
([0-2][0-9])([0-5][0-9])/.*/(.*km)/
        PIPE    -close
        util/readpng -n -l logs/png.log 
/usr1/nawips/metdat/images/sat/\1/\9/\2/\2_\3\4\5\6_\7\8

The pieces of the header that are matched by patters that are sorrounded
by parentheses are what can be used in the action to represent those pieces.
The patterns that begin with the open parenthesis and end with the close
parenthesis are referenced by the \n notation: \1, \2, \3, ..., \9.  There
are nine such allowed references.

Let's examine a portion of the product header specification line:

NIMAGE  ^sat/ch[0-9]/(.*)/(.*)/([12][0-9])([0-9][0-9])([01][0-9])([0-3][0-9])
                      |    |        |           |          |           |
                     \1   \2        \3          \4        \5          \6

continuing:

        ([0-2][0-9])([0-5][0-9])/.*/(.*km)/
              |           |           |
              \7          \8         \9

Comparing this to the example product header above:

sat/ch1/GOES-8/12.0/20020226 0101/PR-REGIONAL/4km/ TIGP03 KNES 260101

we can see that the \ mnemonics expand to the following values:

\1 -> GOES-8
\2 -> 12.0
\3 -> 20
\4 -> 02
\5 -> 02
\6 -> 26
\7 -> 01
\8 -> 01
\9 -> 4km

Given this, the action specified by:

/usr1/nawips/metdat/images/sat/\1/\9/\2/\2_\3\4\5\6_\7\8

will be replaced with:

/usr1/nawips/metdat/images/sat/GOES-8/4km/12.0/12.0_20020226_0101

What you need to do is create a pattern that will result in the
desired output naming.   I thought that you were saying that you
had and still wanted a naming like:

  >My directory structure that I had before ldm was this:
  >
  >/usr1/nawips/metdat/images/sat/GOES-8/4km/IR

that is why I suggested the pattern that I did.

If you want something like:

/usr1/nawips/metdat/images/sat/EAST-CONUS/4km/IR/IR_20020225_2215

Then your pqact.conf action could look like:

# NOAAPORT GINI Images
NIMAGE  ^sat/ch[0-9]/.*/(.*)/([12][0-9])([0-9][0-9])([01][0-9])([0-3][0-9]) 
([0-2][0-9])([0-5][0-9])/(.*)/(.*km)/
        PIPE    -close
        util/readpng -n -l logs/png.log 
/usr1/nawips/metdat/images/sat/\8/\9/\1/\1_\2\3\4\5_\6\7

In this case, the \ mnemonics extract from:

sat/ch1/GOES-8/12.0/20020226 0101/PR-REGIONAL/4km/ TIGP03 KNES 260101

the following:

\1 -> 12.0
\2 -> 20
\3 -> 02
\4 -> 02
\5 -> 26
\6 -> 01
\7 -> 01
\8 -> PR-REGIONAL
\9 -> 4km

If you work through a couple of these, you will get the hang of things.

>The other concern I have (and it is a bigger one)....and one which you
>mentioned earlier is being able to set up a scour procedure.  My usr1
>directory starts getting pretty filled up after 15-18 hours.... so what
>would you recommend as far as setting up a scour procedure?

I recommend grabbing the scripts:

prune_nexrad.csh
prune_gini.csh

from the pub/ldm5 directory of anonymous FTP on our FTP server,
ftp.unidata.ucar.edu, and putting them in the ~ldm/decoders or ~ldm/util
directory (which ever directory you created and put the ldm-mcidas
decoder pnga2area in).

You need to:

1) ftp them
2) put them in a directory in the PATH for the user running the LDM
3) change their read/write/execute permissions so that they are
   executable by the user running the LDM
4) edit them to set them up
5) kick off their invocation through cron

Here are reasonable entries for pruning the NEXRAD and GINI trees:

#
# prune NEXRAD and GINI trees
#
15 * * * * util/prune_nexrad.csh >/dev/null 2>&1
5,20,35,50 * * * * util/prune_gini.csh >/dev/null 2>&1

(assumes that you put the scripts in ~ldm/util; adjust as necessary).

Editing the files requires that you set things to match how many files
you want to keep and setup an internal PATH that matches your setup.

Here is the beginning of prune_gini.csh:

setenv PATH /usr/local/ldm/util:${PATH}
set KEEP=12
if($#argv == 0) then
   set areadir=/usr/local/ldm/data/gempak/nport
   set START_DATE=`date -u`
else
   set areadir=$1
endif

You must set PATH to match your setup.  If you put this script in ~ldm/util,
and if ~ldm/util is in the PATH defined in the ~ldm/.cshrc file, then
you can leave this setting alone.  If not, adjust to match your setup.

Next, you need to decide how many of each kind of GINI image you want
to keep on line.  This is set with the KEEP parameter which is set as 12
above (if you change this, do not add spaces around the '=' sign!!!!!).

The last thing you need to change is the value for areadir.  It appears
like this would need to be set to:

/usr1/nawips/metdat/images/sat

according to the comments that you have made previously.  The value
of areadir is the top level directory under which you store the
ingested GINI files.

On to the NEXRAD tree pruning.  The concept is identical to that for
the GINI pruning.  Edit the header to match your setup:

setenv PATH /usr/local/ldm/util:${PATH}
set KEEP=12
set LOCALKEEP=60
set LOCALSITE="FTG"
if($#argv == 0) then
   set areadir=/usr/local/ldm/data/gempak/nexrad/NIDS
   set START_DATE=`date -u`
else
   set areadir=$1
   if($#argv > 1) set KEEP=$2
endif

This time, you can specify how many NEXRAD images to keep for a site that
is not your local one, and how many to keep for your local one.
These are set with the values of KEEP= and LOCALKEEP=.  You must specify
LOCALSITE as the three letter ID of the radar you want to specify
as your LOCALSITE.

So, when you get data ingestion into the directory structure you want,
you can setup scourning that will walk that directory tree and keep
the number of files for those two types of data under control.

As far as scouring for other data files, you can setup the general
LDM scour with a crontab entry like:

#
# Purge LDM data
#
0 1,4,7,10,13,16,19,22 * * * bin/ldmadmin scour >/dev/null 2>&1

While you are at it, you should also setup a crontab entry that will
mail the LDM statistics back to Unidata:

#
# LDM statistics
#
35 * * * * bin/ldmadmin dostats

Your crontab entries will now look like the following at a minimum:

# LDM statistics
#
35 * * * * bin/ldmadmin dostats
#
# Purge LDM data
#
0 1,4,7,10,13,16,19,22 * * * bin/ldmadmin scour >/dev/null 2>&1
#
# LDM statistics
#
35 * * * * bin/ldmadmin dostats
#
# prune NEXRAD and GINI trees
#
15 * * * * util/prune_nexrad.csh >/dev/null 2>&1
5,20,35,50 * * * * util/prune_gini.csh >/dev/null 2>&1

That should get you going.

>Otherwise things were going pretty good.  I hope to get you the new IP
>address by Thursday or Friday (sooner if luck is with me but I am not
>holding my breath for that)....and then we'll go from there.  But as of
>yesterday it seemed things were functioning smoothly and timely...which
>is good!  I will subscribe to the ldm mailing list as you suggested.

This is essential.  It is through this list that we keep people aware
of what is going on in IDD land.

>I wonder...does this make me a "power ldm user" now?   :-)

Well, no, but you are beefing up ;-)

Tom