[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

20020305: hosting ADDE access to Unidata NEXRAD composite imagery (cont.)

>From: "Thomas L. Mote" <address@hidden>
>Organization: University of Georgia
>Keywords: 200203051400.g25E0BK05321 McIDAS ADDE NEXRAD composite

Tom,

I logged onto cacimbo and setup the decoding and ADDE serving of
the NEXRAD composite images.  This email contains the details of what
I did.

>I just updated my ldmd.conf with your new entries and restarted the 
>LDM. You can make the pqact.conf entries, but please 
>comment where you added entries so I can see what you did. 

OK.

>Just prompted me to look at my log files. The ldm-mcidas log file has 
>numerous entries, but they don't seem to indicate any problems.

I took a look and see a couple of things:

1) the ldm-mcidas.log file is not getting rotated, so it is larger
   than one would expect.  If you wanted to rotate the log file,
   you should add an entry to the 'ldm's crontab that looks more or
   less like:

#
# Shift LDM-McIDAS log files
#
0 17 * * * bin/newlog logs/ldm-mcidas.log

   I also see that you are not rotating your ldmd.log file.  A
   crontab entry that would do this is:

#
# rotate logs
#
0 17 * * * bin/ldmadmin newlog

>I'm 
>still having a problem with the ldmConnect entry (showing up as an 
>error in my ldm log file), but I haven't had a chance to figure out 
>what the problem is. Let me know if it is obvious to you.

I took a quick look, but nothing jumped out at me.  I can look more
later if you like.

>You can go ahead and set up the ADDE feeds and logging, but please 
>send me some description of what you did so I know how to duplicate 
>it. I would volunteer to do it, but this is our advising week.

Here is the procedure:

Overview:

McIDAS side
-----------

1) enable ADDE server logging

2) tell McIDAS where the ADDE log files are/should be

3) setup the ADDE dataset to serve the NEXRAD composite imagery

4) upgrade the McIDAS software to the current version of the GINI
   imager server code so that it can serve the 1 km national N0R composite

LDM side
--------

1) add requests for the composite imagery from the NIMAGE and FNEXRAD
   data streams and then stop and restart the LDM (you already did
   this)

2) modify the pqact.conf file to be able to decode (uncompress actually)
   the NIMAGE and FNEXRAD images that will be received.  This decision
   includes deciding where to put the composite imagery on disk

3) decide how many of each kind of composite image is to be kept on disk
   and setup scouring to achieve this

4) setup rotation of the ADDE server log files


So, there are really 4 steps that need to be taken on both the McIDAS and
LDM sides of the problem.  Here are the details:

McIDAS side
-----------

1) enable ADDE server logging

   Do this by adding the following lines to the ~mcidas/.mcenv file:

   ADDE_LOGGING=YES
   export ADDE_LOGGING

   In practice, the second line was simply combined with the other
   export line already in your ~mcidas/.mcenv file.  The line
   now looks like:

   export MCHOME MCDATA MCPATH MCTABLE_READ MCTABLE_WRITE PATH ADDE_LOGGING

2) tell McIDAS where the ADDE log files are/should be

   Do this by adding a file REDIRECTion:

   cd workdata
   redirect.k ADD SERVER.LO\* \"/data/logs

   This tells McIDAS to look for SERVER.LO* in /data/logs which on your
   system is where the LDM log files are kept.

3) setup the ADDE dataset to serve NEXRAD composite imagery

   This was done by basically deciding where the NEXRAD composite imagery
   should be put (this is done in conjunction with step 2) on the LDM
   side.  In looking at your setup, I found that you had a directory
   called /data/nexrad that you were using to store NEXRAD imagery.
   I decided to stick with this setup and store the images as follows:

   /data/nexrad/NEXRCOMP/6km            <- 6 km national N0R composite
   /data/nexrad/NEXRCOMP/10km           <- 10 km national RCM composite
   /data/nexrad/NEXRCOMP/1km/floater    <- 1 km regional N0R composite
   /data/nexrad/NEXRCOMP/1km/national   <- 1 km national N0R composite

   After deciding this, the NEXRCOMP ADDE dataset was defined using 4
   McIDAS DSSERVE invocations:

   cd workdata
   dsserve.k ADD NEXRCOMP/6KN0R-NAT AREA 1 9999 
DIRFILE=/data/nexrad/NEXRCOMP/6km/BREF\* \"6 km N0R National Composite Radar
   dsserve.k ADD NEXRCOMP/10KRCM-NAT AREA 1 9999 
DIRFILE=/data/nexrad/NEXRCOMP/10km/BREF\* \"10 km RCM National Composite Radar
   dsserve.k ADD NEXRCOMP/1KN0R-FLT AREA 1 9999 
DIRFILE=/data/nexrad/NEXRCOMP/1km/floater/BREF\* \"1 km N0R Retional Composite 
Radar Floater
   dsserve.k ADD NEXRCOMP/1KN0R-NAT GINI 1 9999 
DIRFILE=/data/nexrad/NEXRCOMP/1km/national/rad_\* \"1 km N0R National Composite 
Radar
   
   Be careful if you cut and past these lines as they are long.

4) upgrade the McIDAS software to the current version of the GINI
   image server code so that it can serve the 1 km national N0R composites

   This is non-standard since the GINI server mods are not yet available
   in a McIDAS addendum.  They will be soon, so I will give you the
   process you would follow with an addendum:

   cd mcidas7.8/update
   ftp ftp.unidata.ucar.edu
      <user> umcidas
      <pass> XXXXXX
      cd unix/780/bugfix
      binary
      get mcupdate.tar.Z
      quit
   ./mcunpack
   cd ../src
   make all
   make install.all


LDM side
--------

1) add requests for the composite imagery from the NIMAGE and FNEXRAD
   data streams (you did this)

   The lines added to ~ldm/etc/ldmd.conf were:

request NIMAGE "^rad_" atm.geo.nsf.gov
request FNEXRAD "^pnga2area Q5" atm.geo.nsf.gov

2) modify the pqact.conf file to be able to decode (uncompress actually)
   the NIMAGE and FNEXRAD images that will be received.  This decision
   includes deciding where to put the composite imagery on disk

   Four new actions were added to ~ldm/etc/pqact.conf:

#
# Note: FNEXRAD and NIMAGE actions added by Tom Yoksas/Unidata
#       on 20020305.

#
# NEXRCOMP 6 km National BREF mosaic
FNEXRAD ^pnga2area Q5 (RL) (.*) (.*) (.*) (.*) (........) (....)
        PIPE    -close
        /usr/local/ldm/ldm-mcidas/bin/pnga2area -v
        /data/nexrad/NEXRCOMP/6km/\4_\6_\7
#
# NEXRCOMP 1 km Regional BREF mosaic
FNEXRAD ^pnga2area Q5 (RM) (.*) (.*) (.*) (.*) (........) (....)
        PIPE    -close
        /usr/local/ldm/ldm-mcidas/bin/pnga2area -v
        /data/nexrad/NEXRCOMP/1km/floater/\4_\6_\7

#
# NEXRCOMP 10 km National RCM mosaic
FNEXRAD ^pnga2area Q5 (RN) (.*) (.*) (.*) (.*) (........) (....)
        PIPE    -close
        /usr/local/ldm/ldm-mcidas/bin/pnga2area -v
        /data/nexrad/NEXRCOMP/10km/\4_\6_\7

#
# 1 km National N0R composites in GINI format produced by Unidata
NIMAGE  ^rad_(........)_(....)
        PIPE    -close
        decoders/bin/readpng -n
        /data/nexrad/NEXRCOMP/1km/national/rad_\1_\2

   Always, always, always after making any mods to ~ldm/pqact.conf
   check your editing by running ldmadmin:

   ldmadmin pqactcheck

   If no errors are reported, tell pqact to reread pqact.conf:

   ldmadmin pqactHUP

   Since you were already using the ldm-mcidas decoder pnga2area, I
   did not have to go out and get that executable.  Since you were not
   using the routine 'readpng', I had to grab it from Unidata:

   cd ~ldm/decoders/bin
   ftp ftp.unidata.ucar.edu
      <user> anonymous
      <pass> your_full_email_address
      cd pub/binary/linux_2.2.5-i686
      binary
      get readpng
      quit
   chmod +x readpng

   As I noted in the McIDAS section above, the decision on where to put
   store images is a joint one made between the person setting up
   McIDAS ADDE services and the LDM pqact.conf actions.  You may want
   to change where things are being written so that you can more
   easily use the composites in GEMPAK.  I you do decide to change
   the directory locations, you will have to make mods in three different
   places:

   o McIDAS ADDE serving setup (~mcidas/workdata/RESOLV.SRV)
   o LDM writing of the data (~ldm/etc/pqact.conf)
   o LDM scouring of the data (~ldm/decoders/bin/prune_nexrcomp.csh)

3) decide how many of each kind of composite image is to be kept on disk
   and setup scouring to achieve this

   After setting up grabbing and decoding of imagery, you have to be
   concerned with not filling up your disk.  The mechanism to scour
   the imagery is setup by running a pruning script from cron as the
   user 'ldm'.  First, grab the script from Unidata:

   cd ~ldm/decoders/bin
   ftp ftp.unidata.ucar.edu
      <user> anonymous
      <pass> your_full_email_address
      cd pub/ldm5
      binary
      get prune_nexrcomp.csh
      quit
   chmod +x prune_nexrcomp.csh

   The first thing that has to be set in prune_nexrcomp.csh is the
   PATH variable.  This has to be set so that reentrant invocations
   of prune_nexrcomp.csh can find prune_nexrcomp.csh.  This means
   that one has to add ~ldm/decoders/bin (in your case) to the
   beginning of whatever PATH is in effect when the script runs.

   The next thing that has to be done is the definition of how many
   of the composites to keep on disk.  This is done by setting the
   KEEP= value.  I set this to be 60.

   The last thing that has to be done is setting the top level
   directory under which all of the NEXRAD composites can be found.
   This is done by setting the variable 'areadir'.  I set this to
   /data/nexrad/NEXRCOMP.

   The relevant portion of ~ldm/decoders/bin/prune_nexrcomp.csh now
   looks like:

setenv PATH /usr/local/ldm/decoders/bin:${PATH}
set KEEP=60
if($#argv == 0) then
   set areadir=/data/nexrad/NEXRCOMP
   set START_DATE=`date -u`
else
   set areadir=$1
endif


   After configuring prune_nexrcomp.csh, you have to add a crontab
   entry to kick off the scouring.  I added the following:

#
# prune NEXRAD composite tree
#
15 * * * * decoders/bin/prune_nexrcomp.csh >/dev/null 2>&1


4) setup rotation of the ADDE server log files

   While modifying 'ldm's crontab file, add rotation of the ADDE
   server log files:

#
# Rotate ADDE access log files just past 12 midnight on first day of month
#
1 0 1 * * bin/newlog logs/SERVER.LOG 3; chmod 666 logs/SERVER.LOG


OK, that's it both in a nutshell (Overview) and with a blow-by-blow
description.  Please let me know if you have any questions or see any
problems.

Tom