>From: "Happel, Shelly" <address@hidden>
>Organization: USF
>Keywords: 200301091811.h09IBGt26938 ldm-mcidas nldn2md
Hi Shelly,
>Thanks tom!! I ended that vi session.
OK, thanks.
re: having both a FILE and PIPE action for the same pqact.conf entry
>Yes, I made those changes just yesterday on the NLDN pqact.conf file.
That was bad, but it wasn't the only problem.
>I wasn't finding any of the MD files using the PIPE command, so I placed the
>FILE command in there and it started dumping some data into /home/ldm
This was a good thought, but you should have removed the line with the
PIPE action.
>- what
>the HECK are those files? Are they uncoded lightning data?
If you had a correct FILE action for NLDN products, then yes, you should
see the undecoded lightning data.
>I'm so confused
>about where the LDM puts things (my head is sore from beating it against the
>wall!)
Don't hit it too hard :-)
>Thanks Tom!!
Here goes (I made a LOT of changes/additions):
o One problem was that the ldm-mcidas decoders that you were using were
all bombing out. I took a look at the size of the decoder binaries
in ~ldm/decoders, and compared them with the ones in the binary
ldm-mcidas release for v2002b, and I noticed that yours were way too
small to be the correct ones. So, I downloaded the correct set
of ldm-mcidas binaries (for Linux 2.4) into the ~ldm/ldm-mcidas
directory and unpacked the compressed tar file. I then copied
the decoders that you needed into the ~ldm/decoders directory.
This took care of one problem.
o Another problem was that multiple actions in ~ldm/pqact.conf
were incomplete. You should review the corrections I made
to all of these to get the flavor of what should be done.
o Another problem was the read/write permissions on the directories
/var/data/ldm and /var/data/ldm/mcidas and on the file
/var/data/ldm/mcidas/ROUTE.SYS. The file ROUTE.SYS was owned by
'root', so it could not be written by anyone else. I corrected
this by making the owner 'mcidas' in group 'mcidas'.
While poking around, I noted that the environment for your 'mcidas'
user was incorrect (or, at least did not follow my recommendations).
I made a mod in ~mcidas/.cshrc to:
1) set the McINST_ROOT environment variable to $MCHOME
2) set MCDATA to be $MCHOME/workdata
3) changed MCTABLE_WRITE to match recommendations
The changing of MCDATA changes MCPATH since MCPATH is defined using
the value of MCDATA as the first directory in its list. The value
of MCTABLE_WRITE affects what file will be updated with McIDAS
DATALOC information. The file that will be modified is
/home/mcidas/data/ADDESITE.TXT. I found that this file was owned
by 'root', so I changed it to be owned by 'mcidas' in group 'mcidas'.
I also noticed that while you had built McIDAS-XCD, you had not installed
it. I finished this step (which depended on the correct definition
of McINST_ROOT):
<as 'mcidas'>
cd mcidas2002/src
make install.xcdall VENDOR=-g77
After finishing the XCD installation, I decided to do the setup necessary
to allow XCD data decoding to be done (if you choose to do so):
<as 'mcidas'>
cd data
cp EXAMPLE.NAM LOCAL.NAM
<edit LOCAL.NAM and change directory locations to match your setup>
cd ~/workdata
redirect.k REST LOCAL.NAM
te.k XCDDATA \"/var/data/ldm/mcidas
batch.k XCD.BAT
batch.k XCDDEC.BAT
This created several files in /var/data/ldm/mcidas that will be used
by McIDAS-XCD data monitors/decoders if/when you decided to start
decoding data into McIDAS format. More on this at the end of this
email.
Once ROUTE.SYS was owned by 'mcidas', not 'root', and once a McIDAS
REDIRECTion was setup so that McIDAS knew how to reference the
copy of ROUTE.SYS in /var/data/ldm/mcidas, I could look at the contents
of that copy of ROUTE.SYS. What I saw made me think that the file
was very old since it was missing several entries. I decided to update
it using the template DROUTE.BAT contained in the McIDAS distribution:
<as 'mcidas'>
cd ~mcidas/data
cp DROUTE.BAT USFROUTE.BAT>
<look through USFROUTE.BAT and make sure that there were not changes I
wanted to make>
cd ~/mcidas/workdata
batch.k USFROUTE.BAT
route.k REL C <- enable creating GOES-East/West composites
route.k REL N <- enable creating Mollweide/imagery composites
I then decided to setup ADDE access to the data you are now ingesting:
cd ~mcidas/data
cp DSSERVE.BAT LSSERVE.BAT
<edit LSSERVE.BAT and modify to match USF setup>
cd ~mcidas/workdata
batch.k LSSERVE.BAT
Nex up was to setup the ADDE remote server on metlab:
<as 'root'>
cd /home/mcidas
sh mcinet2002.sh install mcadde
<as 'mcidas'>
create the file ~mcidas/.mcenv
Then,
cd ~mcidas/workdata
dataloc.k ADD CIMSS metlab.cas.usf.edu
dataloc.k ADD RTIMAGES metlab.cas.usf.edu
Now when users access images from the CIMSS or RTIMAGES dataset, they
will be doing so from metlab.cas.usf.edu. I verified that I can
point to your remote ADDE server and access these datasets from
my machine here in Boulder.
Continuing on with XCD setup:
<as 'ldm'>
cd ~ldm/decoders
cp ~mcidas/mcidas2002/src/xcd_run . <- to run XCD data monitors/decoders
cp ~mcidas/mcidas2002/data/mcscour.sh . <- to scour McIDAS data files
<edit xcd_run and mcscour.sh; no changes were needed in xcd_run; I
changed the number of XCD-produced data files to keep in mcscour.sh>
<add crontab entries to: run mcscour.sh, rotate the ldm-mcidas.log files,
rotate the ADDE server access log files (these do not exist yet)>
o add a line to ~ldm/etc/ldmd.conf to run the XCD data monitors:
exec "xcd_run MONITOR"
o add lines to ~ldm/etc/pqact.conf to send text and binary data to
XCD data ingesters:
IDS|DDPLUS ".*" PIPE
xcd_run DDS
HRS ".*" PIPE
xcd_run HRS
The next time the LDM is stopped and restarted the XCD data decoding
will start. If you don't want this, you should comment out the new
lines in ~ldm/etc/ldmd.conf and ~ldm/etc/pqact.conf.
A couple of observations:
1) ingestion and decoding of Unidata-Wisconsin imagery data is now working.
The output AREA files can be found in /var/data/ldm/mcidas.
2) no NLDN data has been decoded since there has been none to decode so
far. As soon as an NLDN data file is received that has some flash
data in it, an MDXX file will be created for it/them in /var/data/ldm/mcidas.
The MDXX files for NLDN lightning will lie in the name space
MDXX0071 - MDXX0080, inclusive.
3) XCD decoding of observational data will start the next time the LDM
is stopped and restarted. GRID decoding will not start until the
user 'mcidas' enables it. This is done as follows:
<login as 'mcidas'>
cd ~mcidas/workdata
decinfo.k SET DMGRID ACTIVE
Since I setup mcscour.sh to run from an 'ldm' cron entry, your disk
won't fill from the XCD-produced data.
4) you are requesting FNEXRAD data from FSU, but they are refusing
to send it to you. You need to contact the IDD admin at FSU and
get him/her to allow your request for FNEXRAD data. I have
setup scouring for FNEXRAD composite data in the cron initiated
C shell script, ~ldm/util/prune_nexrcomp.csh. You should take
a look at this to get an idea of how it works and is configurable.
Also, you are requesting ALL NEXRAD data from FSU, but they also
refusing this request. All of the NEXRAD data is a LOT of data.
You should consider exactly what you want to ingest before you ask
them to allow your feed request. I have, however, setup scouring
for the NEXRAD Level III data (in ~ldm/util/prune_nexrad.csh), so
the disk usage won't go nonlinear.
5) ADDE serving of FNEXRAD and NEXRAD data on metlab may have to be
adjusted once you start getting the FNEXRAD and NNEXRAD data.
Tom
>From address@hidden Wed Jan 15 07:46:51 2003
>Subject: RE: 20030114: thank you!!!
Hey Tom!
Thank you SO much. It's going to take me some time to go through what
you've written here. It's all so complicated and I'm far from qualified
to be at the helm, but I'm trying!!! I feel like kneeling before you in
gratitude. I knew the system was incorrectly functioning but I couldn't
sleuth the solution and my unix pals here are even less familiar than I
am with LDM and mcidas. Thank you thank you thank you thank you!!!
You may be hearing more from me when I get to fully delve into the
system!!
Shelly
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.