>From: "Happel, Shelly" <address@hidden> >Organization: USF >Keywords: 200301091811.h09IBGt26938 ldm-mcidas nldn2md Hi Shelly, >Thanks tom!! I ended that vi session. OK, thanks. re: having both a FILE and PIPE action for the same pqact.conf entry >Yes, I made those changes just yesterday on the NLDN pqact.conf file. That was bad, but it wasn't the only problem. >I wasn't finding any of the MD files using the PIPE command, so I placed the >FILE command in there and it started dumping some data into /home/ldm This was a good thought, but you should have removed the line with the PIPE action. >- what >the HECK are those files? Are they uncoded lightning data? If you had a correct FILE action for NLDN products, then yes, you should see the undecoded lightning data. >I'm so confused >about where the LDM puts things (my head is sore from beating it against the >wall!) Don't hit it too hard :-) >Thanks Tom!! Here goes (I made a LOT of changes/additions): o One problem was that the ldm-mcidas decoders that you were using were all bombing out. I took a look at the size of the decoder binaries in ~ldm/decoders, and compared them with the ones in the binary ldm-mcidas release for v2002b, and I noticed that yours were way too small to be the correct ones. So, I downloaded the correct set of ldm-mcidas binaries (for Linux 2.4) into the ~ldm/ldm-mcidas directory and unpacked the compressed tar file. I then copied the decoders that you needed into the ~ldm/decoders directory. This took care of one problem. o Another problem was that multiple actions in ~ldm/pqact.conf were incomplete. You should review the corrections I made to all of these to get the flavor of what should be done. o Another problem was the read/write permissions on the directories /var/data/ldm and /var/data/ldm/mcidas and on the file /var/data/ldm/mcidas/ROUTE.SYS. The file ROUTE.SYS was owned by 'root', so it could not be written by anyone else. I corrected this by making the owner 'mcidas' in group 'mcidas'. While poking around, I noted that the environment for your 'mcidas' user was incorrect (or, at least did not follow my recommendations). I made a mod in ~mcidas/.cshrc to: 1) set the McINST_ROOT environment variable to $MCHOME 2) set MCDATA to be $MCHOME/workdata 3) changed MCTABLE_WRITE to match recommendations The changing of MCDATA changes MCPATH since MCPATH is defined using the value of MCDATA as the first directory in its list. The value of MCTABLE_WRITE affects what file will be updated with McIDAS DATALOC information. The file that will be modified is /home/mcidas/data/ADDESITE.TXT. I found that this file was owned by 'root', so I changed it to be owned by 'mcidas' in group 'mcidas'. I also noticed that while you had built McIDAS-XCD, you had not installed it. I finished this step (which depended on the correct definition of McINST_ROOT): <as 'mcidas'> cd mcidas2002/src make install.xcdall VENDOR=-g77 After finishing the XCD installation, I decided to do the setup necessary to allow XCD data decoding to be done (if you choose to do so): <as 'mcidas'> cd data cp EXAMPLE.NAM LOCAL.NAM <edit LOCAL.NAM and change directory locations to match your setup> cd ~/workdata redirect.k REST LOCAL.NAM te.k XCDDATA \"/var/data/ldm/mcidas batch.k XCD.BAT batch.k XCDDEC.BAT This created several files in /var/data/ldm/mcidas that will be used by McIDAS-XCD data monitors/decoders if/when you decided to start decoding data into McIDAS format. More on this at the end of this email. Once ROUTE.SYS was owned by 'mcidas', not 'root', and once a McIDAS REDIRECTion was setup so that McIDAS knew how to reference the copy of ROUTE.SYS in /var/data/ldm/mcidas, I could look at the contents of that copy of ROUTE.SYS. What I saw made me think that the file was very old since it was missing several entries. I decided to update it using the template DROUTE.BAT contained in the McIDAS distribution: <as 'mcidas'> cd ~mcidas/data cp DROUTE.BAT USFROUTE.BAT> <look through USFROUTE.BAT and make sure that there were not changes I wanted to make> cd ~/mcidas/workdata batch.k USFROUTE.BAT route.k REL C <- enable creating GOES-East/West composites route.k REL N <- enable creating Mollweide/imagery composites I then decided to setup ADDE access to the data you are now ingesting: cd ~mcidas/data cp DSSERVE.BAT LSSERVE.BAT <edit LSSERVE.BAT and modify to match USF setup> cd ~mcidas/workdata batch.k LSSERVE.BAT Nex up was to setup the ADDE remote server on metlab: <as 'root'> cd /home/mcidas sh mcinet2002.sh install mcadde <as 'mcidas'> create the file ~mcidas/.mcenv Then, cd ~mcidas/workdata dataloc.k ADD CIMSS metlab.cas.usf.edu dataloc.k ADD RTIMAGES metlab.cas.usf.edu Now when users access images from the CIMSS or RTIMAGES dataset, they will be doing so from metlab.cas.usf.edu. I verified that I can point to your remote ADDE server and access these datasets from my machine here in Boulder. Continuing on with XCD setup: <as 'ldm'> cd ~ldm/decoders cp ~mcidas/mcidas2002/src/xcd_run . <- to run XCD data monitors/decoders cp ~mcidas/mcidas2002/data/mcscour.sh . <- to scour McIDAS data files <edit xcd_run and mcscour.sh; no changes were needed in xcd_run; I changed the number of XCD-produced data files to keep in mcscour.sh> <add crontab entries to: run mcscour.sh, rotate the ldm-mcidas.log files, rotate the ADDE server access log files (these do not exist yet)> o add a line to ~ldm/etc/ldmd.conf to run the XCD data monitors: exec "xcd_run MONITOR" o add lines to ~ldm/etc/pqact.conf to send text and binary data to XCD data ingesters: IDS|DDPLUS ".*" PIPE xcd_run DDS HRS ".*" PIPE xcd_run HRS The next time the LDM is stopped and restarted the XCD data decoding will start. If you don't want this, you should comment out the new lines in ~ldm/etc/ldmd.conf and ~ldm/etc/pqact.conf. A couple of observations: 1) ingestion and decoding of Unidata-Wisconsin imagery data is now working. The output AREA files can be found in /var/data/ldm/mcidas. 2) no NLDN data has been decoded since there has been none to decode so far. As soon as an NLDN data file is received that has some flash data in it, an MDXX file will be created for it/them in /var/data/ldm/mcidas. The MDXX files for NLDN lightning will lie in the name space MDXX0071 - MDXX0080, inclusive. 3) XCD decoding of observational data will start the next time the LDM is stopped and restarted. GRID decoding will not start until the user 'mcidas' enables it. This is done as follows: <login as 'mcidas'> cd ~mcidas/workdata decinfo.k SET DMGRID ACTIVE Since I setup mcscour.sh to run from an 'ldm' cron entry, your disk won't fill from the XCD-produced data. 4) you are requesting FNEXRAD data from FSU, but they are refusing to send it to you. You need to contact the IDD admin at FSU and get him/her to allow your request for FNEXRAD data. I have setup scouring for FNEXRAD composite data in the cron initiated C shell script, ~ldm/util/prune_nexrcomp.csh. You should take a look at this to get an idea of how it works and is configurable. Also, you are requesting ALL NEXRAD data from FSU, but they also refusing this request. All of the NEXRAD data is a LOT of data. You should consider exactly what you want to ingest before you ask them to allow your feed request. I have, however, setup scouring for the NEXRAD Level III data (in ~ldm/util/prune_nexrad.csh), so the disk usage won't go nonlinear. 5) ADDE serving of FNEXRAD and NEXRAD data on metlab may have to be adjusted once you start getting the FNEXRAD and NNEXRAD data. Tom >From address@hidden Wed Jan 15 07:46:51 2003 >Subject: RE: 20030114: thank you!!! Hey Tom! Thank you SO much. It's going to take me some time to go through what you've written here. It's all so complicated and I'm far from qualified to be at the helm, but I'm trying!!! I feel like kneeling before you in gratitude. I knew the system was incorrectly functioning but I couldn't sleuth the solution and my unix pals here are even less familiar than I am with LDM and mcidas. Thank you thank you thank you thank you!!! You may be hearing more from me when I get to fully delve into the system!! Shelly
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.