>From: Richard Massa <address@hidden> >Organization: UC Davis >Keywords: 200210150109.g9F19F112387 McIDAS-XCD LDM Hi Richard, >Hello, I'm the new sysadmin for the UC Davis Atmospheric Sciences group, >replacing Erick Lorenz, Did Erick leave UC Davis (just curious)? >and I've taken over the adminstration of the LDM >system, which wasn't (according to my professors) in a working state. So I've >set up a new system running the latest version of the LDM software and I've >been having some problems... Sounds good. >I guess the main thing that I need to ask for is help with pqact (or maybe a >sample configuration file) that places the products into the correct locations >for an ADDE server. OK. There are three packages that come into play here: LDM, ldm-mcidas, and McIDAS-XCD. Both ldm-mcidas and McIDAS-XCD are decoders for products that are ingested by the LDM. >My initial goal to set up the LDM program so the computer >that runs it can function as a McIDAS ADDE server, that way the machines that >run McIDAS don't have to connect to servers outside our network. Sounds good. What kind of a machine is this (operating system, etc.)? >I'm currently feeding IDS, DDPLUS, UNIWISC, and HDS products off of >peridot.atmos.ucla.edu, but I'm also guessing that I need at least the nexrad >and NLDN products in addition to what I have now. The NEXRAD products (Level III products for individual NEXRADs) and FNEXRAD products (national composites for Level III base reflectivities and 1-hour and storm total precipitation) are very useful. They can be gotten from most upstream LDMs. The NLDN data can only be gotten by a point-to-point feed from SUNY Albany. If UC Davis never got the data, you will have to send a note to Dr. David Knight requesting a feed. If you previously got a feed but your machine changed, you will also have to send David an email. If you previously got a feed and you are using the same machine in this incarnation of the LDM (same machine name and IP address), you should be set to go. >I've currently installed the LDM (5.1.2), The current version of the LDM is 5.2.1, but it is not available in binary format. >McIDAS 2002a, and the ldm-mcidas and unidata decoders on the machine. Sounds good. >Is Gempak necessary to have in order for the >ADDE server to function correctly? No. GEMPAK is not needed at all for McIDAS-related things. >I see mention of in different places in the DSSERVE.BAT file. I guess I was not explict enough with my comments in DSSERVE.BAT. The idea is that GEMPAK requires the user to store data in a very specific directory hierarchy. McIDAS is flexible (to the extreme), so it can use that hierarchy with ease. This allows sites running both packages to setup data filing for use with GEMPAK and McIDAS. >Lastly, does the problem with newer kernels and the LDM not working as an ADDE >server still exist or has it been fixed? The problem that you are referring to is serving sounding data through the remote ADDE server on Linux machines. The problem has not been seen on any other operating system that I support McIDAS on (and that includes a variety of PC-UNIX variants). Also, the problem has nothing to do with the LDM itself. Here goes with an overview of what you need to do to get the LDM setup so that ldm-mcidas and McIDAS-XCD can produce data files that you then can use for an ADDE server at your site. For reference, the setup of the ldm-mcidas decoders -- including needed LDM pqact.conf entries -- is discussed in the ldm-mcidas web pages: http://my.unidata.ucar.edu/content/software/mcidas/mcidd/index.html Example pqact.conf entries for ldm-mcidas decoders are found in: http://my.unidata.ucar.edu/content/software/mcidas/mcidd/ldm-mcidas-use.html Discussion of setting up LDM ldmd.conf and pqact.conf entries for McIDAS-XCD use can be found in: McIDAS-X HomePage http://my.unidata.ucar.edu/content/software/mcidas/index.html Installation and Configuration http://my.unidata.ucar.edu/content/software/mcidas/2002/mcidas-x.html Build, Install, and Configure XCD http://www.unidata.ucar.edu/packages/mcidas/2002/mcx/mcidas-xcd.html Unidata McIDAS-XCD Running with the LDM http://www.unidata.ucar.edu/packages/mcidas/2002/mcx/xcd_start.html The "quick and dirty" overview of how things work is: 1) the LDM on the local machine is used to get data from upstream LDMs 2) the LDM startup file, ~ldm/etc/ldmd.conf, must be modified to start a McIDAS-XCD startup shell script whose name is xcd_run. xcd_run is distributed with Unidata McIDAS. It can be found in the ~mcidas/bin directory _after_ McIDAS has been installed. xcd_run MUST be copied to a directory in the PATH of the user running the LDM; typically this is a directory named ~ldm/decoders. the copy of xcd_run copied to the ~ldm/decoders directory MUST be edited and modified to set various enviornment variables to match how McIDAS was installed (mainly, the variable MCHOME). permissions for xcd_run MUST be set so that it is executable by the user running the LDM ~ldm/etc/ldmd.conf MUST be edited and changed to include the XCD startup sequence: exec 'xcd_run MONITOR' 3) the LDM pattern-action file, ~ldm/etc/pqact.conf, MUST be modified to include patterns for McIDAS-XCD execution. These are _very_ simple. They look like: # Entries for XCD decoders DDPLUS|IDS ^.* PIPE xcd_run DDS HRS ^.* PIPE xcd_run HRS WARNING: entries in ~ldm/etc/pqact.conf MUST NEVER have leading spaces. The must also contain tabs as white space in various locations. In the listing above, tabs and no spaces are found inbetween the IDS and ^.*; between the ^.* and PIPE; before 'xcd_run DDS'; between HRS and ^.*; between ^.* and PIPE; before 'xcd_run HRS'. NOTE: any time you modify ~ldm/etc/pqact.conf, you should check your editing for errors by running (as the user 'ldm'): ldmadmin pqactcheck 4) the LDM pattern-action file, ~ldm/etc/pqact.conf, MUST be modified to include entries to run ldm-mcidas decoders. The decoders of interest will be pnga2area (converts PNG compressed imagery in AREA format to McIDAS AREA files), pngg2gini (converts PNG compressed imagery in GINI format to GINI files), proftomd (converts FSL wind profiler data in netCDF format to McIDAS MD files), and nldn2md (converts NLDN lightning data in its broadcast format to McIDAS MD files). Example pqact.conf entries for ldm-mcidas decoders can be found in the file ldm-mcidas-pqact.conf that is included in the ldm-mcidas distribution. This file can also be found on the web at: http://www.unidata.ucar.edu/packages/mcidas/mcidd/ldm-mcidas-pqact.conf The configuration of the McIDAS-XCD decoders is handled in the McIDAS environment (i.e., as the user 'mcidas'). The process for setting up McIDAS-XCD decoding is described in: Build, Install, and Configure -XCD http://www.unidata.ucar.edu/packages/mcidas/2002/mcx/mcidas-xcd.html The process that this user goes through -- and this should be done while the LDM is not running -- is basically: 1) login as 'mcidas' 2) cd workdata 3) define the directory in which you want McIDAS-XCD to create output data files. This should be on a file system that has ample room, especially if your are going to run the McIDAS-XCD GRIB decoder. I typically use a directory named /data/ldm/mcidas. I setup a link in ~ldm so that ~ldm/data/mcidas is /data/ldm/mcidas. I make sure that this directory is readable, writable, and executable by both the user 'mcidas' and the user 'ldm'. These users should be in the same group, so the setup is easy. Assuming that you have also chosen /data/ldm/mcidas as the output directory for McIDAS-XCD decoders, and assuming that this directory exists and is writable by the user McIDAS, you then define a McIDAS string to point to this directory: te.k XCDDATA \"/data/ldm/mcidas 4) define file REDIRECTions to locate McIDAS data files. This step uses the decision you made in step 3): cd ~mcidas/data cp EXAMPLE.NAM LOCAL.NAM <edit LOCAL.NAM and change directory locations for each of the data file name masks to match the directory location you decided on in step 3)> after you have finished editing LOCAL.NAM, make the file REDIRECTions you defined in it active: cd ~/workdata redirect.k REST LOCAL.NAM 5) setup McIDAS XCD decoding: cd ~/workdata batch.k XCD.BAT batch.k XCDDEC.BAT 6) at this point, if you are sure you want to decoded model data (GRIB data) into McIDAS GRID files, turn on the McIDAS GRIB decoder: decinfo.k SET DMGRID ACTIVE At this point, you should be able to turn on the LDM and see your system ingest data and have McIDAS-XCD decode that data. The next step is defining McIDAS ADDE datasets using the products that will ingested by the LDM and files that will be created from those products by ldm-mcidas and McIDAS-XCD decoders. The process is basically: 1) login as 'mcidas' 2) make a local copy of DSSERVE.BAT and edit it to match your system setup: cd ~mcidas/data cp DSSERVE.BAT LSSERVE.BAT <edit LSSERVE.BAT and modify directory paths to match how you setup the LDM to decode and file NNEXRAD and FNEXRAD data>: if you are going to be ingesting FNEXRAD composite radar images, then you need to modify -- if necessary -- the DIRFILE path listed for the FNEXRAD products. These would be the lines: DSSERVE ADD NEXRCOMP/1KN0R-NAT GINI TYPE=IMAGE DIRFILE=/data/ldm/gempak/nport/R ADAR/1km/n0r/n0r_* "1 km N0R US Base Reflectivity Composite DSSERVE ADD NEXRCOMP/2KN1P-NAT GINI TYPE=IMAGE DIRFILE=/data/ldm/gempak/nport/R ADAR/2km/n1p/n1p_* "2 km N1P US 1-hr Precip. Composite DSSERVE ADD NEXRCOMP/4KNTP-NAT GINI TYPE=IMAGE DIRFILE=/data/ldm/gempak/nport/R ADAR/4km/ntp/ntp_* "4 km NTP US Storm Total Precip. Composite and DSSERVE ADD NEXRCOMP/6KN0R-NAT AREA TYPE=IMAGE DIRFILE=/data/ldm/gempak/nexrad/ NEXRCOMP/6KN0R-NAT/BREF_* "6 km N0R National Composite Radar DSSERVE ADD NEXRCOMP/10KRCM-NAT AREA TYPE=IMAGE DIRFILE=/data/ldm/gempak/nexrad/ NEXRCOMP/10KRCM-NAT/BREF_* "10 km RCM National Composite Radar if you are NOT going to be ingesting FNEXRAD composite radar imagery, comment these lines out. A comment in a McIDAS BATCH file starts with 'REM '. if you are going to ingest NEXRAD data from WSI (not likely given that you can get the same data for free in the IDD), you will need to uncomment lines that define the WNEXRAD datasets). Again, it is NOT likely that you would be doing this 3) after you finish modifying LSSERVE.BAT, make the dataset definitions in it active: cd ~/workdata batch.k LSSERVE.BAT The key point in all of the above is that you have to: o decide where to store data products (for both the LDM setup and XCD setup) o "teach" McIDAS where those data products will be found (the file REDIRECTion step) o configure the LDM to ingest and decode data o configure McIDAS to decode data o define ADDE datasets to access the data files that are created by decoders Lastly, since there will be a number of systems accessing the data through McIDAS ADDE, you need to setup the McIDAS ADDE remote server on the machine running the LDM. This is described in: Prepare the mcidas and mcadde Accounts http://www.unidata.ucar.edu/packages/mcidas/2002/mcx/prepare_mcx.html Basically you will be: 1) create the file ~mcidas/.mcenv using the information in the above web page 2) create the account named 'mcadde'. Make the HOME directory for this user the same as for the user 'mcidas'. Put 'mcadde' into the same group as 'mcidas' and 'ldm'. Setup the 'mcadde' account so that it is NOT a login account (default shell will /bin/false) 3) as 'root' run the ADDE setup shell script mcinet2002.sh: <login as 'root'> cd ~mcidas ./mcinet2002.sh install mcadde The setup procedure in mcinet2002.sh _assumes_ that you are not using NIS/NIS+. 4) if there is a firewall running on this machine (or between this machine and others on your network), you will need to configure that firewall to allow access through ports 500 and 503. 5) define ADDE access to the datasets you setup previously: <login as 'mcidas'> cd ~mcidas/data cp DATALOC.BAT LOCDATA.BAT <edit LOCDATA.BAT and replace the phrase 'fully_qualified_host_identifier' with the name of the server that users should go to when wanting to access data through ADDE> Now, there are several datasets defined in LOCDATA.BAT that you will most likely not have on your local machine (e.g., GINICOMP, GINIEAST, GINIWEST). Given this, you would replace 'fully_qualified_host_identifier' with a machine name other than your local machine. Assuming that the name of the machine you are going to run the ADDE remote server on is atm1.ucdavis.edu (substitute the appropriate machine name for this fictitious one), I would suggest defining LOCDATA.BAT entries as follows: DATALOC ADD CIMSS atm1.ucdavis.edu DATALOC ADD GINICOMP pscwx.plymouth.edu DATALOC ADD GINIEAST stratus.al.noaa.gov DATALOC ADD GINIWEST papagayo.unl.edu DATALOC ADD NEXRCOMP atm1.ucdavis.edu DATALOC ADD RTGRIDS atm1.ucdavis.edu DATALOC ADD RTIMAGES atm1.ucdavis.edu DATALOC ADD RTNEXRAD atm1.ucdavis.edu DATALOC ADD RTPTSRC atm1.ucdavis.edu DATALOC ADD RTWXTEXT atm1.ucdavis.edu DATALOC ADD TOPO atm1.ucdavis.edu If you are not going to be ingesting and filing the NEXRAD radar composite images, you will want to change the NEXRCOMP dataset entry above to point at a machine that has the data: DATALOC ADD NEXRCOMP papagayo.unl.edu Lastly, after all of the above is running, you need to setup data file scouring. THIS IS IMPORTANT!!!! Here is the quick and dirty: 1) copy the file mcscour.sh from the ~mcidas/mcidas2002/data/mcscour.sh to a directory in the PATH of the user running the LDM. Edit the just copied version of mcscour.sh and set McIDAS environment variables in the same way that you did for xcd_run. Set entries in mcscour.sh to scour back to the number of days of McIDAS MD and GRID data that you want to keep. mcscour.sh is sent out to keep only 2 days of MD data and 1 day of GRID data (the GRID data is extremely voluminous): qrtmdg.k GRID 5001 6400 1 doqtl.k 1 70 2 doqtl.k 71 80 2 doqtl.k 81 90 2 doqtl.k 91 100 2 delwxt.k 1 10 The number of days of MD data to keep is set by the last argument on the doqtl.k lines. The number of days of GRID data is set by the last argument on the qrtmdg.k line. The delwxt.k line should be OK as it is sent out. This line scours text files created by McIDAS-XCD decoders. 2) if/when you decide to ingest radar composite images (from the FNEXRAD feed) or NEXRAD images from the NNEXRAD feed, please review the discussion of data file scouring in the McIDAS web pages and ask us questions if you need help. There is likely to be some tweeks you will want/need to make to all of the above, but this should basically get you up and working. Tom Yoksas
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.