[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

20001219: big email here (mcidas problems due to my inexperience)

>From: Leigh Orf <address@hidden>
>Organization: Department of Tesselating Kumquats
>Keywords: 200012121654.eBCGsDo24855 McIDAS-X install


re: Heimbach

>Heh, well he has retired and is living in Maine. But he is officially
>emeritus and regularly logs into vortex.atms.unca.edu and I bug him
>every now and then for ldm and mcidas related stuff.


re: update process is easy

>Right, no problem. I have not yet done that. I will soon. I have some
>questions for you that will hopefully clear things up for me
>conceptually. Right now things are in a somewhat broken state, and
>before I start randomly running batch files I thought I'd have you check
>it out. First...


re: problems on Linux related to Slackware

>Hah, earlier this week I nuked the Slackware distro with the ancient
>2.0.24 kernel and put Red Hat 7.0 and built the 2.2.18 kernel on all the
>machines running mcidas-x.

Excellent.  Did you happen to read the Notes and Warnings sections that
relate to Linux users (especially the one about RedHat 7.0):


>So we can disucss this from a relatively
>"mainstream" Linux distro which will simplify things.

I agree.

>From my notes, here are the hurdles I hit and how I solved them:
>When you run make all, the 1st thing that runs is a configure
>script.  For some reason it assumes that the fortran compiler is
>/usr/local/bin/f77 which it is NOT on a RH distro. I know that you can't
>use g77 to compile mcidas; however, it does seem to matter that mcidas
>know where g77 lives, otherwise the build dies.

No, McIDAS does not need to know where g77 lives.

>Or I could be wrong. I
>did originally have a problem because I had gcc linked to kgcc, the
>(redhat kludge) version of gcc to build the kernel. If cc->kgcc the
>build will die when linking wish with GCC lib errors. Linking cc->gcc
>fixes this problem. So that was my mistake but it took some head
>scratching to figure out. Anyway, in my mcidas .profile file (I hate the
>c-shell so and use the k-shell) I export FC=/usr/bin/f77 (which is a
>link to g77) just for posterity. I did install f2c and moved the shared
>f2c libraries to another directory.

This will cause most likely cause you problems down the road.  I suggest
that you define FC to point to the 'mcfc' shell script bundled with
McIDAS instead:


'mcfc' is a shell script that makes running gcc and f2c look like running
a Fortran compiler.

>Hm, come to think of it I think that describe my problems and they were
>all due to mistakes on my side.

I think that you may not be over your problems if some things got built
with g77 and others didn't.

>I will, definitely. I think mcidas is a neat if not somewhat
>fundamentally flawed package (for instance, why are we stuck with a
>640x480 window? Those wind barbs look like crap when they are drawn so

You _aren't_ stuck with a 480x640 window at all!  The size of a McIDAS
image window is definable on a per-user basis.  The first time you run
McIDAS, the file .mcidasrc is created for you in your HOME directory.
This file can be edited and modified to customize your session any
way you like.  One of the simplest things to do is change the default
image size from the default 480x640 to something more reasonable.
You do this by modifying the -f flag values in .mcidasrc.  The default
will look like:

-f 17

This tells McIDAS to start a session with 17 default sized frames (i.e.,
480x640).  If you have a lot of screen real estate, you may choose to
run with settings similar to mine:

-f address@hidden

This starts a session with 17 frames of 720 x 960 LINes x ELEments.

Another thing you can do is start McIDAS though my Tcl/Tk GUI startup:

mcidas config

This pops up an interface through which you can specify how many frames
you want, their sizes, etc.  Give it a try and you will see what I

re: which email list did you sign up to

>Noted. I will subscribe to mcidas-x asap. I get the impression the
>mcidas-users list is not highly populated.

Again, there is no mcidas-users list at Unidata, but there is one at
SSEC and it has all of SSEC's sites (including the Unidata Program Center).
The "deal" we have with SSEC is that our sites come to us for help, not
go to them.

re: ask questions
>I will. You'll be sick of me soon enough ;) Actually, probably not,
>I just dive into things, ask a zillion questions, and once it works
>you don't hear much from me! I am a bit impatient and have more of a
>"type first read docs later later" attitude which sometimes gets me in


re: stack trace when trying to run GUI
>No, MCGUI is not running... I read about that problem. Here you go:
>I type GUI from the mcidas console:
>Error in startup script: Unknown UC location "image_window_id" 
>    while executing
>"error "Unknown UC location \"$name\""" 
>    (procedure "ucLocation" line 16)
>    invoked from within
>"ucLocation $name"
>    (procedure "UCpeek" line 5)
>    invoked from within
>"UCpeek image_window_id"
>    (procedure "initImage" line 4)
>    invoked from within
>"initImage .fmCenter.mcimage         "                                        
>    invoked from within
>"if [running_mcwish] \
>    {
>    initImage .fmCenter.mcimage            ; # Get the Image window pulled int
> o the mcwish window 
>    if {[llength [wm col..."
>    (file "/usr2/mcidas/lib/gui/GUIMain.gui" line 660)
>[yikes, cutting and pasting from the mcidas console is a bit weird...
>much easier from the tcl window]

Right!  The Tcl/Tk GUI console is much more friendly.  Thanks for the
trace: I'll look into this.

re: GUI development will coalesce

>Good! I think in this case, one standard kick-ass interface is better
>than two different good interfaces.

I agree!

re: I want to upgrade to the most current version of Tcl/Tk also

>Cool. I doubt it will really improve performance much; after all most of
>the number crunching is behind the scenes, but it will be good to use
>the latest tcl/tk distro.

I don't think that it will do anthing in terms of speedups, but the later
versions do offer new features that I want to use in my GUI.

re: various tests

>I did not try what you suggested yet since mcidas is 'broken' right now.
>I will though. I am amazed that you were able to use mcidas over a 56 k

It worked well enough.

>Did you do it via NFS?

No, ADDE.  Again, ADDE is the _very_ cool thing about McIDAS: the data
does not have to be local AND you do not need to make it look local with

>NFS is such a dog. It really seems to
>require fast, synchronous communication, i.e. not having bandwidth
>limited in either direction. And low latency, too, which isn't the case
>when you are 15 hops away from the machine you are mounting.
>Ok, here is my current problem. First of all, /data/mcidas/data on
>vortex is where the ldm dumps stuff and is $MCDATA in my mcidas

$MCDATA should NOT be the directory where data is being written.  Each user
must create a mcidas/data subdirectory under their HOME.  MCDATA is
then defined to be this directory.  For instance, if my home directory
is /home/tom and McIDAS is installed in /home/mcidas, then my MCDATA
and MCPATH would be defined to be:


Setting up environment variables for McIDAS is defined in:

for 'mcidas':

Preparing the mcidas and mcadde Accounts

for users other than 'mcidas':
Configuring Unidata McIDAS-X Accounts 
Configuring User Accounts

I made _several_ attempts to get Jim to configure things this way at
UNCA, but was singularly unsuccessful.  Since you are redoing things
please follow my recommendations.  If you do, your life will be easier
in the future.

>Since I see you have an account on vortex and I see your name
>in pqact.conf, I assume you are at least somehwat familiar with our

Yes.  Again, I helped Jim a couple of times, but I could never get
him to do things in a way that would make doing upgrades easier.

>I hate Solaris, admittely mostly because it's not Linux, which I am
>very familiar with. I really wish LDM was ingesting on a Linux box.

My experience is that Solaris works much better than Linux in heavy load

>fact, it may be soon. But the real problem on vortex isn't Solaris, it's
>the fact that /data/mcidas/data is only a bit over 1.2 GB.... WAY too
>small a partition for what we are doing. Soon it will reside on a bigger
>partition. We regularly run out of space and that screws up mcidas and
>the LDM. Anyway, this will happen once I sit down with Bob Benites, the
>sysadmin guy at UNCA who I work with.

Sounds good.  Our comment is that you can never have too much disk for

>Anyway, this is my problem, the above was a tangent. First of all
>/data/mcidas/data is being NFS mounted by our Linux boxes where we run
>mcidas from. While mcidas is installed (7.7 I just installed) on vortex,
>I only run mcidas from vortex when I am setting mcidas-x and xcd up
>since it requires fast disk access.


>When I was having problems with mcidas-x right after I set it up, I
>decided to wipe the entire contents of /data/mcidas/data and start
>anew.  I followed the instructions on setting up and configuring xcd
>up, namely installing SYSKEY.TAB and SCHEMA

And, hopefully, ROUTE.SYS.

>(I really have no idea what
>these files really do but they seem Very Important) running XCD.BAT
>and XCDDEC.BAT. However that didn't seem to be enough to get all the
>data I wanted available to mcidas-x. I ended up grepping .BAT files
>for commands that looked useful and ran a bunch of them. This probably
>explains some of my problems.

The setup for XCD should have been more cookbook that that.  The steps
are pretty simple (after you have done them once):

<as 'ldm'>
ldmadin stop                 <- stop the LDM
cd ~
mkdir decoders
cd decoders
cp ~mcidas/bin/xcd_run .
<edit xcd_run and set the McIDAS environment variables to match your
 installation; make sure that ~ldm/decoders is in the PATH for 'ldm';
 make sure that the file permissions on xcd_run are set so that it is
cp (all of the ldm-mcidas decoders) .
cd ~/etc
<edit ldmd.conf and make sure that you have a line like:
exec    "xcd_run MONITOR"

  in the section of 'exec's
<edit pqact.conf and make sure that you have XCD startup lines that look

# McIDAS XCD decoding
DDPLUS|IDS      ^.*     PIPE
        xcd_run DDS
HRS     ^.*     PIPE
        xcd_run HRS

 NOTE: tabs are _very_ important!
 NOTE: I recommend not turning on McIDAS GRID decoding until you have
       sufficient disk space (you don't yet)

 Given the second note, your pqact.conf entry would look like:

# McIDAS XCD decoding
DDPLUS|IDS      ^.*     PIPE
        xcd_run DDS
#HRS    ^.*     PIPE
#       xcd_run HRS

<always after editing the LDM's pqact.conf file, you should verify its
integrity with:

ldmadmin pqactcheck

<as 'mcidas'>
make sure that MCDATA, MCPATH, etc are defined as I indicate in the
  online McIDAS web pages;  if you deviate from my recommendations
  then things get a lot more difficult
cd data
<edit LOCAL.NAM and set the directory for the various files to be
cp SYSKEY.TAB /data/mcidas/data
cp SCHEMA /data/mcidas/data
cd workdata
cp ROUTE.SYS /data/mcidas/data
redirect.k REST LOCAL.NAM
te.k XCDDATA \"/data/mcidas/data
batch.k XCD.BAT
batch.k XCDDEC.BAT

The last step should have created several files in /data/mcidas/data
(sufficx: *.R*)

<as 'ldm'>
ldmadmin start             <- start the LDM

At this point, the LDM should start feeding xcd_run products.  xcd_run
is just a wrapper around the McIDAS-XCD ingest routines ingetext.k and
ingebin.k.  These are created in the XCD portion of a McIDAS build.
Their rwx permissions should have been set so that 'ldm' can run them
('ldm' and 'mcidas' should be in the same group AND they should have
their umask set to be 002).

>Oh, one thing, what would cause the SCHEMA file to disappear?

If you have setup LDM scouring of the directory in which XCD is decoding
files, then SCHEMA will eventually get wiped off since its time stamp
doesn't change.  This is why I provide the mcscour.sh shell script
that one can modify and run from cron to scour McIDAS data files.

>Out of the blue it disappeared and would appear and
>reappear as an empty file.

The reappearing as an empty file is an artifact of McIDAS.  McIDAS
views files as being virtual.  If a read is done on a file that does
not exist, it is created.  It seems evident that what happened was
that a scouring routine removed SCHEMA, and it was then recreated
the next time an XCD decoder tried to access it.

>This obviously messed things up and led to my
>furthur experimentation which probably messed things up more.


>Another thing that has been really buggging me: Can you explain the
>following from XCDDEC.BAT:
>REM Suspend receipt of the surface (MA), upper air mandatory (RM) and
>REM significant leve (RS) MD files from the Unidata/Wisconsin datastream
>Why are these suspended by default??

The Unidata-Wisconsin datastream used to contain products that were decoded
by these actions.  Those products were removed from the UW stream in July
of 1999.  This line is in XCDDEC.BAT for systems using copies of ROUTE.SYS
from several years ago.

>That's some of the most useful
>stuff. I did RELease MA RM RS and was succesfully receiving that data
>and mcidas-x was displaying it.

Actually, these have nothing to do with XCD decoding.  Once XCD decoding
is setup and you have file REDIRECTions setup to point to the files,
you should be able to access/use the data.

>I think I read somewhere in the docs
>that you are supposed to suspend these so it doesn't conflict with the
>LDM or something. I don't understand.

Back when there were surface and upper air data in the Unidata-Wisconsin
datastream AND there was XCD, there would be interference between
XCD decoding and the products coming from the Unidata-Wisconsin datastream.
Now that there are no surface and upper air data in the UW stream, there
is no possibility of conflict.

>My current problem seems to be related to the fact that my MDX files
>are not being updated in /data/mcidas/data. They were yesterday!

If SCHEMA was deleted and recreated as a zero-length file, then decoding
will stop.

>As I
>understand it, much of the interesting stuff goes on in here. I did just
>discover the CIRCUIT command and ran CIRCUIT SET IDS ACTIVE; CIRCUIT SET
>PPS ACTIVE; CIRCUIT SET HRS ACTIVE (from the online docs) which may or
>man not have helped.

No, no, no!  McIDAS-XCD was designed to be able to ingest data directly
from serial ports (from a satellite data system).  The way it is used
in Unidata McIDAS is by being fed from from the LDM.  Please leave
the CIRCUIT settings as they are sent out in the distribution.

>MDU LIST 1 100
> ----- ------- ---- ---- ---- ---- ------- -----------
>     3 2000352 ISFC    0   72 6000 2000353 SAO/METAR data for   18 DEC 2000
>     4 2000354 ISFC    0   72 6000 2000354 SAO/METAR data for   19 DEC 2000
>    13 2000352 IRAB    0    8 1300 2000353 Mand. Level RAOB for 18 DEC 2000
>    14 2000354 IRAB    0    8 1300 2000354 Mand. Level RAOB for 19 DEC 2000
>    23 2000352 IRSG    0   16 6000 2000353 Sig.  Level RAOB for 18 DEC 2000
>    24 2000354 IRSG    0   16 6000 2000354 Sig.  Level RAOB for 19 DEC 2000
>    33 2000353 ISHP    0   24 2000 2000353 SHIP/BUOY data for   18 DEC 2000
>    43 2000353 FO14    0   38  600 2000353 NGM MOS for day      18 DEC 2000
>    53 2000352 SYN     0    8 6200 2000353 SYNOPTIC data for    18 DEC 2000
>    54 2000353 SYN     0    8 6200 2000354 SYNOPTIC data for    19 DEC 2000
>    63 2000352 PIRP    0   24 1500 2000353 PIREP/AIREP data for 18 DEC 2000
>    74 2000354 NLDN    0 1000 1000 2000354 NLDN data for        19 Dec 2000
>However look at the time stamps on these:
>vortex[1019]:/data/mcidas/data% ls -lt MDX*
>-rw-rw-r--   1 ldm      ldmadmin 31672536 Dec 19 12:01 MDXX0004
>-rw-rw-rw-   1 ldm      ldmadmin   38204 Dec 19 09:30 MDXX0074
>-rw-rw-r--   1 ldm      ldmadmin 2200368 Dec 18 22:26 MDXX0054
>-rw-rw-r--   1 ldm      ldmadmin 5677128 Dec 18 22:26 MDXX0013
>-rw-rw-r--   1 ldm      ldmadmin 1458464 Dec 18 22:26 MDXX0023
>-rw-rw-r--   1 ldm      ldmadmin 43283736 Dec 18 22:26 MDXX0003
>-rw-rw-r--   1 ldm      ldmadmin   16640 Dec 18 22:24 MDXX0024
>-rw-rw-r--   1 ldm      ldmadmin    6656 Dec 18 22:22 MDXX0014
>-rw-rw-r--   1 ldm      ldmadmin 8506752 Dec 18 22:18 MDXX0053
>-rw-rw-r--   1 ldm      ldmadmin 5990920 Dec 18 19:00 MDXX0063
>-rw-rw-r--   1 ldm      ldmadmin 5010916 Dec 18 19:00 MDXX0033
>(I'm writing this at 12:01 PM on Dec 19)
>What complicates matters is the fact that I think data ingestion has
>been somewhat sporadic due to DOS attacks on routers in Wisconsin.

The data coming out of Wisconsin is only imagery.  The imagery is
decoded using the ldm-mcidas pnga2area decoder.  The MD files are
created by XCD decoders that are decoding data being sent in various
IDD streams: DDS, IDS, PPS, HRS.

>Can the LDM/Mcidas recover easily from this kind of thing, or does it
>require a restart of the LMD?

We havn't actually exactly pinpointed what the "thing" it needs to recover
from.  I suspect that the decoding problem is entirely related to the
disappearance of SCHEMA.

>I am coming to realize how intertwined the
>LDM is to all of this and must admit that the LDM is still somewhat of a
>black box. My eyes glaze over when I see all the stuff in pqact.conf
>especially all the stuff that is commented out without explanation.
>It seems the relevant stuff to mcidas is simply:
>DDPLUS|IDS      ^.*     PIPE
>        xcd_run DDS
>HRS     ^.*     PIPE
>        xcd_run HRS

Right.  Also, there will be a line or lines that are related to the MCIDAS
feed type and run pnga2area.

>It seems all the strange uppercase 8.3 [ugh ugh ugh!] files which
>magically appear in $MCDATA are created by processes spawned in the
>xcd_run shell script. Realizing that explained a lot to me.


>[sorry this email is scattered, as you can tell my knowledgs isn't
>complete here]
>I am tempted to start all over from scratch again.

I would recommend this especially since the use of MCDATA is incorrect.

>I did discover some
>setup stuff in the docs that I hadn't found before [such as "configuring
>the mcidas account"]. I admit that I am running mcidas under the mcidas
>account (it complains every time I start it up) because I have been too
>impatient to set things up right.

Running out of the McIDAS account is fine _as long as you realize that what
you do can affect all McIDAS users_.  Think of the 'mcidas' login as 'root'
for McIDAS.

>I think permissions are OK on the
>$MCDATA directory since both ldm and mcidas are able to read and write
>Anyway, I guess the condensed version of this email is "Can you help me
>out and if you want to log on and poke around and fix things, please do
>so but tell me what you did so that I can learn this stuff myself."

Yes.  I am willing to login and setup things IF I am given a free hand
to configure things like the recommendations in the McIDAS web pages.
I could never convince Jim H. about the need for doing this, but I
am hopeful that I can with you.

>I will attempt to refrain from messing with things so that you can see
>the state I'm in and tell me what I did wrong.


>These listings may or may not be helpful to you. This is as of right
>Number of active redirection entries=18
>FOUS14.RAP /data/mcidas/data
>FOUS14.RAT /data/mcidas/data
>GRID5* /data/mcidas/data
>HRS.SPL /data/mcidas/data
>IDXALIAS.DAT /data/mcidas/data
>MDXX00* /data/mcidas/data
>RAOB.RAP /data/mcidas/data
>RAOB.RAT /data/mcidas/data
>SAOMETAR.RAP /data/mcidas/data
>SAOMETAR.RAT /data/mcidas/data
>SCHEMA /data/mcidas/data
>SYNOPTIC.RAP /data/mcidas/data
>SYNOPTIC.RAT /data/mcidas/data
>TERMFCST.RAP /data/mcidas/data
>TERMFCST.RAT /data/mcidas/data
>*.XCD /data/mcidas/data
>*.IDX /data/mcidas/data
>*.IDT /data/mcidas/data

These look fine with the exception that the list is abbreviated.  Given this,
it seems likely that you skipped one or more steps in the installation
configuration process.

>S Pd         Description         Range       Last      Received  Post Process 
> C
>- -- ------------------------- --------- ------------ ---------- ------------ 
> -
>  CA Cloud Top Pressure        1100-1109 AREA1106     00354 1700 CTP.BAT      
> 3
>ROUTE: Done
>Circuit Name: DDS   Ingestor:INGETEXT
>Domestic Data Service
>Configuration File: DDS.CFG
>DDS  ingestor is inactive          ingestor status display number 1
>Circuit Name: PPS   Ingestor:INGETEXT
>Public Products Service
>Configuration File: PPS.CFG
>PPS  ingestor is active            ingestor status display number 2
>Circuit Name: IDS   Ingestor:INGETEXT
>International Data Service
>Configuration File: IDS.CFG
>IDS  ingestor is active            ingestor status display number 3
>Circuit Name: HRS   Ingestor:INGEBIN
>High Resolution Data Service
>Configuration File: HRS.CFG     Spool File: HRS.SPL
>HRS  ingestor is active            ingestor status display number 4

>Note that the above circuits were NOT active until just a few minutes

OK.  The circuits should remain as inactive as you are not going to
be ingesting data from serial ports into McIDAS-XCD.

>OK, enough from me. I promise not to mess with things today. I need to
>get away for a day and enjoy my vacation!

I agree.
>Thanks VERY MUCH for all your help.

No problem.  So, to get onto the machine, I will need logins:

o mcidas
o ldm
o root

Even though you note that I already have an account there, I do not
remember the login information or the machine name.

I suggest that you send the machine name back here to support, and the
passwords for the accounts to my account: address@hidden
Don't mention the machine in the note to me or the account names, just
the passwords.

Also, it seems that you might have been setting up user accounts that
are incorrect.  I can redo one of these accounts and let you do the rest
after explaining what I did (even though, I won't be doing anything
other than what is in the McIDAS web pages).


>From address@hidden Wed Dec 20 19:52:32 2000
>Subject: Re: 20001219: big email here (mcidas problems due to my inexperience) 


Bit by bit I am learning this stuff.

First of all, I moved /data/mcidas/data to a hard drive which vortex
now NFS mounts. There is about 15 GB of data available right now.  It
probably makes most sense to have ldm running on the machine that is
the NFS server (storm2.atms.unca.edu) but I'll have to get upstream
permission. Are you the right person to go to for that permission? I'm
going to keep vortex ingesting text data (the stuff that goes to

I did manage to succesfully create an account at home and run mcidas
without running as user mcidas. I was able to display some satellite
data but not much else. This is probably due to the fact that I just
wiped /data/mcidas/data and started over again (not much data there

I like the fact that I can issue blah.k commands not inside the mcidas
program... neat.

One question, when I pull down MCGUI menus from the Information->Data
Availability-> menu, what files does the software seek to determine what
is current etc.? I'm finding that I'm getting MISSes everywhere under,
for instance, Satellite Imagery even though I can view images and AREA
files are present. Is there a time lag or something? This is under my
new non-mcidas account (my orf account, I just added appropriate stuff
to my .profile file).

I'm really happy with a bigger mcidas image window, that really rocks.
It's like getting a new pair of glasses :)

|   No problem.  So, to get onto the machine, I will need logins:
|   o mcidas
|   o ldm
|   o root

I will send your pasword to your other email address as suggested.  You
are on the sudoers list on vortex.atms.unca.edu. I honestly don't even
know the mcidas or ldm passwords. When you want to assume the identity
of ldm for exmaple, as yoksas type

sudo su - ldm

and then type *your* password (which you of course changed right away
:) and it will be as if you logged in as ldm. If this doesn't work for
you for some reason let me know. I've grown to really like sudo, all
commands are logged and you are less likely to forget you are root and
do something stupid.

|   Also, it seems that you might have been setting up user
|   accounts that are incorrect.  I can redo one of these
|   accounts and let you do the rest after explaining what I did
|   (even though, I won't be doing anything other than what is in
|   the McIDAS web pages).

I've been slogging through the mcidas pages. I think I am getting the
hang of it. But go ahead and create a user account if you want, I can
compare to mine to see if I screwed anything up.

I am still having conceptual problems with what commands should be
issued as a normal user, and what commands should be issued as user
mcidas.  But it's getting slightly clearer.

One thing, I have not gotten the ADDE client/server stuff running yet.
I see there is an appendix in the help pages that goes over this. I
have set up my inetd stuff here at home and look forward to using it.
I imagine that the strings "/data/mcidas/data" will not exist on my
home machine once I get the ADDE stuff running, right, since I won't
need to use NFS? Anyway if you want to set up the ADDE server stuff on
vortex that's fine, but I want to know how to do it espcially since I'll
probably be installing LDM on storm2.

Enough for now, I need to not do this for a while!! (I keep saying


NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.