[
Date Prev][
Date Next][
Thread Prev][
Thread Next][
Date Index][
Thread Index]
20000707: ROUTE PostProcess BATCH invocations and general setup (cont.)
- Subject: 20000707: ROUTE PostProcess BATCH invocations and general setup (cont.)
- Date: Fri, 07 Jul 2000 16:15:12 -0600
>From: "James D. Marco" <address@hidden>
>Organization: Cornell
>Keywords: 200007062021.e66KLJT26042 McIDAS-X ROUTE PP BATCH
Jdm,
>OK. I created a new file: NEWROUTE.BAT and rebuilt the routing.
OK, I see that the various image products now have ROUTE PostProcesses
setup.
>However, there was still no post process stuff defined for the TOPOs
>I added in a couple and rebuilt again.
The compositing of the topography and Unidata-Wisconsin images is done
through the PostProcess BATCH file for the image products coming in
the UW datastream (e.g., GE-VIS.BAT specified for the GOES-East VIS
product, UV, etc.). Your adding execution of PostProcesses for the
created composites created an infinite processing loop. Here is an
example of what happened:
a GOES-East Vis product arrives
it is decoded by lwtoa3
the PostProcess BATCH file GE-VIS.BAT is run by lwtoa3
....>GE-VIS.BAT composites topography and the GOES Vis image
. composites the the vis image with the GOES-West vis image
. (the West image comes in before the East image)
.....you setup the N2 product, GOES-East VIS/TOPO Composite, to also run
GE-VIS.BAT. When the composite got finished, GE-VIS.BAT was run
again
By the time I noticed this, there were about 100 invocations of batch.k
running. I had to SUSpend the N2 product (GOES-East VIS/TOPO Composite)
and wait until all of the processing concluded before proceeding.
I then edited NEWROUTE.BAT and removed the PP= specifications on all
of the topographic composite products (N1-N8), and then remade the
routing table (batch.k NEWROUTE.BAT).
All-in-all, it was pretty humorous seeing the number of batch.k invocations
rise. Your machine is pretty fast, so there were lots of them :-)
>The PP batch files now appear in
>the ROUTE list. I am not sure which post process files belong to which.
>The East/West composite is an example. Do you have a list on the web
>somewhere?
The copy of DROUTE.BAT I send in the distribution has the recommended
setup.
> Any how. THANK YOU for your efforts and time.
No problem.
>I need to go,
>so we'll pick this up later... (Dinner engagement with you-know-who
>as the chief cook on the grill.)
I understand completely.
> I'll set up the mail alarms when I get home.
As I am writing this, I see that Unidata-Wisconsin products are coming
in, and PostProcesses are creating the composites. Here are snippits
from the routing table listing:
route.k LIST
S Pd Description Range Last Received Post Process C
- -- ------------------------- --------- ------------ ---------- ------------ -
CA Cloud Top Pressure 1100-1109 AREA1100 00189 2116 none 3
CB Precipitable Water 1110-1119 AREA1110 00189 2131 none 3
CC Sea Surface Temperature 1120-1129 AREA1120 00189 2133 none 3
CD Lifted Index 1130-1139 AREA1131 00189 2134 none 3
CE CAPE 1140-1149 AREA1141 00189 2135 none 3
CF Ozone 1150-1159 AREA1151 00189 2151 none 3
CI GOES-E/W IR Composite 80-89 AREA0080 00189 2137 none 3
CV GOES-E/W VIS Composite 90-99 AREA0091 00189 2139 none 3
CW GOES-E/W H2O Composite 70-79 AREA0070 00189 2137 none 3
LD NLDN Lightning Flashes 71-71 none none none 3
MA Surface MD data default none none SFC.BAT 3
N1 GOES-East IR/TOPO Composi 220-229 AREA0220 00189 2137 none 3
N2 GOES-East VIS/TOPO Compos 230-239 AREA0230 00189 2139 none 3
N3 GOES-West IR/TOPO Composi 240-249 AREA0240 00189 2117 none 3
N4 GOES-West VIS/TOPO Compos 250-259 AREA0250 00189 2117 none 3
N5 MDR/TOPO Composite 260-269 AREA0260 00189 2106 none 3
N6 Mollweide IR/TOPO Composi 270-279 none none none 3
...
U1 Antarctic IR Composite 190-199 AREA0190 00189 2110 none 3
U2 FSL2 hourly wind profiler default MDXX0089 00189 2117 none 3
U3 Manually Digitized Radar 200-209 AREA0200 00189 2106 MDR.BAT 3
U5 GOES-West US IR Band 4 130-139 AREA0130 00189 2117 GW-IR.BAT 3
U6 FSL2 6-minute Wind profil default MDXX0099 00189 2204 none 3
U9 GOES-West US Visible 120-129 AREA0120 00189 2117 GW-VIS.BAT 3
UA Educational Floater I 160-169 AREA0160 00189 2131 none 3
UB GOES-West US Water Vapor 170-179 AREA0170 00189 2116 GW-WV.BAT 3
UC Educational Floater II 60-69 AREA0060 00189 2134 none 3
UI GOES-East US IR Band 4 150-159 AREA0150 00189 2137 GE-IR.BAT 3
UR Research Floater 180-189 none none none 3
US Undecoded SAO Data default none none none 1
UV GOES-East US Visible 140-149 AREA0140 00189 2139 GE-VIS.BAT 3
UW GOES-East US Water Vapor 210-219 AREA0210 00189 2137 GE-WV.BAT 3
UX Mollweide Composite IR 100-109 none none MOLL.BAT 3
UY Mollweide Composite H2O 110-119 none none none 3
route.k: Done
Now, on to the other things I wanted to touch base on.
First off, you system is working, so none of the changes I am going to
suggest are super critical. I believe, however, that if you make the
changes, you life will be easier in the future.
1) My recommendation is for each McIDAS-X user to have his/her own
McIDAS working directory. For the user 'mcidas' (the super user
for McIDAS-X, -XCD), this would be ~mcidas/workdata; for all other
users, this would be ~<user>/mcidas/data.
The reason that each user should have his/her own McIDAS working
directory is so that they can create/manipulate a set of files
without affecting other users. The most important of these files
is the user's file of McIDAS REDIRECTions, LWPATH.NAM.
When things are configured as per my recommendations, the following
command will result in the finding of only one copy of LWPATH.NAM:
DMAP LWPATH.NAM FORM=ALL
This command run from your 'mcidas' account now results in:
dmap.k LWPATH.NAM FORM=ALL
PERM SIZE LAST CHANGED FROM PATHNAME
---- --------- ------------ ------------ --------
-rw- 4390 Jul 06 15:55 R LWPATH.NAM ./LWPATH.NAM
-rw- 4390 Jul 06 15:55 M #/home/mcidas/data/LWPATH.NAM
-rw- 4390 Jul 06 15:55 M #/home/mcidas/data/LWPATH.NAM
-rw- 4390 Jul 06 15:55 M #/home/mcidas/data/LWPATH.NAM
-rw- 4390 Jul 06 15:55 M #/home/mcidas/data/LWPATH.NAM
-rw- 4369 Apr 19 08:46 M #/unidata/xcd/LWPATH.NAM
-rw- 4369 Apr 19 08:46 M #/var/data/mcidas/LWPATH.NAM
-rw- 4369 Apr 19 08:46 M #/var/data/xcd/LWPATH.NAM
35057 bytes in 8 files
OK, so in this listing there is multiple listings of the same file
(/unidata/xcd/LWPATH.NAM is the same file as /var/data/mcidas/LWPATH.NAM
and /var/data/xcd/LWPATH.NAM), but the point is that there is more
than one LWPATH.NAM that can be found through MCPATH conventions (this
is what the 'M' in the FROM column means). The reason that there is
more than one listing of the same file, /home/mcidas/data/LWPATH.NAM,
is that the /home/mcidas/data directory is specified more than once
in the user 'mcidas' MCPATH.
The copy of LWPATH that is used is the one first in the list, and
that one is found by virtue of a REDIRECTion (the 'R' in the FROM
column). If the REDIRECTion ever got deleted, then the copy in
/home/mcidas/data would get used. If that copy got deleted (there
should never be a copy of LWPATH.NAM in ~mcidas/data), then the
copy in /unidata/xcd would be used, and so on.
Again, each user should have access to one and only one copy of
LWPATH.NAM, and that copy should be in his/her own McIDAS working
directory.
2) Each user should specify their McIDAS working directory in the definition
of their MCDATA environment variable:
setenv MCDATA $HOME/workdata - for the user 'mcidas' (C shell syntax)
setenv MCDATA $HOME/mcidas/data - for all other users (C shell syntax)
3) The MCPATH for each user should contain their McIDAS working directory
as the first directory:
setenv MCHOME home_directory_of_the_user_mcidas
setenv MCPATH ${MCDATA}:$MCHOME/data:$MCHOME/help:...
4) The configuration files for McIDAS-XCD get installed into the workdata
subdirectory of the user 'mcidas'. This is where they should stay.
In particular, they should not be copied to any directory in the
MCPATH of the user 'mcidas'. Right now, you have XCD configuration
files in both ~mcidas/workdata and in /var/data/mcidas (which is a
link to /var/data/xcd).
The reason you don't want these configuration files in any directory
other than ~mcidas/workdata is that you do not want any other user
mucking with the files. Given that the ~mcidas/data and ~mcidas/help
directories should be included in each user's MCPATH, it would be
easy for these users to, intentionally or not, modify one or more
of the configuration files in ~mcidas/data, ~mcidas/help, or
in your case /var/data/mcidas.
5) There should be no McIDAS environment variables defined in the 'ldm'
account. Currently, you have blank definitions for McINST_ROOT, MCDATA,
MCPATH, etc. in your 'ldm' account. I strongly recommend removing
these from your shell definition file (which appears to be .bashrc).
These environment variables get defined in scripts run by the user
'ldm' when needed (batch.k, mcscour.sh, and xcd_run).
6) The scripts batch.k, mcscour.sh, and xcd_run which are located in the
~ldm/decoders directory on your system should be edited to change the
definition of MCDATA to ~mcidas/workdata (instead of ~mcidas/data).
7) I would move all of the BATCH files that you have created in ~mcidas/data
to the ~mcidas/workdata directory IF they are designed to only be
used by the user 'mcidas'. This way, they will not be in any
user's MCPATH (since no user's MCPATH should contain the MCDATA directory
of any other user).
8) After making the above changes, I would do some cleanup in the
~mcidas/data and /var/data/mcidas directories. In particular, I would
remove files in those directories that are also located in the
~mcidas/workdata directory. We can get into specifics about which
these files are if/when you decide to make the changes.
There may be a couple of other small things to address, but I think
that they can wait until after these other changes are made.
The thing that I am most concerned with is the amount of extra work
that you may have to do in order to install new McIDAS versions. The
setup I recommend minimizes such work, so I highly recommend it.
Talk to you later...
Tom
>From address@hidden Fri Jul 7 21:40:31 2000
>Subject: Re: 20000707: ROUTE PostProcess BATCH invocations and general setup
>(cont.)
Tom,
Ooops. 'thought I was getting a handle on this. Sorry for the
mung. Thanks for the cleanup.
At 04:15 PM 7/7/2000 -0600, you wrote:
<deleted...>
>The compositing of the topography and Unidata-Wisconsin images is done
>through the PostProcess BATCH file for the image products coming in
>the UW datastream (e.g., GE-VIS.BAT specified for the GOES-East VIS
>product, UV, etc.). Your adding execution of PostProcesses for the
>created composites created an infinite processing loop. Here is an
>example of what happened:
>
> a GOES-East Vis product arrives
> it is decoded by lwtoa3
> the PostProcess BATCH file GE-VIS.BAT is run by lwtoa3
> ....>GE-VIS.BAT composites topography and the GOES Vis image
> . composites the the vis image with the GOES-West vis image
> . (the West image comes in before the East image)
> .....you setup the N2 product, GOES-East VIS/TOPO Composite, to also run
> GE-VIS.BAT. When the composite got finished, GE-VIS.BAT was run
> again
Ouch.
>By the time I noticed this, there were about 100 invocations of batch.k
>running. I had to SUSpend the N2 product (GOES-East VIS/TOPO Composite)
>and wait until all of the processing concluded before proceeding.
>
>I then edited NEWROUTE.BAT and removed the PP= specifications on all
>of the topographic composite products (N1-N8), and then remade the
>routing table (batch.k NEWROUTE.BAT).
>
>All-in-all, it was pretty humorous seeing the number of batch.k invocations
>rise. Your machine is pretty fast, so there were lots of them :-)
I'm glad you have a sense of humor. It IS funny though. The faster
the machine, the better it can run my mistakes.
>>The PP batch files now appear in
>>the ROUTE list. I am not sure which post process files belong to which.
>>The East/West composite is an example. Do you have a list on the web
>>somewhere?
>
>The copy of DROUTE.BAT I send in the distribution has the recommended
>setup.
OK. I saw that. I need to add some auto-update stuff to the batch files
for the 'static' displays in the Map room.
>> Any how. THANK YOU for your efforts and time.
>As I am writing this, I see that Unidata-Wisconsin products are coming
>in, and PostProcesses are creating the composites.
<deleted...>
GREAT!!!!
>Now, on to the other things I wanted to touch base on.
>
>First off, you system is working, so none of the changes I am going to
>suggest are super critical. I believe, however, that if you make the
>changes, you life will be easier in the future.
I'm sure. I think I finally have the time (till the end of the month) to
go through some of the things correctly.
>1) My recommendation is for each McIDAS-X user to have his/her own
> McIDAS working directory. For the user 'mcidas' (the super user
> for McIDAS-X, -XCD), this would be ~mcidas/workdata; for all other
> users, this would be ~<user>/mcidas/data.
Yup. I started setting this up in /etc/skel so that each new user would
automatically get the correct setup. I installed default $HOME/mcidas/data
in the system startup (/etc/bashrc) but got the new machines in, before the
configuration was stabilized. This results in double paths (mcidas account
for example) for pre-existing accounts. You saw the results below. It works,
but not very well. It needs to be streamlined just for easier administration.
> The reason that each user should have his/her own McIDAS working
> directory is so that they can create/manipulate a set of files
> without affecting other users. The most important of these files
> is the user's file of McIDAS REDIRECTions, LWPATH.NAM.
Yes, This has happened a couple of times in the past. This will be solved
by the above configuration, I hope.
> When things are configured as per my recommendations, the following
> command will result in the finding of only one copy of LWPATH.NAM:
Yes. Much easier to administer and more flexable.
> DMAP LWPATH.NAM FORM=ALL
> This command run from your 'mcidas' account now results in:
<deleted...>
<...deleted>
> OK, so in this listing there is multiple listings of the same file
> (/unidata/xcd/LWPATH.NAM is the same file as /var/data/mcidas/LWPATH.NAM
> and /var/data/xcd/LWPATH.NAM), but the point is that there is more
> than one LWPATH.NAM that can be found through MCPATH conventions (this
> is what the 'M' in the FROM column means). The reason that there is
> more than one listing of the same file, /home/mcidas/data/LWPATH.NAM,
> is that the /home/mcidas/data directory is specified more than once
> in the user 'mcidas' MCPATH.
Yup. Once in the system /etc/bashrc, once in the users local .bashrc.
> The copy of LWPATH that is used is the one first in the list, and
> that one is found by virtue of a REDIRECTion (the 'R' in the FROM
> column). If the REDIRECTion ever got deleted, then the copy in
> /home/mcidas/data would get used. If that copy got deleted (there
> should never be a copy of LWPATH.NAM in ~mcidas/data), then the
> copy in /unidata/xcd would be used, and so on.
> Again, each user should have access to one and only one copy of
> LWPATH.NAM, and that copy should be in his/her own McIDAS working
> directory.
This has been a problem in the past, since it meant 14 iterations of
configuring McIDAS on each of the potential McIDAS servers, for each
user account on a machine. I resisted centralized accounts till I got the
internal network infrastructure built (was AppleTalk, with some 10BT, now
100BTX with some 10BT). This was completed in the middle of the Fall
semester. Funding was made available for 2 dual processor servers and 3
single processor McIDAS machines, and user account upgrades were delayed
till I had the new servers, that is until now. Your advice is invaluable
since I have the time to do it right as I configure the new systems, now
that they are set up. I'll retrofit the old machines when I add the student
accounts in August.
>2) Each user should specify their McIDAS working directory in the definition
> of their MCDATA environment variable:
> setenv MCDATA $HOME/workdata - for the user 'mcidas' (C shell
syntax)
> setenv MCDATA $HOME/mcidas/data - for all other users (C shell
syntax)
Yup.
>3) The MCPATH for each user should contain their McIDAS working directory
> as the first directory:
>
> setenv MCHOME home_directory_of_the_user_mcidas
> setenv MCPATH ${MCDATA}:$MCHOME/data:$MCHOME/help:...
>4) The configuration files for McIDAS-XCD get installed into the workdata
> subdirectory of the user 'mcidas'. This is where they should stay.
> In particular, they should not be copied to any directory in the
> MCPATH of the user 'mcidas'. Right now, you have XCD configuration
> files in both ~mcidas/workdata and in /var/data/mcidas (which is a
> link to /var/data/xcd).
This was a hack to solve other pathing problems.
> The reason you don't want these configuration files in any directory
> other than ~mcidas/workdata is that you do not want any other user
> mucking with the files. Given that the ~mcidas/data and ~mcidas/help
> directories should be included in each user's MCPATH, it would be
> easy for these users to, intentionally or not, modify one or more
> of the configuration files in ~mcidas/data, ~mcidas/help, or
> in your case /var/data/mcidas.
Yes, not good.
>5) There should be no McIDAS environment variables defined in the 'ldm'
> account. Currently, you have blank definitions for McINST_ROOT, MCDATA,
> MCPATH, etc. in your 'ldm' account. I strongly recommend removing
> these from your shell definition file (which appears to be .bashrc).
> These environment variables get defined in scripts run by the user
> 'ldm' when needed (batch.k, mcscour.sh, and xcd_run).
This was done to CLEAR the general system definitions. As I say, this
solved the multiple individual configuration problems, but complicated
McIDAS and LDM configuration. I will need to perform an individual
configuration for the two special accounts. Hmmm, the old csh shouldn't
read the system bashrc. I rarely do anything in LDM or McIDAS (except
some small batch files), sooo, I'll try switching these two accounts
over to csh. Everyone else uses bash. The best of both worlds?
>6) The scripts batch.k, mcscour.sh, and xcd_run which are located in the
> ~ldm/decoders directory on your system should be edited to change the
> definition of MCDATA to ~mcidas/workdata (instead of ~mcidas/data).
Yeah, I saw that.
>7) I would move all of the BATCH files that you have created in ~mcidas/data
> to the ~mcidas/workdata directory IF they are designed to only be
> used by the user 'mcidas'. This way, they will not be in any
> user's MCPATH (since no user's MCPATH should contain the MCDATA directory
> of any other user).
Yup. This is on the agenda. I will probably place user batch files in
another directory (util?) and include it in the path. I originally did a
locate for BAT files and just started building them in the same directory,
knowing I wasn't mucking around with the McIDAS data paths.
>8) After making the above changes, I would do some cleanup in the
> ~mcidas/data and /var/data/mcidas directories. In particular, I would
> remove files in those directories that are also located in the
> ~mcidas/workdata directory. We can get into specifics about which
> these files are if/when you decide to make the changes.
Yup, I had copied the whole ~mcidas/data & ~mcidas/workdata directories
into the /var/data/mcidas directory to solve other pathing problems,
while under the gun to get the program running in time for some classes.
I know its a mess...and I need the wasted disk space.
>There may be a couple of other small things to address, but I think
>that they can wait until after these other changes are made.
>
>The thing that I am most concerned with is the amount of extra work
>that you may have to do in order to install new McIDAS versions. The
>setup I recommend minimizes such work, so I highly recommend it.
I agree. And, I well appreciate your recommendations. Most of this was
already planned, but it is reassuring to get it from the expert! And,
I have gained a much deeper understanding of what goes on in McIDAS as
a fringe.
>Talk to you later...
Absolutly. Once McIDAS is up to spec, there is this thing out there
called McADDE, that looks to be a handy piece of machina ...
How many brews is this gonna' cost me, anyhooo?....Well Worth It!!
jdm
>
>Tom
>***************************************************************************
* <
>Unidata User Support UCAR Unidata
Program <
>(303)497-8644 P.O. Box
3000 <
>address@hidden Boulder, CO
80307 <
>---------------------------------------------------------------------------
- <
>Unidata WWW Service http://www.unidata.ucar.edu/
<
>***************************************************************************
* <
>
James D. Marco, address@hidden, address@hidden
Programmer/Analyst, System/Network Administration,
Computer Support, Et Al.
Office: 1020 Bradfield Hall, Cornell University
Home: 302 Mary Lane, Varna (607)273-9132
Computer Lab: 1125 Bradfield (607)255-5589
