[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

20000416: McIDAS GINI ADDE server installation at Plymouth State

>From: Jim Koermer <address@hidden>
>Organization: Unidata Program Center
>Keywords: Plymouth NOAAPORT GINI ADDE GINI.CFG DSINFO IMGLIST IMGDISP how it 
>works

Jim,

I am CCing this go Gilbert since he might be interested in how ADDE actuall
works.

As promised, the following is a blow-by-blow of how the McIDAS remote
ADDE server is setup on your machine, snow.plymouth.edu.  I present the
information in what seems like the most logical way to me: a synopsis
of what happens in an ADDE transaction:

o a site learns that it may feed GINI imagery by McIDAS ADDE off or your
  system.  A good example of this is Gilbert's request to you and your
  granting him access privilege.  One part of the granting process is you
  providing the end user with the name of your machine running the
  McIDAS remote ADDE server; the other part is you providing that user
  with the name of the dataset that he is allowed to access.  In
  this case, you have provided Gilbert with these two pieces of information:

  machine: snow.plymouth.edu
  dataset: RTGINI

  ADDE has no "discovery mechanism".  This means that a site can not
  interrogate your system to find out what datasets you have available
  through your remote ADDE server.  In addition, there is no general
  protocol for "pinging" random machines on the net to see if they are
  running a McIDAS remote ADDE server.  This is inability to discover
  information can be viewed as one level of security.

o that site sets up their McIDAS or MetAPPS application to "point" to
  your machine.  This means that the user does the following in McIDAS:

  DATALOC ADD RTGINI snow.plymouth.edu

  What this means to McIDAS is that when a user runs any command that
  wants to access data from a dataset called RTGINI, his McIDAS process
  first asks the following question: where do I look for the dataset
  RTGINI?  The answer to this question is controlled in the user's own
  McIDAS session through a table that is created and maintained by
  McIDAS DATALOC commands.  The name of this table is up to the user,
  but is the one that is declared in his/her MCTABLE_WRITE environment
  variable.  Each McIDAS user should have this environment variable
  defined to be the file MCTABLE.TXT which should exist in his/her
  McIDAS-X working directory:

  setenv MCDATA /home/user/mcidas/data
  setenv MCTABLE_WRITE "${MCDATA}/MCTABLE.TXT"

  Again, this is the table that the user 'user' can write.

  To complicate things a little, however, McIDAS has the notion of user
  and site defined DATALOCation tables.  The set of DATALOCation tables
  is defined by each user in the environment variable MCTABLE_READ in
  his/her session.  MCTABLE_READ can contain one or more DATALOCation
  tables, and they will be searched in order, from left to right, when
  trying to find the machine to request data from when the user asks
  for data from a dataset.  The typical setup for MCTABLE_READ is as
  follows:

  setenv MCTABLE_READ "${MCDATA}/MCTABLE.TXT;/home/mcidas/data/ADDESITE.TXT"

  The user's own DATALOCation table is searched first, and then the one
  setup for a site by the McIDAS superuser, mcidas
  (~mcidas/data/ADDESITE.TXT).

o the user asks his/her McIDAS session to list back the contents of
  the dataset RTGINI:

  DSINFO ALL RTGINI

  Data Set INFO (DSINFO) makes the request: list back ALL types of data
  in the dataset called RTGINI.  The first thing that happens in this
  query is a search through files listed in the user's MCTABLE_READ for
  the location of the dataset RTGINI.  Since this was specified by the
  user to be snow.plymouth.edu, your system is contacted and asked.

  Now, which port is contacted on your system is a function of the
  definition of an environment variable named MCCOMPRESS in the user's
  McIDAS-X session.  If this environment variable exists (is defined to
  be anything), then the request goes to port 503 on your machine.  If
  MCCOMPRESS does not exist in the user's session, the request goes to
  port 500.

  The difference in the port access is as you might expect: requests
  sent to port 500 will result in transfer of information that is not
  compressed.  Requests sent to port 503 results in transfers of
  invormation that are compressed on-the-fly.  The great, great
  majority of requests will go to port 503 as the transfer time is
  greatly reduced when the data sent back is compressed.

o your system receives a request on port 503 (assuming MCCOMPRESS
  is defined in the requesting user's session) which inetd has
  to react to.

o inetd looks in /etc/inetd.conf to see what it is supposed to do with
  the request on port 503.  Since the remote ADDE server was setup on
  your machine, and since TCP wrappers were setup on the McIDAS
  inetd services, your inetd.conf will have lines looking like:

mcserv  stream  tcp     nowait  mcadde  /usr/local/libexec/tcpd 
/home/mcidas/bin/mcservsh       -H /home/mcidas
mccompress      stream  tcp     nowait  mcadde  /usr/local/libexec/tcpd 
/home/mcidas/bin/mcservsh       -H /home/mcidas

  These lines would look a little different on different machines since
  the location of the TCP wrapper will be different.

  The TCP wrapper routine, tcpd, is called to evaluate the request on
  port 503.  If the IP number of the machine making the request is not
  allowed in the TCP wrapper configuration file,
  hosts.allow/hosts.deny, nothing more is done.  If the machine making
  the request is NOT denied, then the TCP wrapper runs the McIDAS ADDE
  Bourne shell script, /home/mcidas/bin/mcservsh.

o /home/mcidas/bin/mcservsh tests for the existence of the file .mcenv
  in the McINST_ROOT directory on the machine.  If .mcenv exists, it
  is read so that McIDAS environment variable definitions can become
  active.  All McIDAS routines need certain environment variables set
  in order to run correctly.  .mcenv on your system is located in the
  HOME directory of the user 'mcadde' (which just so happens to be
  the same as the HOME directory of the user 'mcidas'; this is by design).
  On your system, this looks like:

umask 002
MCDATA=/home/mcidas/workdata
MCPATH=${MCDATA}:/home/mcidas/data:/home/mcidas/help
MCGUI=/home/mcidas/bin
MCTABLE_READ="/home/mcidas/mcidas/data/MCTABLE.TXT"
MCTABLE_WRITE="/home/mcidas/mcidas/data/MCTABLE.TXT"
PATH=${MCGUI}:/bin:/usr/bin:/usr/local/bin:/usr/X11R6/bin:/sbin:/usr/sbin
export MCDATA MCPATH MCGUI MCTABLE_READ MCTABLE_WRITE PATH
cd $MCDATA

o /home/mcidas/bin/mcservsh then 'exec's $McINST_ROOT/bin/mcserv, a compiled
  McIDAS executable.  'mcserv' then interprets the request that was sent
  from the user's machine to see what kind of McIDAS data it is asking
  for.  In the example we are working on, the request is to list back
  the types of data in the RTGINI dataset.  So, if RTGINI is not defined
  on your system, an error message will be sent back and the transation
  will be terminated.  On your system, RTGINI is defined, so the return
  will look like:


        Dataset Names of Type: IMAGE in Group: RTGINI

Name         NumPos   Content
------------ ------   --------------------------------------
GAN8KIR       9999    GINI 8 km 10.7 um Alaska National
GAN8KVIS      9999    GINI 8 km Visible Alaska National
GAN8KWV       9999    GINI 8 km WV Alaska National
GAR16KWV      9999    GINI 16 km WV Alaska Regional
GAR2KVIS      9999    GINI 2 km VIS Alaska Regional
GAR8K12       9999    GINI 8 km 12.0 um Alaska Regional
GAR8K39       9999    GINI 8 km 3.9 um Alaska Regional
GAR8KIR       9999    GINI 8 km 10.7 um Alaska Regional
GE1KVIS       9999    GINI 1 km VIS East CONUS
GE2KVIS       9999    GINI 2 km VIS East CONUS
GE4K12        9999    GINI 4 km 12.0 um East CONUS
GE4K39        9999    GINI 4 km 3.9 um East CONUS
GE4KIR        9999    GINI 4 km 10.7 um East CONUS
GE4KVIS       9999    GINI 4 km VIS East CONUS
GE8KWV        9999    GINI 8 km WV East CONUS
GHN14KIR      9999    GINI 14 km IR Hawaii National
GHN14KVIS     9999    GINI 14 km VIS Hawaii National
GHN14KWV      9999    GINI 14 km WV Hawaii National
GHR1KVIS      9999    GINI 1 km VIS Hawaii Regional
GHR4K12       9999    GINI 4 km 12.0 Hawaii Regional
GHR4K39       9999    GINI 4 km 3.9 um Hawaii Regional
GHR4KIR       9999    GINI 4 km 10.7 um Hawaii Regional
GHR8KWV       9999    GINI 8 km WV Hawaii Regional
GNC24K12      9999    GINI 24 km 12.0 um Nhem-Composite
GNC24K39      9999    GINI 24 km 3.9 um Nhem-Composite
GNC24KIR      9999    GINI 24 km 10.7 um Nhem-Composite
GNC24KVIS     9999    GINI 24 km VIS Nhem-Composite
GNC24KWV      9999    GINI 24 km WV Nhem-Composite
GPN8KIR       9999    GINI 8 km 10.7 um Puerto Rico National
GPN8KVIS      9999    GINI 8 km VIS Puerto Rico National
GPN8KWV       9999    GINI 8 km WV Puerto Rico National
GPR1KVIS      9999    GINI 1 km VIS Puerto Rico Regional
GPR4K12       9999    GINI 4 km 12.0 um Puerto Rico Regional
GPR4K39       9999    GINI 4 km 3.9 um Puerto Rico Regional
GPR4KIR       9999    GINI 4 km 10.7 um Puerto Rico Regional
GPR8KWV       9999    GINI 8 km WV Puerto Rico Regional
GSN8K12       9999    GINI 8 km 12.0 um Super-National
GSN8K39       9999    GINI 8 km 3.9 um Super-National
GSN8KIR       9999    GINI 8 km 10.7 um Super-National
GSN8KVIS      9999    GINI 8 km VIS Super-National
GSN8KWV       9999    GINI 8 km WV Super-National
GW1KVIS       9999    GINI 1 km VIS West CONUS
GW2KVIS       9999    GINI 2 km VIS West CONUS
GW4K12        9999    GINI 4 km 12.0 um West CONUS
GW4K39        9999    GINI 4 km 3.9 um West CONUS
GW4KIR        9999    GINI 4 km 10.7 um West CONUS
GW4KVIS       9999    GINI 4 km VIS West CONUS
GW8KWV        9999    GINI 8 km WV West CONUS

    No Datasets found of Type: POINT in Group: RTGINI
    No Datasets found of Type: VPROF in Group: RTGINI
    No Datasets found of Type: GRID  in Group: RTGINI
    No Datasets found of Type: TEXT  in Group: RTGINI
    No Datasets found of Type: NAV   in Group: RTGINI
DSINFO -- done

  The only kind of data that was found to be defined in your RTGINI was
  data of type IMAGE.  No POINT, VPROF, GRID, TEXT, or NAV data was
  found in RTGINI.

  The listing shows not only that there was data of type image
  in RTGINI, but it shows the group names of that data: GAN8KIR, GAN8KVIS,
  GAN8KWV, GAR16KWV, etc.

  The datasets defined on your system and the groups in those datasets
  are defined in the file RESOLV.SRV, which can exist in any directory
  in the MCPATH set of directories defined for the server processes.
  MCPATH is one of the things defined in the .mcenv file that was
  read by the Bourne shell script, mcservsh.  RESOLV.SRV is typically
  found in the first directory in MCPATH.  That directory for Unidata
  sites is defined by the environment variable MCDATA.  This would
  be the /home/mcidas/workdata directory in your case (and 99.99% of
  all others).

o since the user found that:

  o you really are running an ADDE server (i.e., the DSINFO request
    did not time out with no response)

  o you have a dataset named RTGINI defined on your system

  o there is a group named GW1KVIS in RTGINI on your system

  S/he might decide to take a look at what is in that dataset/group.
  S/he could list the contents of that dataset/group:

  IMGLIST RTGINI/GW1KVIS.ALL

  (the .ALL says to list ALL of the contents of RTGINI/GW1KVIS)

  or s/he might decide to display the most recent item in the
  RTGINI/GW1KVIS dataset/group:

  IMGDISP RTGINI/GW1KVIS STA=KDEN EU=IMAGE SF=YES REFRESH='EG;MAP h'

o what actually happens when the user asks for a listing of data that
  is available or a display of the latest item in the dataset is:

  o his/her DATALOC tables are searched to find out which machine to
    send the request to

  o the serving machine is contacted

  o inetd either fires up /home/mcidas/bin/mcservsh directly, or it
    fires up a TCP wrapper that, if the requesting machine is not
    denied/is allowed, fires up /home/mcidas/bin/mcservsh.

  o mcservsh execs /home/mcidas/bin/mcserv

  o mcserv interprets the request to see what kind of data is being
    asked for.  It does this by searching RESOLV.SRV to find the
    requested dataset/group.  If it finds it, it will know what
    type of data is being asked for.  In this case, the type of the
    data is IMAGE

  o mcserv execs either /home/mcidas/bin/adirserv (for lists) or
    /home/mcidas/bin/agetserv (for displays)

  o adirserv/agetserv look at the request to see if the type of
    IMAGE data being requested is of type AREA or something else.
    In this case, it is of type GINI, so adirserv/agetserv know
    that a different process must handle the request

  o adirserv/agetserv execs either giniadir (for lists) or giniaget
    (for displays) since the type of IMAGE data requested was GINI.
    The name of the routine that is execed is put together on the
    fly from the type of data (lower cased) plus adir/aget.  So,
    the name of the routine that is being run is giniadir or giniaget
    depending on the function requested

o giniadir/giniaget startup and interpret the request that was made of
  them.  One part of the request is information from the requesting
  process (i.e., from the user asking for a list/display), the other
  part of the request is information from the dataset definition
  file on the server machine, RESOLV.SRV.  For GINI imagery, one
  piece of the information in RESOLV.SRV is the name of a configuration
  file that giniadir/giniaget must read in order to figure out
  how to satisfy the user's request.

  The configuration file specified in your remote ADDE GINI server is
  /home/mcidas/workdata/GINI.CFG.  GINI.CFG contains the following
  kinds of information:

  o where are data files to be found (DIRMASK)
  o how are data files named (FILEMASH)
  o who is allowed to request data (IPMASK)

  The format of the Unidata ADDE configuration files of which GINI.CFG
  is one example is:

  DIRMASK= <regular expression>
  FILEMASK= <regular expression>
  IPMASK= <regular expression>

  If the location and naming of data files that giniadir/giniaget is
  simple enough, the above three variables would be all that is
  needed.  On your system, for example, the location of GOES-East files
  is /data/noaaport2/sat; the location for GOES-West files is
  /data/noaaport3/sat (I may have these reversed).  The files locations
  can, therefore, be expressed by the regular expression:

  DIRMASK=/data/noaaport[23]/sat
  
  I can also specify the set of IP addresses of machines allowed to
  make requests by specifying a simple or complex regular expression
  for IPMASK.  Right now, IPMASK on your system is defined as:

  IPMASK=*

  All requestors are allowed.

  The names of the files that represent the various groups in the RTGINI
  dataset are not nearly so simple, however.  GOES-East 1 km Visible
  imagery files look like 0004162115_gecvi.sat, GOES-West 3.9 um
  IR images look like 0004162045_gwci3.sat, and so on.
  
  You may think that one could simply specify FILEMASK as '*' and
  find the appropriate files (since '*' matches everything) and you
  would be correct.  The object, however, is to search as few files
  as possible in order to fulfill a user's request.  For this reason,
  I added some shortcuts that can be used in the configuration
  file.  One of the shortcuts is the use of the group portion of
  the dataset/group name of the data requested in the definitions.
  Here is an example:

  GW1KVIS_FILE=*_gwcvi.sat

  This is of the form: \TYPE_FILE where \TYPE represents the group
  portion of the dataset/group name.  This form of file name specification
  takes precedence over the FILEMASK= specifier in the configuration
  file.

  To generalize things, I added the capability to define group specific
  locations, names, and access privilege.  These definitions all take
  precedence over their counterparts DIRMASK, FILEMASK, IPMASK.  What I
  mean by this is the following:  If you chose to store GOES-West 3.9
  um IR images in the /data/gini/goeswest/3.9 directory, and you chose
  to name them YYMMDDHHMM_goesW_3.9.sat, and you chose to make these
  files only available to a machine with IP address 128.117.140.56,
  then you would define the following in GINI.CFG to locate the files:

  GW1KVIS_DIR=/data/gini/goeswest/3.9
  GW1KVIS_FILE=*_goesW_3.9.sat
  GW1KVIS_IP=128.117.140.56
  
  This will be used even though you might have defined:

  DIRMASK=/data/noaaport[23]/sat
  FILEMASK=*
  IPMASK=*

  since the \TYPE_DIR, \TYPE_FILE, and \TYPE_IP forms take precedence
  over the DIRMASK, FILEMASK, and IPMASK forms, respectively.

  With the setup of GINI.CFG on your system, when the request from the
  user comes in to display the most recent RTGINI/GW1KVIS image, the
  GINI ADDE server, giniaget, reads GINI.CFG and finds that:

  o it should look in either /data/noaaport2/sat or /data/noaaport3/sat
    (DIRMASK=/data/noaaport[23]/sat)

  o it should look for files named *_gwcvi.sat (from GW1KVIS_FILE=*_gwcvi.sat)

  A regular expression describing the fully qualified name of the desired
  data is then formed from the combination of DIRMASK and FILEMASK 
  settings:

  name == /data/noaaport[23]/sat/*_gwcvi.sat

  Unix globbing is then used to generate a list of files matching this
  expression in the exact same way that an 'ls' does.  The files in
  this list are then opened, headers read, and date/time sorted so
  that the most recent image can be returned back to the user.

Phew!  Seems kinda complicated, but the entire process is incredibly
flexible.

To answer a simple question like 'how do I limit ADDE access to my
machine' is at least two fold:

1) define the set of IP address allowed to request data from you
   ADDE server.  This will limit this set of IP addresses for ALL
   datasets.

2) define the set of IP addresses allowed to request data from
   your RTGINI dataset through the IPMASK setting in the GINI.CFG
   configuration file.  This is much more flexible that the use
   of TCP wrappers since the control is on a per-dataset basis.

I sincerely hope that the presentation above has not made the entire
process more complicated that it really is.  I felt, however, that it
was necessary to give you the complete overview so you would have
a feel for exactly what is going on behind the scenes on your machine
as users come in and browse your GINI data.

Please let me know if/when you have any questions about the process.

Tom

>From address@hidden  Sun Apr 16 22:22:51 2000
>Subject: Re: 20000416: McIDAS GINI ADDE server installation at Plymouth State

>Tom,

>Thanks! Interesting. But that means that, if I read it correctly, I can
>only be pointed to one RTGINI server at a time. If UNIDATA is faster than
>Plymouth, say...I will have to delete the IP address and add the new one
>(or name). No problem, I was just curious.

>Thanks again. 3 ADDE servers...hmmm. I will check on UIUC, but I doubt if
>they will get one (their own NOAAPORT feed).

>Gilbert

>From address@hidden  Wed Apr 19 09:55:40 2000
>Subject: Re: 20000416: McIDAS GINI ADDE server installation at Plymouth State

Tom,

Thanks for all the info about what you did to install ADDE. Hopefully,
I'll have time to digest all of it. When you do announce this to the
Unidata community, you may want to mention our web site at the following
address:

        http://snow.plymouth.edu/mcidas_sat.html

so that they can get an idea of the all GINI data that are available. It
may entice some non-ADDE sites to get going on ADDE.

Here, we just finished a successful search for a new synoptic person.
I'll be glad when he gets here for the fall semester. It should provide
us with a person that will have a students more involved with the
various weather visualization packages that we have available.

                        Jim

-- 
James P. Koermer             E-Mail: address@hidden
Professor of Meteorology     Office Phone: (603)535-2574
Natural Science Department   Office Fax: (603)535-2723
Plymouth State College       WWW: http://vortex.plymouth.edu/
Plymouth, NH 03264