[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

20030312: partial LDM-6 installation on weather2

>From: Unidata User Support <address@hidden>
>Organization: Unidata Program Center/UCAR
>Keywords: LDM-6

Hi Gilbert,

While tracing down a problem caused by folks feeding UNIWISC imagery
from unidata2.ssec.wisc.edu, I noticed that at some time in the not too
distant past you had such a feed (by virtue of a request for UNIDATA
from unidata2, and UNIDATA == WMO|UNIWISC).  The problem was that
unidata2 has been generating UNIWISC imagery in a test mode for quite
some time (it will become the primary source of UNIWISC data in the
very near future).  In addition to the regular UNIWISC imagery were two
new products that are going to be added to the stream.  The McIDAS
routing table, ROUTE.SYS, was not setup to handle these new images, and
the ldm-mcidas 7.8 decoder pnga2area had a bug that would cause the
routing table to be corrupted for images that it was not setup for.

So, seeing that you had feed UNIWISC images from unidata2, I jumped
onto weather2 and saw that your ROUTE.SYS file had been damaged.  To
correct the problem, I downloaded the McIDAS v2002b addendum and built
and installed it.  I did this mostly to get the latest version of
the routing table creation BATCH file, DROUTE.BAT.

Since the rebuild of McIDAS is better done with the LDM shut down, I
had occasion to also logon to weather2 as 'ldm'.  While 'ldm', I
stopped and restated the LDM, and then did the first steps in building
the most recent LDM-6 release candidate, ldm-6.0.2.tar.Z:

<login as 'ldm'>
cd ~ldm
ftp ftp.unidata.ucar.edu
  <user> anonymous
  <pass> your_full_email_address
  cd pub/ldm/test
  binary
  get ldm-6.0.2.tar.Z
  quit
zcat ldm-6.0.2.tar.Z | tar xvf -

cd ~ldm/ldm-6.0.2/src
./configure
make && make install

The step I couldn't finish was the 'make install_setuids' as this must
be done as 'root'.  So, if you would like to start using the LDM-6
release candidate, you will need to do the following:

<login to weather2 as 'root'>
cd ~ldm/ldm-6.0.2/src
make install_setuids
<logoff as 'root'>

The next phase is the configuration of the LDM-6 version of 'ldmadmin'.
The things to note here are:

1) if running 'uname -n' on your machine does not return back a fully
   qualified hostname, one would need to set the $hostname in ldmadmin.  
   'uname -n' on weather2 does return a fully qualified hostname,
   so you don't have to worry about this.

2) change the size of the queue and pqsurf queue to match the sizes
   already in use on your system.

   Currently, the queue size on weather2 is small: 200MB.  The default
   size sent out in 'ldmadmin' is 400 MB.  If you are using weather2 to
   relay data to other sites, you should consider making this even
   bigger than the default.  I would recommend something like 700 MB
   depending on what data you are trying to relay.  For comparison, we
   use 2 GB queues for IDD relay machines under our control (except for
   thelma where we are using a 4 GB queue, but it is moving a LOT of
   data).

   I have no comment on the pqsurf queue size.

3) setting the number of LDM log files you want to keep online.  Right
   now you have this set to 2.  You may want to increase this.
   We typically keep this at 14 for the IDD relay machines under
   our control.

After adjusting values in ~ldm/ldm-6.0.2/bin/ldmadmin, you will need to
stop your current LDM from running; change the runtime link; and then
restart:

<log on as 'ldm' NOT 'root'!>
cd ~ldm
ldmadmin stop
<wait for all LDM processes to exit>
rm runtime
ln -s ldm-6.0.2 runtime
rehash
ldmadmin start


The following is some information I have provided to top level IDD
relay sites regarding LDM-6:


Notes:

- The online documentation for LDM is still that for LDM-5.  We
  (Steve Emmerson) are working on upgrading it for the general release.


LDM 6 offers a number of new ideas that we are excited about.  Here's
an overview:

o request lines to the same host in ldmd.conf are no longer accumulated
  into a single rpc.ldmd invocation.  This allows a user to split feeds
  without having to create machine name aliases in /etc/hosts.

o the size of the HEREIS - COMINGSOON "fence" is now user configurable.
  In LDM-5, all products smaller than 16384 bytes would be sent using
  the HEREIS protocol.  In LDM-6, this "fence" is settable on a
  per-request line basis.  The default for the "fence" size is now
  essentially infinity (actually, it is size of a uint).  You specify
  the "fence" size as the 4th parameter on an ldmd.conf request line.
  LDM-6 understands a couple of mnemonics for minimim and maximum
  "fence" sizes:

request IDS|DDPLUS ".*" my.upstream.site PRIMARY    (or PRI)
request IDS|DDPLUS ".*" my.upstream.site ALTERNATE  (or ALT)

or equivalently:

request IDS|DDPLUS ".*" my.upstream.site MAXIMIM    (or MAX)
request IDS|DDPLUS ".*" my.upstream.site MINIMUM    (or MIN)

  We think that the PRI (uint_max) and ALT (zero) values are basically
  all that a site will ever need/use, but the values are configurable
  nonetheless.

o products sent with the COMINGSOON/BLKDATA protocol are sent in one
  chunk after a transaction is made between the client and server in
  which the client says that it wants the product.  In LDM-5, the same
  client/server transaction was made and then the product was sent in
  16384 BLKDATA chunks.

o LDM-6 is backward/forward compatible with LDM-5.  LDM-5 clients
  can receive data from LDM-6 servers, and LDM-6 clients can receive
  data from LDM-5 servers.  When LDM-5 talks to LDM-6, it does so using
  LDM-5 protocols.  LDM-6 running LDM-5 protocols is more efficient
  than LDM-5.

o LDM-6 can use the same product queue that LDM-5 uses.  This allows
  a site to install the new version and start running without remaking
  the product queue.

o LDM-6 is much more capable of delivering data to electronically
  remote sites.  One site we have been using for testing is located on
  the edge of the Amazon in Belem, Brazil.

o LDM-6's ldmadmin now does several things for you that you had
  to do by hand in LDM-5:

  o put you in the ~ldm directory at LDM startup
  o create the ~ldm/logs/ldmd.log file if it does not already exist
  o determine your machine's hostname using a 'uname -n'.  If
    this does not result in a fully qualified hostname, ldmadmin
    will complain and not start the LDM.  In this case, you
    need to edit ldmadmin and set the hostname:

change:

chop($hostname = `uname -n`);
# $hostname = "your.hostname.here";

to:

# chop($hostname = `uname -n`);
$hostname = "yourown.hostname.edu";

o LDM-6's ldmadmin now waits until the rpc.ldmd processes have all
  exited before returning you to the Unix command prompt

o LDM-6 now understands CONDUIT and CRAFT as the NMC2 and NEXRD2
  streams.

o Some early tests using synthesized data showed that LDM-6 was capable
  of moving 8.4 times more data than LDM-5.  The increase in "efficiency"
  using real data has, however, not yet been quantified, but qualitatively
  we can assure that the increase in "efficiency" is real.  For reference,
  the early tests that showed the 8.4 times increase was conducted between
  a machine here at the UPC, and a Linux PC running at the Universidade
  Federal do Para in Belem, Brazil using synthetic products that were
  each 200KB in size.  Also, during the test, the client LDM did not
  write into a product queue.

N.B. (nota bene/yo):

During stress testing of the LDM-6, we learned that there is a class of
"pathological" regular expressions that adversly affect the performance
of LDM servers.  Steve sent a note to ldm-users explaining how to
identify pathological cases, and what to do to correct these.
Unfortunately, it is the client requesting data that has control over
these regular expressions, so if you find these in your ldmd.log files,
you will need to contact the downstream host's administrator and ask
him/her to correct the pattern.  Steve has included a new function in
LDM-6, regex, that can be used to test regular expressions for
pathologicalness ;-).  Check the regex man page for more information
after you install LDM-6.

To upgrade to LDM-6, then you should do the following:

1) make backup copies of your ~ldm/etc/ldmd.conf file(s).  While
   you are at it, you could make a backup copy of your ~ldm/etc/pqact.conf
   file, but this is optional since it will not be affected by moving
   to LDM-6.

2) FTP LDM-6 release candidate 6:

<login as 'ldm'>
cd ~ldm
ftp ftp.unidata.ucar.edu
  <user> anonymous
  <pass> your_full_email_address
  cd pub/ldm/test
  binary
  get ldm-6.0.2.tar.Z
  quit
zcat ldm-6.0.2.tar.Z | tar xvf -

3) build LDM-6. Note that LDM-6 is built with '-O' optimization
   turned on.  If you want to build otherwise, you will want to set
   the Unix environment variable CFLAGS _before_ you run configure.

   Also, you should define the Unix environment variable LDMHOME
   _before_ you run configure:

setenv CFLAGS -O
setenv LDMHOME your_ldm_home_directory

cd ~ldm/ldm-6.0.2/src
./configure
make && make install

4) finish the LDM install as 'root':

sudo make install_setuids

5) adjust entries in ldmadmin to match those from your existing
   ldmadin.  This should boil down to you changing the product queue
   size; the number of log files that you want to keep online (we keep
   14 on our IDD machines); uncommenting out the entry for udunits if
   you are running things like gribtonc; filling in your full hostname
   if a 'uname -n' does not return a fully qualified hostname; etc.

6) shutdown your current LDM waiting for all LDM processes to exit.
   This step is easier in LDM-6 since ldmadmin won't return you to
   the Unix/Linux prompt before processes have exited.

7) change the runtime link to point at ldm-6.0.2

cd ~ldm
rm runtime && ln -s ldm-6.0.2 runtime

8) verify that your operating system has finished flushing the LDM
   queue from memory to disk.  I found that this was an important
   step in getting things to work smoothly on thelma.  I did
   the verification by watching the iostat listing of top.  As soon
   as it quites down, you are ready to go.  Our experience is that
   the larger the queue, the longer this step takes.  We even added
   a 'sync' call from ldmadmin to tell the OS to sync to disk before
   shutting down LDM processes.

9) start LDM-6:

rehash                <- C shell only
ldmadmin start

I think I have covered all bases in this note, but you never know.
The good news is that you guys already know all of the upgrade procedure,
so most of the above was unneeded.

Cheers,

Tom