[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: 20030227: LDM-6 now available for toplevel IDD relay nodes


David,

The defaults for the request lines using your ldmd.conf setting from 5.x
will be the same as adding tha MAX" or "PRIMARY" designation. This will
be the desired configuration, unless you are requesting the same
feed from more than one host (a redundant concurent feed).
In that case, you will want to designate one of them as "ALTERNATE"
which will employ the COMINGSOON mechanism to check if you want the
product before sending it.

Steve Chiswell

On Fri, 28 Feb 2003, David Wojtowicz wrote:

> flood.atmos.uiuc.edu is now running v6.0    Using existing ldmd.conf file.
> Not clear to me if any of the new settings should be added to improve
> performance or are the defaults when the new directives are not specified
> OK?
>
> --------------------------------------------------------
> David Wojtowicz, Sr. Research Programmer
> Department of Atmospheric Sciences Computer Services
> address@hidden   (217) 333-8390
>
>
> ----- Original Message -----
> From: "Unidata Support" <address@hidden>
> To: <address@hidden>; <address@hidden>;
> <address@hidden>; <address@hidden>;
> <address@hidden>; <address@hidden>; <address@hidden>
> Cc: <address@hidden>; <address@hidden>;
> <address@hidden>
> Sent: Thursday, February 27, 2003 4:03 PM
> Subject: 20030227: LDM-6 now available for toplevel IDD relay nodes
>
>
> > >From: Unidata Support <address@hidden>
> > >Organization: UCAR/Unidata
> > >Keywords: 200212271733.gBRHXAt25394 LDM 6 release
> >
> > Hi Guys,
> >
> > We have a new version of the LDM, LDM-6, that we like to see you
> > install at your sites before announcing its general availability to the
> > community.  Note: the online documentation for LDM is still that for
> > LDM-5.  We (Steve Emmerson) are working on upgrading it for the general
> > release.
> >
> > LDM 6 offers a number of new ideas that we are excited about.  Here's
> > an overview:
> >
> > o request lines to the same host in ldmd.conf are no longer accumulated
> >   into a single rpc.ldmd invocation.  This allows a user to split feeds
> >   without having to create machine name aliases in /etc/hosts.
> >
> > o the size of the HEREIS - COMINGSOON "fence" is now user configurable.
> >   In LDM-5, all products smaller than 16384 bytes would be sent using
> >   the HEREIS protocol.  In LDM-6, this "fence" is settable on a
> >   per-request line basis.  The default for the "fence" size is now
> >   essentially infinity (actually, it is size of a uint).  You specify
> >   the "fence" size as the 4th parameter on an ldmd.conf request line.
> >   LDM-6 understands a couple of mnemonics for minimim and maximum
> >   "fence" sizes:
> >
> > request IDS|DDPLUS ".*" my.upstream.site PRIMARY
> > request IDS|DDPLUS ".*" my.upstream.site ALTERNATE
> >
> > or equivalently:
> >
> > request IDS|DDPLUS ".*" my.upstream.site MAX        (or MAXIMIM)
> > request IDS|DDPLUS ".*" my.upstream.site MIN        (or MINIMUM)
> >
> >   We think that the MAX (uint) and MIN (zero) values are basically all
> >   that a site will ever need/use, but the values are configurable
> >   nonetheless.
> >
> > o products sent with the COMINGSOON/BLKDATA protocol are sent in one
> >   chunk after a transaction is made between the client and server in
> >   which the client says that it wants the product.  In LDM-5, the same
> >   client/server transaction was made and then the product was sent in
> >   16384 BLKDATA chunks.
> >
> > o LDM-6 is backward/forward compatible with LDM-5.  LDM-5 clients
> >   can receive data from LDM-6 servers, and LDM-6 clients can receive
> >   data from LDM-5 servers.  When LDM-5 talks to LDM-6, it does so using
> >   LDM-5 protocols.  LDM-6 running LDM-5 protocols is more efficient
> >   than LDM-5.
> >
> > o LDM-6 can use the same product queue that LDM-5 uses.  This allows
> >   a site to install the new version and start running without remaking
> >   the product queue.
> >
> > o LDM-6 is much more capable of delivering data to electronically
> >   remote sites.  One site we have been using for testing is located on
> >   the edge of the Amazon in Belem, Brazil.
> >
> > o LDM-6's ldmadmin now does several things for you that you had
> >   to do by hand in LDM-5:
> >
> >   o put you in the ~ldm directory at LDM startup
> >   o create the ~ldm/logs/ldmd.log file if it does not already exist
> >   o determine your machine's hostname using a 'uname -n'.  If
> >     this does not result in a fully qualified hostname, ldmadmin
> >     will complain and not start the LDM.  In this case, you
> >     need to edit ldmadmin and set the hostname:
> >
> > change:
> >
> > chop($hostname = `uname -n`);
> > # $hostname = "your.hostname.here";
> >
> > to:
> >
> > # chop($hostname = `uname -n`);
> > $hostname = "yourown.hostname.edu";
> >
> > o LDM-6's ldmadmin now waits until the rpc.ldmd processes have all
> >   exited before returning you to the Unix command prompt
> >
> > o LDM-6 now understands CONDUIT and CRAFT as the NMC2 and NEXRD2
> >   streams.
> >
> > o Some early tests using synthesized data showed that LDM-6 was capable
> >   of moving 8.4 times more data than LDM-5.  The increase in "efficiency"
> >   using real data has, however, not yet been quantified, but qualitatively
> >   we can assure that the increase in "efficiency" is real.  For reference,
> >   the early tests that showed the 8.4 times increase was conducted between
> >   a machine here at the UPC, and a Linux PC running at the Universidade
> >   Federal do Para in Belem, Brazil using synthetic products that were
> >   each 200KB in size.  Also, during the test, the client LDM did not
> >   write into a product queue.
> >
> > N.B. (nota bene/yo):
> >
> > During stress testing of the LDM-6, we learned that there is a class of
> > "pathological" regular expressions that adversly affect the performance
> > of LDM servers.  Steve sent a note to ldm-users explaining how to
> > identify pathological cases, and what to do to correct these.
> > Unfortunately, it is the client requesting data that has control over
> > these regular expressions, so if you find these in your ldmd.log files,
> > you will need to contact the downstream host's administrator and ask
> > him/her to correct the pattern.  Steve has included a new function in
> > LDM-6, regex, that can be used to test regular expressions for
> > pathologicalness ;-).  Check the regex man page for more information
> > after you install LDM-6.
> >
> > To upgrade to LDM-6, then you should do the following:
> >
> > 1) make backup copies of your ~ldm/etc/ldmd.conf file(s).  While
> >    you are at it, you could make a backup copy of your ~ldm/etc/pqact.conf
> >    file, but this is optional since it will not be affected by moving
> >    to LDM-6.
> >
> > 2) FTP LDM-6 release candidate 6:
> >
> > <login as 'ldm'>
> > cd ~ldm
> > ftp ftp.unidata.ucar.edu
> >   <user> anonymous
> >   <pass> your_full_email_address
> >   cd pub/ldm5/test
> >   binary
> >   get ldm-6.0.tar.Z
> >   quit
> > zcat ldm-6.0.tar.Z | tar xvf -
> >
> > 3) build LDM-6. Note that LDM-6 is built with '-O' optimization
> >    turned on.  If you want to build otherwise, you will want to set
> >    the Unix environment variable CFLAGS _before_ you run configure.
> >
> >    Also, you should define the Unix environment variable LDMHOME
> >    _before_ you run configure:
> >
> > setenv CFLAGS -O
> > setenv LDMHOME your_ldm_home_directory
> >
> > cd ~ldm/ldm-6.0/src
> > ./configure
> > make && make install
> >
> > 4) finish the LDM install as 'root':
> >
> > sudo make install_setuids
> >
> > 5) adjust entries in ldmadmin to match those from your existing
> >    ldmadin.  This should boil down to you changing the product queue
> >    size; the number of log files that you want to keep online (we keep
> >    14 on our IDD machines); uncommenting out the entry for udunits if
> >    you are running things like gribtonc; filling in your full hostname
> >    if a 'uname -n' does not return a fully qualified hostname; etc.
> >
> > 6) shutdown your current LDM waiting for all LDM processes to exit.
> >    This step is easier in LDM-6 since ldmadmin won't return you to
> >    the Unix/Linux prompt before processes have exited.
> >
> > 7) change the runtime link to point at ldm-6.0
> >
> > cd ~ldm
> > rm runtime && ln -s ldm-6.0 runtime
> >
> > 8) verify that your operating system has finished flushing the LDM
> >    queue from memory to disk.  I found that this was an important
> >    step in getting things to work smoothly on thelma.  I did
> >    the verification by watching the iostat listing of top.  As soon
> >    as it quites down, you are ready to go.  Our experience is that
> >    the larger the queue, the longer this step takes.  We even added
> >    a 'sync' call from ldmadmin to tell the OS to sync to disk before
> >    shutting down LDM processes.
> >
> > 9) start LDM-6:
> >
> > rehash                <- C shell only
> > ldmadmin start
> >
> > I think I have covered all bases in this note, but you never know.
> > The good news is that you guys already know all of the upgrade procedure,
> > so most of the above was unneeded.
> >
> > If you would help doing the upgrade, just let us know.
> >
> > Tom Yoksas
> >
> ****************************************************************************
> > Unidata User Support                                    UCAR Unidata
> > (303)497-8644                                                  P.O. Box
> > address@hidden                                   Boulder, CO
> > --------------------------------------------------------------------------
> > Unidata WWW Service                        http://www.unidata.ucar.edu/
> >
> ****************************************************************************
> >
>