News@UnidataUnidata newshttps://www.unidata.ucar.edu/blogs/news/feed/entries/atom2024-03-06T11:18:50-07:00Apache Rollerhttps://www.unidata.ucar.edu/blogs/news/entry/ldm-6-13-14-releasedLDM 6.13.14 ReleasedUnidata News2021-08-04T16:30:41-06:002021-08-04T16:30:41-06:00<p>
LDM 6.13.14 has just been released.
</p>
<p>
LDM 6.13.14 has just been released. You can view a list of the changes in the
<a href="https://www.unidata.ucar.edu/software/ldm/ldm-current/CHANGE_LOG">CHANGE_LOG</a>.
</p>
<ul>
<li><a href="https://www.unidata.ucar.edu/downloads/ldm/">Download Page</a></li>
<li><a href="https://www.unidata.ucar.edu/software/ldm/ldm-6.13.14/">Version-Specific Documentation</a></li>
<li><a href="https://www.unidata.ucar.edu/software/ldm/">LDM Homepage</a></li>
</ul>
<p>This release includes more than 50 improvements across the LDM package. Some notable examples:</p>
<ul>
<li>
Replaced <code>scour(1)</code> script with multi-threaded C program that scours
much faster than the script.
</li>
<li>
In the <code>rpc(3)</code> library, replaced call to <code>bindresvport()</code> in <code>clnttcp_create()</code> with binding to an
ephemeral port to eliminate the necessity for having root privileges.
</li>
<li>
In the <code>noaaportIngester(1)</code>, created "<code>-R <buf_size></code>" option to set the size of the receive
buffer in order to reduce the number of missed packets.
(The operating system must allow the new size.)
</li>
</ul>
<p>See the
<a href="https://www.unidata.ucar.edu/software/ldm/ldm-current/CHANGE_LOG">CHANGE_LOG</a> for a complete list.
</p>
<p>
Please send any questions to <a href="mailto:support-ldm@unidata.ucar.edu">support-ldm@unidata.ucar.edu</a> or
report issues via the <a href="https://github.com/Unidata/LDM/issues">LDM GitHub
site</a>.
</p>
https://www.unidata.ucar.edu/blogs/news/entry/ldm-6-13-12-releasedLDM 6.13.12 ReleasedUnidata News2020-09-30T13:57:23-06:002020-09-30T13:57:23-06:00<p>
LDM 6.13.12 has just been released. You can view a list of the changes in the
CHANGE_LOG.
</p>
<p>
LDM 6.13.12 has just been released. You can view a list of the changes in the
<a href="https://www.unidata.ucar.edu/software/ldm/ldm-current/CHANGE_LOG">CHANGE_LOG</a>.
</p>
<ul>
<li><a href="https://www.unidata.ucar.edu/downloads/ldm/">Download Page</a></li>
<li><a href="https://www.unidata.ucar.edu/software/ldm/ldm-6.13.12/">Version-Specific Documentation</a></li>
<li><a href="https://www.unidata.ucar.edu/software/ldm/">LDM Homepage</a></li>
</ul>
<p>Some notable changes in this release:</p>
<ul>
<li>The use of the <code>IS_ALIVE</code> protocol message has been replaced with a fixed, 5
minute timeout.</li>
<li>The logging level for products that are not inserted into the product queue
because they are either too old or were not requested has been changed from <code>INFO</code>
to <code>WARN</code> in order to raise the visibility of this issue.</li>
<li>A bug in the use of the <code>pqact(1)</code> state-file (which indicates the
insertion-time of the last, successfully-processed product) has been fixed.</li>
<li><code>pqact(1)</code> PIPE and FILE actions that have not been used in 6 hours are now
removed from the set of active actions and their file descriptors are closed. This
has greatly reduced the time it takes to shut down the LDM system for those
instances doing local processing with many such actions lacking the "-close"
option.</li>
<li>Thread-creation within <code>noaaportIngester(1)</code> has been modified to prevent the
program from immediately terminating if the (optional) RHEL Insights Client
package has been installed and prevents a thread from being created with a high
priority. In this case the thread will execute with normal priority and the risk
of missing NOAAPort packets will increase.</li>
</ul>
<p>
Please send any questions to <a
href="mailto:support-ldm@unidata.ucar.edu">support-ldm@unidata.ucar.edu</a> or
report issues via the <a href="https://github.com/Unidata/LDM/issues">LDM GitHub
site</a>.
</p>
<p class="highlight_box">
Interested in working on the LDM? Unidata is hiring! For information, see
<a href="https://www.unidata.ucar.edu/blogs/news/entry/unidata-is-looking-for-a10">Unidata is looking for a Software Developer</a>.
</p>
https://www.unidata.ucar.edu/blogs/news/entry/ldm-6-13-2-releasedLDM 6.13.2 ReleasedUnidata News2016-07-14T17:39:37-06:002017-12-21T12:44:33-07:00<p>LDM 6.13.2 has just been released. It fixes a few bugs and performs better than the previous 6.13 versions. You can view a list of the changes in the <a href="https://www.unidata.ucar.edu/software/ldm/ldm-current/CHANGE_LOG">CHANGE_LOG</a>.</p>
<ul>
<li><a href="ftp://ftp.unidata.ucar.edu/pub/ldm">Download Page</a></li>
<li><a href="https://www.unidata.ucar.edu/software/ldm/ldm-6.13.2">Version-Specific Documentation</a></li>
<li><a href="https://www.unidata.ucar.edu/software/ldm/#home">LDM Homepage</a></li>
</ul>
<p>Please send any questions to <a href="mailto:support-ldm@unidata.ucar.edu">support-ldm@unidata.ucar.edu</a> or report issues via the <a href="https://github.com/Unidata/LDM/issues">LDM GitHub site</a>.</p>
https://www.unidata.ucar.edu/blogs/news/entry/ldm-6-13-0-releasedLDM 6.13.0 ReleasedSteve2016-05-04T13:03:52-06:002017-12-21T12:47:40-07:00<p>LDM Version 6.13.0 is now available. It has a new logging system that, by default, doesn't use the system logging daemon though this change should be invisible. It also has several other improvements and bug fixes.</p>
<p>Version 6.13.0 of the LDM is now available at the following URL-s:</p>
<ul>
<li>LDM Homepage: https://www.unidata.ucar.edu/software/ldm</li>
<li>LDM 6.13.0 Homepage: https://www.unidata.ucar.edu/software/ldm/ldm-6.13.0</li>
<li>LDM Download Page: https://www.unidata.ucar.edu/downloads/ldm/</li>
</ul>
<p>As you can see from the CHANGE_LOG file, the major new feature with this release is a new logging system. The previous system used the system logging daemon by default; the new system doesn't. Instead, it writes directly into the standard LDM log file by default. This change was made because the previous system caused problems and wasn't 100% reliable (see the CHANGE_LOG file for details).</p>
<p>The format of log messages is slightly different. Here's an example of a log message from the previous system:</p>
<pre><code>May 02 2016 20:16:25 chico.unidata.ucar.edu oliver.unidata.ucar.edu(feed)[14078] NOTE: Starting Up(6.12.14/6): 20160502191624.896 TS_ENDT {{ANY, ".*"}}, SIG=b12852828dac75021ac15b7064439754, Primary
</code></pre>
<p>And here's that same type of message in the new format:</p>
<pre><code>20160429T155919.718276Z oliver.unidata.ucar.edu(feed)[21104] NOTE up6.c:445:up6_run() Starting Up(6.13.0/6): 20160429145918.709017 TS_ENDT {{GPS, "(.*)"}}, SIG=f2693a8f53ba6736843a5300e30b68fd, Primary
</code></pre>
<p>Note the timestamp is now in ISO 8601 format (which should make sorting easier). It also has microsecond resolution and is guaranteed to be in the UTC timezone. Note also the field containing the local hostname is missing (it was added by the system logging daemon) and a new field has been added that specifies the location where the message was generated in the form file:line:func().</p>
<p>Aside from slight differences in the format of log messages, you should be able to use the new logging system just like the previous one. In particular, you will still be able to send an LDM process a SIGUSR2 signal to rotate its logging level and the command ldmadmin newlog should still create a new LDM log file and rotate the names of the old ones.</p>
<p>The old logging system still exists in the LDM library for backward-compatibility with third-party developers.</p>
<p>Besides the new logging system, this version also has several other improvements and bug-fixes. See the CHANGE_LOG file for details.</p>
<p>Regards,
Steve Emmerson</p>
https://www.unidata.ucar.edu/blogs/news/entry/ldm_6_12_7_releasedLDM 6.12.7 ReleasedSteve2015-03-11T14:46:53-06:002015-03-11T14:56:52-06:00<p>
Version 6.12.7 of the LDM package has been released. Improvements include:
</p>
<ul>
<li>
Additions to the GEMPAK GRIB2 tables for ingesting the NOAAPORT satellite broadcast. This should
greatly reduce the number of "GB" error-messages in the LDM log file.
</li>
<li>
Support for running <code>rsyslogd(8)</code> on a Solaris
system with configuration-file
<code>/etc/opt/csw/rsyslog.conf</code> and PID-file
<code>/var/run/rsyslogd.pid</code>.
</li>
<li>
Refactored and generally improved code in <code>pqact(1)</code>, hopefully
stopping it from receiving a SIGSEGV when
executing <code>regexec(3)</code>. This has only occurred on
one host and it's not clear if it's due to the code or the
host.
</li>
</ul>
<p>
The complete set of changes can be found <a href='https://www.unidata.ucar.edu/software/ldm/ldm-6.12.7/CHANGE_LOG' title='CHANGE LOG'>here</a>.
</p>
<p>
This new package can be downloaded <a href='ftp://ftp.unidata.ucar.edu/pub/ldm/' title='LDM Download Site'>here</a>.
</p>
<p>
The homepage of the LDM is <a href='https://www.unidata.ucar.edu/software/ldm/' title='LDM Homepage'>here</a>.
</p>https://www.unidata.ucar.edu/blogs/news/entry/ldm_6_12_6_releasedLDM 6.12.6 ReleasedUnidata News2014-09-12T10:55:43-06:002014-09-12T10:55:43-06:00<p>
Version 6.12.6 of the LDM has been released.
</p>
<p>
Besides supporting the new NOAAPORT channels (like
version 6.12.5) this version understands how to handle HRRR
and ESTOFS data-products that will soon be added to the
NOAAPORT broadcast.
</p>
<p>
Support for two additional channels will be added for the
GOES-R satellite sometime before it's launched (currently
scheduled for the second quarter of 2016).
</p>
<p>
Additional information about this release is available on the <a href='https://www.unidata.ucar.edu/software/ldm/ldm-6.12.6/'>LDM
6.12.6 page</a>. You can find a link to the source download area
on the <a href='https://www.unidata.ucar.edu/software/ldm/'>main LDM page</a>.
</p>https://www.unidata.ucar.edu/blogs/news/entry/ldm_6_11_7_releasedLDM 6.11.7 ReleasedUnidata News2014-04-04T17:05:33-06:002014-04-04T17:05:33-06:00<p>
Version 6.11.7 of the LDM has been released. It fixes a
bug that caused the NOAAPORT ingest component to crash.
</p>
<p>
Here's the story:
</p>
<p>
The NOAAPORT Satellite Broadcast System contains a lot of
GRIB-2 messages. Such messages require external tables in
order to be interpreted. The NOAAPORT ingest component of
the LDM uses the tables that come with GEMPAK to interpret
GRIB-2 messages in order to generate meaningful LDM
product-identifiers. Unfortunately, not all tables are
complete and creators of GRIB-2 messages are free to add
data for which no corresponding table-entry exists.
</p><p>
Version 6.11.7 of the LDM has been released. It fixes a
bug that caused the NOAAPORT ingest component to crash.
</p>
<p>
Here's the story:
</p>
<p>
The NOAAPORT Satellite Broadcast System contains a lot of
GRIB-2 messages. Such messages require external tables in
order to be interpreted. The NOAAPORT ingest component of
the LDM uses the tables that come with GEMPAK to interpret
GRIB-2 messages in order to generate meaningful LDM
product-identifiers. Unfortunately, not all tables are
complete and creators of GRIB-2 messages are free to add
data for which no corresponding table-entry exists.
</p>
<p>
The LDM adds a diagnostic log
message to the GEMPAK function that interprets level or
layer data to report when a table entry is missing. In
such a case, it logs the table lookup information and
the last entry in the table (for reference purposes).
Unfortunately, the code made the assumption that a
table would always have <b>some</b>
entries in it.
</p>
<p>
This turns out not to be the case.
</p>
<p>
On February 25th, 2014, a GRIB-2 message was broadcast that
resulted in a completely empty table being given to the
level/layer interpretation function. The subsequent attempt
to log the diagnostic message resulted in the NOAAPORT
ingestion process receiving a segmentation violation signal
(i.e., SIGSEGV). The rest of the LDM system then did what
it's supposed to when a child-process terminates abnormally:
it shut down.
</p>
<p>
The fix was easy once the cause was determined. A diagnostic
message will still be logged, but no SIGSEGV will occur.
</p>
<p>
Those of you who use the LDM to ingest NOAAPORT should
upgrade to this new version before another such GRIB-2
message is broadcast. As always, the relevant URL is
<a href='https://www.unidata.ucar.edu/software/ldm/'>https://www.unidata.ucar.edu/software/ldm/</a>.
</p>https://www.unidata.ucar.edu/blogs/news/entry/ldm_6_11_6_releasedLDM 6.11.6 ReleasedSteve2013-06-24T17:10:41-06:002013-06-27T10:18:34-06:00<p>Version 6.11.6 of the LDM has been released. Continuing the trend of version 6.11.5, this version doesn't have the PNG or ZLIB packages bundled-in -- and for the same reasons: bundling is no longer necessary, security is enhanced by removing them, and the size of the LDM package is reduced. You must have the PNG and ZLIB <b>development</b> packages installed in order to build this version of the LDM.</p><p>Version 6.11.6 of the LDM has been released. Continuing the trend of version 6.11.5, this version doesn't have the PNG or ZLIB packages bundled-in -- and for the same reasons: bundling is no longer necessary, security is enhanced by removing them, and the size of the LDM package is reduced. You must have the PNG and ZLIB <b>development</b> packages installed in order to build this version of the LDM. See <a href='https://www.unidata.ucar.edu/software/ldm/ldm-6.11.6/basics/index.html'>this version's installation instructions</a> for details.</p>
<p>If you're using version 6.11.5 of the LDM without any problems and don't care which PNG or ZLIB library the LDM uses, then you needn't upgrade. </p>
<p>This version of the LDM also has some other minor improvements. For details, please see the file <a href='https://www.unidata.ucar.edu/software/ldm/ldm-6.11.6/CHANGE_LOG'>CHANGE_LOG on this version's homepage</a> or in the top-level source-directory of the distribution.</p>
<p>This version can be downloaded at <a href='https://www.unidata.ucar.edu/downloads/ldm'>https://www.unidata.ucar.edu/downloads/ldm</a>.</p><p>This version's homepage is <a href='https://www.unidata.ucar.edu/software/ldm/ldm-6.11.6'>https://www.unidata.ucar.edu/software/ldm/ldm-6.11.6</a>.</p>
<p>As an experiment, this version also has a binary RPM. While the RPM platform is "fc14.x86_64", it has been successfully installed on an "fc18.x86_64" system. See <a href='https://www.unidata.ucar.edu/software/ldm/ldm-6.11.6/basics/index.html'>this version's installation instructions</a> for details.</p>
<p>Please report any problems to <a href='mailto:support-ldm@unidata.ucar.edu'>support-ldm@unidata.ucar.edu</a>.</p>https://www.unidata.ucar.edu/blogs/news/entry/ldm_6_11_5_releasedLDM 6.11.5 ReleasedSteve2013-05-10T15:24:43-06:002013-05-14T09:34:06-06:00<p><a href='https://www.unidata.ucar.edu/software/ldm/ldm-6.11.5/index.html'>Version 6.11.5 of the LDM</a> has been released. This version doesn't have the "libxml2" package bundled-in; consequently, the "libxml2" <b>development </b>package must be installed on the host system before the LDM package can be built. The development package is necessary because the runtime package doesn't have the required header-files (development packages usually have the word "devel" in their name). If you're using version 6.11.4 of the LDM, have not encountered any problems with it, and don't care which "libxml2" package the LDM uses, then there's no need to upgrade to LDM 6.11.5.</p><p><a href='https://www.unidata.ucar.edu/software/ldm/ldm-6.11.5/index.html'>Version 6.11.5 of the LDM</a> has been released. This version doesn't have the "libxml2" package bundled-in; consequently, the "libxml2" <strong>development </strong>package must be installed on the host system before the LDM package can be built. The development package is necessary because the runtime package doesn't have the required header-files (development packages usually have the word "devel" in their name). If you're using version 6.11.4 of the LDM, have not encountered any problems with it, and don't care which "libxml2" package the LDM uses, then there's no need to upgrade to LDM 6.11.5.</p>
<p>The "libxml2" package was removed from the LDM distribution because:</p>
<ol>
<li>it's become ubiquitous;</li>
<li>older versions of it have security issues;</li>
<li>eliminating it allows it to be independently upgraded; and</li>
<li>eliminating it approximately halves the size of the distribution and the time it takes to build.</li>
</ol>
<p>Please report any problems caused by this removal to <a href='mailto:support-ldm@unidata.ucar.edu'>support-ldm@unidata.ucar.edu</a>.</p>
<p>This version of the LDM also has some minor code improvements. For details, please see the file <a href='https://www.unidata.ucar.edu/software/ldm/ldm-6.11.5/CHANGE_LOG'>CHANGE_LOG on this version's homepage</a> or in the top-level source-directory of the distribution.</p>
<p>This version can be downloaded at <a href='https://www.unidata.ucar.edu/downloads/ldm'>https://www.unidata.ucar.edu/downloads/ldm</a>.</p>https://www.unidata.ucar.edu/blogs/news/entry/ldm_6_11_4_releasedLDM 6.11.4 ReleasedSteve2013-04-03T15:40:32-06:002013-04-03T15:50:31-06:00<p>
Version 6.11.4 of the Local Data Manager (LDM) has been released. This version
fixes and improves the defense against a denial-of-service
attack, which is new with version 6.11. Getting the defense
right was trickier than initially thought.
</p><p>
Version 6.11.4 of the Local Data Manager (LDM) has been released. This version
fixes and improves the defense against a denial-of-service
attack, which is new with version 6.11. Getting the defense
right was trickier than initially thought.
</p>
<p>
A key aspect of the defense is that a request by a
downstream LDM that duplicates an earlier request from
the same downstream host will cause termination of the
upstream LDM process that's servicing the earlier request.
<b>This has ramifications</b> for LDMs running on
distinct computers that request data from the same upstream
LDM from behind a Network Address Translation (NAT) device.
To the upstream LDM, all the requests will appear to come
from the same IP address (that of the NAT device).
Consequently, duplicate requests will interfere with each
other. The solutions to this situation are:
</p>
<ol>
<li>
Have one LDM request all the desired data and have the other LDMs
request from it;
</li>
<li>
Make the patterns in the REQUEST entries distinct by
enclosing them in different numbers of parentheses (e.g.,
".*" on one computer, "(.*)" on the
second, "((.*))" on the third, etc.);
</li>
<li>
Have the LDMs request from each other as well as the
same upstream LDM and just live with the resultant churn in
the connections.
</li>
</ol>
<p>
The
significant new log messages of LDM 6.11.4 are:
</p>
<pre>Terminated obsolete upstream LDM<br />?(addr=<i>ipAddr</i>, pid=<i>pid</i>, vers=<i>vers</i>, type=<i>type</i>, mode=<i>mode</i>, sub=(<i>sub</i>))</pre>
<p>
This means that the upstream LDM that logged the message
terminated the indicated earlier upstream LDM because the
logging LDM will send at least the same data as the earlier
LDM to the same downstream host. <i>ipAddr</i> is the IP
address of the downstream host, <i>pid</i> is the process
identifier of the earlier LDM, <i>vers</i> is the LDM
protocol version (5 or 6), <i>type</i> is the type of
the earlier LDM (feeder or notifier), <i>mode</i> is
the transfer-mode of the earlier LDM (primary or alternate),
and <i>sub</i> is the subscription that the earlier
LDM was satisfying (feedtype, pattern, etc.).
</p>
<p>
If this happens a lot (on the order of once per minute),
then 1) the downstream "host" might actually be
multiple hosts — each running an LDM that makes the same
request — that are hidden behind a NAT device (as described
earlier); and 2) you should contact the LDM user at the
downstream site to make sure that everything is OK.
</p>
<pre>Subscription reduced by one or more existing subscriptions: <i>requested</i> -> <i>allowed</i></pre>
<p>
This means that the upstream LDM that logged the message
reduced the subscription from its associated downstream LDM
by the amount that the subscription overlapped subscriptions
that are being currently serviced by existing,
previously-created upstream LDMs. Note that this can result
in an empty subscription, in which case the downstream LDM
will close the connection and retry sometime later. Here
<i>requested</i> is the initial subscription and <i>allowed</i> is the reduced subscription.
</p>
<p>
You should contact the LDM user at the downstream host and
tell them that they are making redundant requests.
</p>
<p>
If you need to contact the LDM user at the downstream
site, the email address "ldm@<i>host</i>" <b>should</b> work, where <i>host</i> is either the fully-qualified name of the downstream host or its IP
address. It might be necessary to enclosed the IP address in
square brackets.
</p>
<p>
Besides ensuring against the denial-of-service
attacks that we've seen, this new feature of LDM 6.11 should
reduce unnecessary bandwidth usage by enforcing the
requirement that requests for data from the same upstream
LDM be disjoint.
</p>
<p>
This version can be downloaded at <a href='https://www.unidata.ucar.edu/downloads/ldm'>https://www.unidata.ucar.edu/downloads/ldm</a>.
</p>
<p>
For documentation and additional release information, see
<a href='https://www.unidata.ucar.edu/software/ldm/ldm-6.11.4/index.html'>this version's homepage</a>.
</p>