Real-time, self-managing data flows -- Unidata will foster
and support the existence of real-time data flows that encompass a
broad range of Earth-system phenomena, can be accessed with ease by all
constituents, and are self managing in respect to changing contents and
user needs.
-- A goal of Unidata 2008: Shaping the Future of Data Use in the
Geosciences
LDM Status Report
LDM version of previous report was 6.6.3. Current LDM version is 6.6.5.
(Beta LDM version is 6.7.0.1.)
LDM release highlights since previous report:
6.7.0.1 (beta)
Modified "ldmadmin pqactcheck". By default, it now checks the
syntax of every "pqact" configuration-file that is associated with
an EXEC entry for "pqact" in the LDM configuration-file.
6.6.5 2007-06-19
Corrected the "pqact" utility's determination of the month
associated with a data-product from the creation-time of the
data-product and the day-of-the-month field in the
product-identifier. This modification is tested extensively by
executing the command "make check" in the pqact/ subdirectory.
6.6.4 2007-05-17
Modified the function surf_split() in the "pqsurf" program so
that it uses a dynamically allocated buffer instead of a
statically allocated one. This means that "pqsurf" can now
handle arbitrarily large composite bulletins.
Modified the "pqact" utility's determination
of the month associated with a data-product from the
creation-time of the data-product and the day-of-the-month field
in the product-identifier.
Changed the behavior of a downstream LDM upon reception of a
COMINGSOON message whose data-product has zero length. Before,
this would cause the downstream LDM to exit; now the
data-product is simply rejected. Both LDM-6 and LDM-5 code were
modified.
Improved the performance of the "scour" utility.
IDD Status Report
Approximately 440+ machines at 230+ sites are
running LDM-6 and reporting real time statistics. Unidata
staff routinely assist in the installation of LDM-6 at user sites
to help evolve the IDD as rapidly as possible. There are several
organizations/projects that use the LDM to move substantial amounts
of data that do not report statistics to Unidata: NASA, Spain, South
Korea, several private compaines, etc.).
CONDUIT content was transitioned GRIB2 format data files in the
December-January, 2007 timeframe. The current data volume of
CONDUIT is approximately 41.5 GB per day. Since GRIB2 format files
are smaller than the same data in GRIB1, there was a net decrease
in bandwidth usage after completion. Additional ensemble parameters
were added for TIGGE beginning September 25, 2007.
IDD UNIWISC datastream continues to
supply Unidata users with realtime GOES data in native projection.
The addition of 15-minute GOES-South
imager sectors to the Unidata-Wisconsin datastream was endorsed
by the Unidata User's Committee at its Spring 2007 meeting.
The addition of these sectors was postponed due to ongoing
work on GOES-East/West/South activities in Unidata and UCAR.
Real time LDM statistics can be found in the
Real Time IDD Statistics web page
(http://www.unidata.ucar.edu/projects/idd/rtstats). The topology and
data flow information
reflects changes in feed requests within a few minutes of changes.
rtstats continue to be most useful in diagnosing reception
problems at sites, especially when sites have implemented
restrictions on packets volume in and out of their campus without
regard for their user needs.
NB: In order to correctly gauge real-time status of the IDD, it is
important that all participating sites accurately maintain their
system clocks. This is easily done through use of a Network Time
Protocol daemon run on the local machine.
NWS continues to use LDM-6 to collect and relay NEXRAD Level II data.
Results of performance
monitoring studies being conducted by NOAA personnel (Blanchard) are
shared with the UPC, and information gained is being used to guide
code modifications that will be included in future releases.
The NWS has begun modifying the format of NEXRAD Level II products through
deployment of Build 10.0 adding message type 31 in support of higher resolution
1/4 km x 0.5 degree radar operations, as well as future dual polarization
mode data. The LDM product identifiers will be modified to contain additional
information about the data products. The new Level II data format allows for
varaible length records to preserve bandwith. Test products for site NOP3 are
currently being transmitted over the IDD utilizing these changes.
Currently, there are 12 sites known to be running the Unidata
NOAAPORT reception software including Raytheon which is doing
so in support of its use of Unidata GEMPAK which is informing
their development of AWIPS-2. 6 community
sites are acting as IDD injection nodes for NOAAPORT data.
The LDM-6 is being used for
data collection activities in the THORPEX Interactive Grand Global
Ensemble (TIGGE). The UPC continues to provide support as needed in
the use or the LDM by TIGGE participants. The TIGGE stress testing activities
were reported in the February, 2006 CommunitE-letter article
THORPEX/TIGGE and the LDM.
Internet2 bandwidth usage by the LDM protocol reached a
high of approx. 21 TB/week the week of August 15, 2005. Weekly
I2 usage statistics seen at
Internet NetFlow: Weekly
Reports (search for 'UNIDATA LDM' usage statistics in the
Advanced Applications portion of Table 7, Detailed Application
Types (Full Data Set)) indicate that approx. 13.35 TB/week are currently
flowing in I2. The amount of IDD traffic being reported
by I2 netflow statistics fell as UCAR and other organizations are routing
over National Light Rail (NLR) instead of I2.
IDD toplevel relay node, idd.unidata.ucar.edu
The cluster approach to toplevel IDD relay, has been operational
at the UPC since early summer 2005. The cluster, described in
the June 2005 CommunitE-letter article
Unidata's New IDD Cluster, routinely relays data to more
than 440 downstream connections. Data input to the cluster
nodes is approx. 3.6 GB/hr (0.86 TB/day); average data output by
the cluster is approx. 258 Mbps (~2.7 TB/day); peak rates
routinely exceeding 600 Mpbs (~6.2 TB/day).
Currently seven real server nodes operating in two separate locations
on the UCAR Foothills Lab campus (in the UPC offices and in FL-2) and
one director comprise idd.unidata.ucar.edu. A second, director
will be installed in the coming months for added redundancy. In
addition, the cluster is also being used for Spam Assassin processing
on approximately 800,000 inbound email messages to the UPC each month.
The cluster approach to IDD relay has been adopted by NOAA/GSD
(formerly FSL) and Penn State (using funds provided by the
Unidata-administered Equipment Awards program). Unidata staff
have fielded questions on implementing similar clusters at
Unidata community sites that have expressed an
interest in functioning as toplevel IDD relay nodes.
Next Generation LDM Status Report
Steve Emmerson has returned to the development of a Next-Generation LDM.
A request to the user community for potential "users stories"
(i.e., usage scenarios)
to help guide the design and development of a
next-generation LDM
generated significant traffic on the associated mailing-list:
Dear LDM user,
I'm working on a replacement for the LDM and I'd like your help. I need
user-stories (i.e., high-level usage scenarios) so I don't forget
anything or leave out some useful feature. I've created a wiki to help
gather the stories (and for anything else for which it might be useful).
The wiki is at <https://wiki.ucar.edu/display/unidata/LDM+Replacement>.
It contains the link "User Stories" in which I've put one story. Please
read it and if you think something is missing, then please add your own story,
edit any existing one, add comments, etc.
In order to add or edit, I'll have to create an account for you on the
UCAR wiki and add you to the "LDM-Replacement" group. Just reply to
this email with your full name and the email address you'll want to use
as your identifier. Naturally, I reserve the right to be human (i.e.,
picky :-).
Now's your chance!
With regards,
Steve Emmerson
LDM Developer
Unidata Program Center
Our response was to create a
forum,
a
wiki,
and to send a request for continued input to the community:
LDM-6 users,
Everyone here at the Unidata Program Center has been reading ** with
great interest ** the posts regarding our efforts aimed at developing a
Next Generation LDM. We are _very_ pleased that several community
members have shared their thoughts on this development. We sincerely
hope to hear from more of you on this important topic!
In order to provide some background on this development, we offer
the following short overview:
Why develop a Next Generation LDM when the current one works so well?
=====================================================================
The LDM has grown to be a robust, reliable and portable base on which
to build data distribution networks. As design or implementation
limitations were identified, innovative developments have been employed
to keep the LDM viable.
The current implementation of the LDM when coupled with cluster
technology may be able to effectively relay all of the data desired
by the expanding Unidata community for several more years. However,
input on the limitations in the current LDM design from you, the
user community, has convinced us that new, alternate approaches to
data distribution must be thoroughly investigated and pursued.
The need for a more scalable technology is apparent from several
trends:
- the volume of useful data appears to be growing faster than
increases in network bandwidth
- approaches to setting priorities for bandwidth by "shaping"
its use, charging departments for bandwidth, or other institutional
data policies may disrupt the model of essentially free bandwidth
currently in use for the IDD
- competition for network bandwidth from the explosive growth in
video distribution may demand more dynamic routing than is now
possible with current LDM technologies
To avoid interruption in data delivery, future needs must be
anticipated and reliable software must be developed well before the
need is apparent.
Our experience suggests the current LDM-6 may have advanced about as
far as it can given the constraints of its architecture.
Please be assured that the development of a Next Generation LDM
does not imply that incremental improvements to the current LDM-6
will cease!
However, further major advances in the LDM may require a new protocol
that is not tied to the client/server approach but instead users
peer-to-peer concepts.
A new protocol and implementation could allow for the following
improvements:
- more dynamic creation and destruction of data-product streams
- support for access to "one-time" data products (i.e.,
data-products that are not continuously generated)
- better load balancing of communication links
- more adaptive and flexible dynamic routing of data-products with
steady-state results that are relatively independent of the
configuration of initial connections
- a better interface for selecting data-products. For example,
reception of a data-product stream could be started by clicking
on a hyperlink in a web page.
- support for use on computing platforms beyond those being used
in the community for data relay tasks (e.g., Windows)
Naturally, minimizing disruption -- both to individual sites and
to the flow of data -- is a major concern of any new implementation
and deployment. The LDM software is open source, so no one will be
forced to upgrade to different and incompatible software. In fact,
there could be two competing distribution systems running in
parallel for some time, during which the new system will have to
prove itself capable of satisfying users of the current LDM as well
as providing solutions to problems the current LDM cannot handle.
Users interested in the history of the LDM/IDD and that would like
to learn more about ongoing efforts at creating a Next Generation
LDM should take a look at the (14 page) paper we presented at the
AMS 2006 Annual Meeting:
"The Unidata Internet Data Distribution (IDD) System: A Decade of
Development"
http://ams.confex.com/ams/Annual2006/techprogram/paper_105113.htm
http://ams.confex.com/ams/pdfpapers/105113.pdf
Again, we are seeking your input on this new, and exciting development!
In order to make it as easy as possible for you to share your thoughts,
we are providing several ways for you to comment:
- send email to the ldm-users at unidata.ucar.edu list
- post to the ldm-users forum:
Unidata HomePage
http://www.unidata.ucar.edu
Support
http://www.unidata.ucar.edu/support/
Participate in topical forums
http://www.unidata.ucar.edu/forums/index.jspa
LDM Users Forum
http://www.unidata.ucar.edu/forums/forum.jspa?forumID=19
- provide additions-to/comments-for the wiki that Steve Emmerson
mentioned in his request for users stories/high-level usage
scenarios:
https://wiki.ucar.edu/display/unidata/Next-Generation+LDM
- send email to the Unidata User's Committee
- send email to Unidata User Support
- contact any Unidata staff member directly
Please share your thoughts on the development of a Next Generation LDM
and what features you would like to see incorporated in its design!!
Cheers,
The Unidata Staff
--
+----------------------------------------------------------------------------+
* Unidata User Support UCAR Unidata Program *
* (303) 497-8643 P.O. Box 3000 *
* support at unidata.ucar.edu Boulder, CO 80307 *
* Unidata WWW Service http://www.unidata.ucar.edu/ *
+----------------------------------------------------------------------------+
The forum is for threaded discussions and the wiki (which has six
informational pages, currently) is for user-education and the gathering and
sharing of information.
