Real-time, self-managing data flows -- Unidata will foster and support the existence of real-time data flows that encompass a broad range of Earth-system phenomena, can be accessed with ease by all constituents, and are self managing in respect to changing contents and user needs.
--A goal of Unidata 2008: Shaping the Future of Data Use in the Geosciences
Release highlights since 6.4.5:
Solved a problem receiving data from WSI by moving the initialization of the autoshift module so that it accommodates incoming HIYA connections that ignore a RECLASS reply and immediately start sending data-products.
Eliminated reception of a SEGV signal by a downstream LDM under very rare circumstances when the RPC layer closes the connection due to a fault.
Created the "ulog/log" module and added some logging to the RPC module.
Updated distributed RPC library.
Changed error-message "Terminating due to LDM failure..." to "Disconnecting due to LDM failure..." to improve accuracy.
The cluster approach to toplevel IDD relay, first reported in the Spring 2005 status report, has been operational at the UPC since early summer 2005. The cluster, described in the June 2005 CommunitE-letter article Unidata's New IDD Cluster, routinely relays data to between 380 and 400 downstream connections. Data input to the cluster nodes is currently approx. 5 GB/hr (0.12 TB/day); average data output by the cluster nodes is currently approx. 220 Mbps (~2.4 TB/day); peak rates routinely exceeding 410 Mpbs (~4.4 TB/day).
We consider the cluster to be an experiment in progress. Configurations change as we learn more about how the system performs, and new operating system releases are tested for performance.
The cluster approach to IDD relay has been adopted by NOAA/GSD (formerly FSL) and will be implemented at Penn State with funds provided by the Unidata-administered Equipment Awards program (Penn State is waiting on the UPC for configuration guidance). Unidata staff have fielded questions on implementing similar clusters at Unidata community sites (Texas A&M and University of Nebraska-Lincoln) that have expressed an interest in functioning as toplevel IDD relay nodes.
Steve Emmerson continues to work on a replacement system for the LDM (codenamed "Hermes") that will combine the best features of the LDM, INN (i.e., netnews), and more modern peer-to-peer systems such as BitTorrent. The work is currently in the design stage. Design features include
NB: In order to correctly gauge real-time status of the IDD, it is important that all participating sites accurately maintain their system clocks. This is easily done through use of a Network Time Protocol daemon run on the local machine.