•         advanced

Real-time, self-managing data flows -- Unidata will foster and support the existence of real-time data flows that encompass a broad range of Earth-system phenomena, can be accessed with ease by all constituents, and are self managing in respect to changing contents and user needs.

-- A goal of Unidata 2008: Shaping the Future of Data Use in the Geosciences

LDM Status Report

LDM

LDM version of previous report was 6.6.3. Current LDM version is 6.6.5. (Beta LDM version is 6.7.0.0.)

LDM release highlights since previous report:

• 6.7.0.0 (beta)

  Modified "ldmadmin pqactcheck". By default, it now checks the syntax of every "pqact" configuration-file that is associated with an EXEC entry for "pqact" in the LDM configuration-file.

• 6.6.5 2007-06-19

  Corrected the "pqact" utility's determination of the month associated with a data-product from the creation-time of the data-product and the day-of-the-month field in the product-identifier. This modification is tested extensively by executing the command "make check" in the pqact/ subdirectory.

• 6.6.4 2007-05-17

  Modified the function surf_split() in the "pqsurf" program so that it uses a dynamically allocated buffer instead of a statically allocated one. This means that "pqsurf" can now handle arbitrarily large composite bulletins.

  Modified the "pqact" utility's determination of the month associated with a data-product from the creation-time of the data-product and the day-of-the-month field in the product-identifier.

  Changed the behavior of a downstream LDM upon reception of a COMINGSOON message whose data-product has zero length. Before, this would cause the downstream LDM to exit; now the data-product is simply rejected. Both LDM-6 and LDM-5 code were modified.

  Improved the performance of the "scour" utility.

Next Generation LDM

After being diverted by development of a new UDUNITS package (with an XML database, support for logarithmic units, and support for Latin-1 and UTF-8 characters) Steve Emmerson has returned to the development of a next-generation LDM.

IDD Status Report

• Approximately 440+ machines at 230+ sites are running LDM-6 and reporting real time statistics. Unidata staff routinely assist in the installation of LDM-6 at user sites to help evolve the IDD as rapidly as possible. There are several organizations/projects that use the LDM to move substantial amounts of data that do not report statistics to Unidata: NASA, Spain, South Korea, several private compaines, etc.).

• CONDUIT will be transitioning the remaining GRIB1 data sets currently being delivered to GRIB2 format data files over the next 2 months as NCEP will discontinue the production of GRIB1 format data for its FTP servers on Janury 28, 2008. The current data volume of CONDUIT is approximately 60 GB per day. During the transition period, legacy GRIB1 data sets will be removed after a 2 week overlap period where data will be duplicated in GRIB2 format. The data sets will be iteratively replaced to avoid increasing the data volume significantly. Since GRIB2 format files are smaller than the same data in GRIB1, there will be a net decrease in bandwidth usage after completion. Additional ensemble parameters will be added for TIGGE beginning September 25, 2007.

• IDD UNIWISC datastream continues to supply Unidata users with realtime GOES data in native projection.

  Even though GOES-South imagery has successfully been incorporated into the half-hourly GOES-East image sectors in the Unidata-Wisconsin IDD datastream, it is believed that inclusion of the available 15-minute scans would be of use to Unidata community members, especially those in South America. The additions of GOES-South imager sectors to the Unidata-Wisconsin datastream was endorsed by the Unidata User's Committee at its May, 2007 meeting.

  Proposed Unidata-Wisconsin Datastream Changes (http://www.unidata.ucar.edu/committees/polcom/2007summer/statusreports/uniwisc_prop.html) presents examples of GOES-10 imagery that are to be added to the Unidata-Wisconsin IDD datastream and discusses potential impacts to end-users. The target of adding GOES-South imagery to the Unidata-Wisconsin datastream by the end of Summer, 2007 was delayed until the GOES-South dish (located on the southeast side of the Mesa Lab) gets upgraded to be able to track GOES-10. Motorization of the dish should be completed late this fall.

• Real time LDM statistics can be found in the Real Time IDD Statistics web page (http://www.unidata.ucar.edu/software/idd/rtstats). The topology and data flow information reflects changes in feed requests within a few minutes of changes. rtstats continue to be most useful in diagnosing reception problems at sites, especially when sites have implemented restrictions on packets volume in and out of their campus without regard for their user needs.

  NB: In order to correctly gauge real-time status of the IDD, it is important that all participating sites accurately maintain their system clocks. This is easily done through use of a Network Time Protocol daemon run on the local machine.

• NWS continues to use LDM-6 to collect and relay NEXRAD Level II data. Results of performance monitoring studies being conducted by NOAA personnel (Blanchard) are shared with the UPC, and information gained is being used to guide code modifications that will be included in future releases.

• The NWS will be modifying the format of NEXRAD Level II products for Build 10.0 adding message type 31 in support of higher resolution 1/4 km x 0.5 degree radar operations, as well as future dual polarization mode data. The LDM product identifiers will be modified to contain additional information about the data products. The new Level II data format allows for varaible length records to preserve bandwith. Test products for site NOP3 are currently being transmitted over the IDD utilizing these changes.

• The IDD seamlessly transitioned to the use of DVB-S NOAAPORT ingest at the end of March, 2005. Currently, there are 11 sites known to be running the Unidata NOAAPORT reception software. 6 of these sites are acting as IDD injection nodes for NOAAPORT data.

• The LDM-6 was selected for data collection activities in the THORPEX Interactive Grand Global Ensemble (TIGGE). The UPC continues to provide support as needed in the use or the LDM by TIGGE participants. The TIGGE stress testing activities were reported in the February, 2006 CommunitE-letter article THORPEX/TIGGE and the LDM.

• Internet2 bandwidth usage by the LDM protocol reached a high of approx. 21 TB/week the week of August 15, 2005. Weekly I2 usage statistics seen at Internet NetFlow: Weekly Reports (search for 'UNIDATA LDM' usage statistics in the Advanced Applications portion of Table 7, Detailed Application Types (Full Data Set)) indicate that approx. 13.35 TB/week are currently flowing in I2. The amount of IDD traffic being reported by I2 netflow statistics fell as UCAR and other organizations are routing over National Light Rail (NLR) instead of I2.

• IDD toplevel relay node, idd.unidata.ucar.edu

  The cluster approach to toplevel IDD relay, first reported in the Spring 2005 status report, has been operational at the UPC since early summer 2005. The cluster, described in the June 2005 CommunitE-letter article Unidata's New IDD Cluster, routinely relays data to more than 440 downstream connections. Data input to the cluster nodes is approx. 4 GB/hr (0.97 TB/day); average data output by the cluster is approx. 246 Mbps (~2.7 TB/day); peak rates routinely exceeding 500 Mpbs (~5.4 TB/day).

  Currently seven real server nodes operating in two separate locations on the UCAR Foothills Lab campus (in the UPC offices and in FL-2) and one director comprise idd.unidata.ucar.edu. A second, director will be installed in the coming months for added redundancy. In addition, the cluster is also being used for Spam Assassin processing on approximately 800,000 inbound email messages to the UPC each month.

  The cluster approach to IDD relay has been adopted by NOAA/GSD (formerly FSL) and Penn State (using funds provided by the Unidata-administered Equipment Awards program). Unidata staff have fielded questions on implementing similar clusters at Unidata community sites (Texas A&M) that have expressed an interest in functioning as toplevel IDD relay nodes.

Contact Us   •   Site Map   •   Search   •   Terms and Conditions   •   Privacy Policy   •   Participation Policy

Unidata is a member of the UCAR Office of Programs, is managed by the University Corporation for Atmospheric Research, and is sponsored by the National Science Foundation. P.O. Box 3000   •   Boulder, CO 80307-3000 USA   •   Tel: 303-497-8643   •   Fax: 303-497-8690