IDD and NOAAPort

Status Report: March 2012 - September 2012

Mike Schmidt, Jeff Weber, Tom Yoksas

Strategic Focus Areas

The IDD/NOAAPort group's work supports the following Unidata funding proposal focus areas:

  1. Enable widespread, efficient access to geoscience data
    A project like the IDD demonstrates how sites can employ the LDM to move data in their own environments.
  2. Develop and provide open-source tools for effective use of geoscience data
    The IDD is powered by the Unidata LDM-6 which is made freely available to all. The Unidata NOAAPort ingest package is being used by a variety of university and non-university community members. Both the LDM and NOAAPort ingest packages will be bundled by Raytheon with AWIPS-II.
  3. Provide cyberinfrastructure leadership in data discovery, access, and use
    The community-driven IDDs provide push data services to users an ever increasing community of global educators and researchers
  4. Build, support, and advocate for the diverse geoscience community
    Providing access to data in real-time is a fundamental Unidata activity.
    The IDD-Brasil, the South American peer of the North American IDD operated by the UPC, is helping to extend real-time data delivery outside of the U.S. to countries in South America and Africa. The Universidad de Costa Rica is experimenting with relaying data received in the IDD to Colombia.

IDD Activities Since the Last Status Report

  • Rapid Refresh (RAP) grids replaced RUC2 grids in the NOAAPort SBN and in CONDUIT in April
  • NOAA implemented changes to the CONDUIT datastream on August 21. Briefly, the changes removed the Eta and RSM members (which are no longer available) and added the new NMMB members. The changes made resulted in CONDUIT users continuing to get all of the individual members available.

    While working with NOAA on the CONDUIT upgrade, it was learned that the great majority of GRIB2 messages being transmitted were originally created as GRIB1 messages and then converted to GRIB2 format. Over the years, we (Unidata and its CONDUIT user community) has observed that a number of the GRIB2 products had PDS values that do not accurately represent the actual field in the GRIB2 message. This problem is now understood to have been introduced by the routine being to convert GRIB1 messages to GRIB2. NOAA has agreed to work to fix errant GRIB2 fields when we can provide documentation for the problems we are encountering. We view this as being a great opportunity to fix some things that have plagued a variety of Unidata projects for years.

  • Unidata is acting as a toplevel relay in NEXRAD Level II data distribution for university sites and others that were receiving data from the MAX GigaPoP that was decommissioned by the NWS. The other toplevel relay sites for Level II data are the ERC (Education and Research Consortium), IRaDS (Integrated Robust Assured Data Services), and Purdue University.
  • Unidata is now receiving High Resolution Rapid Refresh (HRRR) grids (surface only) in an LDM/IDD feed from NOAA/GSD. The intention is to evaluate making these products available in a new (EXP feedtype) datastream from the Unidata-operated toplevel IDD relay node, idd.unidata.ucar.edu. The challenge in making the data routinely available is its large data volume which is on the order of ~8GB for the pressure level output and ~10 GB/hour for the sigma level output.
  • The UPC continues to relay FNMOC and the CMC data model output directly to the community. FNMOC provides the COAMPS and NOGAPS model output and the CMC provides the GEM model output. Unidata has provided access to these data for the past 8 years, but on a "point-to-point" basis. GEM model output was converted from GRIB1 to GRIB2 in January. The CMC has offered to relay output of there new hi-resolution (15km) GEM model to Unidata by the end of September. The 60 km output will go away permanently by end of year, and likely off Unidata systems once we are getting the 15 km output.

NOAAPort Data Ingest

  • NOAAPort ingest has been functioning near-flawlessly since the NWS transitioned the SBN from DVB-S to DVB-S2 in April/May 2011. The switchover was smooth enough that end-users should have never noticed the change.
  • The NOAAPort ingest package has been bundled with the LDM as of LDM version 6.10+.
  • Raytheon is bundling LDM-6 with AWIPS-II and is actively managing NOAAPort ingest at a variety of NOAA offices using the Unidata NOAAPort ingest package.

Relevant IDD Metrics

  • Approximately 510 machines at 220 sites are running LDM-6 and reporting real time statistics to Unidata. Unidata staff routinely assist in the installation of LDM-6 at user sites as a community service.

     

    A number organizations/projects use the LDM to move substantial amounts of data that do not report statistics to Unidata: NOAA, NASA, USGS, USACE, Governments of Spain, South Korea, private compaines, etc.).
  •  

  • IDD toplevel relay node, idd.unidata.ucar.edu

     

    The cluster approach to toplevel IDD relay, has been operational at the UPC since early summer 2005.

    The cluster, described in the June 2005 CommunitE-letter article Unidata's IDD Cluster, routinely relays data to more than 650 downstream connections. Data input to the cluster nodes now averages over 12.5 GB/hr (0.3 TB/day); average data output is approx. 1.23 Gbps (~13.3 TB/day); peak rates routinely exceed 2.27 Gbps (which would be ~24.5 TB/day if the rate was sustained).

     

    The following shows a snapshot by feedtype of the data being relayed by Unidata's toplevel IDD relay, idd.unidata.ucar.edu.
    Data Volume Summary for idd.unidata.ucar.edu
    
    Maximum hourly volume  18386.051 M bytes/hour
    Average hourly volume  12791.791 M bytes/hour
    
    Average products per hour     300999 prods/hour
    
    Feed                           Average             Maximum     Products
                         (M byte/hour)            (M byte/hour)   number/hour
    NEXRAD2                5143.794    [ 40.212%]     6945.801    69965.250
    CONDUIT                3225.701    [ 25.217%]     5019.001    70255.200
    NEXRAD3                1839.069    [ 14.377%]     2357.928    81251.050
    FSL2                    877.396    [  6.859%]     1643.279     1027.550
    NGRID                   743.965    [  5.816%]     1276.658    16729.900
    HDS                     338.387    [  2.645%]      531.282    18853.000
    FNMOC                   275.755    [  2.156%]     1095.363     2960.300
    NIMAGE                  137.472    [  1.075%]      254.370      179.850
    FNEXRAD                  86.099    [  0.673%]      108.016       69.700
    IDS|DDPLUS               46.155    [  0.361%]       52.688    38724.600
    EXP                      45.278    [  0.354%]       87.771      415.650
    UNIWISC                  22.536    [  0.176%]       29.113       20.200
    LIGHTNING                 4.699    [  0.037%]        9.468      337.100
    DIFAX                     3.829    [  0.030%]       11.884        5.150
    GEM                       1.535    [  0.012%]        8.447      203.900
    GPS                       0.119    [  0.001%]        0.955        1.050
    
    Currently five real server nodes operating in one location on the UCAR campus (in the UCAR co-location facility in FL-2) and two directors comprise idd.unidata.ucar.edu. The cluster approach to IDD relay has been adopted by NOAA/GSD, Penn State and Texas A&M (using funds provided by the Unidata-administered Equipment Awards program).