IDD and NOAAPort
Status Report: April 2014 - September 2014
Mike Schmidt, Jeff Weber, Tom Yoksas
Strategic Focus Areas
The IDD/NOAAPort group's work supports the following Unidata funding proposal focus areas:
- Enable widespread, efficient access to geoscience data
A project like the IDD demonstrates how sites can employ the LDM to move data in their own environments. - Develop and provide open-source tools for effective use of
geoscience data
The IDD is powered by the Unidata LDM-6 which is made freely available to all. The Unidata NOAAPort ingest package is being used by a variety of university and non-university community members. Both the LDM and NOAAPort ingest packages are being bundled by Raytheon in AWIPS-II. - Provide cyberinfrastructure leadership in data discovery,
access, and use
The community-driven IDDs provide push data services to users an ever increasing community of global educators and researchers - Build, support, and advocate for the diverse geoscience
community
Providing access to data in real-time is a fundamental Unidata activity.
The IDD-Brasil, the South American peer of the North American IDD operated by the UPC, is helping to extend real-time data delivery outside of the U.S. to countries in South America and Africa. The Universidad de Costa Rica is experimenting with relaying data received in the IDD to Colombia.
Activities Since the Last Status Report
Internet Data Distribution (IDD)
- Unidata receives High Resolution Rapid Refresh
(HRRR) grids (both 2D and 3D fields) in an LDM/IDD
feed from NOAA/GSD. These products are being made
available from the gale.unidata.ucar.edu. The
challenge in making the data routinely available is its
large data volume which is on the order of ~8 GB for the
pressure level output and ~10 GB/hour for the sigma level
output.
The HRRR is being experimentally served at: http://thredds-jumbo.unidata.ucar.edu/thredds/modelsHrrr.html (.xml for machines)
-
Other data sets we are actively exploring with NOAA/GSD/ESRL are:
- HRRR and ESTOFS data are scheduled to be added to NOAAPort
in mid to late September. The following TINs announced
these additions:
http://www.nws.noaa.gov/os/notification/tin14-28hrrr-cca.htm
http://www.nws.noaa.gov/os/notification/tin13-43estofs_noaaport_aaa.htmBriefly, these additions will be comprised of:
- HRRR: 81 products, hourly F00-15 each hour. CONUS 2.5km grid184. ~44 GB/day
- ESTOFS: 3 products, hourly F00-F180, 00, 06, 12, 18z runs. CONUS 2.5km grid, Puerto Rico 1.25 km grid. ~2 GB/day
HRRR fields and forecasts times that are not included in the NOAAPort expansion will be evaluated as additions to the CONDUIT IDD datastream.
- The UPC continues to relay FNMOC and the CMC data model output
directly to the community. FNMOC provides the COAMPS and
NAVGEM model output and the CMC provides the GEM model
output. Unidata has provided access to these data for the
past 8 years, but on a "point-to-point" basis. GEM model
output was converted from GRIB1 to GRIB2 in January. The
CMC is now relaying output of there new hi-resolution
(15 km) GEM model to Unidata.
NOAAPort Data Ingest
- The NOAAPort SBN, which transitioned from DVB-S to DVB-S2 in April/May 2011, is being upgraded to support just over 60 mbps throughput in aggregate. The UPC has been testing ingest of the high speed broadcast since the onset of a "dual illumination" period (a 45 day window in which existing and new SBN transmissions are active) on August 18.
- Unidata's NOAAPort ingest package is bundled with current versions of the LDM. The current LDM release is v6.12.5.
- Raytheon bundles a version LDM-6 with AWIPS-II and is actively using Unidata's NOAAPort ingest code at a variety of NOAA offices. Raytheon has been providing the UPC code modifications and GRIB table updates needed to support new data to be added to in the NOAAPort expansion. when possible
Relevant IDD Metrics
- Approximately 600 machines at 235 sites are
running LDM-6 and reporting real time statistics to
Unidata. Unidata staff routinely assist in the installation
and tuning of LDM-6 at user sites as a community service.
A number organizations/projects continue use the LDM to move substantial amounts of data that do not report statistics to Unidata: NOAA, NASA, USGS, USACE, Governments of Spain, South Korea, private compaines, etc.).
- IDD toplevel relay node, idd.unidata.ucar.edu
The cluster approach to toplevel IDD relay, has been operational at the UPC since early summer 2005.
The cluster, described in the June 2005 CommunitE-letter article Unidata's IDD Cluster, routinely relays data to more than 700 downstream connections. Data input to the cluster nodes now routinely averages about 20 GB/hr (~0.5 TB/day); average data output from the entire cluster exceeds 1.3 Gbps (~14 TB/day); peak rates routinely exceed 2.2 Gbps (which would be ~24 TB/day if the rate was sustained).
The following shows a snapshot by feedtype of the data being received on one node of the Unidata toplevel IDD relay, idd.unidata.ucar.edu.
Data Volume Summary for uni14.unidata.ucar.edu Maximum hourly volume 27500.168 M bytes/hour Average hourly volume 16285.983 M bytes/hour Average products per hour 308585 prods/hour Feed Average Maximum Products (M byte/hour) (M byte/hour) number/hour NEXRAD2 7042.426 [ 43.242%] 9842.548 71041.318 CONDUIT 2531.718 [ 15.545%] 4401.147 50981.750 NEXRAD3 2228.789 [ 13.685%] 2924.057 97256.909 NGRID 1624.678 [ 9.976%] 3372.235 21758.409 FNMOC 1166.527 [ 7.163%] 6643.485 3242.273 FSL2 835.136 [ 5.128%] 1613.164 1013.523 HDS 358.902 [ 2.204%] 692.099 18245.659 NIMAGE 160.874 [ 0.988%] 292.486 193.727 GEM 81.814 [ 0.502%] 463.448 792.295 FNEXRAD 65.020 [ 0.399%] 110.009 48.318 NOTHER 57.996 [ 0.356%] 365.100 1162.955 IDS|DDPLUS 53.046 [ 0.326%] 66.749 42150.591 EXP 36.403 [ 0.224%] 74.339 326.909 UNIWISC 36.218 [ 0.222%] 84.984 19.591 LIGHTNING 5.817 [ 0.036%] 15.456 348.682 DIFAX 0.512 [ 0.003%] 1.968 0.636 GPS 0.109 [ 0.001%] 1.197 1.045Currently six real server nodes operating in one location on the UCAR campus (in the UCAR co-location facility in FL-2) and two directors comprise idd.unidata.ucar.edu. The cluster approach to IDD relay has been adopted by NOAA/GSD, Penn State and Texas A&M.