October 1, 2010
This effort supports five of the funding proposal's six focus areas:
1. Broadening participation and expanding community services
2. Advancing data services
3. Developing and deploying useful tools
5. Providing leadership in cyberinfrastructure
6. Promoting diversity by expanding opportunities
NB: The NWS plans to stop diseminating the low resolution products on December 1, 2010.
The NWS is proceeding with its plan to distribute dual polarization WSR88D Level 3 products via the NOAAPort SBN and RPCCDS. Implementation is expected to begin in a test mode in the November 2010 timeframe. This addition to the NOAAPort SBN required code modifications to the Unidata NOAAPort ingest package; coordinated deployment of the package; and modifications of existing end-user LDM pattern-action file actions. The code modifications needed have already included in an update to the ingest package and has been adopted by all sites know to be using the package.
NOAA rearchitected the way that NEXRAD Level II data is collected and disseminated. The current Level II Refresh includes collection of all Level II data at a primary site (NWSTG in Silver Springs, MD) with an active backup site (ROC in Norman, OK). All connections are being made over the NOAAnet MPLS backbone. Data collected at the active site (primary or backup) is be relayed to the current set of top level IDD relay nodes (IRADS (OK), Purdue (IN), and the ERC (NC)) over Internet2 and to a small number of additional sites designated by the NWS.
NB: Unidata sites continue to receive the Level II data in the same way that they always have: from one of the top level relay sites or from the top level relay nodes in Boulder, CO (UPC) or Vienna, VA (NSF/AGS).
The Unidata NOAAPort ingest package is known to be used by 16 sites. The number of actual users is undoubtedly higher since SSEC makes the package available to its McIDAS-XCD users.
There are several organizations/projects that use the LDM to move substantial amounts of data that do not report statistics to Unidata: NOAA, NASA, USGS, USACE, Governments of Spain, South Korea, private compaines, etc.).
The cluster approach to toplevel IDD relay, has been operational at the UPC since early summer 2005. The cluster, described in the June 2005 CommunitE-letter article Unidata's IDD Cluster, routinely relays data to more than 650 downstream connections. Data input to the cluster nodes is now over 6.5 GB/hr (0.16 TB/day); average data output is approx. 525 Mbps (~5.7 TB/day); peak rates routinely exceed 1.15 Gbps (which would be ~12.4 TB/day if the rate was sustained).
The following lists the breakdown by feedtype of the data being relayed by Unidata's toplevel IDD relay, idd.unidata.ucar.edu.
Data Volume Summary for idd.unidata.ucar.edu Maximum hourly volume 14155.751 M bytes/hour Average hourly volume 6973.663 M bytes/hour Average products per hour 249825 prods/hour Feed Average Maximum Products (M byte/hour) (M byte/hour) number/hour CONDUIT 2471.870 [ 35.446%] 7245.773 64061.738 NEXRAD2 2175.907 [ 31.202%] 2917.550 62035.357 NEXRAD3 820.510 [ 11.766%] 1048.914 51436.000 NGRID 690.929 [ 9.908%] 1406.051 15980.810 HDS 253.852 [ 3.640%] 393.416 17838.524 FNMOC 226.990 [ 3.255%] 1312.805 1960.190 NIMAGE 134.981 [ 1.936%] 241.821 168.571 FNEXRAD 87.758 [ 1.258%] 109.766 72.405 IDS|DDPLUS 40.365 [ 0.579%] 55.577 35461.214 EXP 37.829 [ 0.542%] 65.342 330.786 UNIWISC 22.599 [ 0.324%] 29.568 26.786 DIFAX 3.305 [ 0.047%] 12.090 4.833 GEM 2.807 [ 0.040%] 26.388 192.476 LIGHTNING 2.129 [ 0.031%] 4.094 232.762 FSL2 1.830 [ 0.026%] 2.076 22.357
Currently seven real server nodes operating in two separate locations on the UCAR campus (in the UPC offices and in FL-2) and one director comprise idd.unidata.ucar.edu. A second director has been added to the cluster to increase redundancy. The cluster approach to IDD relay has been adopted by NOAA/GSD and Penn State (using funds provided by the Unidata-administered Equipment Awards program). The cluster operated by Penn State assumed toplevel relay responsibilities for CONDUIT data in early March, 2010.
Weekly I2 usage statistics can be seen at Internet NetFlow: Weekly Reports (search for 'UNIDATA LDM' usage statistics in the Advanced Applications portion of Table 7).
by Tom Yoksas and Jeff Weber.