March 25, 2011
This effort supports five of the funding proposal's six focus areas:
1. Broadening participation and expanding community services
2. Advancing data services
3. Developing and deploying useful tools
5. Providing leadership in cyberinfrastructure
6. Promoting diversity by expanding opportunities
NB: The NWS stopped disseminating most of the low resolution Level III products on January 12, 2011.
The NWS is proceeding with distribution of dual polarization WSR88D Level 3 products via the NOAAPort SBN and RPCCDS. Implementation is expected to begin in a test mode has begun. This addition to the NOAAPort SBN required code modifications to the Unidata NOAAPort ingest package; coordinated deployment of the package; and modifications of existing end-user LDM pattern-action file actions. The code modifications needed were included in an update to the NOAAPort ingest package and has been adopted by all sites know to be using the package.
NOAA rearchitected its NEXRAD Level II data collection and dissemination. The current Level II Refresh includes collection of all Level II data at a primary site (NWSTG in Silver Springs, MD) with an active backup site (ROC in Norman, OK). All connections are being made over the NOAAnet MPLS backbone. Data collected at the active site (primary or backup) is relayed to the current set of top level IDD relay nodes (IRADS (OK), Purdue (IN), and the ERC (NC)) over Internet2 and to a small number of additional sites designated by the NWS.
NB: Unidata sites continue to receive the Level II data in the same way that they always have: from one of the top level relay sites or from the top level relay nodes in Boulder, CO (UPC) or Vienna, VA (NSF/AGS).
The IDD toplevel relay nodes operated by Unidata redundantly feed Level II data from all of the toplevel relay nodes in order to provide as failsafe a system as possible.
NB: All sites ingesting the NOAAPort SBN feed will need to upgrade their receivers. The UPC is following the Weather Service's lead by adopting use of the Novra S-300 DVB-S2 receiver. The challanges involved in use of the new receiver are 1) availability -- the new units will not be available from Novra until mid-April at the earliest, and 2) cost -- the Novra S-300 costs approx. $649 plus shipping from Canada.
There are several organizations/projects that use the LDM to move substantial amounts of data that do not report statistics to Unidata: NOAA, NASA, USGS, USACE, Governments of Spain, South Korea, private compaines, etc.).
The cluster approach to toplevel IDD relay, has been operational at the UPC since early summer 2005. The cluster, described in the June 2005 CommunitE-letter article Unidata's IDD Cluster, routinely relays data to more than 650 downstream connections. Data input to the cluster nodes is now over 6.5 GB/hr (0.16 TB/day); average data output is approx. 525 Mbps (~5.7 TB/day); peak rates routinely exceed 1.15 Gbps (which would be ~12.4 TB/day if the rate was sustained).
The following shows a snapshot by feedtype of the data being relayed by Unidata's toplevel IDD relay, idd.unidata.ucar.edu.
Data Volume Summary for idd.unidata.ucar.edu Maximum hourly volume 12898.705 M bytes/hour Average hourly volume 5756.894 M bytes/hour Average products per hour 227670 prods/hour Feed Average Maximum Products (M byte/hour) (M byte/hour) number/hour CONDUIT 2468.451 [ 42.878%] 7214.778 65778.822 NEXRAD2 1312.950 [ 22.807%] 1943.615 48699.889 NGRID 708.782 [ 12.312%] 1523.854 15412.422 NEXRAD3 459.403 [ 7.980%] 652.756 40539.711 HDS 254.359 [ 4.418%] 427.890 17985.156 FNMOC 231.792 [ 4.026%] 1521.107 2026.578 NIMAGE 140.420 [ 2.439%] 232.029 182.489 FNEXRAD 68.213 [ 1.185%] 83.200 72.311 IDS|DDPLUS 41.875 [ 0.727%] 51.148 35980.178 EXP 38.806 [ 0.674%] 78.491 428.111 UNIWISC 23.145 [ 0.402%] 32.259 26.200 DIFAX 3.690 [ 0.064%] 12.396 5.044 GEM 2.562 [ 0.045%] 19.542 179.644 FSL2 1.788 [ 0.031%] 1.976 22.289 LIGHTNING 0.586 [ 0.010%] 1.593 327.067 WMO 0.071 [ 0.001%] 0.074 3.889
Currently eight real server nodes operating in two locations on the UCAR campus (in the UPC offices and in FL-2) and one director comprise idd.unidata.ucar.edu. A second director was added to the cluster to increase redundancy. The cluster approach to IDD relay has been adopted by NOAA/GSD and Penn State (using funds provided by the Unidata-administered Equipment Awards program). The cluster operated by Penn State assumed toplevel relay responsibilities for CONDUIT data in early March, 2010.
Weekly I2 usage statistics can be seen at Internet NetFlow: Weekly Reports (search for 'UNIDATA LDM' usage statistics in the Advanced Applications portion of Table 7).
NB: We are unsure why the Internet2 statistics are
not being routinely updated of late!
by Tom Yoksas and Jeff Weber.