Permission to place a copy of this work on this server has been provided by the AMS. The AMS does not guarantee that the copy provided here is an accurate copy of the published work.
At the time of its inception, the IDD was one of the first systems to use the Internet as an underlying framework for delivering data. Much has been learned about using such a system in the two years since the IDD was brought on line. This paper outlines the history and the lessons of those two years.
Another limitation imposed by satellite delivery was that a site had to receive all of the data broadcast for any data streams it was subscribed to. Often this meant using computing resources to receive large amounts of data that were not wanted. While local processing of the data could eliminate unwanted products, they still had to be received first.
To address these and other limitations, Unidata enhanced the LDM software as a client/server application that could communicate over the Internet. The software formed the underpinnings of the IDD system (Domenico, 1994). In creating the IDD, Unidata arranged for basic weather data streams to be delivered to participating universities in the U.S. and Canada[3] either at no charge (other than the costs associated with connecting to the Internet), or at discounted rates. In addition, the LDM software supports data subscription, allowing IDD sites to tailor data stream delivery so they only receive wanted data.
The IDD is a cooperative effort between the Unidata Program Center (UPC) and
Unidata member institutions. Data is ingested at a source site and then
shipped via the LDM over the Internet to one or more downstream LDM servers.
The data is then used locally, and also relayed to other downstream sites,
or nodes. In this fashion, the network load is shared by multiple
participants in the IDD. Presently, there are over 120 participating IDD
nodes, with almost 30 of these acting as relays (Figure 1).
The IDD has been widely viewed as successful, with participating universities gaining unprecedented access to near real-time meteorological data. Data received by Unidata sites include the Family of Services from the National Weather Service (DDS, PPS, IDS and HDS streams) (NWS Home Page, September 1996), GOES satellite imagery from the Space Science and Engineering Center at the University of Wisconsin (SSEC Home Page, September 1996), NEXRAD radar data (NEXRAD Home Page, September, 1996) and data from the National Lightning Detection Network (GAI Home Page, September 1996).
To address the problem of monitoring the Internet, Unidata has developed several monitoring tools, including Perl scripts that tie together several commonly available Unix tools in a way that helps identify problems. These tools include netcheck(1) and syscheck(1), as well as LDM-specific tools such as ldm-ping(1). The scripts are designed to run on a schedule and provide notification if problems are detected.
Once a problem is detected, the process of identifying the source of the problem on the Internet is complicated because end-users do not have access to the various routers and gateways that exist. It is often possible to identify where a problem is, but not the nature of the problem. To further complicate matters, there is usually no direct contact with the entity that controls the source of the problem, and reporting must be done via a roundabout method of dealing with local network organizations who, in turn, report the problem to someone else further up the line, and so on. For data transmission problems to be solved, they must be reported from both ends of the outage. Otherwise, network providers take the position that the problem is with someone else's network, and do not follow through with trouble reporting. Also, many problems are design issues that may take several months to correct. The LDM software allows for a certain amount of elasticity of data, and can handle data latencies approaching one hour, but if the latencies get any larger, then data is lost.
The Internet providers are responding to Internet congestion problems by bringing new routing technologies on line, and adding increased capacity and new network interconnect points. However, it will be a slow process to alleviate all of the current problems.
To address current Internet congestion issues, we have advised sites to limit the amount of data they receive. There is no point in receiving all of the NCEP model output if only a single model is needed. Using the data subscription capabilities of the LDM, this strategy has reduced the number of data products going into sites having reception problems and has significantly reduced, and in some cases eliminated, data loss.
This complicated the task of maintaining an efficient topology. Few valid assumptions could be made to start from. As a result, the process of determining the most efficient topology started as a trial and error exercise. Two further complications were that the routing between any two nodes is asymmetrical, and it changes with time. Thus, any topology could become obsolete at any time, without warning.
As a result of our experience, and given the lack of automated topology configuration capabilities in the current version of the LDM, we have taken a pragmatic approach to the problem. In essence, we assign topologies based on what looks to be the best fit within the constraints of available relay nodes and routing possibilities. We then address individual problems as they are identified and make topology changes accordingly.
All of the data relay nodes, other than the data source sites, operate on a volunteer basis. Site administrators of relay sites agree to take on responsibility for monitoring the IDD (or at least their segment of it), addressing problems of their downstream sites, and responding to topology changes as necessary. Not all sites have the time, personnel, network connectivity, hardware and technical knowledge to perform such duties. In addition, a number of sites simply do not want the added responsibilities. As a result, the IDD has often been faced with a shortage of relay sites to handle new leaf nodes.
To the IDD community's credit, every time this problem has occurred, new sites have stepped up to take on the task of being a relay. However, finding such sites and dealing with the topology issues that arise when a site is promoted to a higher tier in the distribution tree takes time and a large amount of coordination. The fact that the IDD is not centrally administered adds to the complexity of making operational changes.
Failover sites solved one of the outage problems, but did not address the issue of a failure at a data source site. If data is not being injected into the IDD, then no amount of failover will help. Unfortunately, most of the data sources have no redundancy, and downtime at the source results in lost data. However, a few of the IDD sites also receive the Family of Services data via satellite broadcast from Alden Electronics (Alden Home Page, September 1996). Unidata is in the process of arranging for one of these sites to act as a failover node for the IDD first tier relay sites (those sites that get their data from the data source node). This will provide redundancy for one large chunk of the existing data.
The most difficult problem to solve is that of topology specification. There are two main options that Unidata is investigating. The first is the use of reliable multicast protocols. The use of such protocols would eliminate the need for a formal topology and also reduce the amount of data fan-out (and thus the amount of data that has to be transmitted). Unfortunately, there is not yet a single recognized and proven reliable multicast protocol available. Until one exists, multicast is not a viable solution.
The second option is to design into the LDM the ability to automatically configure itself to get the data from the best connection it can find. Using an appropriate metric, such as latencies, the LDM could measure various connection options and choose the best one. If the connection later degraded, another could be found. This is a complex solution, with a number of potential problems that would have to be overcome. The most serious of these would be the question of whether the overall configuration of the IDD topology would stabilize, or if this approach would lead to constant thrashing as LDM servers continuously reconfigure themselves for better connections.
Unidata is looking at both multicast protocols and automated topology configuration. In future releases of the LDM, we hope to implement one of these and thus address the most intractable problems that the IDD faces. Additionally, either of these solutions removes most of the issues of community cooperation. With multicast, no topology is needed, so there is no need for IDD participants to agree to be relay nodes. If the solution is to automate the topology, all nodes in effect become relays, which limits the amount of administrative overhead current relay site administrators deal with. If an upstream site goes down, the downstream LDM would simply reconfigure itself to use another site.
The issue of redundancy in terms of failover sites will begin to be addressed in the next release of the LDM[4]. Unidata is testing a program to automate checking of the upstream node and to provide for automatic reconfiguration of the LDM to switch to its failover, and then back to its primary when able. In future releases, if both multicast and automated topology capabilities prove to be unusable, automatic failover will be built into the LDM. This will remove the need to have external programs and monitors running.
Unidata efforts also continue to focus on including in the IDD new data streams of interest to community members. As such data streams are identified, the ease of adding them to the IDD delivery system will allow us to either make them available, or assist others within the community who wish to do so. Possible data sources include NOAAport, broadband NIDS data, and six minute profiler data.
Another benefit to the IDD concept has been the increasing availability of other, ad hoc, data sets, injected by participating universities and made available to sites interested in them. While not integrated into the IDD topology, the availability of the technology has enabled individuals with interesting data sets to make them available with little effort. The idea of the IDD being a cooperative effort of shared responsibilities has proven to be both viable and efficient.
ftp://ftp.unidata.ucar.edu/pub/ldm5/ldm.tar.ZUniversities interested in participating in the IDD should contact Unidata User Support:
support@unidata.ucar.eduIn addition, all other questions regarding either the LDM software or the IDD should be directed to Unidata User Support.
http://www.alden.com/Domenico, B., S. Bates, D. Fulker, 1994: Unidata Internet Data Distribution (IDD). Proceedings, Tenth International Conference on Interactive Information and Processing Systems for Meteorology, Oceanography and Hydrology, AMS, J15-J20.
Davis, G. and R. Rew, 1994: The Unidata LDM: Programs and Protocols for Flexible Processing of Data Products. Proceedings, Tenth International Conference on Interactive Information and Processing Systems for Meteorology, Oceanography, and Hydrology, AMS, 131-136.
Fulker, D., 1994: Principles Underlying Internet Data Distribution.
http://www.unidata.ucar.edu/projects/idd/plans/principles.htmlGAI Home Page.
http://www.gds.com/IDD Home Page.
http://www.unidata.ucar.edu/projects/idd/IDD System Performance.
http://www.unidata.ucar.edu/projects/idd/status/sitepercent183.gifLDM Home Page.
http://www.unidata.ucar.edu/software/ldm/NEXRAD Home Page.
http://www.nws.noaa.gov/modernize/88dtech.htmNWS Home Page.
http://www.nws.noaa.gov/SSEC Home Page.
http://www.ssec.wisc.edu/Unidata Home Page.
http://www.unidata.ucar.edu/
