Besides the LDM software which moves the data, most of the sites are running a monitoring package which measures the effectiveness of the system at that site. These statistics are sent to the Unidata Program Center (UPC) where they are synthesized and published on the WWW to provide a sense of how the overall system is performing and where there may be problems.

Data Volume

The following charts from a typical 24 hour period show that overall the data from the WSI NIDS Radar data source runs at about 200 Megabytes an hour although only a few sites use the NIDS data whereas the other widely-used data sources combined average out to 20 MB/hour. In total, roughly 220 MB per hour are available from the sources. That approaches 5 Gigabytes of data being shipped out from the data source sites each day.

Chart showing IDD Data Rates Other Than NIDS Over 24 Hours


Chart showing IDD NIDS Data Rates Over 24 Hours
For a variety of reasons, not all sites subscribe to all the datastreams. Nor are all the sites reporting statistics. However, adding up all the data moving to all the sites reporting statistics results in a figure of  about 50 Gigabytes per day being delivered by the entire system.

Reliability

The Unidata Web server includes both long-term and 24-hour charts for each of these perspectives. Early in 1997, we switched from monitoring how many products arrive at each site to measuring the latency, i.e., the time it takes for a product to get from the source to a given site. This product latency is a better measure of impending problems.   As congestion increases, latencies build up.  As the overall system is configured now, latencies can build up to about an hour before products are lost. The charts show the number of "feeds" or separate datastreams that fall into various latency categories for each or the past 24 hours.  In addition to the hourly charts, we have built two long-term histograms reflecting the day-to-day performance of the IDD. In one, the data for all sites is displayed whereas the other only shows the site-feed numbers for those instances where the latencies are over 10 minutes.

Chart showing IDD Reliability Over 24 Hours


Chart showing IDD Relay Site Reliability Over 24 Hours


Chart showing IDD Reliability Over Several Months

Limitations

While the IDD does appear to be satisfying the current data needs of the Unidata community, there are some remaining issues which cause us to believe that we must be vigilant in monitoring and improving the system. These issues fall into three main areas:

• Need for more automated response to failures. At present, manual intervention is needed to switch to an alternate upstream node when necessary. Similarly, one has to manually recover lost data during such outages. Some of these problems will be eliminated with the next release of the LDM software.
• Inadequate resources in terms of people time, hardware and networking at the relay nodes, and a need for more relay nodes. It is likely that this difficulty will never completely go away, although it will lessen as the system becomes more automated and easier to administer--both goals of the next release. Moreover, as the new site administrators gain more experience, they will be in a better position to take over as relays. Also, several of the relay nodes have received grants to upgrade their hardware, which should eliminate problems at a few overloaded sites.
• Congestion in the underlying Internet. This one is difficult to assess, partly because it is an area where we have little control. In the long term, the increasing dependence of the commercial sector on the Internet should lead to higher bandwidth and reliability. In the meantime,

How's the IDD Doing Today?

Up to date versions of these charts and others showing how the IDD is doing are available in Unidata Internet Data Distribution on the Unidata WWW server. A more detailed explanation of the charts and tables can be found in The IDD Statistics: Seeing the Forest and the Trees.