| Date | Number of Institutions | Number of hosts |
|---|---|---|
| October, 2001 | 131 | 244 |
| February, 2002 | 155 | 268 |
| May, 2002 | 165 | 292 |
| October, 2002 | 163 | 305 |
Since last May about ten new sites have expressed interest in participating in the IDD. And, our international reach is growing further. We now have new sites in Mexico, and Hong Kong. Talks are under way regarding establishing a site in Argentina. Also, IDD data is now being fed to Germany in the WMO's pilot project testing the use of the LDM for data relay.
Of 145 sites reporting hourly IDD statistics, only 30, about 20%, are using the latest version. Indeed, 65% are using version 5.1, which was released around spring of 2000.
The IDS, DDS, PPS, and HDS feed types remain the most popular. In decreasing order, the next most popular were NNEXRAD, PROFILER, MCIDAS, NLDN, FNEXRAD, DIFAX, and UNIWISC (a new name for MCIDAS).
| Feed Type | min Daily Prod Cnt | max Daily Prod Cnt | max Avg Hourly Prod Cnt | max Hourly Prod Cnt | min Daily Byte Cnt | max Daily Byte Cnt | max Avg Hourly Byte Cnt | max Hourly Byte Cnt |
|---|---|---|---|---|---|---|---|---|
| CONDUIT | 454620 | 533662 | 21661 | 42818 | 19991.8 | 24455.9 | 979.3 | 1908.8 |
| NNEXRAD | 475138 | 564554 | 22582 | 24983 | 2989.9 | 4010.6 | 160.4 | 207.4 |
| HDS | 213968 | 256965 | 10386 | 24284 | 2089.4 | 2779.2 | 114.0 | 273.5 |
| IDS|DDPLUS | 141363 | 172942 | 6917 | 9824 | 157.5 | 195.4 | 7.8 | 10.6 |
| all feeds (including CRAFT) | 1720076 | 2041189 | N.A. | 130959 | 36608.7 | 44677.3 | N.A. | 5122.6 |
CRAFT stake holders attended a workshop in Norman, OK, on September 26 and 27. The purpose of the meeting was to discuss how to modify the distribution system from a prototype to becoming operational. The meeting was attended by representatives from academia, government (NWS), and commercial enterprises.
In the short term, CRAFT is facing the end of its budget, which was only intended to be seed money. Without further funding, the data may stop flowing sometime this fall, in what is assumed would be temporary gap.
However, the discussion focused mostly on long term planning for a network topology that would meet the needs of the three sectors involved. In particular, the commercial vendors expressed a willingness to share data among themselves. Also, the NWS representatives voiced a willingness to consider relaying the data to three separate networks at the nearest Internet POP to the radar, rather than sending the data to Silver Spring first. If accepted, this would allow each sector to build and maintain their own network and reduce latencies. We are awaiting the outcome of this decision.
netflow statistics (available at the Internet2 Weekly Report list).
However, the amount has been steadily shrinking since that peak. The
last report lists LDM traffic at .53% of traffic, with 1.57
terabytes. It is still, however, the leading advanced application.
The LDM workshop ran August 1 -3. It was attended by 21 people who gave positive evaluations.
More sites are now running rtstats, our new latency
reporting software that reports latencies every hour.
(Real time, graphical representations of latencies can be seen at Real-Time IDD statistics).
rtstats has been helpful in many ways, including in the effort to relay data to Belem,
Brazil.
In this case, simply relaying the IDS|DDPLUS text stream to Belem was
problematic.
This stream consists of many small products, which
is not a problem in relaying within the U.S.
But, rtstats
showed that pushing these
products through a single connection to Belem caused significant
product loss.
Tom Yoksas found that
splitting that feed into three connections allowed the products to
reach their destination with acceptable latencies.
This showed once again that the LDM's reliance on the RPC protocol,
which requires a round trip send and acknowledgment for every
block transmitted, is a bottleneck. Due to this result, the next version of the LDM
will make separate connections for each separate
request entry in the configuration file, allowing
users to easily break requests into multiple connections.
Other efforts are under way to improve the performance of the current LDM. These efforts include Steve Chiswell and Steve Emmerson's experimentation with increasing the block size from the current 16K so that large products could be sent with fewer RPC calls. Initial results showed some improvement, but were limited by the operating system's determination of a maximum block size smaller than the experimental size. Research is continuing in this area.
However, these positive latency results were tempered by the possibility that around 10% of products were missing. Since Usenet is not known to lose articles, the problem most likely lies in the test bed. As analysis tools are still under development, it is possible that these tools are faulty. It is also possible that products were being dropped by other servers. One reason for this is that the newsgroup name inappropriately did not have the word "binaries" in it, and thus may have been canceled by other servers. Also, the naming scheme chosen for the subject line apparently triggered spam filters in some servers. These issues are under investigation.
In September a team of five students from the the Senior Software Engineering class at the University of Colorado, Boulder, selected a Unidata proposal to write LDM Lite, a NNTP based, platform independent, receive-only LDM client. This is a one year project. The students are working with Anne to develop this software, and are currently building a prototype to help guide them in the actual development.