NLDM Status Report
September 25, 2003
We are now testing two versions of NNTP based data relay software. One
is based on INN, a popular open source implementation of the NNTP protocol,
written in C. The other is a Java implementation initially developed by
a team of students but since delegated to our visitor, Mike Linck (who
has written a status report about his work). The Java version, JNLDM, has
been running successfully on a UNIX box, and is now being tested on a Windows
machine. Although originally intended to be a receive only client,
Mike added relay out capability so that it can send statistics. It
is possible that this relay capability could support data relay, although
we have not yet tested that.
Version 1 of the NLDM statistics packages, including both user and developer
documentation, is nearly complete. The statistics have been expanded to
handle multiple reporting hosts and multiple feed types. The web page showing
these statistics is "Current NLDM Statistics" . On that page viewers can select among statistics
for a variety of machines, including the Windows machine, domain.unidata.ucar.edu.
Statistics are calculated for a five different window and bin size combinations:
5 minute window, 1 second bin size
1 hour window, 15 second bin size
2 hour window, 30 second bin size
1 day window, 5 minute bin size
2 day window, 10 minute bin size
The statistics show the following information:
Cumulative latencies: percentages of binned latencies
Reception statistics: percentage of products sent
that were received since a particular point in time
Products received over time
Bytes received over time
Maximum latencies over time
Minimum latencies over time
Connections: number of inbound connections from
all peers over time
Paths: percentages of paths taken by inbound products
Like the products, statistics are relayed using news server technology.
The statistics will be extremely valuable in configuring and testing
this network. For example, we'll be able to tune local configurations,
such as the minimum and maximum number of connections between two peers.
(In both versions of the software connections are created and destroyed
dynamically in response to some queue parameters.) Also, we'll be
able to evaluate peering as the path statistics will illustrate relative
speed between two hosts.
We have been relaying CONDUIT data to atm.geo.nsf.gov in Washington, D.C.,
since January. As of this writing only the CONDUIT feed type is being
relayed, but it is expected that the full feed will be relayed soon.
We had a successful although unplanned demonstration of the automated
routing provided by the NNTP flooding algorithm on the night of the 18th
of September (when Isabelle hit Washington, D.C.). At that time atm.geo.nsf.gov
lost Internet2 connectivity, although it maintained connectivity to the
commodity internet. Normally the vast majority of products, 98 percent
or better, take a direct path from imogene.unidata.ucar.edu to atm.geo.nsf.gov.
But during this difficulty, the statistics clearly showed that the percentage
of products taking the direct path dropped to approximately 50% to 80%,
while the other paths available rose in frequency. Significantly, there
was no observable change in product latencies. Some relevant graphs
from the statistics page at that time are shown here.
Percent of Products Taking Path imogene.unidata.ucar.edu -> atm.geo.nsf.gov
Percent of Products Taking Path imogene.unidata.ucar.edu -> canoe.uregon.edu
Percent of Products Taking Path imogene.unidata.ucar.edu -> arclight.uoregon.edu
We are building a prototype network using imogene as the ingest machine.
Currently all hosts in the network are Unidata machines, although installation
on a machine at Texas A & M is underway at the time of this writing.
We will be soliciting volunteers to be beta sites for this network.
If anyone on the User Committee or the Policy Committee is interested in
participating in this prototype, please contact Anne Wilson.