NLDM Progress Report
October 12, 2004
The NLDM network is successfully relaying seven major IDD data feeds
among six relay machines. Two additional machines use the same
relay technology to receive statistics for display. The network
has been robust and reliable.
The network currently consists of the following eight machines:
|ingest and relay
|ingest and relay
|Iowa City, IO
|College Station, TX
|San Jose, CA
The network is relaying the CONDUIT, CRAFT,
NEXRAD, IDS|DDPLUS, UNIWISC and NIMAGE data feeds. This selection
of feed types shows that the network can handle a variety of types of
streams. The CONDUIT stream is a bursty, high volume binary
stream. CRAFT, HDS, and NEXRAD are more continuous, high volume,
binary streams. IDS|DDPLUS is a continuous stream of very small
text products. UNIWISC is a sparse, relatively low volume, binary
stream of products of around 3MB. NIMAGE is also a sparse binary
stream, but contains the largest data
products in the IDD, with product sizes up to around 20MB.
has been augmented to allow comparison of the same statistics across
feed types for a particular machine. Bringing up multiple
the statistics pages allows comparison across machines and provides a
window to network performance as a whole.
Latencies and reception are good. A
recent examination showed 99% of all products arriving at all sites
5 seconds for the most part. The exceptions were for machines
having sporadic network issues or periods of high load.
Also, the large NIMAGE products typically take around 45 -55
seconds. I suspect this is due to having to push 20MB over a
single connection, as opposed to pushing the same volume over multiple
connections, as occurs at times with the CONDUIT stream without the
higher latencies. (This would also be true for LDM.)
I have been working on a comprehensive white paper describing the
research results and features of INN and NNTP, which is now about 35
pages long. The final section of the paper, which remains to be
"Recommendations". However, I will make the general case here.
INN relays data at least as well as LDM. With latencies for both
generally quite small and with statistics being calculated differently
for each, it is difficult to argue at this point that one is better in
this regard than the other.
However, INN has additional features that the LDM does not have, which
could be useful to us:
We are discussing how to proceed. There is no urgent need to act
as LDM6 is well positioned to handle data flow in the near
future. Steve Emmerson and I are researching the question of what
an ideal data relay system would do and which technology would move us
best in that direction.
- The flooding algorithm has been shown to provide good
- INN dynamically creates and destroys connections between peers.
- INN can switch automatically between CHECK and
no-CHECK streaming transmission modes.
- Working with newgroups is a big improvement over the
current limited number of LDM feed types.
- INN newsgroup lists support metacharacter expansion
similar to that used in shell file name expansion.
- INN supports "negative subscriptions", i.e., users can specify
what newsgroups they don't want to receive.
- With its backlog handling, INN can handle outages on
the order of weeks, which could be useful as we migrate into areas
beyond real time data distribution.
- For similar reasons, the pull based retrieval
provided by NNTP could also be beneficial.
- INN supports multiple methods of article storage yielding a range
of article retention times.
- The article storage method most similar to the LDM product queue
supports multiple buffers and the logical grouping of physical buffers
into higher level "meta" buffers.
- In addition to streaming, INN supports batch transmission.
- Arbitrary header information can be attached to a product.
This can serve as metadata.
- INN provides high level server administration tools, including
resource monitoring and email notification of problems.
- NNTP supports some network-level administration via control
messages, particularly notification of newsgroup changes.
- INN is IPV6 enabled.
- INN peers can be reconfigured dynamically without restarting the
- Users can communicate with a remote INN server directly via
telnet. This has been useful in testing and can also be used for
retrieval. (Although one could telnet to a LDM port, the LDM
protocol doesn't support such interactive sessions.)
- Due to the lengthy history and popularity of news, there is a lot
of NNTP based software available.
With the NLDM network already in place, another approach would be to
give the community a choice of LDM or NLDM. After some
improvement of the release engineering, with INN being robust and
reliable it is possible that this could be accomplished with fairly
minimal committment of resources in the short term.