Relaying Data Via NNTP
Unidata is investigating the use of NNTP for the purpose of data relay.
An early report of the reasons for this effort and its viability is available
at On
the Use of Internet News for Delivering Data .
Some
preliminary testing results are available at Exploring
Usenet as an Alternative for Distributing Geoscientific Data Via the
Internet
As an experiment, Unidata is currently relaying high resolution model output from Boulder to
Washington, D.C. via INN. This is a high volume stream
ranging from 23 to 25 gigabytes of data per day. A
test starting on January 11 and extending for over 24 hours
resulted in the successful relay of over one half million products with 99%
arriving within 4 seconds or less:
Product Latencies
Total values read: 549622
secs #prods % cumulative%
---- ------ ---- ------------
-1: 1572 0.29 0.29
0: 364983 66.41 66.69
1: 149892 27.27 93.96
2: 21588 3.93 97.89
3: 5440 0.99 98.88
4: 913 0.17 99.05
5: 413 0.08 99.12
10: 652 0.12 99.24
20: 1699 0.31 99.55
30: 1416 0.26 99.81
40: 975 0.18 99.99
50: 79 0.01 100.00
Graphs showing current 5 minute statistics are available at NNTP Relay Stats. These four graphs show (1) average latency per second, (2) products received per second, (3) minimum latency per second, and (4) maximum latency per second.
Products relayed via Usenet for this test are under the group unidata.*
with a 'Distribution: unidata' header. Currently all unidata.binaries.*
articles will be sourced with a path header entry of imogene.unidata.ucar.edu.
Newsgroup Hierarchy
A newsgroup hierarchy for data relay is under development for this purpose.
That hierarchy will be posted here as groups are created. Currently,
the group being used for data relay is called unidata.binaries.grid.grib
Group Security
These are moderated groups to help ensure that the data products received
are what they are represented to be. Data products are currently
checksummed to ensure proper transmission.
Data Encoding
A variety of data encodings are being evaluated for efficiency. One,
labeled as X-UnidataEncoding under the Content-Transfer-Encoding header,
is a non proprietary
encoding that produces output significantly smaller than some of the
more standard encodings.
Viewing the Data
Unidata provides several packages for viewing atmospheric data, such as
the Integrated Data Viewer (IDV), GEMPAK, and MCIDAS. Please see
our software
page for more information about acquiring and using these packages.
However, note that some packages require a priori "decoding" into
a format suitable for viewing. Thus, to view the data you may also
need the appropriate decoders, which we also provide. For further
information about decoders please see the same page.
Note that to receive a data product via NNTP and view it, the product
would need to be decoded from its transmission format and then subsequently
decoded into a format compatible with the chosen visualization package.
Contact Information
Please send questions or comments to:
<anne@unidata.ucar.edu>
This document is maintained by
Anne Wilson<anne@unidata.ucar.edu>
and was updated on
.
