Relaying Data Via NNTP
Unidata is investigating the use of NNTP for the purpose of data relay.
An early report of the reasons for this effort and its viability is available
at "On the Use of Internet News for Delivering Data" .
preliminary testing results are available at "Exploring Usenet as an Alternative for Distributing Geoscientific Data Via the Internet"
As an experiment, Unidata is currently relaying high resolution model output from Boulder to
Washington, D.C. via INN. This is a high volume stream
ranging from 23 to 25 gigabytes of data per day. A
test starting on January 11 and extending for over 24 hours
resulted in the successful relay of over one half million products with 99%
arriving within 4 seconds or less:
Total values read: 549622
secs #prods % cumulative%
---- ------ ---- ------------
-1: 1572 0.29 0.29
0: 364983 66.41 66.69
1: 149892 27.27 93.96
2: 21588 3.93 97.89
3: 5440 0.99 98.88
4: 913 0.17 99.05
5: 413 0.08 99.12
10: 652 0.12 99.24
20: 1699 0.31 99.55
30: 1416 0.26 99.81
40: 975 0.18 99.99
50: 79 0.01 100.00
Graphs showing current 5 minute statistics are available "NNTP Relay Stats". These four graphs show (1) average latency per second, (2) products received per second, (3) minimum latency per second, and (4) maximum latency per second.
Products relayed via Usenet for this test are under the group unidata.*
with a 'Distribution: unidata' header. Currently all unidata.binaries.*
articles will be sourced with a path header entry of imogene.unidata.ucar.edu.
A newsgroup hierarchy for data relay is under development for this purpose.
That hierarchy will be posted here as groups are created. Currently,
the group being used for data relay is called unidata.binaries.grid.grib
These are moderated groups to help ensure that the data products received
are what they are represented to be. Data products are currently
checksummed to ensure proper transmission.
A variety of data encodings are being evaluated for efficiency. One,
labeled as X-UnidataEncoding under the Content-Transfer-Encoding header,
is a non proprietary encoding that produces output significantly smaller than some of the
more standard encodings.
Viewing the Data
Unidata provides several packages for viewing atmospheric data, such as
the Integrated Data Viewer (IDV), GEMPAK, and MCIDAS. Please see
for more information about acquiring and using these packages.
However, note that some packages require a priori "decoding" into
a format suitable for viewing. Thus, to view the data you may also
need the appropriate decoders, which we also provide. For further
information about decoders please see the same page.
Note that to receive a data product via NNTP and view it, the product
would need to be decoded from its transmission format and then subsequently
decoded into a format compatible with the chosen visualization package.