using netCDF for storing data from realtime source

[Timothy Hume]

I guess I must have written my software inefficiently. Also, I had to
deal with data coming in out of sequence (sometimes a message from the
GTS for an old observation would come in after a message for a new
observation). This is probably why I found that setting up a fixed
length array was quicker, because I could insert each observation in a
time slot that had been reserved for it, and eliminate the need to sort
the data into temporal order.

-----Original Message-----
From: owner-netcdfgroup at unidata.ucar.edu on behalf of Thaden, John J
Sent: Tue 2007-01-30 04:50
To: netcdfgroup at unidata.ucar.edu
Subject: using netCDF for storing data from realtime source

I must say this topic has caught me by surprise.  I'd gotten the
impression
from reading that the netCDF binary file type was created specifically
to
gracefully handle real time acquisition.  Those variables having as one
of
their dimensions the unlimited dimension are, to my understanding,
located
physically at the bottom of the file so that new data can be appended to
them
without rewriting existing portions of the file, that, if one such
variable
is to receive new data that will cause it to exceed the current actual
length
of the unlimited dimension, then all variables sharing the unlimited
dimension will be lengthened, the one with an actual datum, others with
a
value representing "missing data", to be replaced in later if real data
became available for those variables as well.  The processing overhead
of
writing missing values would seem to be a small price to pay for having
a
file that nicely organizes a collection of variables, and array-type
variables, such that all can be addressably accessed.  Can this be done
with
text files?    

John Thaden
Research Assistant Professor of Geriatrics
College of Medicine
University of Arkansas for Medical Sciences
Little Rock

"Timothy Hume" <T.Hume at bom.gov.au> wrote...
>
> I once set up a system to store real-time meteorological data in
NetCDF
> files. One of the difficulties is that you can never be sure of the
> temporal frequency of the data; some stations report hourly, others
> three hourly, and others report irregularly. To get around this issue,
I
> used the unlimited dimension for time, and simply "grew" the dimension
> as new data became available. However, "growing" the unlimited
dimension
> seemed to take a lot of computer time compared to slotting the data
into
> an array where all the dimensions were fixed length. It may be that I
> wrote my software inefficiently, but I suspect the slowness of
"growing"
> the unlimited dimension is partly due to the structure of a NetCDF
file.
> If it is possible, it may be more efficient to predefine the length of
> your time dimension (making it large enough to hold all the data you
> receive), rather than using an unlimited dimension. The downside of
this
> approach is that you are likely to be left with some "empty cells" in
> your data array.

> Also, the option of using a text file, as suggested by someone else,
may
> be better. You can trivially append new data to a text file using the
> shell's ">>" feature.


Confidentiality Notice: This e-mail message, including any attachments,
is for the sole use of the intended recipient(s) and may contain
confidential and privileged information.  Any unauthorized review, use,
disclosure or distribution is prohibited.  If you are not the intended
recipient, please contact the sender by reply e-mail and destroy all
copies of the original message.


=========================
=========================
=========================
====
To unsubscribe netcdfgroup, visit:
http://www.unidata.ucar.edu/mailing-list-delete-form.html
=========================
=========================
=========================
====