using netCDF for storing data from realtime source

• To: netcdfgroup@xxxxxxxxxxxxxxxx
• Subject: using netCDF for storing data from realtime source
• From: "Thaden, John J" <ThadenJohnJ@xxxxxxxx>
• Date: Mon, 29 Jan 2007 09:50:42 -0600

I must say this topic has caught me by surprise.  I'd gotten the impression
from reading that the netCDF binary file type was created specifically to
gracefully handle real time acquisition.  Those variables having as one of
their dimensions the unlimited dimension are, to my understanding, located
physically at the bottom of the file so that new data can be appended to them
without rewriting existing portions of the file, that, if one such variable
is to receive new data that will cause it to exceed the current actual length
of the unlimited dimension, then all variables sharing the unlimited
dimension will be lengthened, the one with an actual datum, others with a
value representing "missing data", to be replaced in later if real data
became available for those variables as well.  The processing overhead of
writing missing values would seem to be a small price to pay for having a
file that nicely organizes a collection of variables, and array-type
variables, such that all can be addressably accessed.  Can this be done with
text files?    

John Thaden
Research Assistant Professor of Geriatrics
College of Medicine
University of Arkansas for Medical Sciences
Little Rock

"Timothy Hume" <T.Hume@xxxxxxxxxx> wrote...
>
> I once set up a system to store real-time meteorological data in NetCDF
> files. One of the difficulties is that you can never be sure of the
> temporal frequency of the data; some stations report hourly, others
> three hourly, and others report irregularly. To get around this issue, I
> used the unlimited dimension for time, and simply "grew" the dimension
> as new data became available. However, "growing" the unlimited dimension
> seemed to take a lot of computer time compared to slotting the data into
> an array where all the dimensions were fixed length. It may be that I
> wrote my software inefficiently, but I suspect the slowness of "growing"
> the unlimited dimension is partly due to the structure of a NetCDF file.
> If it is possible, it may be more efficient to predefine the length of
> your time dimension (making it large enough to hold all the data you
> receive), rather than using an unlimited dimension. The downside of this
> approach is that you are likely to be left with some "empty cells" in
> your data array.

> Also, the option of using a text file, as suggested by someone else, may
> be better. You can trivially append new data to a text file using the
> shell's ">>" feature.


Confidentiality Notice: This e-mail message, including any attachments, is for 
the sole use of the intended recipient(s) and may contain confidential and 
privileged information.  Any unauthorized review, use, disclosure or 
distribution is prohibited.  If you are not the intended recipient, please 
contact the sender by reply e-mail and destroy all copies of the original 
message.


==============================================================================
To unsubscribe netcdfgroup, visit:
http://www.unidata.ucar.edu/mailing-list-delete-form.html
==============================================================================

• Follow-Ups:
  • Re: using netCDF for storing data from realtime source
    • From: Ed Hartnett