using netCDF for storing data from realtime source

[Gerry Creager]

Depending on resources, I'd consider this application for a relational 
database like PostgreSQL, MySQL or Oracle, depending on your preferences 
and hexadecimal religion.  A simple insert without reindexing is pretty 
quick, and if you're certain you don't have duplications it can be 
blindingly fast, in my experience.

We do this for surface observations coming from a variety of sources 
with a variety of frequencies of update, although no one source at 1000 
Hz...

gerry

Timothy Hume wrote:
> Hi Magnus,
> 
> I once set up a system to store real-time meteorological data in NetCDF 
> files. One of the difficulties is that you can never be sure of the 
> temporal frequency of the data; some stations report hourly, others 
> three hourly, and others report irregularly. To get around this issue, I 
> used the unlimited dimension for time, and simply "grew" the dimension 
> as new data became available. However, "growing" the unlimited dimension 
> seemed to take a lot of computer time compared to slotting the data into 
> an array where all the dimensions were fixed length. It may be that I 
> wrote my software inefficiently, but I suspect the slowness of "growing" 
> the unlimited dimension is partly due to the structure of a NetCDF file. 
> If it is possible, it may be more efficient to predefine the length of 
> your time dimension (making it large enough to hold all the data you 
> receive), rather than using an unlimited dimension. The downside of this 
> approach is that you are likely to be left with some "empty cells" in 
> your data array.
> 
> Also, the option of using a text file, as suggested by someone else, may 
> be better. You can trivially append new data to a text file using the 
> shell's ">>" feature.
> 
> Tim Hume
> Bureau of Meteorology Research Centre
> Melbourne
> Australia
> 
> -----Original Message-----
> From: owner-netcdfgroup at unidata.ucar.edu on behalf of Magnus Hagdorn
> Sent: Fri 2007-01-26 05:44
> To: netcdfgroup at unidata.ucar.edu
> Subject: using netCDF for storing data from realtime source
> 
> Hello world,
> we are currently considering to use the netCDF library to store data
> coming from our instruments. The data represents multiple variables
> together with a timestamp of the measurement. Some variables are updated
> more often than others. Sample rates are of the order of 100Hz to
> 1000Hz. I was thinking of storing the data as a table with 3 columns,
> timestamp, ID and value together with some appropriate header. Do you
> think this is a reasonable idea, are there other more suitable standard
> formats?
> Regards
> magnus
> 
> ==============================================================================
> To unsubscribe netcdfgroup, visit:
> http://www.unidata.ucar.edu/mailing-list-delete-form.html
> ==============================================================================
> 

-- 
Gerry Creager -- gerry.creager at tamu.edu
Texas Mesonet -- AATLT, Texas A&M University	
Cell: 979.229.5301 Office: 979.458.4020 FAX: 979.862.3983
Office: 1700 Research Parkway Ste 160, TAMU, College Station, TX 77843

==============================================================================
To unsubscribe netcdfgroup, visit:
http://www.unidata.ucar.edu/mailing-list-delete-form.html
==============================================================================