[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: 970610: data concept NetCDF question (mime attatchments)

>To: address@hidden
>cc: address@hidden
>From: Kurt Brandl <address@hidden>
>Subject: data concept
>Organization: TU-Wien
>Keywords: 199706101549.JAA19182

Hi Kurt,

> A couple of years ago, we decided to develop a software package not 
> only for transfer but much more for storage of surface analytical data 
> of various instruments in a standard form, in order to enable 
> standardized image and signal processing. During the time of 
> development the acronym MAXMIND (Management And Exchange of Method 
> Independent Data) was introduced as project name. We have now released 
> the 1.0 version of the MAXMIND software including a first version of 
> documentation into the WWW at URL: 
> "http://www.iap.tuwien.ac.at/www/maxmind/";. 
>  
> We are using HDFVsets for the transfer and the storage of surface 
> analytical data for multi-method analysis of whole measurement series 
> (file size a few 100 MB). These files include measurement 
> descriptions, instrument descriptions and a sample history as well as 
> raw data of parameter values, images and spectra in one physical file.
> 
> 
> We would like to ask you a few questions:
> 
> - - is there any layers on the netCDF with files system like functions
> 
>       - deleting data

There are no netCDF functions for deleting variables (which contain most
of the data) or dimensions (which may be shared by multiple variables).
There is a function for deleting an attributes (containing metadata).
Hence, deleting data can only be accomplished by copying the undeleted data.

>       - grouping data (directories) like HDFVsets e.g.

The only means for grouping data with netCDF involve use of conventions
such as a "group" attribute to specify the group(s) of which a variable
is a member, or global attributes that name the variables in each group.


>       - linking

Not yet, though this is in our eventual plans for netCDF-4.

>       - copying

There is an nc_var_copy() function available, though it is not part of
the documented interface.  Various higher-level packages (e.g. FAN,
ncks, ...) support copying selected variables or extracting subsets of
selected variables.

>       - changing data size

Without copying the data, the only size change that is permitted is to
increase the "unlimited" dimension, allowing data variables that use
that dimension to grow together.  This resembles record-oriented I/O, if
you think of the unlimited dimension as the record number.

> - - is there support for including of raw data files into netCDF, e.g. 
> including
>   of the XY intern binary file into netCDF

You can store an array of opaque bytes into a netCDF variable, but you
must know the variable's size when it is created.

> - - how scale netCDF performance with size of file (e.g. 500 MB HDF 
> file)

All netCDF data access uses direct access (seek to an offset) rather
than sequential access, so it scales very well.  Accessing a small
subset of a large dataset takes no longer than accessing a small subset
of a small dataset, in general.

> Please find enclosed one gif-file of the data concept.

Thanks.

--Russ

_____________________________________________________________________

Russ Rew                                         UCAR Unidata Program
address@hidden                     http://www.unidata.ucar.edu