[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: NetCDF

> Organization: IBM
> Keywords: 199411231658.AA25072

Hi Steve,

>   Lloyd Treinish suggested I discuss this with you. . .
> I am in the process of selecting a database format to use with our device/
> process simulation tools (which use unstructured meshes).  He has a NetCDF
> specification which allows storage of unstructured mesh data on NetCDF files,
> and he can read these files using Data Explorer.  I am concerned, however,
> about our need for limited database-type operations.  In particular,
> we will be incrementally constructing large files, as each time step or
> bias point is solved, and Lloyd thought that the current NetCDF implimentation
> requires a file copy operation for each 'object insert' operation.  Is this
> true?  Also, efficient object deletion is important.  How does NetCDF support
> deletion?

Adding data for a time step to a netCDF file can be done efficiently if you
use the netCDF unlimited dimension for time.  This will not require a file
copy, but merely a seek and write to the end of an existing file, and an
update to the record size near the beginning of the file.

If by `object insert' you mean adding a new variable, however, then Lloyd is
right: you cannot build a netCDF file incrementally by interspersing the
creation of variables with the writing of data.  Each addition of a variable
(or dimension or attribute) is like updating the schema of a database, and
requires work equivalent to copying the data to another file.  This can be
avoided if you define all needed netCDF objects at the beginning and write
data into the objects only after they have all been created.  If you cannot
anticipate which of a small set of objects will be needed, it may not waste
much space to create them all and only add data to those that are needed.
If, however, the objects needed cannot be anticipated, adding them to
existing netCDF files will be expensive.

NetCDF does not support deletion very well.  Only attributes can be deleted.
There is no interface in the l9library for deletion of variables or
dimensions, so this can only be done by copying data into a new file defined
without the deleted objects.  Of course data in existing objects can be
overwritten by different data, but only by accessing the same object.

> We could discuss this by phone if you prefer;  if so, please email me your
> phone number and suggested times to call.

I much prefer email, since I can't predict very well when I'll be near my
phone.

--
Russ Rew                                                UCAR Unidata Program
address@hidden                                          P.O. Box 3000
http://www.unidata.ucar.edu/                          Boulder, CO 80307-3000