[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: 20010731: netCDF file size

>From: "Reggie Chang" <address@hidden>
>Subject: file size
>Organization: ZONA Technology
>Keywords: 200107310123.f6V1NH117343 netCDF file size

Reggie,

> Yesterday I also figured out the cause of the larger
> size file written in NetCDF format.  As you mentioned
> in the email, all the items with unlimited declared will be
> extended to the maximum 'time'.
>
> I've been wondering if there is a way to get around with
> that.  Since there is only one unlimited dimension in a
> NetCDF file, it would not be possible to introduce another
> unlimited dimension.  However, can one change the 'fixed'
> dimension of an array as the program execution continues?
> (Say change 100x100xN to 100x100x2N later)   Or is it
> possible to make the second array bigger to contain earlier
> array and then remove the first written array.  (I checked
> the User's Guide but only saw nf_del_att with the delete
> feature.)

No, sorry, netCDF avoids any kind of garbage collection in order to
keep the format and access code simple and fast.  If data were moved
around in the file to fill up holes created by arrays dynamically
growing and shrinking, netCDF would have to be more like a database
system than a small interface for array-oriented data access.  Issues
of free space management, fragmentation, and algorithms for efficient
coalescing of adjacent free blocks would all be involved.

HDF5 from NCSA comes closer to permitting that kind of flexibility, at
the cost of a more complex interface and implementation.  I think it
permits multiple resizable dimensions and supports more sophisticated
data structures.  If you really need to be able to grow data long any
dimension efficiently, you probably should look at HDF5.  If we had
the resources, we would like to eventually implement netCDF 4 using
something like the HDF 5 format to support multiple resizable
dimensions, parallelism, larger files, tile/chunked access, and packed
data.

--Russ

_____________________________________________________________________

Russ Rew                                         UCAR Unidata Program
address@hidden                     http://www.unidata.ucar.edu