Re: overcoming netcdf3 limits

On Tue, Apr 24, 2007 at 02:30:39PM -0600, Ed Hartnett wrote:
> However, another question: have you considered using netCDF-4? It does
> not have the limits of the 64-bit offset format, and support parallel
> I/O, as well as a number of other features (groups, compound data
> types) which might be helpful in organizing really large data sets.

Hi Ed

The CDF-1 and CDF-2 file formats appear to be quite robust in the face
of client failures.  Greg S at least has observed file corruption with
the HDF5 file format during parallel I/O if a client dies at a
particular time.   As I understand it, it's hard to devise a solution
for the HDF5 file format that is both rock-solid robust *and* delivers

In parallel-netcdf land we also like the CDF-1 and CDF-2 file formats:
they are easy to work with from an MPI-IO perspective.    Also, by
writing out all the metadata in define mode, there is very little
chance of file corruption for any failure in data mode. 


Rob Latham
Mathematics and Computer Science Division    A215 0178 EA2D B059 8CDF
Argonne National Lab, IL USA                 B29D F333 664A 4280 315B

To unsubscribe netcdfgroup, visit: