Re: overcoming netcdf3 limits

On Tue, Apr 24, 2007 at 11:31:20AM -0600, John Caron wrote:
> Could you give use case(s) where the limits are being hit?
> I'd be interested in actual dimension sizes, number of variables, whether 
> you are using a record dimension, etc.

I'm just the intermediary, but I've copied two users who ran into this
limit.

Here's what Greg had to say:
   As a quick answer to the question, we use netcdf underneath our
   exodusII file format for storing finite element results data.  If
   the mesh contains #nodes nodes and #elements elements, then there
   will be a dataset of the size #elements*8*4 (assuming a hex element
   with 8 nodes, 4 bytes/int) to store the nodal connectivity of each
   hex element in a group of elements (element block).  Assuming 4GiB,
   this limits us to ~134 Million elements per element block which is
   large, but not enough to give us more than a few months breathing
   room.                               

In both cases, they want to stick with a single netcdf file as they
scale to very large machines and problems.   They also like the
stability of the CDF-2 format: there's no metadata corruption if a
process dies in the middle of a write. 

If we (parallel-netcdf) make changes to either the API or file format
to overcome CDF-2 limitations, would the serial-netcdf folks be
interested in that work?  

Thanks
==rob

-- 
Rob Latham
Mathematics and Computer Science Division    A215 0178 EA2D B059 8CDF
Argonne National Lab, IL USA                 B29D F333 664A 4280 315B

==============================================================================
To unsubscribe netcdfgroup, visit:
http://www.unidata.ucar.edu/mailing-list-delete-form.html
==============================================================================