overcoming netcdf3 limits

[John Caron]

Greg Sjaardema wrote:
> John Caron wrote:
>>
>> Greg Sjaardema wrote:
>>> As a quick answer to the question, we (Sandia Labs) use netcdf
>>> underneath our exodusII
>>> file format for storing finite element results data.
>>>
>>> If the mesh contains #nodes nodes and #elements elements, then there
>>> will be a dataset of the size #elements*8*4 (assuming a hex element with
>>> 8 nodes, 4 bytes/int) to store the nodal connectivity of each hex
>>> element in a group of elements (element block). Assuming 4GiB, this
>>> limits us to ~134 Million elements per element block (using CDF-2) which
>>> is large, but not enough to give us more than a few months breathing
>>> room.    Using CDF-1 format, we top out at about 30 million elements or
>>> less which is hit routinely.
>> Im not sure if I understand the problem yet:
>>
>> In the file you sent me, you use time as the record variable.
> Yes.
>> Each record variable must be less than 2^32, not counting the record
>> dimension. So you can have about 2^29 elements, assuming each element
>> is 8 bytes. And you can have 2^32 time steps.
> Yes, but the record variables are currently not the limiting area.
>> The non-record variables are dimensioned (num_elements, num_nodes).
>> Number of nodes seems to be 8, and these are ints, so you have 32
>> bytes * num_elements, so you can have a max of 2^27 elements = 134
>> million elements. 
> Yes, this is where we are hitting the limit.
>> Currently the largest you have is 33000. Do you need more than 2^27 ?
> The file I sent was representative.  As I think I mentioned in that
> email, we have models ranging in size from 1 element up to 250 million
> elements and everything in between.  The header I sent was just to show
> how the models are set up, not a model that is the largest we typically
> use.  Yes, we definitely need more than 2^27 elements.
>> I think Im not sure what you mean by "a few months breathing room".
> In a few more months if we do not change either the underlying netcdf
> limits or the way that our exodusII format is using netcdf, we will have
> to tell users that the models they require cannot be stored.

ok, thanks for clarifying that.

do you expect to need more than 2^31 = 2.147 * 10^9 elements ?? that is the usual limit for array indices, and it would be difficult (at least in Java) to read more than that at once. reading sections at a time would be ok. 

is there any "natural" way to break up the element array into more than one dimension?

==============================================================================
To unsubscribe netcdfgroup, visit:
http://www.unidata.ucar.edu/mailing-list-delete-form.html
==============================================================================