overcoming netcdf3 limits

[Greg Sjaardema]

John Caron wrote:
> Hi Greg:
>
> Can you send me the output from ncdump -h on a typical file ?
Attached is the output from one of our files.  The file it comes from is
approximately 407 MB. Files can range from a few hundred bytes up to
100's GB.
>
> How does the parallel part work? do certain sections of the file get
> reserved for different nodes, or is there a master node that
> essentially serializes the writing?
During mesh generation, we generate the model in anywhere from 1 to M
separate pieces and then join these pieces into a single mesh model
prior to the analysis.  We then have another tool which will determine
how to decompose the model across the number of processors being used
for the analysis run.  It then creates N subpieces of the model which
are then read by the analysis code. The analysis code then runs and each
processor writes its portion of the model to a results file. After the
job is finished, then we can combine those pieces into a single piece
for visualization.  We also have the option to join into subpieces (join
4,000 individual pieces into 10 groups) or some of the visualizers will
read the individual pieces without joining.   An old description of what
we did back in 1999 is available at
http://endo.sandia.gov/SEACAS/Documentation/Parallel_Instruction.pdf. 
The basic operation is the same today, but the sizes of models have
increased greatly and some of the processes and codes have changed.

--Greg
>
> Greg Sjaardema wrote:
>> As a quick answer to the question, we (Sandia Labs) use netcdf
>> underneath our exodusII
>> file format for storing finite element results data.
>>
>> If the mesh contains #nodes nodes and #elements elements, then there
>> will be a dataset of the size #elements*8*4 (assuming a hex element with
>> 8 nodes, 4 bytes/int) to store the nodal connectivity of each hex
>> element in a group of elements (element block). Assuming 4GiB, this
>> limits us to ~134 Million elements per element block (using CDF-2) which
>> is large, but not enough to give us more than a few months breathing
>> room.    Using CDF-1 format, we top out at about 30 million elements or
>> less which is hit routinely.
>>
>> There is a pdf file at
>> http://endo.sandia.gov/SEACAS/Documentation/exodusII.pdf that shows
>> (starting at page 177) how we map exodusII onto netcdf.  There have been
>> some changes since the report was written to reduce some of the dataset
>> sizes.  For example, we split the "coord" dataset into 3 separate
>> datasets now and we also split the vals_nod_var into a single dataset
>> per nodal variable.
>>
>> --Greg
>>
>>
>> John Caron wrote:
>>> Hi Rob:
>>>
>>> Could you give use case(s) where the limits are being hit?
>>> I'd be interested in actual dimension sizes, number of variables,
>>> whether you are using a record dimension, etc.
>>>
>>> Robert Latham wrote:
>>>> Hi
>>>>
>>>> Over in Parallel-NetCDF land we're running into users who find even
>>>> the CDF-2 file format limitations, well, limiting.
>>>> http://www.unidata.ucar.edu/software/netcdf/docs/netcdf/NetCDF-64-bit-Offset-Format-Limitations.html
>>>>
>>>>
>>>>
>>>> http://www.unidata.ucar.edu/software/netcdf/docs/faq.html#Large%20File%20Support10
>>>>
>>>>
>>>>
>>>> If we worked up a CDF-3 file format for parallel-netcdf (off the top
>>>> of my head, maybe a 64 bit integer instead of an unsigned 32 bit
>>>> integer could be used to describe variables), would the serial netcdf
>>>> folks be interested, or are you looking to the new netcdf-4 format to
>>>> take care of these limits?
>>>>
>>>> Thanks
>>>> ==rob
>>>>
>>> ==============================================================================
>>>
>>>
>>> To unsubscribe netcdfgroup, visit:
>>> http://www.unidata.ucar.edu/mailing-list-delete-form.html
>>> ==============================================================================
>>>
>>>
>>>
>>>
>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: exo.header
Url: http://mailman.unidata.ucar.edu/mailing_lists/archives/netcdfgroup/attachments/20070425/e99918ca/attachment.ksh