ncdigest V1 #1077

[Quincey Koziol]

On May 1, 2007, at 10:25 PM, ncdigest wrote:

>
> Date: Tue, 01 May 2007 17:12:45 -0700
> From: Katie Antypas <kantypas at lbl.gov>
> Subject: 4GigB variable size limit
>
> Hi Everyone,
>
> I'm jumping into the discussion late here, but coming from a  
> perspective
> of trying to find and develop an IO strategy which will work at the
> petascale level, the 4 GigB variable size limitation is a major
> barrier.  Already a 1000^3 grid variable can not fit into a single
> netcdf variable.  Users at NERSC and other supercomputing centers
> regularly run problems of this size or greater and IO demands are only
> going to get bigger.  We don't believe chopping up data structures  
> into
> pieces is a good long term solution or strategy.  There isn't a  
> natural
> way to break up the data and chunking eliminates the elegance, ease  
> and
> purpose of a parallel IO library.  Besides the direct code changes,
> analytics and visualization tools become more complicated as datafiles
> from the same simulation but of different sizes would not have the  
> same
> number variables.  Restarting a simulation from a checkpoint file on a
> different number of processors would also become more convoluted.
>
> The view from NERSC is that if Parallel-NetCDF is to be viable option
> for users running large parallel simulations, this is a limitation  
> that
> must be lifted...
>
> Katie Antypas
> NERSC User Services Group
> Lawrence Berkeley National Lab

	I'm certain that the netCDF team will speak up, but I think one of  
their goals in moving to use HDF5 as the underlying format for the  
netCDF-4 release is to benefit from the parallel I/O features in HDF5  
as well as the (essentially) unlimited dataset sizes that HDF5  
provides.  Have you considered using the netCDF-4 release instead of  
Parallel-netCDF?

	Quincey Koziol
	The HDF Group
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2448 bytes
Desc: not available
Url : http://mailman.unidata.ucar.edu/mailing_lists/archives/netcdfgroup/attachments/20070502/15570ab4/attachment.bin