Re: another chunking question - when should chunking NOT be used

• To: netcdf-hdf@xxxxxxxxxxxxxxxx
• Subject: Re: another chunking question - when should chunking NOT be used
• From: John Caron <caron@xxxxxxxxxxxxxxxx>
• Date: Tue, 16 Dec 2003 15:19:43 -0700

Muqun (Kent) Yang wrote:

Hi,
At 02:23 PM 12/16/2003 -0700, John Caron wrote:

Ed Hartnett wrote:

Howdy all!

Another question relating to chunking - if we don't need it (i.e. for
a dataset with no unlimited dimensions), do we still chunk it?

Or is it better to leave it contiguous?

(With the mental reservation that only chunked datasets will be able
to take advantage of compression, when we get to that feature.)

Thanks!

Ed

i would be inclined to say that the default should be the old netcdf layout (continuous) just because that is optimal for sequential processing, which is a reasonnable assumption. but the previously suggested 3 modes could still be used, regardless of whether unlimited dimesnions are used or not.

HDF5 supports compression features only when chunking is turned on. So with only contiguous storage, the compression can not be done. That should be a factor to consider.

Kent

good point. so compression is a fourth mode.

• References:
  • Re: another chunking question - when should chunking NOT be
    • From: Muqun (Kent) Yang