[netcdfgroup] NetCDF4 MPI-IO and compression

[Rob Ross]

In defense of the HDF team, this is difficult capability to enable in  
practice. This is because prior to compression you don't know how much  
space a given write will require, so parallelizing writes (especially  
if they aren't performed collectively) is not straightforward and  
requires a significant amount of coordination.

Rob

On Feb 13, 2008, at 12:07 PM, Ed Hartnett wrote:

> Thomas Green <thomas.green at metoffice.gov.uk> writes:
>
>> NetCDF users,
>>
>> Just a quick question.  Can NetCDF4 use MPI-IO if also using
>> compression?  I read on the HDFgroup website that MPI-IO and  
>> compression
>> are not compatible but I am not sure if this still applies and how it
>> affects NetCDF4.
>>
>> The HDFgroup FAQ related to MPI-IO and compression:
>> http://www.hdfgroup.com/hdf5-quest.html#p5comp
>>
>> Many thanks,
>>
>> Tom
>>
>
> No, sorry, that is not allowed by HDF5.
>
> If you want parallel I/O you must use uncompressed data.
>
> I have asked the HDF5 team whether this might be changed in a future
> HDF5 release...
>
> Ed
>
> -- 
> Ed Hartnett  -- ed at unidata.ucar.edu
> _______________________________________________
> netcdfgroup mailing list
> netcdfgroup at unidata.ucar.edu
> For list information or to unsubscribe,  visit: http://www.unidata.ucar.edu/mailing_lists/
>