[netcdfgroup] NetCDF4 MPI-IO and compression

[Rob Ross]

That makes sense. -- Rob

On Feb 13, 2008, at 2:03 PM, Ed Hartnett wrote:

> Rob Ross <rross at mcs.anl.gov> writes:
>
>> In defense of the HDF team, this is difficult capability to enable in
>> practice. This is because prior to compression you don't know how  
>> much
>> space a given write will require, so parallelizing writes (especially
>> if they aren't performed collectively) is not straightforward and
>> requires a significant amount of coordination.
>>
>> Rob
>>
>
> Also I was mistaken when I wrote that parallel I/O and the deflate
> filter can't be used together. This is only true for writing the data,
> not reading it.
>
> So you can read compressed data in parallel.
>
> Thanks,
>
> Ed
> -- 
> Ed Hartnett  -- ed at unidata.ucar.edu
>