Unidata - To provide the data services, tools, and cyberinfrastructure leadership that advance Earth system science, enhance educational opportunities, and broaden participation. Unidata
         
  advanced  
 

Re: [netcdfgroup] NetCDF4 MPI-IO and compression

Rob Ross <rross@xxxxxxxxxxx> writes:

> In defense of the HDF team, this is difficult capability to enable in
> practice. This is because prior to compression you don't know how much
> space a given write will require, so parallelizing writes (especially
> if they aren't performed collectively) is not straightforward and
> requires a significant amount of coordination.
>
> Rob
>

Also I was mistaken when I wrote that parallel I/O and the deflate
filter can't be used together. This is only true for writing the data,
not reading it.

So you can read compressed data in parallel.

Thanks,

Ed
-- 
Ed Hartnett  -- ed@xxxxxxxxxxxxxxxx


 
 
  Contact Us     Site Map     Search     Terms and Conditions     Privacy Policy     Participation Policy
 
National Science Foundation (NSF) UCAR Community Programs   Unidata is a member of the UCAR Community Programs, is managed by the University Corporation for Atmospheric Research, and is sponsored by the National Science Foundation.
P.O. Box 3000     Boulder, CO 80307-3000 USA     Tel: 303-497-8643     Fax: 303-497-8690