Due to the current gap in continued funding from the U.S. National Science Foundation (NSF), the NSF Unidata Program Center has temporarily paused most operations. See NSF Unidata Pause in Most Operations for details.

Re: [netcdfgroup] NetCDF4 MPI-IO and compression

Rob Ross <rross@xxxxxxxxxxx> writes:

> In defense of the HDF team, this is difficult capability to enable in
> practice. This is because prior to compression you don't know how much
> space a given write will require, so parallelizing writes (especially
> if they aren't performed collectively) is not straightforward and
> requires a significant amount of coordination.
>
> Rob
>

Also I was mistaken when I wrote that parallel I/O and the deflate
filter can't be used together. This is only true for writing the data,
not reading it.

So you can read compressed data in parallel.

Thanks,

Ed
-- 
Ed Hartnett  -- ed@xxxxxxxxxxxxxxxx


  • 2008 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the netcdfgroup archives: