On Wed, Dec 14, 2016 at 9:13 PM, Dave Allured - NOAA Affiliate <
dave.allured@xxxxxxxx> wrote:
> So I think you have a read cacheing failure, due to interaction between
> the ncdump read pattern, and your chunking scheme.
>
...
> A sampling tool found that ncdump was spending more than 96% of its time
> inside an HDF5 chunk reader with decompression. Every time an HDF5 chunk
> is physically read from disk, the *entire* chunk must be decompressed, even
> to access a single value. You see why chunk cacheing is important.
>
Does HDF5 not do any chunk caching itself? or for that matter, netcdf4? Is
is really up to the application level to manage the caching? that seems
like handling it at the wrong level to me.
-CHB
--
Christopher Barker, Ph.D.
Oceanographer
Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception
Chris.Barker@xxxxxxxx
