Due to the current gap in continued funding from the U.S. National Science Foundation (NSF), the NSF Unidata Program Center has temporarily paused most operations. See NSF Unidata Pause in Most Operations for details.
Due to the current gap in continued funding from the U.S. National Science Foundation (NSF), the NSF Unidata Program Center has temporarily paused most operations. See NSF Unidata Pause in Most Operations for details.
Dear Christopher, HDF5 does provide a chunk cache, but I presume in this case it is simply too small to fit. You can imagine if you increase the number of columns further at some point the 2D chunks will exceed (any) sized cache. This is however, suboptimal in that case as ncdump outputs row after row. One way to fix this would be to adjust ncdump to increase the chunk cache to a reasonable amount of main memory, potentially offering users a way to influence this value from command line. See https://support.hdfgroup.org/HDF5/doc/RM/RM_H5P.html#Property-SetChunkCache There is quite a good doku here about general aspects of Chunking: https://support.hdfgroup.org/HDF5/doc/Advanced/Chunking/ Julian 2016-12-15 20:17 GMT+01:00 Chris Barker <chris.barker@xxxxxxxx>: > On Wed, Dec 14, 2016 at 9:13 PM, Dave Allured - NOAA Affiliate > <dave.allured@xxxxxxxx> wrote: > >> >> So I think you have a read cacheing failure, due to interaction between >> the ncdump read pattern, and your chunking scheme. > > ... >> >> A sampling tool found that ncdump was spending more than 96% of its time >> inside an HDF5 chunk reader with decompression. Every time an HDF5 chunk is >> physically read from disk, the *entire* chunk must be decompressed, even to >> access a single value. You see why chunk cacheing is important. > > > Does HDF5 not do any chunk caching itself? or for that matter, netcdf4? Is > is really up to the application level to manage the caching? that seems like > handling it at the wrong level to me. > > -CHB > > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker@xxxxxxxx > > _______________________________________________ > NOTE: All exchanges posted to Unidata maintained email lists are > recorded in the Unidata inquiry tracking system and made publicly > available through the web. Users who post to any of the lists we > maintain are reminded to remove any personal information that they > do not want to be made public. > > > netcdfgroup mailing list > netcdfgroup@xxxxxxxxxxxxxxxx > For list information or to unsubscribe, visit: > http://www.unidata.ucar.edu/mailing_lists/ -- http://wr.informatik.uni-hamburg.de/people/julian_kunkel
netcdfgroup archives: