Hi John: On Apr 25, 2011, at 12:42 PM, John Caron wrote: > On 4/25/2011 1:37 PM, Roy Mendelssohn wrote: >> yes, internal compression. All the files were made from netcdf3 files using >> NCO with the options: >> >> ncks -4 -L 1 >> >> The results so far show a decrease in file size from 40% of original to >> 1/100 th of the original file size. If the internally compressed data >> requests are cached differently than request to netcdf3 files, we want to >> take that into account when we do the tests, so that we do not just see the >> affect of differential cacheing. >> >> When we have done tests on just local files, the reads where about 8 times >> slower from a compressed file. But Rich Signell has found that the >> combination of serialization/bandwidth is the bottleneck, and you hardly >> notice the difference in a remote access situation. That is what we want to >> find out, because we run on very little money and with compression as >> mentioned above our RAIDS would go a lot farther, as long the hit to the >> access time is not too great. >> >> Thanks, >> >> -Roy > > in netcdf4/hdf5, compression is tied to the chunking. Each chunk is > individually compressed, and must be completely decompressed to retrieve even > one value from that chunk. So the trick is to make your chunks correspond to > your "common cases" of data access. If thats possible, you should find that > compressed access is faster than non-compressed access, because IO is > smaller. but it will be highly dependent on that. Hi John: But do you cache the entire chunk that you decompress, or do you toss it? So if I make a second request that has data in that chunk i it saved or is it reread from the file. -Thanks, -Roy ********************** "The contents of this message do not reflect any position of the U.S. Government or NOAA." ********************** Roy Mendelssohn Supervisory Operations Research Analyst NOAA/NMFS Environmental Research Division Southwest Fisheries Science Center 1352 Lighthouse Avenue Pacific Grove, CA 93950-2097 e-mail: Roy.Mendelssohn@xxxxxxxx (Note new e-mail address) voice: (831)-648-9029 fax: (831)-648-8440 www: http://www.pfeg.noaa.gov/ "Old age and treachery will overcome youth and skill." "From those who have been given much, much will be expected"