Re: [thredds] How are compressed netcdf4 files handled in TDS

Hi John:

On Apr 25, 2011, at 12:42 PM, John Caron wrote:

> On 4/25/2011 1:37 PM, Roy Mendelssohn wrote:
>> yes, internal compression.  All the files were made from netcdf3 files using 
>> NCO with the options:
>> 
>> ncks -4 -L 1
>> 
>> The results so far show a decrease in file size from 40% of original to 
>> 1/100 th of the original file size.   If the internally compressed data 
>> requests are cached differently than request to netcdf3 files, we want to 
>> take that into account when we do the tests, so that we do not just see the 
>> affect of differential cacheing.
>> 
>> When we have done tests on just local files, the reads where about  8 times 
>> slower from a compressed file.  But Rich Signell has found that the 
>> combination  of serialization/bandwidth is the bottleneck, and you hardly 
>> notice the difference in a remote access situation.  That is what we want to 
>> find out, because we run on very little money and with compression as 
>> mentioned above our RAIDS would go a lot farther, as long the hit to the 
>> access time is not too great.
>> 
>> Thanks,
>> 
>> -Roy
> 
> in netcdf4/hdf5, compression is tied to the chunking. Each chunk is 
> individually compressed, and must be completely decompressed to retrieve even 
> one value from that chunk. So the trick is to make your chunks correspond to 
> your "common cases" of data access. If thats possible, you should find that 
> compressed access is faster than non-compressed access, because IO is 
> smaller. but it will be highly dependent on that.


Hi John:

But do you cache the entire chunk that you decompress, or do you toss it?  So 
if I make a second request that has data in that chunk i it saved or is it 
reread from the file.

-Thanks,

-Roy



**********************
"The contents of this message do not reflect any position of the U.S. 
Government or NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
1352 Lighthouse Avenue
Pacific Grove, CA 93950-2097

e-mail: Roy.Mendelssohn@xxxxxxxx (Note new e-mail address)
voice: (831)-648-9029
fax: (831)-648-8440
www: http://www.pfeg.noaa.gov/

"Old age and treachery will overcome youth and skill."
"From those who have been given much, much will be expected"