Re: [thredds] THREDDS Cache

On 5/2/2011 12:27 PM, jerry y pan wrote:
Hi John,
Our TDS (4.2) uses some compressed netcdf files (*.nc.gz) and it works fine, except that the very first access to them were slow (relatively large files, about 400 MB each). The subsequent accesses would be much faster, but it would become slow again after a while of non-activity. I can see that TDS uncompress these files to the temp data location, my question is that if TDS cleans up these temp files, which leads to the work to decompress them next time and hence the subsequent slowness? If so, is there a way to keep the cache there permanently? Or, perhaps the faster response right after the first access is due to in memory cache? Any configuration I could twist the cache?

-Jerry Pan

Hi Jerry:

Yes, compressed files are uncompressed the first time they are seen, and likely thats why you see the slowdown.

To control how these files are cached, see:

I would suggest that you use

    <*scour*>1 hour</scour>
    <*maxSize*>10 Gb</maxSize>

and choose maxSize carefully. The default directory is {tomcat}/content/thredds/cache/cdm/ by default, or set it in the above xml.

monitor the cache directory closely to see what files are uncompressed, perhaps test accessing the datasets with and without compression and time the difference.

esentially this is a space / time tradeoff. I assume you dont want to store the files uncompressed, so you have to pay the price of that. The trick is to make maxSize big enough to keep the "working set" uncompressed, ie if there is a reletively small "hot" set of files that get accessed a lot, you want to give enough cache space to keep them uncompressed.

  • 2011 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the thredds archives: