Re: [thredds] Static aggregations

David Robertson wrote:
Hi John,

Im guessing that you have caching disabled, or is ineffective for some reason. Can you send me your threddsConfig.xml file to verify that ? If thats true, is that deliberate?

NetcdfFileCache is deliberately disabled. We have a several datasets that get replaced with identical file names on a regular basis (hourly, daily, etc.). This setup dates back to TDS 3.16 so perhaps this is better handled in 4.1 and I can turn the cache back on.
Try turning it back on a test server and see if the change to the identical file(s) get discovered .

I have a couple more questions about NetcdfFileCache. (1)What format will the NetCDF cache files have. The dataset in question is a collection of NetCDF-4 files because we needed to compress them to save space. (2)Will TDS 4.1 cache the entire dataset in question? Each yearly aggregation is ~3000 files and if the files are cached in NetCDF-3 format our scratch space will fill up quite quick.
NetcdfFileCache is an object cache, it keeps Java objects, including the aggregation lists, in memory for fast response. The data itself is not cached so the file size or format doesn't matter.


Under those circumstances, any access to an aggregation has to rebuild the aggregation, no matter what the recheckEvery setting is. TDS 4.1 now has a file system cache using ehcache, which will only do an OS file scan when the directory changes.

This could be what's causing the shorter delays (10 seconds) as that is about how long it takes to list the directory in question (~29,000 files) when it hasn't been accessed in a while.
Are all 29,000 files in the same directory as the updating files?

If so, you have a large number of files (29,000) and some of them change without actually updating the directory entry (since a new file is not created). We probably cant optimize both problems at the same time. But if you can force the directory "last modified" to update (write a dummy file?) perhaps things will work correctly.

Also, break up your files into separate directories, with most files in unchanging directories. This should help a lot.



  • 2009 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the thredds archives: