[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: TDS Questions

Hi Estanislao :

On 5/11/2011 2:25 PM, Estanislao Gonzalez wrote:

Hi John,

as we've already spoken I'll try to summarize here the questions:
1) To minimize the memory footprint of the TDS, is it possible to limit the cache amount used for storing catalogs in memory? Is that configurable somehow? 
not currently but i will add asap.
2) Is it possible to move the catalog to a DB? directly (read from DB) or indirectly (read from Catalog but saved into DB for intermediate caching)?
we dont want the overhead of administering a DB, and it wouldnt add any functionality in this case i think, do you?.
3) I'll have to describe the problem here a little as I can't think about the proper question:
I've developed a stager for staging files served by the TDS that are on tape, or actually in any remote repo (http://esgf.org/esgf-stager-site/ - this is part of esgf, BSD; I promise to write a proper documentation but the code is pretty much well documented, go to the repo link). The stager sits on a tomcat filter that is configured to catch some urls and map them into the expected location of the file in the file system. Using the user-agent attribute in the http head its decided if the stager goes into interactive (user is redirected to a waiting page) or non-interactive mode (call blocks, in the worst case client will time out, but a new call doesn't trigger the retrieval again). For this to work the stager recreates the complete directory structure as stored on the remote server (e.g. tape library), in this way, for example, the OpenDAP will find the file where it should be. The problem happens when the file is buried into a directory structure that wasn't there before. If only the file is retrieved, i.e. the directory structure is already in the cache area, the application works perfectly. However if a directory structure is being generated the TDS answers with a 404. Restarting the TDS shows the file is there and can be accessed properly.
You were right and the modification timestamp of the directory is being updated *iff* something in it gets changed. *However* this is not the case for sub-directories. 

So, if I have:
root/dir1
and the stager downloads dir2/file1 to root/dir1 (root/dir1/dir2/file1) the file stamp of the root doesn't change, only that of dir1 does. So this might be the problem. 

.. I'm missing the question so: is that so? :-)
if the directory.lastModified() date doesnt change when a subdirectory changes, then my caching assumptions are wrong and i need to fix it. what OS have you tested this behavior on? 




Thanks,
Estani