[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Updating Aggregations in TDS

Hi Roy:

Sorry for taking so long:

Roy Mendelssohn wrote:
Hi All:

We have a question as to exactly how the update of aggregation of files occurs in concert with the recheckEvery setting. Some background will help. We update a lot of aggregated datasets quite often, but not at exactly regular times (say roughly hourly but all over the map). When a new files has been added, here is a place where a flag is set that says this is the case. Some of our programs read this, and therefore expect the new file to be included in the time aggregation. However, when programs go to the TDS we find that the actual update only occurs after we make the new request.

Am I correct that this is the way that the aggregation (with caching) works in concert with recheckEvery:

1. The first time the dataset is requested, the dimensional information is read and is stored in cache.

2. Nothing happens again, regardless of the value in recheckEvery and what is in the directory, until a new request is made on that dataset.

3. If a new request is made, and the recheckEvery time has not been exceeded, then the cached values are used.

4. If a new request is made, and the recheckEvery time has been exceeded, then the directory is rescanned.


Is this a correct understanding of the process? If so, where our problem occurs is that some of our directories have lots of files, so the rescan takes awhile and it can take 2-3 requests to get the new values showing up.

So my question is there a way to have the directory rescanned when we have added files, other than sending a request? If we reinitialize the TDS from the Tomcat manager through a command, would that do it (which strikes me as overkill since the catalogs themselves are not changing).

basically you want to do the recheck in the background, and then accept whatever is there when a request comes in? Or you want to detect whenever the directory changes?



"The contents of this message do not reflect any position of the U.S. Government or NOAA."
Roy Mendelssohn
Supervisory Operations Research Analyst
Environmental Research Division
Southwest Fisheries Science Center
1352 Lighthouse Avenue
Pacific Grove, CA 93950-2097

e-mail: address@hidden (Note new e-mail address)
voice: (831)-648-9029
fax: (831)-648-8440
www: http://www.pfeg.noaa.gov/

"Old age and treachery will overcome youth and skill."
"From those who have been given much, much will be expected"

NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.