[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Updating Aggregations in TDS



Hi John:

while it is a little more complicated then that, for now we can assume we are just doing additions in time (basically we do deletes infrequently enough that we can do a restart even if needed).

-Roy

On Oct 31, 2008, at 1:09 PM, John Caron wrote:

Are files also deleted, or only added? Do files ever get appended to or modified?

Roy Mendelssohn wrote:
When a file is added to a directory, a flag is set in a table that says
it has been added. We would rather poll that table (much faster as it
doesn't need to open a read a slew of netcdf files). I can have Bob
write a program in Java to do the required recheck operations if you can
point us to the appropriate steps to be taken for the netcdf-java library.


In the near future, both the information to generate our catalog.xml and
what files are present and/or have been added will be in a database.


-Roy

On Oct 30, 2008, at 9:55 AM, John Caron wrote:

We are going to add this kind of functionality to 4.0, currently it
doesnt exist.

How do you detect when data changes?

Roy Mendelssohn wrote:
Hi John:

I am glad we understand how recheckEvery works. It is consistent with
what we were experiencing.


basically you want to do the recheck in the background, and then
accept whatever is there when a request comes in? Or you want to
detect whenever the directory changes?


Because other things, such as web page displays, need to be updated when
data is added, we already have a system where we know when data have
been added to a dataset. What we would like to do is at that point
force a recheck for that dataset, and hopefully only for that dataset,
rather than the entire catalog. If we can do that, we probably will
turn off the recheckEvery because we know every dataset will be up to
date.


Does this makes sense? If need be, we can write a script in Java to do
the above, if we can get advice on the best way to do this.


Thanks,

-Roy

**********************
"The contents of this message do not reflect any position of the U.S.
Government or NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
1352 Lighthouse Avenue
Pacific Grove, CA 93950-2097


e-mail: address@hidden (Note new e-mail address)
voice: (831)-648-9029
fax: (831)-648-8440
www: http://www.pfeg.noaa.gov/

"Old age and treachery will overcome youth and skill."
"From those who have been given much, much will be expected"

********************** "The contents of this message do not reflect any position of the U.S. Government or NOAA." ********************** Roy Mendelssohn Supervisory Operations Research Analyst NOAA/NMFS Environmental Research Division Southwest Fisheries Science Center 1352 Lighthouse Avenue Pacific Grove, CA 93950-2097

e-mail: address@hidden (Note new e-mail address)
voice: (831)-648-9029
fax: (831)-648-8440
www: http://www.pfeg.noaa.gov/

"Old age and treachery will overcome youth and skill."
"From those who have been given much, much will be expected"

**********************
"The contents of this message do not reflect any position of the U.S. Government or NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
1352 Lighthouse Avenue
Pacific Grove, CA 93950-2097


e-mail: address@hidden (Note new e-mail address)
voice: (831)-648-9029
fax: (831)-648-8440
www: http://www.pfeg.noaa.gov/

"Old age and treachery will overcome youth and skill."
"From those who have been given much, much will be expected"


NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.