[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Ncwms-users] Still some cache or time issues for our curvilinear FMRC OpenDAP data?

That is sufficient in all cases, but may not be necessary.

In an aggregation, files are rechecked whenever the recheck_every time has been exceeded. If there is a request happening at the same time, one can see the data values shift, and an inconsistent data model.

We need to make sure to only delete files when a request is not happening. To 
do that correctly, we have to integrate the TDS with the scouring program. We 
are looking into that. Meanwhile, if there is a time when requests are unlikely 
to happen (eg 2 am), delete then. If you cant afford to allow even the 
possibility of an incorrect response, then you should stop the server, delete, 
and restart.

Note that adding files at the end of the aggregation is not a problem. Any 
current request will just not see the new files.

Richard Signell wrote:

Please confirm:
For now, if providers want to make sure that their time-changing
aggregations from TDS are properly handled by clients like ncWMS, they
need to make sure that files are not removed from the start of the
aggregation.   And if they are, thredds needs to be restarted.


On Wed, May 20, 2009 at 3:00 PM, Richard Signell <address@hidden> wrote:

I know ALL the details of this dataset!  ;-)

This machine is running a TDS4, and the FMRC aggregation is working on
a directory that that is updated each day (if everything goes well)
with a new forecast file.   There is a cron job that runs once a day
that moves any files in this directory to an archive location.

Right now the files in this directory look like this:
address@hidden his]$ ls
his_20090513.nc  his_20090515.nc  his_20090517.nc
his_20090514.nc  his_20090516.nc  his_20090520.nc

So there are a few days missing here because things *did* go wrong.
(problems with A/C in the cluster room).

Tomorrow there should be a new "his_20090521.nc" file around 9:15 am,
and at 10:00 the cronjob runs and will remove the "his_20090513.nc"
file.     The aggregation does a "recheckEvery=10min", and the ncWMS
which points to the "best time series" from the FMRC also has a
recheck of every 10 minutes.

If I know that deleting the old files is causing a problem, we can
take an alternate approach of keeping all the files for a month, not
deleting any old ones, and then start new the next month (restarting
THREDDS and NcWMS, I suppose).

Is that enough info?


On Wed, May 20, 2009 at 10:28 AM, John Caron <address@hidden> wrote:
Jon Blower wrote:
Hi Rich,

I think what's happened here is the following:
1) ncWMS synced with your OPeNDAP server and noticed a time dimension
of length, say, 100
2) The data under the OPeNDAP server changed, reducing the length of
the time dimension to, say 80
3) Before the next sync with ncWMS, you requested a map that
corresponded to a time index of, say, 84, which was off the end of the
actual data but valid as far as ncWMS knew.

I guess this is a potentially-serious problem.  Even if you didn't get
an error, you might get the wrong data.  This could happen at anytime
when you add or remove data from the beginning or middle of a dataset.
 It won't happen if you add new data on to the end of a dataset (which
is what I assumed would almost always happen).

Any ideas as to how this could be fixed?  We could sync with the
OPeNDAP server with every single GetMap request, but this would
seriously affect performance.  Does OPeNDAP have a versioning system
whereby you can find out whether a dataset has changed without
downloading the whole metadata record?

Cheers, Jon
This is a deep problem. Opendap considers itself a stateless protocol, like 
HTTP. But for datasets that can change, this is a problem. Aggregation datasets 
are particularly susceptible.

Like HTPP, we need to use sessions to prevent state changes that cause these 
kinds of errors. We are looking to redesign the TDS to deal with this, but we 
need to be able to control the adding and removing of files, make guesses as to 
session timeout, get clients to return cookies, etc. So its a rather big and 
messy job. And that's just the TDS, dunno what other opendap servers might do. 
In a way, we've done all the easy stuff and now we're on to the harder stuff.

Rich, if you know any of the details of this dataset, how/why/when it changes, 
what kind of opendap server it is, etc, that would be useful.

Crystal Reports - New Free Runtime and 30 Day Trial
Check out the new simplified licensing option that enables
unlimited royalty-free distribution of the report engine
for externally facing server and web deployment.
Ncwms-users mailing list

Dr. Richard P. Signell   (508) 457-2229
USGS, 384 Woods Hole Rd.
Woods Hole, MA 02543-1598

NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.