[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Problem with aggregation and stateless DAP



Hi John,

So long as you don't break simple stateless requests, I'm happy with whatever you choose in order to provide stateful behaviour also -- I can see the need for this now. The rest of this email contains my thoughts about other ways to look at the problem.

I wonder why this is the case. What is the new file being added? Could you please explain exactly what's being added to me? Why are new files being added?

As for the deletion at 0Z, I would ask whether the request is for "Latest" or for a specific date. I don't see why files for a specific date would be removed, for example.

However, I'll just assume you do need to do what you're doing for the moment. I guess a session is the only way. However, not all clients are going to be aware, so you might still need some way to handle things on the server when people *do* ask for things without a session.

What about introducing an "unlimited" vector into the aggregation? If you joined along a new dimension, then requests (for example) the first 10 records would always come back with the same thing, even though more data might now be available "at the other end". If you know your data is going to be highly dynamic, then this doesn't seem unreasonable. You might even implement a new kind of request for new additions to an old aggregation.

I've quoted a bit from your other email, which prompted me to go an re-examine this one.

>The problem is; everytime you do a data request, will you examine the entire DDX and possibly some of the data like coordinate systems to make sure nothing has changed?

I wouldn't bother usually, but if I was setting something up to track the changes I *could* do it. I'm also thinking of the scenario where a user is asking for the file as NetCDF and saving to local disk. If the information only exists in the DDX, might not that be a problem?

Here's another option -- what if each DDX contained a unique identifier -- such as the date and time to high precision? Further requests would include this as a "currency" indicator. The server would then only aggregate files which themselves have a creation date before that indicator.

Cheers,
-T

John Caron wrote:

Greetings dods-techies:

Im reworking the agg server, and trying to solve all the complex problems that i ignored in the first version.

So I have an aggregation that contains, say 700 files, each is one time point. Every 15 minutes another file is added. Once a day at 0Z, the previous days files (about 100) are deleted.

The problem is that a client might come in, gets the DDS and sees a variable with dimension 700. Then they make a request for the first 10 values. Meanwhile, 100 files have been deleted, so they get the wrong 10 back. Other problems also are possible.

Because DAP is stateless, I dont see any way for the server to realize that the request is based on an old DAP that is now wrong.

I dont see any way around this except to add a notion of a "session". A server could optionally send a session cookie, which the client must pass back on each request. Only servers that need sessions would have to implement this.

With a session, I could keep track of when the aggregation dataset has been modified, and do something intelligent.

Sessions would also help make authentication more efficient.

Anyone have any thoughts?



NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.