[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Problem with aggregation and stateless DAP



I talked with Benno about this at AGU (I think) and forgot to relay that conversation to this thread. What I remember was that Benno argued for use of an ETag and/or Last-Modified along with Expires headers. This has the benefit that the DAP stays stateless and it would work.

However... It seems to me that state is ultimately an optimization. I'd like to reopen the discussion and make sure that we get a cogent statement about the situation written somewhere. The answer might be in my email somewhere, but I was going to write a quick summary on the wiki and after scanning the messages for a bit, I couldn't. So...

John: Can you send a description of your idea about adding state and
Benno: Can you send me a description of your idea about using ETag, et cetera?

This would help me greatly. Thanks!

James

On Dec 5, 2005, at 5:00 PM, John Caron wrote:



Tennessee Leeuwenburg wrote:
Hi John,
So long as you don't break simple stateless requests, I'm happy with whatever you choose in order to provide stateful behaviour also -- I can see the need for this now. The rest of this email contains my thoughts about other ways to look at the problem. I wonder why this is the case. What is the new file being added? Could you please explain exactly what's being added to me? Why are new files being added?

Its a realtime data feed from the IDD. In this particular dataset, its satellite files that come in every 15 minutes, constituting a time series of satellite images.


As for the deletion at 0Z, I would ask whether the request is for "Latest" or for a specific date. I don't see why files for a specific date would be removed, for example.

We only keep 7 days worth, so each night we delete the oldest day's worth.

However, I'll just assume you do need to do what you're doing for the moment. I guess a session is the only way. However, not all clients are going to be aware, so you might still need some way to handle things on the server when people *do* ask for things without a session.

If the client doesnt assume any state, I can easily fulfill the request with the dataset's state at the moment the request comes in.

What about introducing an "unlimited" vector into the aggregation? If you joined along a new dimension, then requests (for example) the first 10 records would always come back with the same thing, even though more data might now be available "at the other end". If you know your data is going to be highly dynamic, then this doesn't seem unreasonable. You might even implement a new kind of request for new additions to an old aggregation.

yes, thats exactly the situation, i have an "unlimited" time dimension, and usually the file is just growing, so the client with the "old" DDX doesnt get any bad data.

However, in principle the data might arrive out of order, and for sure we have the problem when the files get deleted. Its really these cases, that happen less frequently, that need special attention.

I may implement an operation that says "check to see if the dataset has grown along the unlimited dimension".


I've quoted a bit from your other email, which prompted me to go an re-examine this one. >The problem is; everytime you do a data request, will you examine the entire DDX and possibly some of the data like coordinate systems to make sure nothing has changed? I wouldn't bother usually, but if I was setting something up to track the changes I *could* do it.


Yes, you could do it, but most clients wont.


I'm also thinking of the scenario where a user is asking for the file as NetCDF and saving to local disk. If the information only exists in the DDX, might not that be a problem?

Well, you culd ask for the data in a single request, but it will likely fail because currently things get buffered in memory. So if you break the data requests into seperate requests, you have the possibility of things changing while youre bringing the data over.


Here's another option -- what if each DDX contained a unique identifier -- such as the date and time to high precision? Further requests would include this as a "currency" indicator. The server would then only aggregate files which themselves have a creation date before that indicator.

hmmm, sounds like a variation of the checksum idea. ill have to think about that.

thanks for your ideas!


--
James Gallagher                jgallagher at opendap.org
OPeNDAP, Inc                   406.723.8663