Re: Problem with aggregation and stateless DAP

• To: James Gallagher <jhrg@xxxxxxx>
• Subject: Re: Problem with aggregation and stateless DAP
• From: John Caron <caron@xxxxxxxxxxxxxxxx>
• Date: Mon, 23 Jan 2006 10:52:26 -0700

James Gallagher wrote:

I talked with Benno about this at AGU (I think) and forgot to relay that conversation to this thread. What I remember was that Benno argued for use of an ETag and/or Last-Modified along with Expires headers. This has the benefit that the DAP stays stateless and it would work.

However... It seems to me that state is ultimately an optimization. I'd like to reopen the discussion and make sure that we get a cogent statement about the situation written somewhere. The answer might be in my email somewhere, but I was going to write a quick summary on the wiki and after scanning the messages for a bit, I couldn't. So...

John: Can you send a description of your idea about adding state and

Benno: Can you send me a description of your idea about using ETag, et cetera?

This would help me greatly. Thanks!

James

Heres a summary of where things stand.

The problem is to deal with datasets that are continually changing. In that 
case, simply detecting that they have changed is inadequate, since then you 
have to notify the application/user, invalidate the application state, etc. 
This is ok if it happens occasionally, but what if it happpens all the time? 
You have an unuseable application for those datasets.

Putting (optional) state on the server is a way to deal with it. It is an 
optimization.



-------- Original Message --------
Subject: Re: More thoughts about datasets that change
Date: Thu, 15 Dec 2005 18:30:34 -0700
From: John Caron <caron@xxxxxxxxxxxxxxxx>
Organization: UCAR/Unidata
To: Tech DODS <dods-tech@xxxxxxxxxxxxxxxx>
References: <4390E8B5.7040208@xxxxxxxxxxxxxxxx> 
<0E6B7ECA-21E8-4C03-9D50-9F894E9A9D91@xxxxxxxxxxx> <439371D0.3060303@xxxxxxxxxxxxxxxx> 
<F632DE7B-8875-49E4-941D-0DC0D227AAEA@xxxxxxx> <4394BF8B.7040701@xxxxxxxxxxxxxxxx>



John Caron wrote:

James Gallagher wrote:

Suppose we introduce the idea that a client MAY get a cookie from a server, indicating that a session as been created by it's request. It MAY include that cookie with future requests, indicating to a server that it requests the server honor the previous session, making the current response relative to the data source's state at the time the cookie was issued (I know, I used the word ;-). In this case the server MUST either honor the request OR return an error. There's no requirement that a server support sessions and no requirement that a client support them either. The DAP still has fundamentally stateless behavior, but also has the capability to tell certain servers, 'Hey, I was here before and this is what things looked like then.' Most servers won't support this, and neither will most clients (my guess) but some will and it looks like an important capability.

Yes, this would solve my problem in the Agg Server, thanks for summarizing it concisely. You might add that if the session is established, the server should make a best effort not to let the dataset change during the duration of the session.

I will go ahead and try to implement this in the Agg Server, and report on any further problems encountered before we make a final decision on it. Im pretty sure it will be easy enough to return cookies in the client libraries, so I will likely also modify the java netcdf/opendap client library. That way I will have a complete test of the idea.

Heres a summary of where I am on this project:


If you remember, the problem is to keep the opendap dataset metadata from 
changing in a way that gives erroneous results silently, for the case that a 
client has a stateful view of the dataset, e.g. the netcdf-opendap clients.

In my server, I had already implemented caching of datasets, so that repeated 
requests to the same dataset would be efficient. Normally I would lock the 
dataset for the duration of the request, then immediately release it. What I do 
now is to reserve the dataset object for that particular client by not 
releasing it until the session expires. I then know what metadata the client 
assumes, and can satisfy that if possible, and give an error message if not.

It was quite easy to modify Java DODS DConnect class to add support for session 
cookies. Just a few lines of code, and it only happens if the client enables it.

In the normal session negotiation, the server offers a cookie when it gets the 
first client request, and if the client returns the cookie, the server 
establishes the session. In the Tomcat framework, every request generates a 
session object, which times out if the cookie is not returned. I was worried 
somewhat about the overhead of this. More importantly, it meant that I couldnt 
immediately lock the dataset, but had to wait to see if the client returned the 
cookie. For those 2 reasons, I decided to have the client send a header to 
indicate that  it would, indeed accept cookies. So when the client sends a 
request, it adds the header:

X-Accept-Session: true

This allows me to immediately lock the dataset, and to not bother creating a 
session if the header is absent. I put the dataset object into the session 
object, and retrieve it every request. The session automatically times out 
after 30 minutes.

It seemed silly to have the dataset stay locked an extra 30 minutes when most of the time 
the client library knows exactly when its done, namely when the client calls close() on 
the dataset. So I wanted to send a message when close() was called. I decided to just add 
a new suffix ".close" to the dataset name on a GET call. When the server sees 
this, it unlocks the dataset and terminates the session.

So in summary:

2. Look for any cookies on the response. Return them on subsequest requests.
3. A new method close() was added. If its called, send a message to server with dataset name 
and ".close" suffix".

TDS Server
1. Look for X-Accept-Session: true header on each request.
2. If exists, establish a session for the client, return a session id cookie.
3. Cache the dataset object for the duration of the session.
4. Detect any changes to the dataset and give an error back to the client if 
needed.
5. On a "close" message, close the session and release the dataset.
6. Timeout any session that hasnt been used for 30 minutes.

I am willing to change any of this if needed, but this is what I found to give 
me the best results in my server and client.

• References:
  • Re: Problem with aggregation and stateless DAP
    • From: James Gallagher