NOTE: The wcsplus mailing list is no longer active. The list archives are made available for historical reasons.
Hi Jon,Thanks for these details on your DEWS WCS. I'd seen some of these details before in a paper Jeremy gave us but after all this asynchronous discussion it is making a bit more sense to me.
Comments below. Jon Blower wrote:
Hi all, As Adit said in an earlier post, we designed and built a WCS with asynchronous capability in DEWS. Thought it might be useful to summarize on this list the key features of the design, which borrows from the WPS spec. The asynchronous behaviour is specified by two parameters, STORE and STATUS, which both default to false (meaning that we are backward-compatible with WCS1.0.0). STORE=true means "Give me a URL to the data instead of the data itself" STATUS=true means "Let me monitor the extraction process" There are three possible behaviours (and one that makes no sense and is disallowed): (1) STORE=false and STATUS=false. "Fully-synchronous." The server waits until the data have been extracted and then replies to the client with the data as a direct response to the request. (2) STORE=true and STATUS=false. "Semi-synchronous." The server waits until the data have been extracted and responds to the client with a URL to the data file. (3) STORE=true and STATUS=true. "Asynchronous." The server replies *immediately* with a document containing a unique job ID. The client can then poll the server using this job ID to discover information about the progress of the extraction. When the extraction is complete the polling results in the URL to the data file being returned to the client.
One case that seems important to me but is not satisfied by the above is when the client would prefer the data returned synchronously but still wants the data even if the server can't return it till later. Or should that be handled by a combination of the above? Which makes me think that some kind of negotiation is needed. For instance, what about the client that wants the data now or within an hour but doesn't want it if it will take a day.
Perhaps a fully-synchronous request that returns an exception (maybe with a new "AsynchronousResponseRequired" code) followed by an asynchronous request. It would be nice if the client could get an estimate of how long the asynchronous response might take but the exception reports don't really support structured information beyond the code and location.
Though this make me think of "use exceptions only for exceptional situations" (from "Effective Java", Josh Bloch). So maybe a different kind of negotiation mechanism would be better.
By the way, what HTTP response code do you guys use for the above store/status situations? And do you use the 400 (Bad Request) when you return an exception? As I work on implementations, I keep wishing the WCS (and other OGC) spec(s) were more clear about HTTP response codes. So, as we continue with this asynchronous response discussion, I'd like to make sure we have some detail on the HTTP response codes.
Jon, this leads me to an earlier comment of yours, "I'm not sure about the use of HTTP response codes and RESTful paradigms to manage the asynchronous download (I'm a fan of REST in general by the way). I would recommend thinking carefully about the complexities that this design would impose on the design of clients (the same goes for a "serverDecide" option in the asynchronous parameter of the WCS request)."Can you explain this a bit more? Since we are working on top of HTTP, this seems like a natural way to go. Do you see the complexity arising from: 1) the lack of detail on the content of a 202 (Accepted) response, 2) the client having to check the response code before deciding how to deal with the response, or 3) something else?
STORE=false, STATUS=true makes no sense and is disallowed (server responds with an error). The server can respond with an error if it does not wish to satisfy a fully- or semi-synchronous request (because the data extraction will take too long, for example). We chose this design instead of a "serverDecide" option because it simplifies the client (the client always knows what kind of response to expect).
For me, the three opposing forces, in order of importance, are: 1) make sure the spec is clear; 2) allow for simple, easy to implement, clients; 3) allow for simple, easy to implement, servers. I worry that striving for a simple client could make the spec less understandable by those that will be
As Adit says, the format of the status documents was inspired by the WPS ExecuteResponse document. I think we diverged from this for reasons of expediency, but I think the ER format is logical: a large data extraction is conceptually the same as a long-running processing job.
I haven't looked at the ExecuteResponse document yet but I'm all for using an existing spec.
There will be many possible designs but at least we know this one works (at least for us!) and it is close to an existing OGC spec (WPS). The DEWS WCS is a "reference implementation" for this design and we're happy to share the code (Java web app).
Thanks again for going into some details on the DEWS WCS implementation. Thanks everyone for a great discussion. Ethan
Hope this helps, Jon
-- Ethan R. Davis Telephone: (303) 497-8155 Software Engineer Fax: (303) 497-8690 UCAR Unidata Program Center E-mail: edavis@xxxxxxxx P.O. Box 3000 Boulder, CO 80307-3000 http://www.unidata.ucar.edu/ ---------------------------------------------------------------------------
wcsplus archives: