Re: [thredds] nco as a web service

Hi Roy,

On WPS I'm not really in disagreement -- WPS suffers from the usual OGC 
premature standardisation.  However in our experience the lightweight 
"processing as HTTP GET" approach quickly runs into scalability problems:

 1. Many functions on our data aren't going to complete within the timeout of 
an HTTP GET so some sort of asynchronous pattern is needed
 2. The results may be big.  It makes much more sense to create the output 
resources (in the Restful sense) then let the user download them, or even 
interrogate them with OPeNDAP (now possible with COWS-WPS).
 3. An HTTP URL can only hold so much information.  Google Charts have moved 
from a HTTP GET based system to an AJAX API, I'm guessing partly for this 
reason.

I should also warn that I've seen several nonconstructive debates about WPS vs 
WCPS vs other workflow-ish systems on the OGC lists.  Maybe they've sorted out 
a strategy now, I don't follow it close enough to know, but the impression is 
no-ones quite sure which is the best solution.  On this I'm with Roy's 80% 
approach.

Cheers,
Stephen.

On 19 Jun 2012, at 16:27, Roy Mendelssohn wrote:

> Hi All:
> 
> Ah, as usual I get to be the wet blanket.  I am not as excited about WPS, or 
> for that matter most OGC standards, as others, because I find since OGC tries 
> to answer the general question, the resultant services have to deal with the 
> most complex case, therefore the 80% of uses cases that aren't that complex 
> are burdened with overly complex services, that are difficult for the average 
> user, and require constant update of software, and software that is hard to 
> embed into existing applications.
> 
> Most of what a lot of us do are essentially read-only services.  For these, I 
> much prefer the basic approach  presently used by F-TDS and GDS, though there 
> can be discussion of how best to connote an expression, and what the syntax 
> should be.  Even more, as has been noted, WPS does not solve the key question 
> of the syntax of the expressions.
> 
> So I know this will start a lot of flames, but I just look at the service 
> that we developed and use, ERDDAP.  All sorts of people, who only know a 
> little programming, successful use ERRDAP in R, Matlab, shell scripts, 
> python, java etc., and it is easy to use from within applications, because 
> any environment that can send a URL and receive a file can use it.  Moreover, 
> since only methods internal to the applications are used, the scripts or 
> programs or whatever do not break as both the application and operating 
> environment change.
> 
> Our experience over many years of serving data is that in most cases, simple 
> and fast are to be preferred.  Clearly people can come up with counter cases 
> that are not covered, and I am all for developing services for those cases, 
> but not for burdening simpler cases with all that baggage.
> 
> My $0.02.
> 
> -Roy
> 
> 
> On Jun 19, 2012, at 12:14 AM, <stephen.pascoe@xxxxxxxxxx> wrote:
> 
>> We are working on wrapping CDO operators in our WPS implementation COWS-WPS. 
>> This work is being done as part of the IS-ENES an ExArch projects with with 
>> DKRZ, KNMI and others. 
>> 
>> WPS doesn't define your functions for you but gives you a protocol and 
>> framework for asynchronous server-side tasks. 
>> 
>> http://cows.ceda.ac.uk/cows_wps/intro.html
>> 
>> --
>> Stephen Pascoe from iPhone
>> 
>> On 18 Jun 2012, at 23:07, "John Cartwright" <john.c.cartwright@xxxxxxxx> 
>> wrote:
>> 
>>> Could the the OGC's Web Processing Service spec
>>> (http://en.wikipedia.org/wiki/Web_Processing_Service) serve that
>>> function?
>>> 
>>> --john
>>> 
>>> 
>>> On Mon, Jun 18, 2012 at 3:56 PM, Roy Mendelssohn
>>> <roy.mendelssohn@xxxxxxxx> wrote:
>>>> But for a useful service, the form and syntax of the URL should be 
>>>> independent of the mechanism that does the server side calculations  
>>>> (which rules out SWAMP).  So for example, both Grads an F-TDS use the same 
>>>> format in the URL to say that "this is an expression", but the expressions 
>>>> themselves are platform specific.  That is not the way to get overarching 
>>>> services.
>>>> 
>>>> Instead, we need agreement on how in the URL request we signal a 
>>>> server-side function, the syntax of that function (independent of the 
>>>> engine underneath) and a few simple functions to start  (say simple data 
>>>> transformations, differencing and averaging on a dimension(s)).  Then the 
>>>> server back-end can parse the request and use Ferret or Grads or NCO or 
>>>> Python or whatever is desired, and like with any good service, the back 
>>>> end could change  without having any affect on the URL or the user.
>>>> 
>>>> I know Matthew Arrott at least used to like the approach in Chapter 12 of 
>>>> "Python Scripting for Computation Science".  But a lot of that is the 
>>>> engine underneath.  I am more interested in the form of the URL.  Get some 
>>>> agreement on that, and some real implementation could proceed.
>>>> 
>>>> -Roy
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> On Jun 18, 2012, at 2:37 PM, Russ Rew wrote:
>>>> 
>>>>> Jeff,
>>>>> 
>>>>>> However, we have to keep in mind performance ramifications. It still 
>>>>>> takes
>>>>>> a long time to move gigabytes of data across a network. This brings up 
>>>>>> the
>>>>>> importance of moving the computation to the data, instead of moving the
>>>>>> data to the computation. For some data sets and many use cases remote
>>>>>> access to data works very well so things like brokering are tractable.
>>>>>> However, for *big* data sets (e.g., climate model output) we need to come
>>>>>> up with richer mechanisms (like the NCO on local data) to bring 
>>>>>> computation
>>>>>> to the data.
>>>>> 
>>>>> See Daniel Wang's SWAMP (the Script Workflow Analysis for
>>>>> MultiProcessing), built on top of NCO:
>>>>> 
>>>>> https://code.google.com/p/swamp/
>>>>> 
>>>>> --Russ
>>>>> 
>>>>> _______________________________________________
>>>>> thredds mailing list
>>>>> thredds@xxxxxxxxxxxxxxxx
>>>>> For list information or to unsubscribe,  visit: 
>>>>> http://www.unidata.ucar.edu/mailing_lists/
>>>> 
>>>> **********************
>>>> "The contents of this message do not reflect any position of the U.S. 
>>>> Government or NOAA."
>>>> **********************
>>>> Roy Mendelssohn
>>>> Supervisory Operations Research Analyst
>>>> NOAA/NMFS
>>>> Environmental Research Division
>>>> Southwest Fisheries Science Center
>>>> 1352 Lighthouse Avenue
>>>> Pacific Grove, CA 93950-2097
>>>> 
>>>> e-mail: Roy.Mendelssohn@xxxxxxxx (Note new e-mail address)
>>>> voice: (831)-648-9029
>>>> fax: (831)-648-8440
>>>> www: http://www.pfeg.noaa.gov/
>>>> 
>>>> "Old age and treachery will overcome youth and skill."
>>>> "From those who have been given much, much will be expected"
>>>> "the arc of the moral universe is long, but it bends toward justice" -MLK 
>>>> Jr.
>>>> 
>>>> _______________________________________________
>>>> thredds mailing list
>>>> thredds@xxxxxxxxxxxxxxxx
>>>> For list information or to unsubscribe,  visit: 
>>>> http://www.unidata.ucar.edu/mailing_lists/
>>> 
>>> _______________________________________________
>>> thredds mailing list
>>> thredds@xxxxxxxxxxxxxxxx
>>> For list information or to unsubscribe,  visit: 
>>> http://www.unidata.ucar.edu/mailing_lists/ 
>> -- 
>> Scanned by iCritical.
>> 
>> _______________________________________________
>> thredds mailing list
>> thredds@xxxxxxxxxxxxxxxx
>> For list information or to unsubscribe,  visit: 
>> http://www.unidata.ucar.edu/mailing_lists/ 
> 
> **********************
> "The contents of this message do not reflect any position of the U.S. 
> Government or NOAA."
> **********************
> Roy Mendelssohn
> Supervisory Operations Research Analyst
> NOAA/NMFS
> Environmental Research Division
> Southwest Fisheries Science Center
> 1352 Lighthouse Avenue
> Pacific Grove, CA 93950-2097
> 
> e-mail: Roy.Mendelssohn@xxxxxxxx (Note new e-mail address)
> voice: (831)-648-9029
> fax: (831)-648-8440
> www: http://www.pfeg.noaa.gov/
> 
> "Old age and treachery will overcome youth and skill."
> "From those who have been given much, much will be expected" 
> "the arc of the moral universe is long, but it bends toward justice" -MLK Jr.
> 

--
Scanned by iCritical.



  • 2012 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the thredds archives: