Re: [thredds] nco as a web service

The maximum URL length is an application server (e.g., Tomcat) setting. See, e.g.,
http://serverfault.com/questions/56691/whats-the-maximum-url-length-in-tomcat
If that setting is increased, there may be other things that limit the max allowed length, but this probably gets us far enough (e.g., a 64KB limit). Since the administrator of the service that Roy is proposing will also be the administrator of the application server, s/he can increase the maximum URL length allowed. So this should be a manageable problem.

On 7/1/2012 11:25 AM, Dennis Heimbigner wrote:
Roy-

 > ...  One comment.  I think you misunderstood my point about
 > Matlab and R.  I am not interested in Matlab specific
 > implementations.  The point was because the URL completely
 > defines the request, I can implement scripts in any application
 > that can send an URL and receive a file in terms of functions
 > built-in to that application - that is my clients do not break as
 > the application or operating system change.

Not quite sure I understand. This phrase "...receive a file in
terms of functions built-in to that application" sounds
like you are creating an association between functions defined
on the client side and functions defined on the server side.
Can you elaborate?

 > Why I strongly prefer, if it is at all reasonable, services that
 > only use GET, not POST.

Again, that is only possible if you keep your requests
short enough to not violate the URL length restrictions.

=Dennis Heimbigner
  Unidata



Roy Mendelssohn wrote:
Hi Dennis:

Thanks.  One comment.  I think you misunderstood my point about Matlab
and R.  I am not interested in Matlab specific implementations.  The
point was because the URL completely defines the request, I can
implement scripts in any application that can send an URL and receive
a file in terms of functions built-in to that application  - that is
my clients do not break as the application or operating system change.

While I understand why this occurred, a few years ago we had straight
OPeNDAP implementations.  We had a lot of users using scripts we
developed for Matlab, running under Windows.  Due to updates in both
Windows and Matlab, the OPeNDAP files for Windows stopped working (at
least for Matlab).  We had a lot of users that were left stranded and
stranded for quite a long time.  Developing and maintaining clients,
particularly clients that are working within an application for which
you have to write code, very quickly becomes a non-trivial exercise.

Since we switched to a service where the URL completely defines the
request, our Matlab and R scripts have survived quite nicely any
number of updates both to the applications themselves and to the
operating systems.  That is because the clients now only use functions
built into the applications.

Why I strongly prefer, if it is at all reasonable, services that only
use GET, not POST.

-Roy




On Jun 28, 2012, at 1:03 PM, Dennis Heimbigner wrote:

I am old and slow, but suppose I am in OpeNDAP, are you proposing
to separate say constraint expressions and server-side function
requests basically the same (ie I just scan what is after each
comma) or do you propose some method that signifies in the URL
that what follows is an expression?  In F-TDS and GDS the form of
the URL is:
First, I am proposing to subsume DAP constraints.
Second, I am proposing, like DAP, to put the expressions
in the query part of the URL (i.e. after the '?').

http://machine:port/thredds/dodsC/dataset_expr_{dataset2,dataset3,...}{expression1;expression2;...}.URLsuffix?constraint

So, I would rewrite this as something more-or-less like this:
http://machine.../dataset?expression1,expression2,...
Where the expressions would include the references to dataset2,
dataset3,
and the constraint.

BTW, the reason I have asked about the experience of people who
are using F-TDS and GDS on whether synchronous requests can cover
the large majority of cases, is because I am very partial to
systems where the URL completely defines the request, and hence
essentially use GET as the verb.
The synchronous/asynchronous issue is, for me, a separable issue.
I should note that GET has a limit on the size of URLS, so
there needs to be ways to deal with that. Two possibilities are
1) use POST or PUT, or 2) provide a way to upload a long expression
in parts USING multiple GETs.

The reason for this is long
experience.  where client code has broken with changes in
operating system and/or application, fixes were slow in coming,
so many users were left with nothing working.  In a system where
the URL completely defined the request, say ERDDAP, in Matlab:

link='http://coastwatch.pfeg.noaa.gov/erddap/griddap/erdBAsstamday.mat?sst[(2010-01-16T12:00:00Z):1:(2010-01-16T12:00:00Z)][(0.0):1:(0.0)][(30):1:(50.0)][(220):1:(240.0)]';

F=urlwrite(link,'cwatch.mat');

Will get the related file, and the entire command is in Matlab,
no extra code required.  The same in R is:

download.file(url="http://coastwatch.pfeg.noaa.gov/erddap/griddap/erdBAsstamday.nc?sst[(2010-01-16T12:00:00Z):1:(2010-01-16T12:00:00Z)][(0.0):1:(0.0)][(30):1:(50.0)][(220):1:(240.0)]",
destfile="AGssta.nc",mode='wb')

again, "download.file" is an R command.
I think that we do not want to be R/MATLAB specific
in a proposal to put stuff in URLs. I would rather
propose to allow uploading of R/MATLAB scripts to serve
as additional, user-defined functions.

I would prefer to
maintain this simplicity and cover 80% of the cases if possible,
than cover the rest but where more complex, application specific
code would have to be developed and maintained.
Agreed. However my assumption is the the output of any function that
is not assigned to a single-assignment variable will be returned as part
of the response; but other ways of specifying this are possible within
the functional framework I am proposing.

=Dennis Heimbigner
Unidata

**********************
"The contents of this message do not reflect any position of the U.S.
Government or NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
1352 Lighthouse Avenue
Pacific Grove, CA 93950-2097

e-mail: Roy.Mendelssohn@xxxxxxxx (Note new e-mail address)
voice: (831)-648-9029
fax: (831)-648-8440
www: http://www.pfeg.noaa.gov/

"Old age and treachery will overcome youth and skill."
"From those who have been given much, much will be expected" "the arc
of the moral universe is long, but it bends toward justice" -MLK Jr.

_______________________________________________
thredds mailing list
thredds@xxxxxxxxxxxxxxxx
For list information or to unsubscribe,  visit:
http://www.unidata.ucar.edu/mailing_lists/

--
Sincerely,

Bob Simons
IT Specialist
Environmental Research Division
NOAA Southwest Fisheries Science Center
1352 Lighthouse Ave
Pacific Grove, CA 93950-2079
Phone: (831)658-3205
Fax:   (831)648-8440
Email: bob.simons@xxxxxxxx

The contents of this message are mine personally and
do not necessarily reflect any position of the
Government or the National Oceanic and Atmospheric
Administration.
<>< <>< <>< <>< <>< <>< <>< <>< <><




  • 2012 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the thredds archives: