Adding a service to a remotely served dataset
Roy Mendelssohn
Roy.Mendelssohn at noaa.gov
Mon Jul 9 13:59:50 MDT 2007
I just wanted to add that this is something we have been looking at
in a more general framework - providing "proxy" services to other
existing services (or gateways if you will). That way any one site
will only have to provide the data using a single service, while
"assembly" or "aggregation" centers aggregate these services and
provide proxy services for a larger class of web services (I hope
this is clear).
-Roy
On Jul 9, 2007, at 12:53 PM, Ethan Davis wrote:
> John,
>
> Is this capability available with the netCDF subset service as well?
>
> Ethan
>
> John Caron wrote:
>> We made an extension of the WCS Dataset URL format to allow the
>> TDS to serve remote datasets. Identify the dataset by adding the
>> parameter dataset whose value is a URL:
>>
>> http://servername:8080/thredds/wcs?dataset=datasetURL&
>>
>> The URL must be a dataset readable by the NetCDF-Java library,
>> typically an OPeNDAP dataset on another server. It must have
>> gridded data, with identifiable coordinate systems, etc. For
>> example, an OPeNDAP URL might be
>>
>> http://las.pfeg.noaa.gov/cgi-bin/nph-dods/data/oceanwatch/nrt/gac/
>> AG14day.nc
>> This can be served remotely as a WCS dataset with this URL:
>>
>> http://servername:8080/thredds/wcs?dataset=http://
>> las.pfeg.noaa.gov/cgi-bin/nph-dods/data/oceanwatch/nrt/gac/
>> AG14day.nc&
>>
>>
>> This is non-standard WCS; we havent emphasized this feature until
>> we have a chance to review security implications. The WCS service
>> now if off by default, and must explicitly be turned on in
>> threddsConfig.xml, mostly because of this feature. We should
>> probably seperately allow users to enable/disable "remote WCS
>> access".
>>
>>
>> Oliver Newell wrote:
>>> Hi Roy -
>>>
>>> FYI - We're currently experimenting with implementing WCS
>>> services, and the same requirement (wish?) as what you're
>>> describing has been raised. Basically, the need to discover a
>>> data set via a cataloging mechanism (THREDDS, an OGC catalog, a
>>> DOD NCES catalog) and then be able to access it via a WCS that
>>> resides somewhere on the network. My initial take on it is that
>>> since WCS data access requests work with unique identifiers (not
>>> URLs), this limits the capabilities of the service in the sense
>>> that the Id-to-resource (data file or db) mapping is managed
>>> locally by the WCS. In other words, the WCS has to 'know about'
>>> the resource in advance of the request, in order to do the mapping.
>>>
>>> I think the WCS service should support a mode of operation where
>>> a resource's URL is included in the requests to the service, in
>>> order to support the external WCS concept. I think it is still
>>> important to maintain the Id as a separate entity, so it is not
>>> just a question of using a URL for the coverage identifier,
>>> though that idea might be tempting.
>>>
>>> The idea needs flushing out a bit, but it is still early enough
>>> in WCS's development that changes like this can be incorporated
>>> into the spec via the OGC working group (my opinion). It
>>> definitely seems like it would be a useful capability to address
>>> the situation you describe.
>>>
>>> -Oliver
>>>
>>> Roy Mendelssohn wrote:
>>>> Hi Glenn:
>>>>
>>>> No problem. But we were interested in the general problem, as
>>>> there are other datasets which also do not have WCS and which we
>>>> would like to add it, rather than trying to get each of them to
>>>> add it (Some just have an OPeNDAP server, not a THREDDS
>>>> server). Yours allowed me to give Ethan a concrete example so
>>>> he could look at the catalogs.
>>>>
>>>> Thanks,
>>>>
>>>> -Roy
>>>> On Jul 9, 2007, at 7:52 AM, Glenn.Rutledge wrote:
>>>>
>>>>> Hello Roy-
>>>>> That particular dataset will be turned on shortly via WCS.
>>>>> That was an oversight. Nice to have users telling us where we
>>>>> need stuff! Glenn
>>>>>
>>>>> Ethan Davis wrote the following on 7/6/2007 6:02 PM:
>>>>>> Hi Roy,
>>>>>>
>>>>>> Looks like you are referencing NCDC catalogs rather than
>>>>>> serving the datasets yourself. So, users would get redirected
>>>>>> to the NCDC server and access the data through the NCDC
>>>>>> server. So, as things stand, NCDC would have to add the WCS
>>>>>> service.
>>>>>>
>>>>>> You could point to each NCDC dataset with an NcML wrapper and
>>>>>> actually serve the data from your server rather than
>>>>>> redirecting to NCDC. If you did that, you could add the WCS
>>>>>> service. However, the performance wouldn't be as good as
>>>>>> direct from NCDC.
>>>>>>
>>>>>> I'm afraid I don't know about the WCS request for a time
>>>>>> range. So, I'll leave that for others to answer.
>>>>>>
>>>>>> Ethan
>>>>>>
>>>>>> Roy Mendelssohn wrote:
>>>>>>> Hi Ethan:
>>>>>>>
>>>>>>> I am not certain I follow. To be more specific, look at:
>>>>>>>
>>>>>>> http://oceanwatch.pfeg.noaa.gov/thredds/catalog.html
>>>>>>>
>>>>>>> and scroll down to the datasets " Remote test to Nomads
>>>>>>> THREDDS catalog", which is actually on a THREDDS server on an
>>>>>>> NCDC computer. They are remotely served through catalogs.
>>>>>>> Can we add the WCS service on our end, or will it only work
>>>>>>> if NCDC adds the WCS service.
>>>>>>>
>>>>>>> While I have your attention, I have not been able to get a
>>>>>>> straight answer if there is a way to send the WCS request to
>>>>>>> get a time range, rather than a single time, and if so what
>>>>>>> is the syntax for that.
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> -Roy
>>>>>>>
>>>>>>> On Jul 6, 2007, at 11:24 AM, Ethan Davis wrote:
>>>>>>>
>>>>>>>> Hi Roy,
>>>>>>>>
>>>>>>>> If you are actually serving the dataset through your server
>>>>>>>> (which it sounds like you are since you mention existing
>>>>>>>> NcML for the dataset) rather than providing a link in your
>>>>>>>> catalog to the remote server, I believe you can simply add
>>>>>>>> the WCS service to your TDS configuration catalog and serve
>>>>>>>> the data via WCS. It shouldn't require any changes to the
>>>>>>>> NcML, just to the service element in the catalog.
>>>>>>>>
>>>>>>>> Of course, this depends on the dataset meeting the WCS
>>>>>>>> server requirements, e.g., evenly spaced grid.
>>>>>>>>
>>>>>>>> I'm not sure how it would perform. But I'll leave it for
>>>>>>>> John to respond on that.
>>>>>>>>
>>>>>>>> Ethan
>>>>>>>>
>>>>>>>> Roy Mendelssohn wrote:
>>>>>>>>> Hi:
>>>>>>>>>
>>>>>>>>> The question for today is if our TDS is "serving" a dataset
>>>>>>>>> remotely (ie. the remote site has their own TDS and we
>>>>>>>>> just link indirectly to it), can we add a service to our
>>>>>>>>> version that they don't have (just like we can aggregate a
>>>>>>>>> remote dataset that is not aggregated). In particular the
>>>>>>>>> remote dataset only has an OPeNDAP response, but we would
>>>>>>>>> like to be able to respond to both an OPeNDAP and a WCS
>>>>>>>>> response.
>>>>>>>>>
>>>>>>>>> If we change our NcML appropriately, will that work?
>>>>>>>>>
>>>>>>>>> Thanks in advance,
>>>>>>>>>
>>>>>>>>> -Roy
>>>>>>>>>
>>>>>>>>> **********************
>>>>>>>>> "The contents of this message do not reflect any position
>>>>>>>>> of the U.S. Government or NOAA."
>>>>>>>>> **********************
>>>>>>>>> Roy Mendelssohn
>>>>>>>>> Supervisory Operations Research Analyst
>>>>>>>>> NOAA/NMFS
>>>>>>>>> Environmental Research Division Southwest Fisheries
>>>>>>>>> Science Center
>>>>>>>>> 1352 Lighthouse Avenue
>>>>>>>>> Pacific Grove, CA 93950-2097
>>>>>>>>>
>>>>>>>>> e-mail: Roy.Mendelssohn at noaa.gov (Note new e-mail address)
>>>>>>>>> voice: (831)-648-9029
>>>>>>>>> fax: (831)-648-8440
>>>>>>>>> www: http://www.pfeg.noaa.gov/
>>>>>>>>>
>>>>>>>>> "Old age and treachery will overcome youth and skill."
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ==============================================================
>>>>>>>>> ================
>>>>>>>>> To unsubscribe thredds, visit:
>>>>>>>>> http://www.unidata.ucar.edu/mailing-list-delete-form.html
>>>>>>>>> ==============================================================
>>>>>>>>> ================
>>>>>>>>
>>>>>>>> --Ethan R. Davis Telephone:
>>>>>>>> (303) 497-8155
>>>>>>>> Software Engineer Fax:
>>>>>>>> (303) 497-8690
>>>>>>>> UCAR Unidata Program Center E-mail:
>>>>>>>> edavis at ucar.edu
>>>>>>>> P.O. Box 3000
>>>>>>>> Boulder, CO 80307-3000 http://
>>>>>>>> www.unidata.ucar.edu/
>>>>>>>> ---------------------------------------------------------------
>>>>>>>> ------------
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> **********************
>>>>>>> "The contents of this message do not reflect any position of
>>>>>>> the U.S. Government or NOAA."
>>>>>>> **********************
>>>>>>> Roy Mendelssohn
>>>>>>> Supervisory Operations Research Analyst
>>>>>>> NOAA/NMFS
>>>>>>> Environmental Research Division Southwest Fisheries Science
>>>>>>> Center
>>>>>>> 1352 Lighthouse Avenue
>>>>>>> Pacific Grove, CA 93950-2097
>>>>>>>
>>>>>>> e-mail: Roy.Mendelssohn at noaa.gov (Note new e-mail address)
>>>>>>> voice: (831)-648-9029
>>>>>>> fax: (831)-648-8440
>>>>>>> www: http://www.pfeg.noaa.gov/
>>>>>>>
>>>>>>> "Old age and treachery will overcome youth and skill."
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>> --Glenn K. Rutledge
>>>>> Services Team Leader
>>>>> Remote Sensing and Applications Division
>>>>> NOMADS Project Manager
>>>>> National Oceanic and Atmospheric Administration
>>>>> National Climatic Data Center
>>>>> Asheville NC 28801
>>>>> Phone: (828) 271-4097
>>>>> Fax: (828) 271-4328
>>>>>
>>>>> NOMADS: http://nomads.ncdc.noaa.gov/
>>>>>
>>>>
>>>> **********************
>>>> "The contents of this message do not reflect any position of the
>>>> U.S. Government or NOAA."
>>>> **********************
>>>> Roy Mendelssohn
>>>> Supervisory Operations Research Analyst
>>>> NOAA/NMFS
>>>> Environmental Research Division Southwest Fisheries Science
>>>> Center
>>>> 1352 Lighthouse Avenue
>>>> Pacific Grove, CA 93950-2097
>>>>
>>>> e-mail: Roy.Mendelssohn at noaa.gov (Note new e-mail address)
>>>> voice: (831)-648-9029
>>>> fax: (831)-648-8440
>>>> www: http://www.pfeg.noaa.gov/
>>>>
>>>> "Old age and treachery will overcome youth and skill."
>>>>
>>>>
>>>>
>>>> ===================================================================
>>>> ===========
>>>> To unsubscribe thredds, visit:
>>>> http://www.unidata.ucar.edu/mailing-list-delete-form.html
>>>> ===================================================================
>>>> ===========
>>>
>>> ====================================================================
>>> ==========
>>> To unsubscribe thredds, visit:
>>> http://www.unidata.ucar.edu/mailing-list-delete-form.html
>>> ====================================================================
>>> ==========
>>
>> =====================================================================
>> =========
>> To unsubscribe thredds, visit:
>> http://www.unidata.ucar.edu/mailing-list-delete-form.html
>> =====================================================================
>> =========
>
> --
> Ethan R. Davis Telephone: (303)
> 497-8155
> Software Engineer Fax: (303)
> 497-8690
> UCAR Unidata Program Center E-mail:
> edavis at ucar.edu
> P.O. Box 3000
> Boulder, CO 80307-3000 http://
> www.unidata.ucar.edu/
> ----------------------------------------------------------------------
> -----
>
>
> ======================================================================
> ========
> To unsubscribe thredds, visit:
> http://www.unidata.ucar.edu/mailing-list-delete-form.html
> ======================================================================
> ========
>
**********************
"The contents of this message do not reflect any position of the U.S.
Government or NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
1352 Lighthouse Avenue
Pacific Grove, CA 93950-2097
e-mail: Roy.Mendelssohn at noaa.gov (Note new e-mail address)
voice: (831)-648-9029
fax: (831)-648-8440
www: http://www.pfeg.noaa.gov/
"Old age and treachery will overcome youth and skill."
==============================================================================
To unsubscribe thredds, visit:
http://www.unidata.ucar.edu/mailing-list-delete-form.html
==============================================================================
More information about the Thredds
mailing list