NCDC Web Services
David Maidment
maidment at mail.utexas.edu
Thu Sep 7 08:03:48 MDT 2006
Glenn:
OK, good to get this cleared up. I am glad this is not some inherent issue with OpenDAP,
which I had earlier suspected it might be.
David
________________________________
From: Glenn.Rutledge [mailto:Glenn.Rutledge at noaa.gov]
Sent: Thu 9/7/2006 9:01 AM
To: David Maidment
Cc: John Caron; support-thredds; Ilya Zaslavsky
Subject: Re: NCDC Web Services
Dave-
Yes- you are indeed correct. SOme organizations (NSF I guess), require a standard pot 80 and block all others unless special arrangement have been made. Because your's is not the first complaint (maybe the 3rd), NOMADS is going to set up a Tomcat/Apache interface where we will present standard port to users. We had delayed that because we are doing a complete overhaul of our front end (to a load balanced arrangement). Anyway- the hardware got delayed and so to did the action to standardize that access for you. So- you should have access within several weeks..
Sorry for the hassle.... Glenn
David Maidment wrote the following on 9/6/2006 8:56 AM:
Glenn:
I think that what the information below means from John Caron is that OpenDAP
services need to be accessible through Port 80 even if that is not the default
with the Tomcat server or otherwise access to them will be blocked by firewalls
in some instances. Can you check to see whether in publishing NARR data
access through OpenDAP you have used Port 80 or not. This might be why
we and others are having trouble with firewalls when accessing NARR from within
some organizations, but not with others.
David
________________________________
From: John Caron [mailto:caron at unidata.ucar.edu]
Sent: Sun 9/3/2006 7:25 PM
To: David Maidment
Cc: Glenn.Rutledge; support-thredds; Ilya Zaslavsky
Subject: Re: NCDC Web Services
David Maidment wrote:
John:
I feel like Dr Watson when Sherlock Holmes says "Elementary my dear Watson".
I am sure what you are telling me is elementary but it requires more knowledge of
server technology than I possess.
First of all -- what is a port? How is a port defined or established? By the server? If
so how? Can a server have multiple ports? Why would you want to do that?
A port is just an abstraction to establish various communication channels to the same host machine. I mention it because thats generally how firewalls work.
Is Tomcat a server technology built with open source components?
Yes, its entirely open source.
Is it the java equivalent of a Windows based server?
Its a jaav-based HTTP server.
What is the significance of the default port on Tomcat being 8080 and on Windows (?) servers 80?
Tomcat can run under port 80 if you want. Again, its only relevent because standard firewall policy is to allow HTTP traffic on port 80. Often other ports are not allowed. This doesnt particularly solve security problems, but its what people do anyway.
Are SOAP/XML services similarly restricted by port numbers? It seems like our
SOAP/XML services are not similarly restricted by firewalls as what we've done with
OpenDAP, but that may be because we are mediating them through SDSC that may
be using Port 80.
Most servers can use any port you tell them. But they have to use a specific port.
Ah -- the glories of computer technology!
SOrry i cant give you a simpler answer.
David
________________________________
From: John Caron [mailto:caron at unidata.ucar.edu]
Sent: Wed 8/23/2006 5:12 PM
To: David Maidment
Cc: Glenn.Rutledge; 'support-thredds'
Subject: Re: NCDC Web Services
Im not sure what the issue is, but its not really a server thing. Firewalls live on the client. Generally HTTP traffic on port 80 is let through, and not much else. Tomcat default port is 8080, it can be changed to port 80 is need be, but you cant expect THREDDS servers to do that for the convenience of some site that has some restrictive firewall.
A proxy server can be a good solution, but i dont know what the details of use are.
David Maidment wrote:
Glenn:
Yes, please find out what we have to do. This looks like a significant
issue to me.
John -- could you look at the email trail below and comment on the
issues of OpenDAP and
firewalls. Is there any difference if we use your netCDF server?
David
------------------------------------------------------------------------
*From:* Glenn.Rutledge [mailto:Glenn.Rutledge at noaa.gov]
*Sent:* Wednesday, August 23, 2006 10:09 AM
*To:* David Maidment
*Subject:* Re: NCDC Web Services
Systems folks would do this...and as I understand it- is not at all
uncommon. It's a representation of a service within a firewall I am
out of my realm but I do plan to investigate further since NASA had the
exact comment you did just last week.
I'll let you know more if you like. GR
David Maidment wrote the following on 8/23/2006 10:54 AM:
Glenn:
What is a proxy server? How would I or any other regular user know
how to use one of these?
David
------------------------------------------------------------------------
*From:* Glenn.Rutledge [mailto:Glenn.Rutledge at noaa.gov]
*Sent:* Tuesday, August 22, 2006 12:53 PM
*To:* David Maidment
*Subject:* Re: NCDC Web Services
Hi David-
A response from OPeNDAP:
our code (libdap, libnc-dap, our clients - with the possible exception
of the ODC) can all be told to use a proxy server. Open ports to that
machine and you're off and running. If someone can use the Internet,
they can use our clients and read from your (or any) server. Point
these folks toward the documentation on the .dodsrc file. If they
still have questions (and I know some of our docs are getting stale),
then have them contact support at undata.ucar.edu with their questions
(or the newly renamed opendap or opendap-tech emails lists - see
www.opendap.org <http://www.opendap.org/> and look under support for information about those).
I'm glad you asked me, because this is an important feature of the
client-side software!
Hope this helps--- Glenn
David Maidment wrote the following on 8/21/2006 9:13 PM:
Glenn:
I have some concerns about how useful OpenDAP is. It seems sensitive
to being blocked by firewalls.
Our direct calls to NARR that relied on using OpenDAP worked fine
>from here at UT but from inside of
NSF with all their firewalls in place, the data were blocked, and the
same is true when trying to do this
>from within ESRI.
I think the question of how to do these things is linked up with
computer security and that needs to be
factored into the debate.
David
------------------------------------------------------------------------
*From:* Glenn.Rutledge [mailto:Glenn.Rutledge at noaa.gov]
*Sent:* Monday, August 21, 2006 9:13 AM
*To:* David Maidment
*Subject:* Re: NCDC Web Services
Thanks David-
I agree- but somewhat concerned whether SOAP/REST will have
capabilities across the other data forms you cite - although based on
your individual needs- it doesn't seem to matter. But for the larger
whole- we need to test this out. Glenn
David Maidment wrote the following on 8/21/2006 9:47 AM:
Glenn:
I see observations data as time series as one kind of service, and weather
model, remote sensing and Nexrad data as another. For the weather and
climate "field" information, some variant of what John Caron is doing seems
appropriate. For the observations data, I think we have a fairly good scheme
figured out in the work we are doing with CUAHSI. I can see the netCDF
style of things for the weather and climate fields. I think the time series stuff
could be handled that way but is easier and perhaps more appropriate just to
do it in XML via SOAP or REST.
What I would like to see developed is a "Weather and Climate Server" so that
the data users can go to one system and get into the various information sources
without having to go separately to NCDC, Unidata, NCAR, NWS, etc. This is
like WaterOneFlow that we are attempting to do in CUAHSI for water observation
data.
David
________________________________
From: Glenn.Rutledge at noaa.gov [mailto:Glenn.Rutledge at noaa.gov]
Sent: Fri 8/18/2006 6:39 PM
To: David Maidment
Subject: Re: RE: NCDC Web Services
Hi David,
I agree. Here's the note I sent Sharon earlier this week after your
request. I want to plan for the long term and work toward the benefit
of all. We are meeting on Monday. Don't worry about the lack of the
attachment I reference for the moment. Regards, Glenn
__________________________________
Sharon et al,
I think we need to have a set of meetings to define our plan for
exposing NCDC services. I'm not aware of the status of this kind of
activity already, but there are several efforts underway within the
Center and we're already on a similar track - we just need to solidify
our goals- and maybe generate a requirements document from which to work
toward. btw- NOAA's GEO-IDE would benefit from just such an activity.
Rich's efforts in OGC compliant (GIS) services are the way to go, and
NOMADS team have implemented a THREDDS Data Server (TDS) for OGC WCS
services and provides for OPeDNAP enabled requests handled under TDS.
Of course SteveD's group has advanced even higher levels of standards
using SOAP for more generalized services descriptions. There are
limitations to generalities with such standards, and not community based
standards such as OPeNDAP at the present time.
I'm co-PI on a NASA, Unidata, OPeNDAP and GMU effort called ACCESS that
has defined and are building just such a "Gateway" using NetCDF files
initially- using the COARDS and CF compliant OGC Catalog Services for
the Web (CSW). This cross- community effort has many players including
OGC, CEOS, GALEON, GO-ESSP and many others. It builds on the work of
the Common Data Model and TDS work at Unidata and OGC and International
services. There is work to do on my end: Grib/grib2/BUFR issues but
we've already made significant headway in this regard and participated
in the last GEO demo last May in Beijing. NCDC will be the benefactor of
this effort and RSAD will be installing this capability (in test mode)
when completed later this year.
The attached could be a staring point for these Center-wide discussions,
and I suggest that the attached be reviewed and considered for expansion
into the center when fully developed. The reason that I'm co-PI on the
effort is that NOMADS would install and provide this service for models.
It applies to other datasets as well but would need to be adapted for
Neal's/Steve's data. Comments? Glenn
What David Maidmont seeks of course is a unified NCDC web presence to
all of NCDC's resources.
----- Original Message -----
From: David Maidment <maidment at mail.utexas.edu> <mailto:maidment at mail.utexas.edu>
Date: Friday, August 18, 2006 7:28 pm
Subject: RE: NCDC Web Services
Rich:
There are a lot of questions here that require some reflection.
What I want to do
in the short term is stand up a web service to NCDC that is as
simple as possible
that shows that we have a serious commitment to work together.
That is why CRN
is attractive to me -- its high quality data thus good for
research, its not very many
stations, so not too difficult to build the catalog even if we do
it station by station
using the web services that you created (do they access CRN?), and
hopefullyyour services will work onto the archive reasonably well.
I am trying to create a path that points the way to the future in a
methodicalmanner that step by step we figure out what comes next
and implement it
and we start with some fairly straightforward, confidence building
steps that
solidify the relationships which already exist on a technical level
betweenyou and your colleagues and us. We need to learn how you
think about
data at NCDC. We need to understand what databases you maintain, how
many services you actually offer and which ones we should focus on
first, which
ones leave to later.
Our first priority is access to historical NCDC data archives not
near real-time
data.
I want to understand what datasets you are publishing in netCDF-
accessibleform and how those dovetail with what is available from
NCAR and Unidata.
I don't know enough about NIDIS yet to comment on it in an informed
way.
Thanks for your feedback.
David
________________________________
From: Rich.Baldwin [mailto:Rich.Baldwin at noaa.gov]
Sent: Fri 8/18/2006 8:32 AM
To: David Maidment
Cc: Cedric David; Ilya Zaslavsky; Glenn Rutledge; Sharon Le Duc;
Neal Lott
Subject: Re: NCDC Web Services
Hello David and Ilya,
I was glad to meet you all last week. I had a productive meeting with
your teams at UTexas and SDSC. It's good to be back from the
conferenceand vacation. I have a conflict for the scheduled
Tuesday meeting, but
I'll see if I can break away. Just in case I can't make it, let
me try
to clear away some debris.
There are 3 main issues to deal with
1) Data Access
I appreciate your concern regarding the potential server impact
resulting from use of our web services (particularly in light of the
NWIS problems). Accessing the ASOS stations within ISD for the last
days data through our current web services shouldn't produce an undue
impact on our servers. I have made some preliminary tests which bare
this out. As I mentioned in our meeting last week, NCDC has begun a
major effort to assemble an element inventory (data catalog) for all
datasets. The initial inventory will be from the COOP (NWS
cooperativeobserver network) dataset; available early 2007. An
inventory/catalogfor ISD will follow that effort. There are 3
scenarios that present
themselves.
a) The existing web services point to a dataset (global hourly)
which is
updated daily with the last days observations. You all could use
theseweb services to extract current data and build a
catalog/inventory of
observations. Basic station observations of temperature, wind, dew
point, pressure will be recorded and observations like precipitation
will be present when observed. Among the additional station
observations, there will be a good deal of variance day to day,
stationto station. This would give you a current archive, and the
finalmodifications will be done well before the 10/30 target. A
historicalrecord could be built from our planned web service
inventory/catalogavailable next year.
b) You all could wait for us to build element inventories (catalog)
fordatasets of interest and then use web services to those tables
to update
information. This would probably take around 6 months.
c) You could access the archived data through files on our ftp server,
create your own catalog/inventory. Then plug into our web services
catalog/inventory for updates.
During last weeks meeting, I discussed our current global hourly data
inventory from which I can implement a web service for access to an
inventory of the number of observations for each station by year and
month. This would take a day to put together. Note that this doesn't
make a distinction of what was observed only that something was
observed.
2) NIDIS
As I mentioned last week, NCDC is heavily involved in the NIDIS effort
(National Integrated Drought Information System), a collaborative
effortof serveral agencies and institutions. The scope and
requirements for
the NIDIS portal are still being sorted out so in that regard... What
requirements might CUAHSI find useful in a drought portal? What web
service products might NCDC be able to consume from CUAHSI? At this
point, I don't believe the pieces are in place for something like
watershed flux analysis, but this and similar products would be
the goals.
3) NetCDF
The resource links you sent were very useful; I have already begun
on a
mock up for implementation. At our meeting last week, you were
satisfied w/ the time line which I laid out concerning netcdf
development/implementation (post 10/30). If that has changed, please
let me know.
Take care, Rich
PS. My laptop which has the slides from my presentation is getting a
s/w and h/w update (potential exploding battery), I'll send the
slides ASAP.
David Maidment wrote:
Rich:
Thanks so much for spending time with us on Thursday at SDSC. I
appreciate you presenting
to us the work that you've done with web services and I
especially appreciate that you have tried
to make them CUAHSI style. As you know, I have asked Cedric
David to work with you to test your
services and keep me informed as to which databases they provide
access to and how we can best
use what you have created. Could you please send me a copy of
the slides that you used last Thursday.
I want to study the nomenclature that you employed since its
reflective of how things are called at NCDC
and I need to understand that.
Cedric -- what we need to do is to use these services to create
an observations catalog for each
of the NCDC databases that NCDC is exposing using these services.
I am not sure if this should
be done at Texas or at SDSC.
Ilya -- what do you think about the above?
Rich --- the website with the tutorial on how to access the
Unidata NetCDFServer is at:
http://www.crwr.utexas.edu/gis/gishydro06/SpaceAndTime/NetCDF/Animating%20netCDF%20Data%20in%20ArcMap.htm>
The actual server itself is at:
http://motherlode.ucar.edu:8080/thredds/idd/nam_model.html>
This is really hard to find if you just search the Unidata web site.
I'd like to discuss with you and your colleagues how best to
develop a "weather and climate server"
that uses time series web services and also netCDF services.
Perhaps that is something we can
discuss with Sharon LeDuc.
David
H
--
Glenn K. Rutledge
Services Team Leader
Remote Sensing and Applications Division
NOMADS Project Manager
National Oceanic and Atmospheric Administration
National Climatic Data Center
Asheville NC 28801
Phone: (828) 271-4097
Fax: (828) 271-4328
NOMADS: http://nomads.ncdc.noaa.gov/
--
Glenn K. Rutledge
Services Team Leader
Remote Sensing and Applications Division
NOMADS Project Manager
National Oceanic and Atmospheric Administration
National Climatic Data Center
Asheville NC 28801
Phone: (828) 271-4097
Fax: (828) 271-4328
NOMADS: http://nomads.ncdc.noaa.gov/
--
Glenn K. Rutledge
Services Team Leader
Remote Sensing and Applications Division
NOMADS Project Manager
National Oceanic and Atmospheric Administration
National Climatic Data Center
Asheville NC 28801
Phone: (828) 271-4097
Fax: (828) 271-4328
NOMADS: http://nomads.ncdc.noaa.gov/
--
Glenn K. Rutledge
Services Team Leader
Remote Sensing and Applications Division
NOMADS Project Manager
National Oceanic and Atmospheric Administration
National Climatic Data Center
Asheville NC 28801
Phone: (828) 271-4097
Fax: (828) 271-4328
NOMADS: http://nomads.ncdc.noaa.gov/
More information about the Thredds
mailing list