[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: NCDC Web Services



Im not sure what the issue is, but its not really a server thing. Firewalls 
live on the client. Generally HTTP traffic on port 80 is let through, and not 
much else. Tomcat default port is 8080, it can be changed to port 80 is need 
be, but you cant expect THREDDS servers to do that for the convenience of some 
site that has some restrictive firewall.

A proxy server can be a good solution, but i dont know what the details of use are.
David Maidment wrote:
Glenn:
Yes, please find out what we have to do. This looks like a significant issue to me. John -- could you look at the email trail below and comment on the issues of OpenDAP and
firewalls.   Is there any difference if we use your netCDF server?
David
------------------------------------------------------------------------
*From:* Glenn.Rutledge [mailto:address@hidden]
*Sent:* Wednesday, August 23, 2006 10:09 AM
*To:* David Maidment
*Subject:* Re: NCDC Web Services

Systems folks would do this...and as I understand it- is not at all uncommon. It's a representation of a service within a firewall I am out of my realm but I do plan to investigate further since NASA had the exact comment you did just last week.
I'll let you know more if you like. GR

David Maidment wrote the following on 8/23/2006 10:54 AM:

Glenn:
What is a proxy server? How would I or any other regular user know how to use one of these? David
------------------------------------------------------------------------
*From:* Glenn.Rutledge [mailto:address@hidden]
*Sent:* Tuesday, August 22, 2006 12:53 PM
*To:* David Maidment
*Subject:* Re: NCDC Web Services

Hi David-
A response from OPeNDAP:

our code (libdap, libnc-dap, our clients - with the possible exception of the ODC) can all be told to use a proxy server. Open ports to that machine and you're off and running. If someone can use the Internet, they can use our clients and read from your (or any) server. Point these folks toward the documentation on the .dodsrc file. If they still have questions (and I know some of our docs are getting stale), then have them contact address@hidden with their questions (or the newly renamed opendap or opendap-tech emails lists - see www.opendap.org and look under support for information about those).

I'm glad you asked me, because this is an important feature of the client-side software!


Hope this helps--- Glenn

David Maidment wrote the following on 8/21/2006 9:13 PM:

Glenn:
I have some concerns about how useful OpenDAP is. It seems sensitive to being blocked by firewalls. Our direct calls to NARR that relied on using OpenDAP worked fine from here at UT but from inside of NSF with all their firewalls in place, the data were blocked, and the same is true when trying to do this
from within ESRI.
I think the question of how to do these things is linked up with computer security and that needs to be
factored into the debate.
David
------------------------------------------------------------------------
*From:* Glenn.Rutledge [mailto:address@hidden]
*Sent:* Monday, August 21, 2006 9:13 AM
*To:* David Maidment
*Subject:* Re: NCDC Web Services

Thanks David-
I agree- but somewhat concerned whether SOAP/REST will have capabilities across the other data forms you cite - although based on your individual needs- it doesn't seem to matter. But for the larger whole- we need to test this out. Glenn

David Maidment wrote the following on 8/21/2006 9:47 AM:

Glenn:

I see observations data as time series as one kind of service, and weather
model, remote sensing and Nexrad data as another.   For the weather and
climate "field" information, some variant of what John Caron is doing seems
appropriate.   For the observations data, I think we have a fairly good scheme
figured out in the work we are doing with CUAHSI.   I can see the netCDF
style of things for the weather and climate fields.  I think the time series 
stuff
could be handled that way but is easier and perhaps more appropriate just to
do it in XML via SOAP or REST.

What I would like to see developed is a "Weather and Climate Server" so that
the data users can go to one system and get into the various information sources
without having to go separately to NCDC, Unidata, NCAR, NWS, etc.   This is
like WaterOneFlow that we are attempting to do in CUAHSI for water observation data.

David


________________________________

From: address@hidden [mailto:address@hidden]
Sent: Fri 8/18/2006 6:39 PM
To: David Maidment
Subject: Re: RE: NCDC Web Services



Hi David,
I agree.  Here's the note I sent Sharon earlier this week after your
request.  I want to plan for the long term and work toward the benefit
of all.  We are meeting on Monday.  Don't worry about the lack of the
attachment I reference for the moment.  Regards,  Glenn
__________________________________
Sharon et al,
I think we need to have a set of meetings to define our plan for
exposing NCDC services.  I'm not aware of the status of this kind of
activity already, but there are several efforts underway within the
Center and we're already on a similar track - we just need to solidify
our goals- and maybe generate a requirements document from which to work
toward. btw- NOAA's GEO-IDE would benefit from just such an activity.

Rich's efforts in OGC compliant (GIS) services are the way to go, and
NOMADS team have implemented a THREDDS Data Server (TDS) for OGC WCS
services and provides for OPeDNAP enabled requests handled under TDS.
Of course SteveD's group has advanced even higher levels of standards
using SOAP for more generalized services descriptions.  There are
limitations to generalities with such standards, and not community based
standards such as OPeNDAP at the present time.
I'm co-PI on a NASA, Unidata, OPeNDAP and GMU effort called ACCESS that
has defined and are building just such a "Gateway" using NetCDF files
initially- using the COARDS and CF compliant OGC Catalog Services for
the Web (CSW).  This cross- community effort has many players including
OGC, CEOS, GALEON, GO-ESSP and many others.  It builds on the work of
the Common Data Model and TDS work at Unidata and OGC and International
services.  There is work to do on my end:  Grib/grib2/BUFR issues but
we've already made significant headway in this regard and participated
in the last GEO demo last May in Beijing. NCDC will be the benefactor of
this effort and RSAD will be installing this capability (in test mode)
when completed later this year.

The attached could be a staring point for these Center-wide discussions,
and I suggest that the attached be reviewed and considered for expansion
into the center when fully developed.  The reason that I'm co-PI on the
effort is that NOMADS would install and provide this service for models.
It applies to other datasets as well but would need to be adapted for
Neal's/Steve's data.  Comments? Glenn

What David Maidmont seeks of course is a unified NCDC web presence to
all of NCDC's resources.
----- Original Message -----
From: David Maidment <address@hidden>
Date: Friday, August 18, 2006 7:28 pm
Subject: RE: NCDC Web Services

Rich:

There are a lot of questions here that require some reflection. What I want to do
in the short term is stand up a web service to NCDC that is as
simple as possible
that shows that we have a serious commitment to work together. That is why CRN
is attractive to me -- its high quality data thus good for
research, its not very many
stations, so not too difficult to build the catalog even if we do
it station by station
using the web services that you created (do they access CRN?), and
hopefullyyour services will work onto the archive reasonably well.

I am trying to create a path that points the way to the future in a
methodicalmanner that step by step we figure out what comes next
and implement it
and we start with some fairly straightforward, confidence building
steps that
solidify the relationships which already exist on a technical level
betweenyou and your colleagues and us.   We need to learn how you
think about
data at NCDC.  We need to understand what databases you maintain, how
many services you actually offer and which ones we should focus on
first, which
ones leave to later.

Our first priority is access to historical NCDC data archives not
near real-time
data.

I want to understand what datasets you are publishing in netCDF-
accessibleform and how those dovetail with what is available from
NCAR and Unidata.

I don't know enough about NIDIS yet to comment on it in an informed
way.
Thanks for your feedback.

David



________________________________

From: Rich.Baldwin [mailto:address@hidden]
Sent: Fri 8/18/2006 8:32 AM
To: David Maidment
Cc: Cedric David; Ilya Zaslavsky; Glenn Rutledge; Sharon Le Duc;
Neal Lott
Subject: Re: NCDC Web Services



Hello David and Ilya,

I was glad to meet you all last week.  I had a productive meeting with
your teams at UTexas and SDSC.  It's good to be back from the
conferenceand vacation.   I have a conflict for the scheduled
Tuesday meeting, but
I'll see if I can break away.   Just in case I can't make it, let
me try
to clear away some debris.

There are 3 main issues to deal with

1) Data Access
I appreciate your concern regarding the potential server impact
resulting from use of our web services (particularly in light of the
NWIS problems).  Accessing the ASOS stations within ISD for the last
days data through our current web services shouldn't produce an undue
impact on our servers.   I have made some preliminary tests which bare
this out.  As I mentioned in our meeting last week, NCDC has begun a
major effort to assemble an element inventory (data catalog) for all
datasets.  The initial inventory will be from the COOP (NWS
cooperativeobserver network) dataset; available early 2007.  An
inventory/catalogfor ISD will follow that effort.  There are 3
scenarios that present
themselves.

a) The existing web services point to a dataset (global hourly)
which is
updated daily with the last days observations.  You all could use
theseweb services to extract current data and build a
catalog/inventory of
observations.   Basic station observations of temperature, wind, dew
point, pressure will be recorded and observations like precipitation
will be present when observed.  Among the additional station
observations, there will be a good deal of variance day to day,
stationto station.  This would give you a current archive, and the
finalmodifications will be done well before the 10/30 target.  A
historicalrecord could be built from our planned web service
inventory/catalogavailable next year.

b) You all could wait for us to build element inventories (catalog)
fordatasets of interest and then use web services to those tables
to update
information.  This would probably take around 6 months.

c) You could access the archived data through files on our ftp server,
create your own catalog/inventory.  Then plug into our web services
catalog/inventory for updates.

During last weeks meeting, I discussed our current global hourly data
inventory from which I can implement a web service for access to an
inventory of the number of observations for each station by year and
month.  This would take a day to put together.  Note that this doesn't
make a distinction of what was observed only that something was
observed.
2) NIDIS

As I mentioned last week, NCDC is heavily involved in the NIDIS effort
(National Integrated Drought Information System), a collaborative
effortof serveral agencies and institutions.  The scope and
requirements for
the NIDIS portal are still being sorted out so in that regard... What
requirements might CUAHSI find useful in a drought portal?  What web
service products might NCDC be able to consume from CUAHSI?  At this
point, I don't believe the pieces are in place for something like
watershed flux analysis, but this and similar products  would be
the goals.

3) NetCDF
The resource links you sent were very useful; I have already begun
on a
mock up for implementation.  At our meeting last week, you were
satisfied w/ the time line which I laid out concerning netcdf
development/implementation (post 10/30).  If that has changed, please
let me know.

Take care, Rich

PS.  My laptop which has the slides from my presentation is getting a
s/w and h/w update (potential exploding battery), I'll send the
slides ASAP.


David Maidment wrote:
Rich:

Thanks so much for spending time with us on Thursday at SDSC.  I

appreciate you presenting
to us the work that you've done with web services and I

especially appreciate that you have tried
to make them CUAHSI style.   As you know, I have asked Cedric

David to work with you to test your
services and keep me informed as to which databases they provide

access to and how we can best
use what you have created.  Could you please send me a copy of

the slides that you used last Thursday.
I want to study the nomenclature that you employed since its

reflective of how things are called at NCDC
and I need to understand that.

Cedric -- what we need to do is to use these services to create

an observations catalog for each
of the NCDC databases that NCDC is exposing using these services.

I am not sure if this should
be done at Texas or at SDSC.

Ilya -- what do you think about the above?

Rich  --- the website with the tutorial on how to access the

Unidata NetCDFServer is at:

http://www.crwr.utexas.edu/gis/gishydro06/SpaceAndTime/NetCDF/Animating%20netCDF%20Data%20in%20ArcMap.htm>

The actual server itself is at:

http://motherlode.ucar.edu:8080/thredds/idd/nam_model.html>
This is really hard to find if you just search the Unidata web site.

I'd like to discuss with you and your colleagues how best to

develop a "weather and climate server"
that uses time series web services and also netCDF services.

Perhaps that is something we can
discuss with Sharon LeDuc.

David






H





--
Glenn K. Rutledge
Services Team Leader
Remote Sensing and Applications Division
NOMADS Project Manager
National Oceanic and Atmospheric Administration
National Climatic Data Center
Asheville NC 28801
Phone: (828) 271-4097
Fax: (828) 271-4328

NOMADS: http://nomads.ncdc.noaa.gov/


--
Glenn K. Rutledge
Services Team Leader
Remote Sensing and Applications Division
NOMADS Project Manager
National Oceanic and Atmospheric Administration
National Climatic Data Center
Asheville NC 28801
Phone: (828) 271-4097
Fax: (828) 271-4328

NOMADS: http://nomads.ncdc.noaa.gov/


--
Glenn K. Rutledge
Services Team Leader
Remote Sensing and Applications Division
NOMADS Project Manager
National Oceanic and Atmospheric Administration
National Climatic Data Center
Asheville NC 28801
Phone: (828) 271-4097
Fax: (828) 271-4328

NOMADS: http://nomads.ncdc.noaa.gov/