[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

RE: NCDC Web Services



Glenn:
 
OK, good to get this cleared up.  I am glad this is not some inherent issue 
with OpenDAP,
which I had earlier suspected it might be.
 
David
 

________________________________

From: Glenn.Rutledge [mailto:address@hidden
Sent: Thu 9/7/2006 9:01 AM
To: David Maidment
Cc: John Caron; support-thredds; Ilya Zaslavsky
Subject: Re: NCDC Web Services


Dave-
Yes- you are indeed correct.  SOme organizations (NSF I guess), require a 
standard pot 80 and block all others unless special arrangement have been made. 
 Because your's is not the first complaint (maybe the 3rd), NOMADS is going to 
set up a Tomcat/Apache interface where we will present standard port to users.  
We had delayed that because we are doing a complete overhaul of our front end 
(to a load balanced arrangement).  Anyway- the hardware got delayed and so to 
did the action to standardize that access for you.  So- you should have access 
within several weeks..   

Sorry for the hassle....  Glenn

David Maidment wrote the following on 9/6/2006 8:56 AM: 

        Glenn:
         
        I think that what the information below means from John Caron is that 
OpenDAP
        services need to be accessible through Port 80 even if that is not the 
default
        with the Tomcat server or otherwise access to them will be blocked by 
firewalls
        in some instances.   Can you check to see whether in publishing NARR 
data
        access through OpenDAP you have used Port 80 or not.   This might be why
        we and others are having trouble with firewalls when accessing NARR 
from within
        some organizations, but not with others.
        
        David
         
        
        ________________________________
        
        From: John Caron [mailto:address@hidden
        Sent: Sun 9/3/2006 7:25 PM
        To: David Maidment
        Cc: Glenn.Rutledge; support-thredds; Ilya Zaslavsky
        Subject: Re: NCDC Web Services
        
        
        
        
        
        David Maidment wrote:
          

                John:
                
                I feel like Dr Watson when Sherlock Holmes says "Elementary my 
dear Watson".
                I am sure what you are telling me is elementary but it requires 
more knowledge of
                server technology than I possess.
                
                First of all -- what is a port?  How is a port defined or 
established?  By the server?  If
                so how?   Can a server have multiple ports?   Why would you 
want to do that?
                    

        A port is just an abstraction to establish various communication 
channels to the same host machine. I mention it because thats generally how 
firewalls work.
        
          

                Is Tomcat a server technology built with open source 
components?  
                    

        Yes, its entirely open source.
        
        Is it the java equivalent of a Windows based server?  
        
        Its a jaav-based HTTP server.
        
          

                What is the significance of the default port on Tomcat being 
8080 and on Windows (?) servers 80?
                    

        Tomcat can run under port 80 if you want. Again, its only relevent 
because standard firewall policy is to allow HTTP traffic on port 80. Often 
other ports are not allowed. This doesnt particularly solve security problems, 
but its what people do anyway.
        
          

                Are SOAP/XML services similarly restricted by port numbers?  It 
seems like our
                SOAP/XML services are not similarly restricted by firewalls as 
what we've done with
                OpenDAP, but that may be because we are mediating them through 
SDSC that may
                be using Port 80.
                    

        Most servers can use any port you tell them. But they have to use a 
specific port.
        
          

                Ah -- the glories of computer technology!
                    

        SOrry i cant give you a simpler answer.
        
          

                David
                
                
                
                
                ________________________________
                
                From: John Caron [mailto:address@hidden
                Sent: Wed 8/23/2006 5:12 PM
                To: David Maidment
                Cc: Glenn.Rutledge; 'support-thredds'
                Subject: Re: NCDC Web Services
                
                
                
                Im not sure what the issue is, but its not really a server 
thing. Firewalls live on the client. Generally HTTP traffic on port 80 is let 
through, and not much else. Tomcat default port is 8080, it can be changed to 
port 80 is need be, but you cant expect THREDDS servers to do that for the 
convenience of some site that has some restrictive firewall.
                
                A proxy server can be a good solution, but i dont know what the 
details of use are.
                
                David Maidment wrote:
                
                    

                        Glenn:
                        
                        Yes, please find out what we have to do.  This looks 
like a significant
                        issue to me.
                        
                        John -- could you look at the email trail below and 
comment on the
                        issues of OpenDAP and
                        firewalls.   Is there any difference if we use your 
netCDF server?
                        
                        David
                        
                        
                        
------------------------------------------------------------------------
                        *From:* Glenn.Rutledge [mailto:address@hidden
                        *Sent:* Wednesday, August 23, 2006 10:09 AM
                        *To:* David Maidment
                        *Subject:* Re: NCDC Web Services
                        
                        Systems folks would do this...and as I understand it- 
is not at all
                        uncommon.  It's a representation of a service within a 
firewall   I am
                        out of my realm but I do plan to investigate further 
since NASA had the
                        exact comment you did just last week.
                        
                        I'll let you know more if you like. GR
                        
                        David Maidment wrote the following on 8/23/2006 10:54 
AM:
                        
                        
                              

                                Glenn:
                                
                                What is a proxy server?  How would I or any 
other regular user know
                                how to use one of these?
                                
                                David
                                
                                
                                
------------------------------------------------------------------------
                                *From:* Glenn.Rutledge [mailto:address@hidden
                                *Sent:* Tuesday, August 22, 2006 12:53 PM
                                *To:* David Maidment
                                *Subject:* Re: NCDC Web Services
                                
                                Hi David-
                                A response from OPeNDAP:
                                
                                our code (libdap, libnc-dap, our clients - with 
the possible exception
                                of the ODC) can all be told to use a proxy 
server. Open ports to that
                                machine and you're off and running. If someone 
can use the Internet,
                                they can use our clients and read from your (or 
any) server. Point
                                these folks toward the documentation on the 
.dodsrc file. If they
                                still have questions (and I know some of our 
docs are getting stale),
                                then have them contact address@hidden with 
their questions
                                (or the newly renamed opendap or opendap-tech 
emails lists - see
                                www.opendap.org <http://www.opendap.org/>  and 
look under support for information about those).
                                
                                I'm glad you asked me, because this is an 
important feature of the
                                client-side software!
                                
                                
                                Hope this helps--- Glenn
                                
                                David Maidment wrote the following on 8/21/2006 
9:13 PM:
                                
                                
                                        

                                        Glenn:
                                        
                                        I have some concerns about how useful 
OpenDAP is.  It seems sensitive
                                        to being blocked by firewalls.
                                        Our direct calls to NARR that relied on 
using OpenDAP worked fine
                                                  

                                >from here at UT but from inside of
                                        

                                        NSF with all their firewalls in place, 
the data were blocked, and the
                                        same is true when trying to do this
                                                  

                                >from within ESRI.
                                        

                                        I think the question of how to do these 
things is linked up with
                                        computer security and that needs to be
                                        factored into the debate.
                                        
                                        David
                                        
                                        
                                        
------------------------------------------------------------------------
                                        *From:* Glenn.Rutledge 
[mailto:address@hidden
                                        *Sent:* Monday, August 21, 2006 9:13 AM
                                        *To:* David Maidment
                                        *Subject:* Re: NCDC Web Services
                                        
                                        Thanks David-
                                        I agree- but somewhat concerned whether 
SOAP/REST will have
                                        capabilities across the other data 
forms you cite - although based on
                                        your individual needs- it doesn't seem 
to matter.  But for the larger
                                        whole- we need to test this out.  Glenn
                                        
                                        David Maidment wrote the following on 
8/21/2006 9:47 AM:
                                        
                                        
                                                  

                                        Glenn:
                                        
                                        I see observations data as time series 
as one kind of service, and weather
                                        model, remote sensing and Nexrad data 
as another.   For the weather and
                                        climate "field" information, some 
variant of what John Caron is doing seems
                                        appropriate.   For the observations 
data, I think we have a fairly good scheme
                                        figured out in the work we are doing 
with CUAHSI.   I can see the netCDF
                                        style of things for the weather and 
climate fields.  I think the time series stuff
                                        could be handled that way but is easier 
and perhaps more appropriate just to
                                        do it in XML via SOAP or REST.
                                        
                                        What I would like to see developed is a 
"Weather and Climate Server" so that
                                        the data users can go to one system and 
get into the various information sources
                                        without having to go separately to 
NCDC, Unidata, NCAR, NWS, etc.   This is
                                        like WaterOneFlow that we are 
attempting to do in CUAHSI for water observation
                                        data.
                                        
                                        David
                                        
                                        
                                        ________________________________
                                        
                                        From: address@hidden 
[mailto:address@hidden
                                        Sent: Fri 8/18/2006 6:39 PM
                                        To: David Maidment
                                        Subject: Re: RE: NCDC Web Services
                                        
                                        
                                        
                                        Hi David,
                                        I agree.  Here's the note I sent Sharon 
earlier this week after your
                                        request.  I want to plan for the long 
term and work toward the benefit
                                        of all.  We are meeting on Monday.  
Don't worry about the lack of the
                                        attachment I reference for the moment.  
Regards,  Glenn
                                        __________________________________
                                        Sharon et al,
                                        I think we need to have a set of 
meetings to define our plan for
                                        exposing NCDC services.  I'm not aware 
of the status of this kind of
                                        activity already, but there are several 
efforts underway within the
                                        Center and we're already on a similar 
track - we just need to solidify
                                        our goals- and maybe generate a 
requirements document from which to work
                                        toward. btw- NOAA's GEO-IDE would 
benefit from just such an activity.
                                        
                                        Rich's efforts in OGC compliant (GIS) 
services are the way to go, and
                                        NOMADS team have implemented a THREDDS 
Data Server (TDS) for OGC WCS
                                        services and provides for OPeDNAP 
enabled requests handled under TDS.
                                        Of course SteveD's group has advanced 
even higher levels of standards
                                        using SOAP for more generalized 
services descriptions.  There are
                                        limitations to generalities with such 
standards, and not community based
                                        standards such as OPeNDAP at the 
present time.
                                        
                                        I'm co-PI on a NASA, Unidata, OPeNDAP 
and GMU effort called ACCESS that
                                        has defined and are building just such 
a "Gateway" using NetCDF files
                                        initially- using the COARDS and CF 
compliant OGC Catalog Services for
                                        the Web (CSW).  This cross- community 
effort has many players including
                                        OGC, CEOS, GALEON, GO-ESSP and many 
others.  It builds on the work of
                                        the Common Data Model and TDS work at 
Unidata and OGC and International
                                        services.  There is work to do on my 
end:  Grib/grib2/BUFR issues but
                                        we've already made significant headway 
in this regard and participated
                                        in the last GEO demo last May in 
Beijing. NCDC will be the benefactor of
                                        this effort and RSAD will be installing 
this capability (in test mode)
                                        when completed later this year.
                                        
                                        The attached could be a staring point 
for these Center-wide discussions,
                                        and I suggest that the attached be 
reviewed and considered for expansion
                                        into the center when fully developed.  
The reason that I'm co-PI on the
                                        effort is that NOMADS would install and 
provide this service for models.
                                        It applies to other datasets as well 
but would need to be adapted for
                                        Neal's/Steve's data.  Comments? Glenn
                                        
                                        What David Maidmont seeks of course is 
a unified NCDC web presence to
                                        all of NCDC's resources.   
                                        
                                        ----- Original Message -----
                                        From: David Maidment <address@hidden> 
<mailto:address@hidden> 
                                        Date: Friday, August 18, 2006 7:28 pm
                                        Subject: RE: NCDC Web Services
                                        
                                        
                                        
                                        
                                                    

                                        Rich:
                                        
                                        There are a lot of questions here that 
require some reflection.
                                        What I want to do
                                        in the short term is stand up a web 
service to NCDC that is as
                                        simple as possible
                                        that shows that we have a serious 
commitment to work together.
                                        That is why CRN
                                        is attractive to me -- its high quality 
data thus good for
                                        research, its not very many
                                        stations, so not too difficult to build 
the catalog even if we do
                                        it station by station
                                        using the web services that you created 
(do they access CRN?), and
                                        hopefullyyour services will work onto 
the archive reasonably well.
                                        
                                        I am trying to create a path that 
points the way to the future in a
                                        methodicalmanner that step by step we 
figure out what comes next
                                        and implement it
                                        and we start with some fairly 
straightforward, confidence building
                                        steps that
                                        solidify the relationships which 
already exist on a technical level
                                        betweenyou and your colleagues and us.  
 We need to learn how you
                                        think about
                                        data at NCDC.  We need to understand 
what databases you maintain, how
                                        many services you actually offer and 
which ones we should focus on
                                        first, which
                                        ones leave to later.
                                        
                                        Our first priority is access to 
historical NCDC data archives not
                                        near real-time
                                        data.
                                        
                                        I want to understand what datasets you 
are publishing in netCDF-
                                        accessibleform and how those dovetail 
with what is available from
                                        NCAR and Unidata.
                                        
                                        I don't know enough about NIDIS yet to 
comment on it in an informed
                                        way.
                                        Thanks for your feedback.
                                        
                                        David
                                        
                                        
                                        
                                        ________________________________
                                        
                                        From: Rich.Baldwin 
[mailto:address@hidden
                                        Sent: Fri 8/18/2006 8:32 AM
                                        To: David Maidment
                                        Cc: Cedric David; Ilya Zaslavsky; Glenn 
Rutledge; Sharon Le Duc;
                                        Neal Lott
                                        Subject: Re: NCDC Web Services
                                        
                                        
                                        
                                        Hello David and Ilya,
                                        
                                        I was glad to meet you all last week.  
I had a productive meeting with
                                        your teams at UTexas and SDSC.  It's 
good to be back from the
                                        conferenceand vacation.   I have a 
conflict for the scheduled
                                        Tuesday meeting, but
                                        I'll see if I can break away.   Just in 
case I can't make it, let
                                        me try
                                        to clear away some debris.
                                        
                                        There are 3 main issues to deal with
                                        
                                        1) Data Access
                                        I appreciate your concern regarding the 
potential server impact
                                        resulting from use of our web services 
(particularly in light of the
                                        NWIS problems).  Accessing the ASOS 
stations within ISD for the last
                                        days data through our current web 
services shouldn't produce an undue
                                        impact on our servers.   I have made 
some preliminary tests which bare
                                        this out.  As I mentioned in our 
meeting last week, NCDC has begun a
                                        major effort to assemble an element 
inventory (data catalog) for all
                                        datasets.  The initial inventory will 
be from the COOP (NWS
                                        cooperativeobserver network) dataset; 
available early 2007.  An
                                        inventory/catalogfor ISD will follow 
that effort.  There are 3
                                        scenarios that present
                                        themselves.
                                        
                                        a) The existing web services point to a 
dataset (global hourly)
                                        which is
                                        updated daily with the last days 
observations.  You all could use
                                        theseweb services to extract current 
data and build a
                                        catalog/inventory of
                                        observations.   Basic station 
observations of temperature, wind, dew
                                        point, pressure will be recorded and 
observations like precipitation
                                        will be present when observed.  Among 
the additional station
                                        observations, there will be a good deal 
of variance day to day,
                                        stationto station.  This would give you 
a current archive, and the
                                        finalmodifications will be done well 
before the 10/30 target.  A
                                        historicalrecord could be built from 
our planned web service
                                        inventory/catalogavailable next year.
                                        
                                        b) You all could wait for us to build 
element inventories (catalog)
                                        fordatasets of interest and then use 
web services to those tables
                                        to update
                                        information.  This would probably take 
around 6 months.
                                        
                                        c) You could access the archived data 
through files on our ftp server,
                                        create your own catalog/inventory.  
Then plug into our web services
                                        catalog/inventory for updates.
                                        
                                        During last weeks meeting, I discussed 
our current global hourly data
                                        inventory from which I can implement a 
web service for access to an
                                        inventory of the number of observations 
for each station by year and
                                        month.  This would take a day to put 
together.  Note that this doesn't
                                        make a distinction of what was observed 
only that something was
                                        observed.
                                        2) NIDIS
                                        
                                        As I mentioned last week, NCDC is 
heavily involved in the NIDIS effort
                                        (National Integrated Drought 
Information System), a collaborative
                                        effortof serveral agencies and 
institutions.  The scope and
                                        requirements for
                                        the NIDIS portal are still being sorted 
out so in that regard... What
                                        requirements might CUAHSI find useful 
in a drought portal?  What web
                                        service products might NCDC be able to 
consume from CUAHSI?  At this
                                        point, I don't believe the pieces are 
in place for something like
                                        watershed flux analysis, but this and 
similar products  would be
                                        the goals.
                                        
                                        3) NetCDF
                                        The resource links you sent were very 
useful; I have already begun
                                        on a
                                        mock up for implementation.  At our 
meeting last week, you were
                                        satisfied w/ the time line which I laid 
out concerning netcdf
                                        development/implementation (post 
10/30).  If that has changed, please
                                        let me know.
                                        
                                        Take care, Rich
                                        
                                        PS.  My laptop which has the slides 
from my presentation is getting a
                                        s/w and h/w update (potential exploding 
battery), I'll send the
                                        slides ASAP.
                                        
                                        
                                        David Maidment wrote:
                                        
                                        
                                        
                                                      

                                        Rich:
                                        
                                        Thanks so much for spending time with 
us on Thursday at SDSC.  I
                                          
                                                        

                                        appreciate you presenting
                                        
                                        
                                        
                                                      

                                        to us the work that you've done with 
web services and I
                                          
                                                        

                                        especially appreciate that you have 
tried
                                        
                                        
                                        
                                                      

                                        to make them CUAHSI style.   As you 
know, I have asked Cedric
                                          
                                                        

                                        David to work with you to test your
                                        
                                        
                                        
                                                      

                                        services and keep me informed as to 
which databases they provide
                                          
                                                        

                                        access to and how we can best
                                        
                                        
                                        
                                                      

                                        use what you have created.  Could you 
please send me a copy of
                                          
                                                        

                                        the slides that you used last Thursday.
                                        
                                        
                                        
                                                      

                                        I want to study the nomenclature that 
you employed since its
                                          
                                                        

                                        reflective of how things are called at 
NCDC
                                        
                                        
                                        
                                                      

                                        and I need to understand that.
                                        
                                        Cedric -- what we need to do is to use 
these services to create
                                          
                                                        

                                        an observations catalog for each
                                        
                                        
                                        
                                                      

                                        of the NCDC databases that NCDC is 
exposing using these services.
                                          
                                                        

                                        I am not sure if this should
                                        
                                        
                                        
                                                      

                                        be done at Texas or at SDSC.
                                        
                                        Ilya -- what do you think about the 
above?
                                        
                                        Rich  --- the website with the tutorial 
on how to access the
                                          
                                                        

                                        Unidata NetCDFServer is at:
                                        
                                        
                                        
                                                      

                                          
                                                        

                                        
http://www.crwr.utexas.edu/gis/gishydro06/SpaceAndTime/NetCDF/Animating%20netCDF%20Data%20in%20ArcMap.htm>
                                        
                                        
                                        
                                                    

                                        The actual server itself is at:
                                          
                                                        

                                        
http://motherlode.ucar.edu:8080/thredds/idd/nam_model.html>
                                        
                                        
                                        
                                                      

                                        This is really hard to find if you just 
search the Unidata web site.
                                        
                                        I'd like to discuss with you and your 
colleagues how best to
                                          
                                                        

                                        develop a "weather and climate server"
                                        
                                        
                                        
                                                      

                                        that uses time series web services and 
also netCDF services.
                                          
                                                        

                                        Perhaps that is something we can
                                        
                                        
                                        
                                                      

                                        discuss with Sharon LeDuc.
                                        
                                        David
                                        
                                        
                                        
                                        
                                        
                                          
                                                        

                                        H
                                        
                                                      

                                        
                                                    

                                        --
                                        Glenn K. Rutledge
                                        Services Team Leader
                                        Remote Sensing and Applications Division
                                        NOMADS Project Manager
                                        National Oceanic and Atmospheric 
Administration
                                        National Climatic Data Center
                                        Asheville NC 28801
                                        Phone: (828) 271-4097
                                        Fax: (828) 271-4328
                                        
                                        NOMADS: http://nomads.ncdc.noaa.gov/
                                        
                                                  

                                --
                                Glenn K. Rutledge
                                Services Team Leader
                                Remote Sensing and Applications Division
                                NOMADS Project Manager
                                National Oceanic and Atmospheric Administration
                                National Climatic Data Center
                                Asheville NC 28801
                                Phone: (828) 271-4097
                                Fax: (828) 271-4328
                                
                                NOMADS: http://nomads.ncdc.noaa.gov/
                                
                                        

                        --
                        Glenn K. Rutledge
                        Services Team Leader
                        Remote Sensing and Applications Division
                        NOMADS Project Manager
                        National Oceanic and Atmospheric Administration
                        National Climatic Data Center
                        Asheville NC 28801
                        Phone: (828) 271-4097
                        Fax: (828) 271-4328
                        
                        NOMADS: http://nomads.ncdc.noaa.gov/
                              

                    

          


-- 
Glenn K. Rutledge
Services Team Leader
Remote Sensing and Applications Division
NOMADS Project Manager
National Oceanic and Atmospheric Administration
National Climatic Data Center
Asheville NC 28801
Phone: (828) 271-4097
Fax: (828) 271-4328

NOMADS: http://nomads.ncdc.noaa.gov/


NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.