Glenn: OK, good to get this cleared up. I am glad this is not some inherent issue with OpenDAP, which I had earlier suspected it might be. David ________________________________ From: Glenn.Rutledge [mailto:address@hidden Sent: Thu 9/7/2006 9:01 AM To: David Maidment Cc: John Caron; support-thredds; Ilya Zaslavsky Subject: Re: NCDC Web Services Dave- Yes- you are indeed correct. SOme organizations (NSF I guess), require a standard pot 80 and block all others unless special arrangement have been made. Because your's is not the first complaint (maybe the 3rd), NOMADS is going to set up a Tomcat/Apache interface where we will present standard port to users. We had delayed that because we are doing a complete overhaul of our front end (to a load balanced arrangement). Anyway- the hardware got delayed and so to did the action to standardize that access for you. So- you should have access within several weeks.. Sorry for the hassle.... Glenn David Maidment wrote the following on 9/6/2006 8:56 AM: Glenn: I think that what the information below means from John Caron is that OpenDAP services need to be accessible through Port 80 even if that is not the default with the Tomcat server or otherwise access to them will be blocked by firewalls in some instances. Can you check to see whether in publishing NARR data access through OpenDAP you have used Port 80 or not. This might be why we and others are having trouble with firewalls when accessing NARR from within some organizations, but not with others. David ________________________________ From: John Caron [mailto:address@hidden Sent: Sun 9/3/2006 7:25 PM To: David Maidment Cc: Glenn.Rutledge; support-thredds; Ilya Zaslavsky Subject: Re: NCDC Web Services David Maidment wrote: John: I feel like Dr Watson when Sherlock Holmes says "Elementary my dear Watson". I am sure what you are telling me is elementary but it requires more knowledge of server technology than I possess. First of all -- what is a port? How is a port defined or established? By the server? If so how? Can a server have multiple ports? Why would you want to do that? A port is just an abstraction to establish various communication channels to the same host machine. I mention it because thats generally how firewalls work. Is Tomcat a server technology built with open source components? Yes, its entirely open source. Is it the java equivalent of a Windows based server? Its a jaav-based HTTP server. What is the significance of the default port on Tomcat being 8080 and on Windows (?) servers 80? Tomcat can run under port 80 if you want. Again, its only relevent because standard firewall policy is to allow HTTP traffic on port 80. Often other ports are not allowed. This doesnt particularly solve security problems, but its what people do anyway. Are SOAP/XML services similarly restricted by port numbers? It seems like our SOAP/XML services are not similarly restricted by firewalls as what we've done with OpenDAP, but that may be because we are mediating them through SDSC that may be using Port 80. Most servers can use any port you tell them. But they have to use a specific port. Ah -- the glories of computer technology! SOrry i cant give you a simpler answer. David ________________________________ From: John Caron [mailto:address@hidden Sent: Wed 8/23/2006 5:12 PM To: David Maidment Cc: Glenn.Rutledge; 'support-thredds' Subject: Re: NCDC Web Services Im not sure what the issue is, but its not really a server thing. Firewalls live on the client. Generally HTTP traffic on port 80 is let through, and not much else. Tomcat default port is 8080, it can be changed to port 80 is need be, but you cant expect THREDDS servers to do that for the convenience of some site that has some restrictive firewall. A proxy server can be a good solution, but i dont know what the details of use are. David Maidment wrote: Glenn: Yes, please find out what we have to do. This looks like a significant issue to me. John -- could you look at the email trail below and comment on the issues of OpenDAP and firewalls. Is there any difference if we use your netCDF server? David ------------------------------------------------------------------------ *From:* Glenn.Rutledge [mailto:address@hidden *Sent:* Wednesday, August 23, 2006 10:09 AM *To:* David Maidment *Subject:* Re: NCDC Web Services Systems folks would do this...and as I understand it- is not at all uncommon. It's a representation of a service within a firewall I am out of my realm but I do plan to investigate further since NASA had the exact comment you did just last week. I'll let you know more if you like. GR David Maidment wrote the following on 8/23/2006 10:54 AM: Glenn: What is a proxy server? How would I or any other regular user know how to use one of these? David ------------------------------------------------------------------------ *From:* Glenn.Rutledge [mailto:address@hidden *Sent:* Tuesday, August 22, 2006 12:53 PM *To:* David Maidment *Subject:* Re: NCDC Web Services Hi David- A response from OPeNDAP: our code (libdap, libnc-dap, our clients - with the possible exception of the ODC) can all be told to use a proxy server. Open ports to that machine and you're off and running. If someone can use the Internet, they can use our clients and read from your (or any) server. Point these folks toward the documentation on the .dodsrc file. If they still have questions (and I know some of our docs are getting stale), then have them contact address@hidden with their questions (or the newly renamed opendap or opendap-tech emails lists - see www.opendap.org <http://www.opendap.org/> and look under support for information about those). I'm glad you asked me, because this is an important feature of the client-side software! Hope this helps--- Glenn David Maidment wrote the following on 8/21/2006 9:13 PM: Glenn: I have some concerns about how useful OpenDAP is. It seems sensitive to being blocked by firewalls. Our direct calls to NARR that relied on using OpenDAP worked fine >from here at UT but from inside of NSF with all their firewalls in place, the data were blocked, and the same is true when trying to do this >from within ESRI. I think the question of how to do these things is linked up with computer security and that needs to be factored into the debate. David ------------------------------------------------------------------------ *From:* Glenn.Rutledge [mailto:address@hidden *Sent:* Monday, August 21, 2006 9:13 AM *To:* David Maidment *Subject:* Re: NCDC Web Services Thanks David- I agree- but somewhat concerned whether SOAP/REST will have capabilities across the other data forms you cite - although based on your individual needs- it doesn't seem to matter. But for the larger whole- we need to test this out. Glenn David Maidment wrote the following on 8/21/2006 9:47 AM: Glenn: I see observations data as time series as one kind of service, and weather model, remote sensing and Nexrad data as another. For the weather and climate "field" information, some variant of what John Caron is doing seems appropriate. For the observations data, I think we have a fairly good scheme figured out in the work we are doing with CUAHSI. I can see the netCDF style of things for the weather and climate fields. I think the time series stuff could be handled that way but is easier and perhaps more appropriate just to do it in XML via SOAP or REST. What I would like to see developed is a "Weather and Climate Server" so that the data users can go to one system and get into the various information sources without having to go separately to NCDC, Unidata, NCAR, NWS, etc. This is like WaterOneFlow that we are attempting to do in CUAHSI for water observation data. David ________________________________ From: address@hidden [mailto:address@hidden Sent: Fri 8/18/2006 6:39 PM To: David Maidment Subject: Re: RE: NCDC Web Services Hi David, I agree. Here's the note I sent Sharon earlier this week after your request. I want to plan for the long term and work toward the benefit of all. We are meeting on Monday. Don't worry about the lack of the attachment I reference for the moment. Regards, Glenn __________________________________ Sharon et al, I think we need to have a set of meetings to define our plan for exposing NCDC services. I'm not aware of the status of this kind of activity already, but there are several efforts underway within the Center and we're already on a similar track - we just need to solidify our goals- and maybe generate a requirements document from which to work toward. btw- NOAA's GEO-IDE would benefit from just such an activity. Rich's efforts in OGC compliant (GIS) services are the way to go, and NOMADS team have implemented a THREDDS Data Server (TDS) for OGC WCS services and provides for OPeDNAP enabled requests handled under TDS. Of course SteveD's group has advanced even higher levels of standards using SOAP for more generalized services descriptions. There are limitations to generalities with such standards, and not community based standards such as OPeNDAP at the present time. I'm co-PI on a NASA, Unidata, OPeNDAP and GMU effort called ACCESS that has defined and are building just such a "Gateway" using NetCDF files initially- using the COARDS and CF compliant OGC Catalog Services for the Web (CSW). This cross- community effort has many players including OGC, CEOS, GALEON, GO-ESSP and many others. It builds on the work of the Common Data Model and TDS work at Unidata and OGC and International services. There is work to do on my end: Grib/grib2/BUFR issues but we've already made significant headway in this regard and participated in the last GEO demo last May in Beijing. NCDC will be the benefactor of this effort and RSAD will be installing this capability (in test mode) when completed later this year. The attached could be a staring point for these Center-wide discussions, and I suggest that the attached be reviewed and considered for expansion into the center when fully developed. The reason that I'm co-PI on the effort is that NOMADS would install and provide this service for models. It applies to other datasets as well but would need to be adapted for Neal's/Steve's data. Comments? Glenn What David Maidmont seeks of course is a unified NCDC web presence to all of NCDC's resources. ----- Original Message ----- From: David Maidment <address@hidden> <mailto:address@hidden> Date: Friday, August 18, 2006 7:28 pm Subject: RE: NCDC Web Services Rich: There are a lot of questions here that require some reflection. What I want to do in the short term is stand up a web service to NCDC that is as simple as possible that shows that we have a serious commitment to work together. That is why CRN is attractive to me -- its high quality data thus good for research, its not very many stations, so not too difficult to build the catalog even if we do it station by station using the web services that you created (do they access CRN?), and hopefullyyour services will work onto the archive reasonably well. I am trying to create a path that points the way to the future in a methodicalmanner that step by step we figure out what comes next and implement it and we start with some fairly straightforward, confidence building steps that solidify the relationships which already exist on a technical level betweenyou and your colleagues and us. We need to learn how you think about data at NCDC. We need to understand what databases you maintain, how many services you actually offer and which ones we should focus on first, which ones leave to later. Our first priority is access to historical NCDC data archives not near real-time data. I want to understand what datasets you are publishing in netCDF- accessibleform and how those dovetail with what is available from NCAR and Unidata. I don't know enough about NIDIS yet to comment on it in an informed way. Thanks for your feedback. David ________________________________ From: Rich.Baldwin [mailto:address@hidden Sent: Fri 8/18/2006 8:32 AM To: David Maidment Cc: Cedric David; Ilya Zaslavsky; Glenn Rutledge; Sharon Le Duc; Neal Lott Subject: Re: NCDC Web Services Hello David and Ilya, I was glad to meet you all last week. I had a productive meeting with your teams at UTexas and SDSC. It's good to be back from the conferenceand vacation. I have a conflict for the scheduled Tuesday meeting, but I'll see if I can break away. Just in case I can't make it, let me try to clear away some debris. There are 3 main issues to deal with 1) Data Access I appreciate your concern regarding the potential server impact resulting from use of our web services (particularly in light of the NWIS problems). Accessing the ASOS stations within ISD for the last days data through our current web services shouldn't produce an undue impact on our servers. I have made some preliminary tests which bare this out. As I mentioned in our meeting last week, NCDC has begun a major effort to assemble an element inventory (data catalog) for all datasets. The initial inventory will be from the COOP (NWS cooperativeobserver network) dataset; available early 2007. An inventory/catalogfor ISD will follow that effort. There are 3 scenarios that present themselves. a) The existing web services point to a dataset (global hourly) which is updated daily with the last days observations. You all could use theseweb services to extract current data and build a catalog/inventory of observations. Basic station observations of temperature, wind, dew point, pressure will be recorded and observations like precipitation will be present when observed. Among the additional station observations, there will be a good deal of variance day to day, stationto station. This would give you a current archive, and the finalmodifications will be done well before the 10/30 target. A historicalrecord could be built from our planned web service inventory/catalogavailable next year. b) You all could wait for us to build element inventories (catalog) fordatasets of interest and then use web services to those tables to update information. This would probably take around 6 months. c) You could access the archived data through files on our ftp server, create your own catalog/inventory. Then plug into our web services catalog/inventory for updates. During last weeks meeting, I discussed our current global hourly data inventory from which I can implement a web service for access to an inventory of the number of observations for each station by year and month. This would take a day to put together. Note that this doesn't make a distinction of what was observed only that something was observed. 2) NIDIS As I mentioned last week, NCDC is heavily involved in the NIDIS effort (National Integrated Drought Information System), a collaborative effortof serveral agencies and institutions. The scope and requirements for the NIDIS portal are still being sorted out so in that regard... What requirements might CUAHSI find useful in a drought portal? What web service products might NCDC be able to consume from CUAHSI? At this point, I don't believe the pieces are in place for something like watershed flux analysis, but this and similar products would be the goals. 3) NetCDF The resource links you sent were very useful; I have already begun on a mock up for implementation. At our meeting last week, you were satisfied w/ the time line which I laid out concerning netcdf development/implementation (post 10/30). If that has changed, please let me know. Take care, Rich PS. My laptop which has the slides from my presentation is getting a s/w and h/w update (potential exploding battery), I'll send the slides ASAP. David Maidment wrote: Rich: Thanks so much for spending time with us on Thursday at SDSC. I appreciate you presenting to us the work that you've done with web services and I especially appreciate that you have tried to make them CUAHSI style. As you know, I have asked Cedric David to work with you to test your services and keep me informed as to which databases they provide access to and how we can best use what you have created. Could you please send me a copy of the slides that you used last Thursday. I want to study the nomenclature that you employed since its reflective of how things are called at NCDC and I need to understand that. Cedric -- what we need to do is to use these services to create an observations catalog for each of the NCDC databases that NCDC is exposing using these services. I am not sure if this should be done at Texas or at SDSC. Ilya -- what do you think about the above? Rich --- the website with the tutorial on how to access the Unidata NetCDFServer is at: http://www.crwr.utexas.edu/gis/gishydro06/SpaceAndTime/NetCDF/Animating%20netCDF%20Data%20in%20ArcMap.htm> The actual server itself is at: http://motherlode.ucar.edu:8080/thredds/idd/nam_model.html> This is really hard to find if you just search the Unidata web site. I'd like to discuss with you and your colleagues how best to develop a "weather and climate server" that uses time series web services and also netCDF services. Perhaps that is something we can discuss with Sharon LeDuc. David H -- Glenn K. Rutledge Services Team Leader Remote Sensing and Applications Division NOMADS Project Manager National Oceanic and Atmospheric Administration National Climatic Data Center Asheville NC 28801 Phone: (828) 271-4097 Fax: (828) 271-4328 NOMADS: http://nomads.ncdc.noaa.gov/ -- Glenn K. Rutledge Services Team Leader Remote Sensing and Applications Division NOMADS Project Manager National Oceanic and Atmospheric Administration National Climatic Data Center Asheville NC 28801 Phone: (828) 271-4097 Fax: (828) 271-4328 NOMADS: http://nomads.ncdc.noaa.gov/ -- Glenn K. Rutledge Services Team Leader Remote Sensing and Applications Division NOMADS Project Manager National Oceanic and Atmospheric Administration National Climatic Data Center Asheville NC 28801 Phone: (828) 271-4097 Fax: (828) 271-4328 NOMADS: http://nomads.ncdc.noaa.gov/ -- Glenn K. Rutledge Services Team Leader Remote Sensing and Applications Division NOMADS Project Manager National Oceanic and Atmospheric Administration National Climatic Data Center Asheville NC 28801 Phone: (828) 271-4097 Fax: (828) 271-4328 NOMADS: http://nomads.ncdc.noaa.gov/
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.