•         advanced

Goals

The THREDDS initiative's overarching goal is to provide students, educators, and researchers with coherent access to a large collection of real-time and archived data sets from a variety of environmental data sources at a number of distributed server sites. The data sets will be conveniently accessible from an arsenal of data analysis and display tools ranging from "thin" web-based clients that allow the learner to browse and manipulate data using the processing power on the servers, to "thick" clients that harness the computing power and flexibility of the user's own workstation while accessing data from a collection of remote servers. THREDDS (THematic Real-time Environmental Distributed Data Services) will provide real-time data delivery via reliable, event-driven "push" technology as well as transparent access to data sets using "pull" systems that make it possible to access data on remote servers as if they were on the users' own computer system. The system will be built on a set of components and data servers, most of which either are already in operation or are under development.

With the resulting system, the user will be able to find Earth science data of interest on a collection of servers that have a multidisciplinary set of themes, e.g., seismic data, radar observations, marine data, oceanographic observations, satellite imagery, and climate data, among others. Several ports of entry will be available. For example, the user will be able to point a Web browser at the central DLESE site to search for data from that site. Alternatively, it will be possible to browse the data on all the servers from any one of them. Furthermore, this work will result in prototype applications that allow the user to search for relevant data from within analysis and display applications running on local workstations.

Integrating Existing Components

Many of the system's technological components are already in place on a number of servers or are under development, independent of this proposal:

• The Distributed Oceanographic Data System (DODS) and the Abstract Data Distribution Environment (ADDE) protocols for client/server data access. These have been developed at the University of Rhode Island and the Space Science and Engineering Center (SSEC) at the University of Wisconsin-Madison respectively. These transport protocols have already been implemented on a large number of data servers nationwide. We propose to build on the current deployment, add to it, and enhance it with a coordinated discovery system. Adding the GrADS (Gridded Analysis and Display System) capability to the DODS server gives the user additional data manipulation tools on the server.
• The Live Access Server (LAS) for browsing, analysis, and display of data via the Web. LAS was developed at the Pacific Marine Environment Lab and is utilized on many data servers using DODS as a transport protocol. Again, THREDDS will augment and coordinate already-completed work.
• A suite of Earth system data analysis and display client applications programs based on a generalized version of the Unidata MetApps, based on Java and VisAD (a component library for interactive and collaborative visualization and analysis of numerical data). The client data access components developed for MetApps will be made available to other data analysis and visualization tool developers:
  • WXWise applets at the University of Wisconsin,
  • the Virtual Exploratorium, U of Illinois, West Chester State, NCAR, DLESE
  • EDMI (Earth Data Multimdia Instrument) from New Media Studio
  • WorldWatcher from Northwestern University
• tools will also be adapted as data access, analysis and display facilities. Our goal is to demonstrate that the data discovery and usage metadata systems are usable through tools already under development.
• The Internet Data Distribution (IDD) system based on the Local Data Manager (LDM) for automated real-time data delivery to the distributed servers. The LDM, developed at the Unidata Program Center, is in use on nearly 200 computer systems at the 90 institutions comprising the IDD.

The Metadata Challenge

The technological core of the proposal, the crucial component that has yet to be developed, is a system for adding the semantic description of scientific data sets necessary for data manipulation and discovery. It must interoperate with data providers, data servers, data clients, catalog servers, discovery systems, and other middleware components. Investigators will select key scientific data sets and semantic descriptions developed for an end-to-end demonstration of the utility of this approach. Unidata staff will work closely with the Digital Library for Earth System Education (DLESE) to ensure that the resulting metadata system will interoperate effectively with the National SMETE Digital Library (NSDL). Other probable partners are:

• The Earth System Markup Language (ESML) at the University of Alabama Huntsville,
• The DIstributed MEtadata System (DIMES) at George Mason University,
• The aggregation data catalog that is part of DODS, and
• The Earth System Grid initiative at Argonne National Laboratories and NCAR.

The first three initiatives are part of the Earth System Information Partners (ESIP) Federation. The Earth Ssytem Grid is sponsored by the DOE and focuses more on the needs of the high performance computing community. Thus the resulting THREDDS metadata facility, insofar as possible, will interoperate with collections (NASA data for the most part) of the ESIP Federation and those of the Earth System Grid.

This work will contribute to several recommendations in the National Science Board (NSB) report "Environmental Science and Engineering for the 21st Century," however we view it as a major step toward "a coordinated, digital, environmental network" in recommendation 9 in the NSB report.

Data Server Partners

At this time, we are engaging the following institutions as THREDDS sites. The sites, themes noted parenthetically, are:

• NCDC, the National Climatic Data Center (climate)
• LDEO, the Lamont Doherty Earth Observatory (climate, oceanographic)
• PMEL, the Pacific Marine Environment Laboratory (oceanographic, marine)
• NCAR, the National Center for Atmospheric Research (atmospheric, oceanographic)
• CDC, the Climate Diagnostic Center (climate)
• University of Wisconsin SSEC (satellite)
• University of Oklahoma (radar)
• GMU/COLA, George Mason University/Center for Oceans Land Atmosphere (hydrologic)
• IRIS DMC, Incorporated Research Institutes for Seismology Data Management Center (seismic)
• NGDC, National Geophysical Data Center (geophysical)
• FNMOC (Fleet Numerical Meteorological and Oceanographic Center)

Some of the sites already have many of the THREDDS components in operation: DODS and/or ADDE for client/server access to the data; LAS for browsing, analyzing and visualizing the data on the server; and LDM/IDD for updating the data holdings in real-time. Others only have some of the components installed, but are committed to installing the others. A few are committed to becoming THREDDS sites, but have not yet installed a server or any of the THREDDS components. At this point, none of the sites has a metadata system that interoperates with DLESE or with all the others sites. So, in addition to developing the interoperable metadata system, a substantial segment of the proposed work over the two year duration of the proposal will be to provide training and consultation required to get all the sites up and running with all the components necessary to make them full-blown, interoperable THREDDS servers.

Commitments from Server Sites Partners

The THREDDS sites themselves, with support from Unidata through this grant, are committed to:

• Supplying the hardware and system administration required to run the THREDDS servers;
• Providing access to data sets characteristic of the data server at their site via DODS and/or ADDE client/server protocols in addition to the traditional FTP, tapes, and so on;
• Providing browser/server analysis access to certain data sets via Web-based thin clients such as the Live Access Server (LAS) where appropriate;
• Working with Unidata to incorporate systems for expanded metadata to make it easier for users to find needed data sets and to use them once found;
• Using LDM/IDD technology (where appropriate) to make real-time data available on the server.
  << These commitments are in from some sites, but we need similar statements from the others.>>

Educational Users from a Broad Range of Institutions

Unidata software developers are working collaboratively with representatives of the user community to develop a new, platform-independent set of environmental data analysis and display applications programs called MetApps. Part of this proposed work will be to ensure that the visualization tools are general enough for use in an Earth system education environment. Developers are using an online collaboration tool (based on D3E, the Distributed Document Discourse Environment) to facilitate interactions with the user community. For this effort, the D3E community will be expanded to include representatives of a broader range of the Earth system sciences as well as faculty from two-year colleges and teacher training institutions. << Ben follows up with Susan Wood and Alexander Sadowski in this one.>>

We also plan to work with the WXWise and Virtual Exploratorium developers to ensure that their applets and educational application programs can access, analyze, and display real-time data from the THREDDS servers. << Ben follows up with Tom W., Steve A., Raj P. on this one.>>

Deliverables Summary

The THREDDS server sites provide the server hardware and staff needed for ongoing maintenance and administration of the system as noted earlier:

With resources from this proposal, the UPC will provide:

• A server-side metadata inventory to enable users to find and use data on the collection of THREDDS servers (coordinating with the DIMES, ESML, DODS, OpenGIS Web Mapping Server, and Earth System Grid initiatives);
• A client-side metadata tool providing access to data housed on the collection of servers;
• A metadata system demonstrably interoperable with the DLESE discovery system;
• The training, consultation, and support needed by server sites to install the THREDDS components on their servers;
• A set of client applications which demonstrate the use of THREDDS metadata (in conjunction with the MetApps, WeatherWise and Virtual Exploratorium initiatives);
• A complete end-to-end data discovery, analysis and display system that has been tested by selected representatives of a wide range of undergraduate institutions (including two-year colleges and teacher training institutions) to ensure the system is useful in those environments.

t

Contact Us   •   Site Map   •   Search   •   Terms and Conditions   •   Privacy Policy   •   Participation Policy

Unidata is a member of the UCAR Office of Programs, is managed by the University Corporation for Atmospheric Research, and is sponsored by the National Science Foundation. P.O. Box 3000   •   Boulder, CO 80307-3000 USA   •   Tel: 303-497-8643   •   Fax: 303-497-8690