Among the motivations for developing THREDDS are:
While Unidata's primary focus has always been on providing real-time data and tools for analyzing that data on computer systems in university departments, there has long been a community desire for "seamless" access to retrospecitive data as well. The idea is that researchers, educators, and students in the Unidata community would benefit from being able to access archived data using the same local tools they use for analysis and display of real-time data. As early as 1989, Unidata had articulated a goal to "maximize the effectiveness of Unidata systems for analyzing historical data from the major national archive centers." As detailed in later sections, hundreds of data provider sites (NCAR, NCDC, and FNMOC among them) have implemented THREDDS technologies to make their data available to users with THREDDS-enabled applications programs such as the Unidata Integrated Data Viewer (IDV).
In addition to the collections in data centers, many Unidata sites were establishing their own collections and sought to share their own data in a more convenient fashion than the "needdata" email list. The Unidata 2003 proposal which was actually written in 1997 includes a goal of “utilizing the aggregate data holdings of all Unidata sites as a common resource, accessed via the Internet." This not only allows other sites to gain access to data they may have missed, but allows some institutions to access the data without having their own IDD systems as noted in the following paragraph.
Unidata's IDD serves large number and wide range of departments, it was also recognized that there were departments and institutions that simply did not need and could not support the full power of the IDD "firehose." In some cases, a professor might only teach the meteorology course every other semester. In other cases, the need to administer a Unix computer in order to become a full-blown node on the Unidata IDD was a substantial barrier. Client/server technologies provided a simpler "buy in" that allows participation by wider range of institutions. As an example, California University of Pennsylvania doesn't have an LDM/IDD feed, but uses the Unidata IDV to access data served on remote sites. The IDV display below renders one type of data available via THREDDS services. In this case, the GFS numerical weather prediction model output of mean sea level pressure (as color-shaded image and contour lines) and 50 m/s wind speed isosurfaces showing the jet streams.
IDV Globe Display of GFS numerical weather prediction model output
Finally there was a realization that the geosciences were moving toward more interdisciplinary research and education programs where an Earth system approach emphasized the difficult research topics at the boundaries of the traditional disciplines. It quickly became apparent that these disciplines had widely different data systems and, in fact, had fundamentally different ways of thinking about their datsets. In particular, the solid Earth and hydrology communities made extensive use of GIS (Geographic Information Systems) which are built upon relational database technology. In order to facilititate data sharing with the hydrology, coastal oceans, and human impacts communities, it would be necessary to establish interoperability with their data systems. To gain a sense of power afforded by integrating the tools of both communities, the ESRI arcGIS screen shot below shows demographic information as an overlay on real-time, high resolution, local forecast models being run in regions of high precipitation probability and served via THREDDS technologies.
Figure 4: ArcMap Rendering Schools and Hospitals in Region of High Forecast Precipitation
The forecast was generated as part of the LEAD (Linked Environments for Atmospheric Discovery) Large ITR project by the WRF (Weather Research Forecast) model on a LEAD node at Unidata. ArcMap tools were used to determine which schools and hospitals were in the areas where the high resolution WRF forecast storm total precipitation in excess of 7 inches. This illustrates the power of combining real-time (hourly) weather forecast data with the database tools of the GIS community. Similar overlays could be done with GIS data representing watershed and drainage basins to combine the atmospheric predictions with hydrological measurements.
All these motivations have come into play during the evolution of THREDDS.
THREDDS (Thematic Real-time Environmental Distributed Data Services) is middleware to bridge the gap between data usere and remote data providers. The goal is to simplify the discovery and use of scientific data and to allow scientific publications and educational materials to reference scientific data.
THREDDS’ initial focus was to allow data users to find datasets that are pertinent to their specific education and research needs, access the data, and use them without necessarily downloading the entire file to their local system. To achieve this, we needed a way for data providers to publish lists of what data are available and to describe their data to enable discovery and use.
Catalogs are the heart of the THREDDS concept. They are XML documents that describe on-line datasets. Catalogs can contain arbitrary metadata, and we have also defined a standard set of metadata to bridge to discovery centers like GCMD, DLESE and NSDL.
The THREDDS Catalog Generator produces THREDDS catalogs by scanning or crawling one or more local or remote dataset collections. Catalogs can be generated periodically or on demand, using configuration files that control what directories get scanned, and how the catalogs are created.
THREDDS was initially funded as a National Science Digital Library (NSDL) Collections project. The idea was to develop a technology that would complement the client/server approach to data access that had been developed by Distributed Oceanographic Data System (DODS at that time, now OPeNDAP). OPeNDAP is an internet client/server protocol that allows client application programs (like IDL, Matlab and the IDV) to access datasets on remote servers as if the datasets were on the local disk of the workstation. Where a client application program would normally take the name of a local file, the user simply has to supply a URL pointing to a dataset on a remote OPeNDAP server. The client program then operates on the remote dataset as if it were a local file.
At about the same time, the McIDAS development team at the University of Wisconsin Madison's SSEC (Space Science and Engineering Center) were augmenting the McIDAS system with a client/server interface called ADDE (Abstract Data Distribution Environment), a remote data access protocol originally developed for geolocated data that communicates requests from client applications to servers, which then return data objects back to the client. Since the both the client and server interfaces were part of the McIDAS distribution, it was easy for Unidata sites to simply turn on the McIDAS ADDE server and make their datasets available to sites with the McIDAS client software. Shortly after this version of McIDAS was distributed by Unidata, several dozen Unidata sites became data servers via McIDAS ADDE.
Thus remote access protocols such as OPeNDAP and ADDE made it possible for data provider sites to make their datasets available via the internet to users at other sites whose client software were instrumented to use the protocol for accessing the data from remote machines. The general idea was that, instead of specifying the name file containing a dataset on the local network the user of the client software could specify an internet URL that represented on a remote server machine somewhere on the internet. The advent of these protocols was a significant step toward giving Unidata users "seamless access to retrospective data" as well as the ability to "share their own data."
This worked out great for people who happened to know the URL names of datasets on remote servers. However, there was still a problem in that there was no internet equivalent of the local network file system of folders and directories that enables users to find locally stored datasets. These client/server systems needed some form of cataloging system that allowed users to browse data collections on remote servers.
The initial role of THREDDS was to supply tools that would create catalogs
of, and provide client access to, the collections of data on remote
servers. These catalogs are machine readable lists of datasets
available on OPeNDAP servers with enough user-readable metadata to allow
users of THREDDS-enabled clients to browse catalogs of remote datasets just
as they browse the file system on their own workstations. The
inventory level catalogs also supplied the use metadata required to enable
client software to do reasonable things with the data once it was
accessed. Early on in the project, it was recognized that the
the simple inventory list catalogs were not sufficient. In fact, a
hierarchy of catalogs is needed so that groups of inventories could be
catalogued at a higher level. For example, all the inventory catalogs
for output from NCEP forecast models can be grouped into an NCEP model
catalog. LEAD model output can be grouped into a LEAD model output
catalog, etc. These catalogs of catalogs can be grouped at higher
levels. To get a sense of how this works on practice, you can browse
the THREDDS
Top Level Catalog on Motherlode:
http://motherlode.ucar.edu:8080/thredds/catalog.html
Below is a screenshot of the top level THREDDS catalog on the Unidata motherlode
server.
Similar THREDDS catalogs are available at many other sites, including the NCDC NOMADS (NOAA National Operational Model Archive & Distribution System)
site
http://nomads.ncdc.noaa.gov:8085/thredds/catalog.html
Fleet Numerical Meteorlogocial and Oceanographic Data Center's US GODAE (Global Ocean Data Assimilation Experiment) site
and the NCAR Community Data Portal
Another way to look at these catalogs is that they are textual documents
that have special pointers to binary datasets that can be accessed via
client application software using special protocols like OPeNDAP. Two
important THREDDS capabilities relate to this characteristic.
NetCDF-Java 2.2 is a 100% Java library which includes a prototype implementation of the Common Data Model (CDM ). This netCDF API supports access several file formats:
and provides access to THREDDS catalogs.
Common Data Model (data access layer) UML Diagram
The overall GALEON target is this general goal of interoperability via standards-based web services interfaces. But there is one rather more specific objective that involves using these interfaces as the basis for a gateway between traditional GIS applications and datasets available in existing servers in the FES community. These servers which number in the hundreds are based on a set of client-server protocols that have evolved in the FES community over the last decade. The basic building blocks are NetCDF, OPeNDAP, ADDE, and THREDDS technologies. But there are other services built on these, for example LAS, GDS, and INGRID. There are already several hundred of these servers making a wide-variety and large volume of data available to existing client applications. So a key aim of GALEON is to expand the usefulness of these servers by adding a standards-based interface to provide a gateway so that WCS clients can access the datasets.
The diagram below is a schematic of this gateway implementation as it was envisioned at the time GALEON was initiated. Since that time, development work has integrated the underlying THREDDS/OPenDAP services into a package called the THREDDS Data Server (TDS) which has a rudimentary WCS interface built in.
Initial Concept of WCS-interface as a Gateway to Existing FES Services
The GALEON experiments have resulted in many recommended changes to OGC WCS specification. Among the most important to the users of atmospheric and oceanographic netCDF datasets are:
The THREDDS Data Server is implemented in 100% Java, and is contained in a
single war file, which allows very easy installation into the open-source
Tomcat web server. This means
that users can implement the entire package on nearly any computing system. The OAI Harvester refers to a system which can gather metadata using a standard interface defined by the Open Archives Initiative (OAI) -- another example of open standards that enable different groups to collaborate with others by implementing standard web interfaces. In this case the OAI protocol allows data discovery sites to gather metadata from TDS sites.
http://www.unidata.ucar.edu/projects/THREDDS/GALEON/Reports/RelatedTechnologies.html
An overview of GALEON (Geo-interface for Air, Land, Environmental, Oceans NetCDF) is also available
http://www.unidata.ucar.edu/projects/THREDDS/GALEON/Reports/GALEONoverview.htm