CommuniteE-Letter Volume 5, Number 9, January 2010
  previous newlsetter navigation next newlsetter navigation

GRIDSPEC, LibC--The NetCDF CF Library

AMS 90th Annual Meeting
The Time Series Data Server
  Past Issues

AMS 90th Annual Meeting

The AMS annual meeting heads back to Atlanta this year and so do several Unidata staff members. We've compiled a table of sessions and presentations related to Unidata, and we invite you to browse that for posters and presentations that might interest to you.

The IIPS session featuring N-AWIPS and AWIPS II presentations is Tuesday January 18 from 3:30-5:30 in room B218. Based on interactions within the Unidata community during the previous year, we believe that this will be of significant interest to many of you.

Unidata will be providing live data via the IDD to the Raytheon/NOAA booth (#628) for use in demonstrating AWIPS II capabilities.

In addition, we call your attention to the Town Hall Meeting on Data Stewardship which will be Monday, 18 January, from 12:15 - 1:15 p.m.; Room B2112 This Town Hall Meeting will discuss the Data Stewardship Prospectus that has been developed by the AMS Ad Hoc Committee. The prospectus discusses AMS's possible role in data stewardship, and it includes statements of need and an overarching recommendation to form a standing Data Stewardship Committee under the Scientific and Technological Activities Commission. This meeting will provide a forum for gathering additional feedback from the AMS membership on the prospectus, the scope of activities to be undertaken by the standing committee, and the views of the members with regard to the AMS's role in data stewardship.

We look forward to seeing many of you in Atlanta.

The Time Series Data Server

By Doug Lindholm, Laboratory for Atmospheric and Space Physics, University of Colorado, Boulder

The Time Series Data Server (TSDS) is part of an effort within the Heliophysics community. Working collaboratively with Bob Weigel at George Mason University, we designed and developed an OPeNDAP server that specializes in serving time series data. The TSDS is built on Java Servlet technology with a flexible, modular framework that allows for easy customization of input and output data.

OPeNDAP code is necessarily complex and can be inflexible when a data provider wants to alter its behavior to suit a specific function. It can also be difficult to install and maintain, while the TSDS  is a completely independent implementation of the DAP 2.0 specification. Implemented as a Java Servlet as noted above, it is compiled into a war file that can simply be dropped into a Servlet container like Tomcat. Its modular design also simplifies adding functionality.

The core of the TSDS is Unidata's Common Data Model (CDM). On the back end, each data set to be served (be it from a file, relational database, or service) is described by a netCDF markup language (NcML) file. The server uses netCDF Java to read the data into the CDM via the NcML. Some generalized IO Service Provider implementations have been provided and others can be customized so data provider can serve their existing data without changing their existing data formats or management infrastructure. Filter and Writer plug-ins enable data providers to give their users more selection and output options.
The TSDS starts with the simplifying assumption that every data set is a Sequence that represents Variables that are a function of time. Those Variables can be scalars (e.g. temperature), Structures (e.g. magnetic field vector), or Sequences (e.g. spectrum). With this narrowed scope and a rapidly iterative agile development process, a simple, elegant design has emerged.

While some current OpeNDAP server implementations are tied to a data file centric model, the TSDS emphasizes the idea of virtual data sets.

The TSDS defines Java interfaces for Writers and Functions. A Java properties file maps the OpeNDAP URL's dataset extension to an implementation of the Writer that produces the requested output, and maps function names in the constraint expression to implementations of the Function interface.

The Writer interface essentially is a write(dataset).  If a data provider wants to serve an unsupported format, they simply have to implement the code that writes the data in the desired format given a basic double array for each time sample. These new Writer implementations can then be contributed to the user community through the SourceForge web site so that others may use them as plugins.

The Function similarly acts on a dataset. Existing and customized Functions can be plugged in by the provider to give users the option to request data manipulation on the server before downloading.

On the other side of the fence, NcML exposes the data to be served in terms of the CDM. I have created generic ASCII, flat binary, and relational database IOServiceProvider implementations that data providers can easily reuse by mapping columns to variables and such and providing other bits of metadata. The requirements are small, but the possibilities (e.g. enhanced metadata) are large. I have an abstract  IO ServiceProvider implementation that makes it relatively easy for data providers to write a little code to read their native data into scalar, Structure, or Sequence Variables. Again, these can be shared as plugins and reused within the community.

The TSDS source code and easily deployable war file and plug-ins soon will be available as Open Source..


A cubed-sphere with embedded nests

To advance the state of the art in climate science, Unidata is happy to participate in the development of infrastructure to enable more accurate distribution of gridded data sets such as those produced by climate models. The SW originally developed at GFDL has been included in the libCF library.

GRIDSPEC's goal is to allow producers of gridded data sets to exactly specify their grids, in a way that allows for conservative regridding.

The GRIDSPEC code in libcf is based on the code of Zhi Liang, at GFDL, who is now directly contributing to libcf development, with an account in the cvs repository.

All of this GRIDSPEC work has prompted NOAA's Global Interoperability Program to send some support funding to Unidata, which is very welcome here.

Follow Us:

WAYS YOU CAN GET US: RSS | Newsletters and Alerts |
Facebook Twitter


Please send comments to
The CommuniteE-letter is produced by editor, Jo Hansen, and production manager, Tina Campbell
printer friendly icon NSF Logo    UCAR Logo