CADIS Report: October 2008

Mohan Ramamurthy and Jeff Weber

To maintain a vibrant program, the UPC, from time to time, participates in certain projects that are tied to its overall mission but funded separately from the core program.  Such synergistic activities are both essential and complementary to the core effort, and both the Policy Committee and NSF encourage them. Such projects have played a vital role in advancing the program in new directions, creating new capabilities for the community, enhancing interoperability of Unidata software, providing new datasets to the community, and entraining and diffusing innovative ideas and technologies into the community.

The Cooperative Arctic Data and Information Service (CADIS) is such a project, supporting the Arctic Observing Network (AON) community. AON is an NSF Office of Polar Programs funded initiative for the International Polar Year (IPY) to improve observational capabilities in the Arctic and leave a long-term legacy for the benefit of science and society. AON data will contribute to scientific research leading to (1) increased knowledge and understanding of the regional and global causes and consequences of present-day environmental Arctic Change, (2) scenarios for and prediction of the course of future Arctic Change and its regional and global consequences, and (3) the development of adaptive responses to Arctic Change.


CADIS is intended as a development project to bring community standards, data submission, archival and stewardship expertise, analysis and visualization tools, and vision to develop a comprehensive data management capability to support AON and other IPY-related projects.

The CADIS team is made up of 23 people from NCAR (EOL and CISL), Unidata, and NSIDC.  Additional information about the CADIS project and data provided by CADIS can be found at:

Unidata is providing middleware (THREDDS), data transport (LDM) for real-time AON data transmission, and analysis and visualization capabilities (IDV). Unidata's Network Common Data Form with CF conventions is a strong contender for use as the "compatible format," so Unidata is assisting in format conversion to netCDF.

The main development areas and accomplishments to date pertaining to Unidata’s work are described below:


Metadata: members from all CADIS partners institutions participated in a metadata working group, which was tasked with identifying the requirements for capturing all information relevant to discover, describe and access data for the Arctic domain. The result of this effort was the definition of a CADIS-specific metadata profile, which is based on the IPY guidelines, and includes additional elements either to meet the specific requirements of the CADIS project, or to facilitate export to other schemas such as ISO and DIF. The CADIS metadata profile is meant as a specification of the general THREDDS schema, which is used as container and interoperability framework.


Metadata Authoring Application: CISL and EOL developers have been collaborating towards the creation of a web-based metadata authoring tool, and its integration into the current CDP publication infrastructure, which is meant to be used by CADIS PIs to capture and share the information to describe their data holdings. A first prototype of this application has been completed and will be released to the PIs for feedback in early 2008.


Prototype CADIS Web Portal: a CADIS-specific area has been established within the existing CDP web portal in order to prototype hierarchical browsing of CADIS data collections, data access, and integration with the metadata authoring tool. This work included the definition of science-based categories to organize the data holdings, as well as branding of the CDP user interface to expose a CADIS-specific look-and-feel.


Dataset format translation: Unidata developers have provided data format translation between native formats such as GRIB, EASE Grid, and Office Note X to netCDF so that data sets in those formats can be accessed via the CADIS portal. Thus far, six candidate datasets have been identified. Unidata tools have been applied to perform those translations confirming the efficiency of this method to standardize CADIS data.


Data Visualization: As an Interim measure, EOL has implemented a GIS Mapserver capability for locating all AON measurement locations on a polar project map. This is the first and modest step to providing interactive perusal and access via the CADIS Portal. Unidata has been working to develop visualizations of those translated datasets using the IDV. IDV will be one of the tools available to users for browsing CADIS data in the future. VETS engineers have engaged in exploratory work aimed at providing visualization and sub-setting capabilities for CADIS data, based on integration of community standard applications like LAS (Live Access Server from NOAA/PMEL) and THREDDS Data Server from Unidata. A few examples of CADIS datasets (either gridded or in-situ data) have been successfully configured and visualized, thus showing the long-term viability of this approach. As a precondition for data access, gridded datasets had to be converted from GRIB format to NetCDF, and in-situ data ingested from ASCII files into a relational database. Unidata staff are working to have example visualizations of all six types of data (mentioned above), using the IDV


Real time data ingest: Unidata has also been working with the CADIS team and in consultation a couple of the AON PIs to determine the need and process for setting up a real-time data distribution system using the LDM. As the need arises, Unidata is ready to help AON PIs to install the LDM to move real-time data from Arctic observatory sites to the CADIS system in Boulder and to other users in the community.


Current Activities

  • As the data portal gets populated with more data, the UPC continues to aid in providing visualization when data are compatible with the IDV and providing guidance for data use with other, non-Unidata clients.
  • Continue to work with the highly heterogeneous data and provide tools/documentation to help transform .xls, CSV, data into netCDF-CF files. This will include efforts by both the THREDDS and IDV teams and should provide added functionality to both projects for the Unidata community.
  • Investigate how IDV "bundle technology" and RAMADDA can be incorporated as "services" into the new data portal/gateway.