NetCDF

Status Report: October 2010 - April 2011

Russ Rew, Ed Hartnett, Dennis Heimbigner

Strategic Focus Areas

The netCDF group's work supports the following Unidata funding proposal focus areas:

  1. Broadening participation and expanding community services
    by providing effective support to a growing netCDF user community, including users of satellite products, GIS, climate, and ocean data.
  2. Advancing data services
    by developing and supporting a key infrastructure element for data providers and users of climate, atmosphere, and ocean data.
  3. Developing and deploying useful tools
    by engineering general-purpose software and utilities supporting the analysis, visualization, and management of model output and observational data.
  4. Enhancing user support services
    by maintaining comprehensive documentation, providing annual workshops, responding to support questions, maintaining example programs and files, and keeping online FAQs, best practices, and the netCDF web site up to date.
  5. Providing leadership in cyberinfrastructure
    by seeking endorsement of standards bodies for netCDF formats and for Climate and Forecast (CF) model metadata conventions based on netCDF.
  6. Promoting diversity by expanding opportunities
    using an open source development process, accepting user contributions, and democratizing access to valuable data archives.

Activities Since the Last Status Report

Workshops

We presented two Unidata netCDF workshops on October 28-29, 2010, in the Damon room of NCAR's Mesa Lab. In addition to presentations by Unidata staff Ed, Russ, Dennis, John, Ethan, Yuan, and Jeff, we also had invited presentations on NCAR Climate Data by Gary Strand (NCAR CGD), HDF5 for netCDF Developers by Quincey Kozial (The HDF Group), and GRIDSPEC by Alex Pletzer (Tech-X). With over fifty participants for both days, things went surprisingly well even though Four Mile Canyon caught fire during the workshop, requiring Ed to evacuate his family between two of the presentations!

The workshops were preceded by a useful one-day "Data Summit" with HDF5 developers as well as grad students and faculty with the Damasc research project at UCSC.


Releases

After several beta test releases, netCDF version 4.1.2 of the C-based software was announced on March 29, 2011. The most important changes in this release were:

  • changes for building cleanly on Windows;
  • building shared libraries by default, instead of static libraries;
  • greatly speeding up opening of large or complex netCDF-4 files, writing a large number of attributes, or lookups by name in files with lots of variables or dimensions;
  • adding security authorization to the C OPeNDAP client for Earth System Grid use and for standard HTTP authorization using passwords; and
  • adding useful compression and chunking capabilities to the nccopy utility.

Windows DLLs are now being built as part of the daily snapshot build/test cycle; separate Visual Studio build files are no longer required. Testing of the resulting DLLs is underway.

The 4.1.2 release also included a number of bug fixes as well as improvements in portability, extensibility, maintainability, and additional new features described in the Release Notes.


Standards progress

A new web page tracks standards endorsements related to netCDF. The most recent is an OGC endorsement for use netCDF as an encoding standard, which is complete except for the imminent press release.


Presentations

Ed coauthored an AGU poster on GRIDSPEC/libCF. Russ presented an AGU talk on "Advances in the NetCDF Data Model, Format, and Software". Russ also accepted an invite to Trieste, Italy, to present several talks on scientific data management and netCDF at an event hosted by the International Centre for Theoretical Physics. The Trieste meeting unfortunately conflicted with Russ's participation in this Users Committee meeting, but he claims this is the first Users Committee meeting he's missed since 1948.

Planned Activities

Ongoing Activities

We plan to continue the following activities:

  • Respond to C- and Fortran-based netCDF user questions and run netCDF workshops.
  • Incorporate successful features of netCDF-Java into C-based libraries.
  • Improve support for evolving Climate and Forecast (CF) conventions.
  • Oversee development of GRIDSPEC in libCF as part of a collaboration with NOAA's Global Interoperability Project, PCMDI, GFDL, and Tech-X, anticipating that this will become part of the CF conventions.
  • Deal with needs of a growing user community for representing observational data, satellite products, and geoinformatics data.
  • Add support for additional security authorization mechanisms as needed.

New Activities

We plan to undertake the following:

  • Create new netCDF-4 test file collection to support netCDF-4 tool developers.
  • Refactor netCDF Fortran-90 API to use Fortran-2003 C-interoperability layer.
  • Convert documentation from Texinfo to Doxygen.
  • Add support for additional remote access protocols.

Relevant Metrics

During the last year, there were about [TODO] downloads from [TODO] countries of the C-based netCDF software from Unidata, in addition to downloads from mirror sites, package management systems, and incorporation into other software packages. More detailed metrics are available.

Other metrics that may be useful include:

  • Number of open-source software packages that can use netCDF data: 76
  • Number of commercial or licensed software packages that can use netCDF data: 21
  • Number of Google hits for "netcdf": 667,000
  • Number of Google images for "netcdf": 20,300
  • Number of Google scholar entries for "netcdf": 6,920
  • Number of blog mentions of netCDF: 9,940
  • Google count of number of books containing the term "netcdf": 2,820
  • Amazon count of number of books relevant to the subject "netcdf": 19
  • Number of videos tagged as netCDF-relevant: 16