Work in maintaining, supporting, and developing the netCDF data model and software is associated with Endeavor 6: Improved scientific data access infrastructure from the Unidata 2008 proposal. NetCDF has become a key infrastructure element for data providers and users of oceanographic and atmospheric science data, as well as data in other geosciences.
Recent netCDF development, both at Unidata and at other institutions, aims at generalizing the netCDF data model, improving interoperability with other representations for scientific data, making the netCDF interface more suitable for use on high-end parallel platforms with high-resolution models, and providing netCDF software on a wider range of platforms.
In October 2005, we hosted the NCO (netCDF Operators) developers Charlie Zender and Harry Mangalam to collaborate and discuss parallelism, benchmarks, OPeNDAP compatibility with netCDF-4, realistic test files, and upgrading the C++ interface.
NetCDF-4 development has continued, with six alpha releases of netCDF-4.0 improving testing, support for new platforms and compilers, performance, functionality, and documentation. All planned features are now supported in netCDF-4.0. We added significant testing of parallel I/O in netCDF-4, and found no bugs, which was heartening. The NCO developers have successfully built and tested NCO using the last few alpha releases. New functionality includes variable-length types, enumerations, per variable compression, and netCDF-4 support in the Fortran interfaces. An alpha version of netCDF 4.1 implements the Common Data Model coordinate system layer.
NetCDF-4 depends on HDF5 1.8, now tentatively scheduled for release in October 2006, assuming there are no unanticipated problems. The surprising delay in HDF5 1.8 development has been an obstacle to making netCDF-4 functionality available to developers and ultimately to users. Some of the delay may be the result of problems in transition from NCSA to the new HDF Group nonprofit that is responsible for the continued development and support of HDF.
Since the last status report, we tested and released netCDF 3.6.1 and 3.6.2, with continued improvements to the netCDF configure and build system, fixes for some file size problems, and a new XML output option for the ncdump utility. Documentation enhancements included a new netCDF tutorial.
Work continued on developing the Common Data Model and version 2.2 of the Netcdf-Java library that implements it, with several stable releases fixing bugs and adding various improvements and support for new formats, including HDF5 and GRIB, through netCDF interfaces. The latest netCDF-Java software supports an updated version of NcML (netCDF Markup Language). NcML is an XML representation of netCDF metadata that can also be used to create netCDF files, to create "virtual" netCDF datasets, to add or modify metadata without modifying the underlying data, and to handle aggregation.
Ed Hartnett, the primary developer for both netCDF-4 and significant improvements to netCDF-3, accepted a position working in NCAR SCD on the Earth Systems Modeling Framework (ESMF) project, beginning April 10. Unidata has negotiated a joint appointment for 25% of Ed's time until netCDF-4 is officially released. At some point we will be recruiting a new netCDF developer to fill Ed's large shoes, although that will be difficult.
Russ has recently had a diminished role in netCDF development and support due to increasing involvement with NOAA and IOOS activities.
We will be mentoring a SOARS student this summer to design and implement some improvements to the netCDF C++ interface.
At the December AGU meeting, Russ presented a poster on "Use of an Enhanced NetCDF Data Model and Interface for Scientific Data Access and Sharing". Russ, Ed, and John also wrote a paper on recommendations for using netCDF-4 for data providers and developers, delivered an associated presentation at the AMS meeting in Atlanta, and Russ is coauthor on a white paper, Maintaining and Advancing the CF Standard for Earth System Science Community Data proposing mechanisms for supporting recommended conventions for netCDF data.