NetCDF FAQ
NetCDF (network Common Data Form) is a set of interfaces for
array-oriented data access and a freely-distributed collection of
data access libraries for C, Fortran, C++, Java, and other
languages. The netCDF libraries support a machine-independent
format for representing scientific data. Together, the interfaces,
libraries, and format support the creation, access, and sharing of
scientific data.
NetCDF data is:
- Self-Describing. A netCDF file includes information
about the data it contains.
- Portable. A netCDF file can be accessed by computers
with different ways of storing integers, characters, and
floating-point numbers.
- Direct-access. A small subset of a large dataset may
be accessed efficiently, without first reading through all the
preceding data.
- Appendable. Data may be appended to a properly
structured netCDF file without copying the dataset or redefining
its structure.
- Sharable. One writer and multiple readers may
simultaneously access the same netCDF file.
- Archivable. Access to all earlier forms of netCDF data
will be supported by current and future versions of the
software.
The netCDF software was developed by Glenn Davis, Russ Rew,
Steve Emmerson, John Caron, Harvey Davies, and Ed Hartnett at the
Unidata Program Center in Boulder, Colorado, with contributions from many other
netCDF users.
The latest source distribution, which includes the C, Fortran,
and C++ interfaces, is available for downloading as a compressed tar
file or a gzipped tar file.
Installation instructions are available with the distribution or online.
The Java interface for netCDF is available separately from the
NetCDF
Java site.
Binary distributions for some platforms are available from the
directory ftp://ftp.unidata.ucar.edu/pub/binary.
Version 3.6 includes improved large file support, simplified
installation, bug fixes, and portability enhancements to version
3.5. The original netCDF file format is still supported as the
default, so files written with version 3.6 can be read with
previous versions and vice versa. Existing source code requires no
code changes.
Version 3.6 also supports a new variant of the file format with
64-bit offsets for very large files. Minor code changes are
required to create such files by using a non-default flag on
creation, and files that use the new 64-bit offset variant will not
be readable by previous versions of the netCDF library.
Applications built with the version 3.6 library will be able to
access both formats transparently.
Users are discouraged from using the new format variant unless
required for files too large for the classic netCDF format.
Third-party software that supports netCDF data access should
eventually be upgraded to version 3.6 or later to support both the
classic and the new large-file format variant.
The netCDF User's Guides for C, FORTRAN-77, and FORTRAN-90 have
been converted from a proprietary format to Texinfo, used in the
GNU documentation system. There is also a new Language-neutral
User's Guide that documents the netCDF data model and other
language-independent concepts. These changes will support better
maintenance of the documentation int the future.
Version 3.6 incorporates improved configuration and installation
support, so that on most platforms, the netCDF library and
utilities can be built without specifying environment variables,
and the library will be built with large file support by default,
if available. Better Windows support is also available now,
included with the standard distribution.
Performance enhancements and bug fixes include a speed-up of
Fortran-90 array access interfaces, addition of a "-x" option to
ncgen to specify use of fast "no fill" mode for netCDF file
creation, fixes to ncdump and ncgen utilities to properly handle
large dimensions, and removal of an unintended constraint on record
sizes.
For more details, see the version 3.6.0
release notes.
There are many ways to represent the same information in any
general-purpose data model. Choices left up to the user in the case
of netCDF include which information to represent as variables or as
variable attributes; what names to choose for variables,
dimensions, and attributes; what order to use for the dimensions of
multidimensional variables; what variables to include in the same
netCDF file; and how to use variable attributes to capture the
structure and meaning of data. We provide some guidelines in the
NetCDF User's Guide (e.g., the section on
``Differences between Attributes and Variables'') and in a new
web document ``Writing NetCDF
Files: Best Practices'', but we've found that a little
experience helps. Occasionally we have decided it was useful to
change the structure of netCDF files after experience with how the
data is used.
NetCDF files should have the file name extension ".nc". The
recommended extension for netCDF files was changed from ".cdf" to
".nc" in 1994 in order to avoid a clash with the NASA CDF file
extension, and now it also avoids confusion with "Channel
Definition Format" files.
The netcdfgroup@unidata.ucar.edu mailing-list is intended for
discussions and announcements about netCDF interfaces, software,
and use. The volume of this list varies widely, from one message
per month to a dozen messages per day (especially after a new
release). A message posted to this mailing-list will be seen by
several hundred people, so it's usually not appropriate for asking
simple questions about use. Such questions should instead be sent
to support@unidata.ucar.edu.
If you would prefer to get only a single daily digest of the
postings to the netcdfgroup mailing-list, subscribe instead to the
ncdigest@unidata.ucar.edu mailing-list. It's a digested form of the
netcdfgroup mailing-list, containing the same messages but
appearing at most once per day instead of whenever anyone sends a
message to the group.
To subscribe or unsubscribe to either of these mailing lists,
use
this form.
Here are some example netCDF files.
Discussions of conventions for representing time and handling
time-dependent data have been a past topic of discussion on the
netcdfgroup mailing list. When the subject comes up, interesting
discussions often result, so we've archived past discussions on
this subject at http://www.unidata.ucar.edu/packages/netcdf/time/.
A summary of Unidata's recommendations is available from
http://www.unidata.ucar.edu/software/netcdf/time/recs.html.
Briefly, we recommend use of the units conventions supported by the
udunits library for time
and other units attributes.
Other groups have established more specific conventions that
include the representation of time in netCDF files. For more
information on such conventions, see the NetCDF Conventions Page at
http://www.unidata.ucar.edu/packages/netcdf/conventions.html.
The netCDF mailing list has over 500 addresses (some of which
are aliases to more addresses) in thirty countries. Several groups
have adopted netCDF as a standard way to represent some forms of
scientific data.
A somewhat dated description of some of the projects and groups
that have used netCDF is available from http://www.unidata.ucar.edu/packages/netcdf/usage.html.
A primary reference is the User's Guide:
Rew, R. K., G. P. Davis, S. Emmerson, and H. Davies, NetCDF
User's Guide for C, An Interface for Data Access, Version 3,
April 1997.
Current online and downloadable documentation is available from
the documentation
directory.
Other references include:
Brown, S. A, M. Folk, G. Goucher, and R. Rew, "Software for
Portable Scientific Data Management," Computers in Physics,
American Institute of Physics, Vol. 7, No. 3, May/June 1993, pp.
304-308.
Fulker, D. W., "Unidata Strawman for Storing Earth-Referencing
Data," Seventh International Conference on Interactive Information
and Processing Systems for Meteorology, Oceanography, and
Hydrology, New Orleans, La., American Meteorology Society, January
1991.
Jenter, H. L. and R. P. Signell, 1992. "NetCDF: A
Freely-Available Software-Solution to Data-Access Problems for
Numerical Modelers". Proceedings of the American Society of Civil
Engineers Conference on Estuarine and Coastal Modeling. Tampa,
Florida.
Kuehn, J.A., "Faster Libraries for Creating Network-Portable
Self-Describing Datasets", Proceedings of the 37th Cray User Group
Meeting, (Barcelona, Spain, March 1996), Cray User Group, Inc.
Rew, R. K. and G. P. Davis, "NetCDF: An Interface for Scientific
Data Access," IEEE Computer Graphics and Applications, Vol. 10, No.
4, pp. 76-82, July 1990.
Rew, R. K. and G. P. Davis, "The Unidata netCDF: Software for
Scientific Data Access," Sixth International Conference on
Interactive Information and Processing Systems for Meteorology,
Oceanography, and Hydrology, Anaheim, California, American
Meteorology Society, pp. 33-40, February 1990.
Rew, R. K. and G. P. Davis, " Unidata's netCDF Interface
for Data Access: Status and Plans," Thirteenth International
Conference on Interactive Information and Processing Systems for
Meteorology, Oceanography, and Hydrology, Anaheim, California,
American Meteorology Society, February 1997.
Yes, the NetCDF User's Guide contains an appendix
that specifies the netCDF file format.
Also, the
"NetCDF File Structure and Performance" chapter explains the
physical structure of netCDF data at a high enough level to make
clear the performance implications of different data
organizations.
If users only access netCDF data through the documented
interfaces, future changes to the format will be transparent.
We tested the 3.6 release on the following operating systems with various compilers:
- AIX
- HPUX
- IRIX, IRIX64
- Linux
- MacOS X
- OSF1
- Solaris
- Windows
The NetCDF Installation and
Porting Guide explains how to build netCDF from source on
various platforms. Often, it's as easy as running
./configure
make test
make install
If you have an unusual combination of compilers or platform, you may
have to set a few environment variables before invoking
./configure
make test
make install
For details, see the
NetCDF Installation and
Porting Guide.
You should also check reports we maintain of successful
builds in other environments. If you have success with a
combination useful to others, please send it for addition to the list.
We make build output from various
platforms available for comparison with your output. In
general, you can ignore compiler warnings if the
"make test" step is successful. Lines that begin with "***" in the
"make test" output indicate results from tests.
The C and Fortran-77 interfaces are tested extensively, but only
rudimentary tests are currently used for the C++ and Fortran-90
interfaces.
If you invoke
ncdump --version
the last line of the resulting output will identify the version
associated with the ncdump utility. You can also call one
of the functions
nc_inq_libvers(),
nf_inq_libvers(), or
nf90_inq_libvers()
from C, Fortran-77, or Fortran-90 programs to get a version string.
Yes, but there are significant restrictions on the structure of
large netCDF files that result from the 32-bit relative offsets
that are part of the classic netCDF format. For details, see
NetCDF Classic Format Limitations in the User's Guide.
Large File Support (LFS) refers to operating system and C
library facilities to support files larger than 2 GiB. On many
32-bit platforms the default size of a file offset is still a
4-byte signed integer, which limits the maximum size of a file to 2
GiB. Using LFS interfaces and the 64-bit file offset type, the
maximum size of a file may be as large as 263 bytes, or
8 EiB. For many current platforms, large file macros or appropriate
compiler flags have to be set to build a library with support for
large files. This is handled automatically in netCDF 3.6.
More information about Large File Support is available from
Adding
Large File Support to the Single UNIX Specification.
When the netCDF format was created in 1988, 4-byte fields were
reserved for file offsets, specifying where the data for each
variable started relative to the beginning of the file or the start
of a record boundary.
This first netCDF format variant, the only format supported in
versions 3.5.1 and earlier, is referred to as the netCDF
classic format. The 32-bit file offset in the classic
format limits the total sizes of all but the last non-record
variables in a file to less than 2 GiB, with a similar limitation
for the data within each record for record variables. The netCDF
classic format is also identified as version 1 or
CDF1 in reference to the format label at the start of a
file.
With netCDF 3.6, a second variant of netCDF format is now
supported in addition to the classic format. The new variant is
referred to as the 64-bit offset format, version
2, or CDF2. The primary difference from the classic
format is the use of 64-bit file offsets instead of 32-bit offsets,
but it also supports larger variable and record sizes.
No, version 3.6 of the netCDF library detects which variant of
the format is used for each file when it is opened for reading or
writing, so it is not necessary to know which variant of the format
is used. The version of the format will be preserved by the library
on writing. If you want to modify a classic format file to use the
64-bit offset format so you can make it much larger, you will have
to create a new file and copy the data to it.
Yes, the 3.6 library and all planned future versions of the
library will continue to support reading and writing files using
the classic (32-bit offset) format as well as the new 64-bit offset
format. There is no need to convert existing archives from the
classic to the 64-bit offset format. Even netCDF-4, which will
introduce a third variant of the netCDF format based on HDF5, will
continue to support accessing classic format netCDF files as well
as 64-bit offset netCDF files.
No, we discourage users from making use of the new format unless
they need it for very large files. It may be some time until
third-party software that uses the netCDF library is upgraded to
3.6 or later versions that support the new large file facilities,
so we advise continuing to use the classic netCDF format for data
that doesn't require huge file offsets. The library makes this
recommendation easy to follow, since the default for file creation
is the classic format.
The short answer is that under most circumstances, you should
not care, if you use version 3.6.0 or later of the netCDF library.
But the difference is indicated in the first four bytes of the
file, which are 'C', 'D', 'F', '\001' for the classic netCDF format
and 'C', 'D', 'F', '\002' for the new 64-bit offset format. On a
Unix system, one way to display the first four bytes of a file, say
foo.nc, is to run the following command:
od -An -c -N4 foo.nc
which will output
C D F 001
or
C D F 002
depending on whether foo.nc is a classic or 64-bit offset netCDF
file, respectively.
The application will indicate an error trying to open the file
and present an error message equivalent to "not a netCDF file".
This is why it's a good idea not to create 64-bit offset netCDF
files until you actually need them.
Yes, by specifying the appropriate file creation flag you can
create 64-bit offset netCDF files the same way on 32-bit platforms
as on 64-bit platforms.
With netCDF version 3.6.0 or later, use the NC_64BIT_OFFSET flag
when you call nc_create(), as in:
err = nc_create("foo.nc",
NC_NOCLOBBER | NC_64BIT_OFFSET,
&ncid);
In Fortran-77, use the NF_64BIT_OFFSET flag when you call
nf_create(), as in:
iret = nf_create('foo.nc',
IOR(NF_NOCLOBBER,NF_64BIT_OFFSET),
ncid)
In Fortran-90, use the NF90_64BIT_OFFSET flag when you call
nf_create(), as in:
iret = nf90_create(path="foo.nc",
cmode=or(nf90_clobber,nf90_64bit_offset),
ncid=ncFileID)
In C++, use the Offset64Bits enum in the NcFile constructor, as
in:
NcFile nc("foo.nc",
FileMode=NcFile::New,
FileFormat=NcFile::Offset64Bits);
A new flag, '-v', has been added to ncgen to specify the file
format variant. By default or if '-v 1' or '-v classic' is
specified, the generated file will be in netCDF classic format. If
'-v 2' or '-v 64-bit-offset' is specified, the generated file will
use the new 64-bit offset format. To permit creating very large
files quickly, another new ncgen flag, '-x', has been added to
specify use of nofill mode when generating the netCDF file.
No, there are still some limits on sizes of netCDF objects, even
with the new 64-bit offset format. Each fixed-size variable and the
data for one record's worth of a record variable are limited in
size to a little less that 4 GiB, which is twice the size limit in
versions earlier than netCDF 3.6.
The maximum number of records remains 232-1.
While most platforms support a 64-bit file offset, many
platforms only support a 32-bit size for allocated memory blocks,
array sizes, and memory pointers. In C developer's jargon, these
platforms have a 64-bit off_t type for file offsets,
but a 32-bit size_t type for size of arrays. Changing
netCDF to assume a 64-bit size_t would restrict
netCDF's use to 64-bit platforms.
There are several possible reasons why creating a large file can
fail that are not related to the netCDF library:
If you get the netCDF library error "One or more variable sizes
violate format constraints", you are trying to define a variable
larger than permitted for the file format variant. This error
typically occurs when leaving "define mode" rather than when
defining a variable. The error status cannot be returned when a
variable is first defined, because the last fixed-size variable
defined is permitted to be larger than other fixed-size variables
(when there are no record variables). Similarly, the last record
variable may be larger than other record variables. This means that
subsequently adding a small variable to an existing file may be
invalid, because it makes what was previously the last variable now
in violation of the format size constraints. For details on the
format size constraints, see the Users Guide sections
NetCDF Classic Format Limitations and
NetCDF 64-bit Offset Format Limitations.
If you get the netCDF library error "Invalid dimension size" for
a non-negative size, you are exceeding the size limit of netCDF
dimensions, which must be less than 2,147,483,644 for classic files
with no large file support and otherwise less than
4,294,967,292.
No, except that 32-bit applications should link with a 32-bit
version of the library and 64-bit applications should link with a
64-bit library, similarly to use of other libraries that can
support either a 32-bit or 64-bit model of computation.
No, classic files created with the new library should be
compatible with all older applications, both for reading and
writing, with one minor exception. The exception is due to a
correction of a netCDF bug that prevented creating records larger
than 4 GiB in classic netCDF files with software linked against
versions 3.5.1 and earlier. This limitation in total record size
was not a limitation of the classic format, but an unnecessary
restriction due to the use of too small a type in an internal data
structure in the library. If you want to always make sure your
classic netCDF files are readable by older applications, make sure
you don't exceed 4 GiBytes for the total size of a record's worth
of data. (All records are the same size, computed by adding the
size for a record's worth of each record variable, with suitable
padding to make sure each record begins on a byte boundary
divisible by 4.)
Utilities available in the current netCDF distribution from
Unidata are ncdump, for converting netCDF files to an ASCII
human-readable form, and ncgen for converting from the ASCII
human-readable form back to a binary netCDF file or a C or FORTRAN
program for generating the netCDF file. Software for Manipulating or Displaying NetCDF
Data provides a list of other software useful for access,
visualization, and analysis of netCDF data and data represented in
other forms. Another useful
guide to netCDF utilities is available from NOAA's Geophysical
Fluid Dynamics Laboratory.
The Scientific
Data Format Information FAQ provides a somewhat dated
description of other access interfaces and formats for scientific
data, including CDF and HDF. A brief comparison of CDF,
netCDF, and HDF is available in the CDF FAQ. Another
comparison is in Jan Heijmans'
An Introduction to Distributed Visualization. John May's book
Parallel
I/O for High Performance Computing includes a chapter on
Scientific Data Libraries that describes netCDF and HDF5, with
example source code for reading and writing files using both
interfaces.
CDF
was developed at the NASA Space Science Data Center at Goddard, and
is freely available. It was originally a VMS FORTRAN interface for
scientific data access. Unidata reimplemented the library from
scratch to use XDR for a
machine-independent representation, designed the CDL (network
Common Data form Language) text representation for netCDF data, and
added aggregate data access, a single-file implementation, named
dimensions, and variable-specific attributes.
NetCDF and CDF have evolved independently. CDF now supports many
of the same features as netCDF (aggregate data access, XDR
representation, single-file representation, variable-specific
attributes), but some differences remain (netCDF doesn't support
native-mode representation, CDF doesn't support named dimensions).
There is no compatibility between data in CDF and netCDF form, and
as yet no translation software exists to convert data in one form
to data in the other form. For a more detailed description of
differences between CDF and netCDF, see the CDF FAQ.
The National Center for Supercomputing Applications (NCSA)
developed HDF and makes it
freely available. HDF is a data model and extensible data format
for self-describing files that was developed independently of
netCDF. HDF supports both C and Fortran interfaces, and it has been
successfully ported to a wide variety of machine architectures and
operating systems. HDF emphasizes a single common format for data,
on which many interfaces can be built.
NCSA implemented software that provides a netCDF interface to
HDF version 4. With this software, it was possible to use the
netCDF calling interface to place data into an HDF4 file.
HDF5 provides a richer data model, with emphasis on efficiency
of access, parallel I/O, and support for high-performance
computing. The netCDF-4 project is implementing an enhanced netCDF
interface on the HDF5 storage layer to preserve the desirable
common characteristics of netCDF and HDF5 while taking advantage of
their separate strengths: the widespread use and simplicity of
netCDF and the generality and performance of HDF5.
Yes, as part of the OPeNDAP framework, developers have
implemented a client-server system for access to remote data that
supports use of the netCDF interface for clients. The software is
available from the OPeNDAP download
site. After linking your netCDF application with the OPeNDAP
netCDF library, you can use URL's to access data from other sites
running an OPeNDAP server. This supports accessing small subsets of
large datasets remotely through the netCDF interfaces, without
copying the datasets.
The GrADS Data
Server provides subsetting and analysis services across the
Internet for any GrADS-readable dataset, including suitable netCDF
datasets.
Several programs and packages have been developed that convert between
GRIB
and netCDF data: degrib, CDAT, CDO, and GrADS,
among others. The netCDF Java Library (version 2.2 and
later) will suport reading GRIB data (and some other kinds of data)
through a netCDF interface, as if
it had been converted to a netCDF dataset.
I have some netcdf files which have data in them and were
apparently not properly closed. When I examine them using ncdump
they report zero data points, although the size is a few megabytes.
Is there a way of recovering them?
Yes, see the
archived message for a way to do this.
Yes, the document Known
problems with the netCDF Distribution describes reported
problems and workarounds in the latest version and some earlier
releases.
If you find a bug, send a description to
support@unidata.ucar.edu. This is also the address to use for
questions or discussions about netCDF that are not appropriate for
the entire netcdfgroup mailing list.
A search link is available at the bottom of the netCDF homepage, providing a full-text search
of the support questions and answers about netCDF provided by
Unidata support staff.
The netCDF distribution comes with interfaces for C, Fortran77,
Fortran90, and C++. Other languages for which interfaces are
available separately include:
It provides all the functionality of the C interface (except for
the generalized mapped access of ncvarputg() and ncvargetg()) and
is somewhat simpler to use than the C interface. With the C++
interface, no IDs are needed for netCDF components, there is no
need to specify types when creating attributes, and less
indirection is required for dealing with dimensions. However, the
C++ interface is less mature and less-widely used than the C
interface, and the documentation for the C++ interface is less
extensive, assuming a familiarity with the netCDF data model and
the C interface. Recently development of the C++ interface has
languished as resources have been redirected to enhancing the Java
interface.
It provides all the functionality of the C interface. The
Fortran interface uses Fortran conventions for array indices,
subscript order, and strings. There is no difference in the on-disk
format for data written from the different language interfaces.
Data written by a C language program may be read from a Fortran
program and vice-versa. The Fortran-90 interface is much smaller
than the FORTRAN 77 interface as a result of using optional
arguments and overloaded functions wherever possible.
They provide all the functionality of the C interface, using
appropriate language conventions. There is no difference in the
on-disk format for data written from the different language
interfaces. Data written by a C language program may be read from
programs that use other language interfaces, and vice-versa.
NetCDF version 4.0 will allow users to use HDF5 as their
underlying data format, while continuing to use the familiar netCDF
interface. As HDF5 includes data compression, this feature will be
available to netCDF users.
For more information about netCDF-4, see the netCDF-4
web site.
Another alternative, written and made available by Bill Noon, is
a set of modifications to the netCDF 3.3.1 library source that
allows transparent access to both compressed and uncompressed
netCDF files. A more complete explanation, the source changes,
instructions on how to patch the netcdf-3.3.1 source, and some
performance numbers are available from the znetcdf
site.
In version 4.0 the netCDF API will be extended and implemented
on top of the HDF5 data format, for users that have the appropriate
HDF5 libraries installed. (Those users without HDF5 will still be
able to use netCDF, version 4.0, to create "classic" format netCDF
files, as well as 64-bit offset netCDF files.)
NetCDF users with HDF5 will be able to create netCDF-4/HDF5
files with benefits not available with the netCDF format, such as
multiple unlimited dimensions. Backward compatibility in accessing
old netCDF files will be supported.
The combined library will preserve the desirable common
characteristics of netCDF and HDF5 while taking advantage of their
separate strengths: the widespread use and simplicity of netCDF and
the generality and performance of HDF5.
For more information about netCDF-4, see the netCDF-4
web site.
This page is maintained by Russ Rew.
Questions or comments can be sent to <support@unidata.ucar.edu>.
