NetCDF is a data abstraction for array-oriented data access and a software library that provides a concrete implementation of the interfaces that support that abstraction. The implementation provides a machine-independent format for representing arrays. Although the netCDF file format is hidden below the interfaces, some understanding of the implementation and associated file structure may help to make clear which netCDF operations are expensive and why.
For a detailed description of the netCDF format, see section File Format Specification. It is not needed to read and write netCDF files or understand efficiency issues. Programs that use only the documented interfaces and that make no other assumptions about the format will continue to work even if the netCDF format is changed in the future, because any such change will be made below the documented interfaces and will support earlier versions of netCDF data.
This chapter describes the structure of a netCDF file and some characteristics of the XDR layer that provides network transparency in enough detail to understand netCDF performance issues.
A netCDF dataset is stored as a single file comprising two parts:
All the data are represented in XDR form to make them machine-independent.
The descriptive header at the beginning of the netCDF file is an XDR encoding of a high-level data structure that represents information about the dimensions, variables, and attributes in the file. The variable descriptions in this header contain offsets to the beginning of each variable's data or the relative offset of a variable within a record. The descriptions also contain the dimension size and information needed to determine how to map multidimensional indices for each variable to the appropriate offsets.
This header has no usable extra space; it is only as large
as it needs to be for the dimensions, variables, and attributes in each netCDF
file. This has the advantage that netCDF files are compact, requiring very little
overhead to store the ancillary data that makes the files self-describing. A
potential disadvantage of this organization is that any operation on a netCDF
file that requires expanding the header, for example adding new dimensions and
new variables to an existing netCDF file, will be as expensive as copying the
file. This expense is incurred when
ncendef() is called, after
a call to
ncredef(). If you create all necessary dimensions, variables,
and attributes before writing variable data, and avoid later additions
and renamings of netCDF components that require more space in the header part
of the file, you avoid the cost associated with expanding the header.
The fixed-size data part that follows the header contains all the variable data for variables that do not employ the unlimited (record) dimension. The data for each variable is stored contiguously in this part of the file. If there is no unlimited dimension, this is the last part of the netCDF file.
The record-data part that follows the fixed-size data consists of a variable number of records, each of which contains data for all the record variables. The record data for each variable is stored contiguously in each record.
The order in which the data in the fixed-size data part and in each record appears is the same as the order in which the variables were defined, in increasing numerical order by netCDF variable ID. This knowledge can sometimes be used to enhance data access performance, since the best data access is currently achieved by reading or writing the data in sequential order.
XDR is a standard for describing and encoding data and a library of functions for external data representation, allowing programmers to encode data structures in a machine-independent way. NetCDF employs XDR for representing all data, in both the header part and the data parts. XDR is used to write portable data that can be read on any other machine for which the XDR library has been implemented.
Many vendors provide an XDR library along with other C run-time libraries. The netCDF software distribution also includes Sun's portable implementation of XDR for platforms that don't already have a vendor-supplied XDR library.
An I/O layer implemented much like the C standard I/O (stdio)
library is used by the XDR layer to read and write XDR-encoded data to netCDF
files. Hence an understanding of the standard I/O library provides answers to
most questions about multiple processes accessing data concurrently, the use
of I/O buffers, and the costs of opening and closing netCDF files. In particular,
it is possible to have one process writing a netCDF file while other processes
read it. Data reads and writes are no more atomic than calls to stdio
ncsync() call (
for FORTRAN) is analogous to the
fflush() call in the standard
I/O library, writing unwritten buffered data so other processes can read it;
ncsync() also brings header changes up-to-date (e.g., changes to
As in the stdio library, flushes are also performed when "seeks" occur to a different area of the file. Hence the order of read and write operations can influence I/O performance significantly. Reading data in the same order in which it was written within each record will minimize buffer flushes.
There is one unusual case where the situation is more complex: when a writer enters define mode to add some additional dimensions, variables, or attributes to an existing netCDF file that is also open for reading by other processes. In this case, when the writer leaves define mode, a new copy of the file is created with the new dimensions, attributes, or variables and the old data, but readers that still have the file open will not see the changes, unless they close and reopen the file.
You should not expect netCDF data access to work with multiple writers having the same file open for writing simultaneously.
It is possible to tune an implementation of netCDF for some platforms by replacing the I/O layer beneath XDR with a different platform-specific I/O layer. This has been done for Crays, for example. This may change the similarities between netCDF and standard I/O, and hence characteristics related to data sharing, buffering, and the cost of I/O operations.
The cost of using a canonical representation for data like XDR varies according to the type of data and whether the XDR form is the same as the machine's native form for that type. XDR is especially efficient for byte, character, and short integer data.
For some data types on some machines, the time required to convert data to and from XDR form can be significant. The best case is byte arrays, for which very little conversion expense occurs, since the XDR library has built-in support for them. The netCDF implementation includes similar support added to XDR for arrays of short (16-bit) integers. The worst case is reading or writing large arrays of floating-point data on a machine that does not use IEEE floating-point as its native representation. The XDR library incurs the expense of a function call for each floating-point quantity accessed. On some architectures the cost of a function invocation for each floating-point number can dominate the cost of netCDF access to floating-point fields.
The distributed netCDF implementation is meant to be portable. Platform-specific ports that further optimize the implementation for better I/O performance or that unroll the loops in the XDR library to optimize XDR conversion of long integer and floating-point arrays are practical and desirable in cases where higher performance for data access is necessary.
As was mentioned in the previous section, it is possible to replace the I/O layer that is used by XDR in order to increase I/O efficiency. This has been done for UNICOS, the operating system of Cray computers (e.g. the Cray Y-MP).
Additionally, it is possible for the user to obtain even greater I/O efficiency
through appropriate setting of the
NETCDF_FFIOSPEC environment variable. This
variable specifies the Flexible File I/O buffers
for netCDF I/O when executing under the UNICOS operating system (the variable
is ignored on other operating systems). An appropriate specification can greatly
increase the efficiency of netCDF I/O -- to the extent that it can rival and
actually surpass default FORTRAN binary I/O. Possible specifications include
bufa:336:22, asynchronous, I/O buffers of 336 blocks each (i.e. double buffering). This is the default specification and favors sequential I/O.
cache:256:88, synchronous, 256-block buffers. This favors larger random accesses.
cachea:256:8:28, asynchronous, 256-block buffers with a 2 block read-ahead/write-behind factor. This also favors larger random accesses.
cachea:8:256:0256, asynchronous, 8-block buffers without read-ahead/write-behind. This favors many smaller pages without read-ahead for more random accesses as typified by slicing netCDF arrays.
cache:8:256,cachea.sds:1024:4:1This is a two layer cache. The first (synchronous) layer is composed of 256 8-block buffers in memory, the second (asynchronous) layer is composed of 4 1024-block buffers on the SSD. This scheme works well when accesses proceed through the file in random waves roughly 2x1024-blocks wide.
All of the options/configurations supported in CRI's FFIO library are available through this mechanism. We recommend that you look at CRI's I/O optimization guide for information on using FFIO to it's fullest. This mechanism is also compatible with CRI's EIE I/O library.
NETCDF_FFIOSPEC variable to a program's I/O pattern
can dramatically improve performance. Speedups of two orders of magnitude have