4.3 Large File Support
It is possible to write netCDF files that exceed 2 GiByte on platforms
that have "Large File Support" (LFS). Such files are
platform-independent to other LFS platforms, but trying to open them
on an older platform without LFS yields a "file too large" error.
Without LFS, no files larger than 2 GiBytes can be used. The rest of
this section applies only to systems with LFS.
The original binary format of netCDF limits the size of data files by
using a signed 32-bit offset within its internal structure. Files larger
than 2 GiB can be created, with certain limitations. See NetCDF Classic Format Limitations.
In version 3.6.0, netCDF includes it's first-ever variant of the underlying
data format. The new format introduced in 3.6.0 uses 64-bit file offsets
in place of the 32-bit offsets. There are still some limits on the sizes
of variables, but the new format can create very large
datasets. See NetCDF 64 bit Offset Format Limitations.
The original netCDF format is identified using four bytes in the file
header. All files in this format have “CDF\001” at the beginning of the
file. In this documentation this format is referred to as “netCDF
classic format.”
Files with the 64-bit offsets are identified with a “CDF\002” at the
beginning of the file. In this documentation this format is called
“64-bit offset format.”
The original data format (netCDF classic), is still the default data
format for the netCDF library. Create files in 64-bit offset format
by specifying the 64 bit offset flag when creating a file.
In C, use flag NC_64BIT_OFFSET in function
nc_create. See nc_create. In Fortran,
use flag nf_64bit_offset in function NF_CREATE. See NF_CREATE.)
When opening an existing netCDF file the netCDF library will
transparently detect its format and adjust accordingly. However,
netCDF library versions earlier than 3.6.0 cannot read 64-bit offset
format files. NetCDF classic format files (even if created by version
3.6.0 or later) remain compatible with older versions of the netCDF
library. Users are encouraged to use netCDF classic format to
distribute data, for maximum portability.
It is also possible to change the default creation format, to convert
a large body of code without changing every create call. C programmers
see nc_set_default_format. Fortran
programs see NF_SET_DEFAULT_FORMAT.
The following table summarizes the size limitations of various
permutations of LFS support, netCDF version, and data format. Note
that 1 GiB = 2^30 bytes or about 1.07e+9 bytes, 1 EiB = 2^60 bytes or
about 1.15e+18 bytes. Note also that all sizes
are really 4 bytes less than the ones given below. For example the
maximum size of a fixed variable in netCDF 3.6 classic format is
really 2 GiB - 4 bytes.
| Limit | No LFS | v3.5 | v3.6/classic | v3.6/64-bit
offset
|
| Max File Size | 2 GiB | 8 EiB | 8 EiB | 8 EiB
|
| Max Number of Fixed Vars > 2 GiB | 0 | 1 (last) | 1
(last) | 2^32
|
| Max Record Vars w/ Rec Size > 2 GiB | 0 | 1 (last) | 1
(last) | 2^32
|
| Max Size of Fixed/Record Size of Record Var | 2 GiB | 2
GiB | 2 GiB | 4 GiB
|
| Max Record Size | 2 GiB/nrecs | 4 GiB | 8 EiB/nrecs
| 8 EiB/nrecs
|
