[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: 20010406: netcdf 3.4 question

>To: address@hidden
>From: David Wong <address@hidden>
>Subject: netcdf 3.4 question
>Organization: Lockheed Martin
>Keywords: 200104061441.f36Ef1L14819

Hi Davis,

>     I wonder there is any hard file size limit in netCDF 3.4. My code
> died with netCDF error -31 and the output file size is about 2.14G.

Yes, the netCDF format uses 32-bit file offsets and extents, so in
general the size of a netCDF file is limited to about 2.14 Gbytes
(2^31, since off_t is a signed type).  However, it's possible to keep
more data in a single netCDF file, as described below.

On systems that support 64-bit offsets and a large file environment
with files exceeding 2^31 Gbytes (IRIX64, SunOS 5.x for SPARC v9,
OSF1/Alpha, ...) it is possible to create and access very large netCDF
files.  The remaining size constraints are that the file offset to the
beginning of the record variables (if any) must be less than 2^31 Gbytes,
and the relative offset to the start of each fixed length variable or
each record variable within a record must be less than 2^31 Gbytes.
Hence, a very large netCDF file might have

  * no record variables, some ordinary fixed-length variables, and one
    very large (exceeding 2^31 Gbytes) fixed-length variable; or
  * some ordinary fixed-length and record variables, and one very large
    record variable; or
  * some ordinary fixed-length and record variables and a huge number
    of records.

If you create very large netCDF files, they will only be usable on
other systems that support very large files.  The netCDF file format
has not changed, so files less than 2^31 Gbytes in size are still
writable and readable on all systems on which netCDF is supported.

To eliminate the above weaker file size constraints would require a
new netCDF format.  So far the original format (version 1, since 1987)
has been sufficient for all versions of the software through the
latest netCDF 3.5 release.  Implementing software that would support a
new format (based on HDF-5) but that would also continue to permit
access to files in the previous format has been in our long-term plans
for netCDF.  But there will be no release in the near future that
doesn't have the above restrictions on variable size or number of
records in a netCDF file.

--Russ

_____________________________________________________________________

Russ Rew                                         UCAR Unidata Program
address@hidden                     http://www.unidata.ucar.edu