[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: 19990909: NetCDF limitations

>To: address@hidden
>From: "M. Lautenschlager" <address@hidden>
>Subject: NetCDF limitations
>Organization: DKRZ - Deutsches Klimarechenzentrum
>Keywords: 199909091415.IAA27783, netCDF file size, huge files

Hi Michael,

> at the German Climate Computing Centre we are discussing to store
> climate model output in a new format. Up to now we preferably used GRIB1
> but for climate model data there are limitations in storage flexibility.
> One of the alternative formats we are discussing is NetCDF. 
>
> But there is one limitation mentioned in the "NetCDF User's Guide for C"
> Version 3 that strikes me:
>
> "With the current netCDF file format, no more than 2 gigabytes of data
> can be stored in a single netCDF dataset. This limitation is a result of
> 32-bit offsets currently used for storing positions within a file."
> (sec. 1.9)
>
> A 2 GByte file size is probably too small for the next generation
> climate model output.
>
> Is this file size limitation still valid or will be a new release in the
> near future available which does not contain this limit?

With version 3.4, we reduced some of the file size constraints.  Here
is an extract from the announcement of netCDF 3.4 that provides more
details for how large a netCDF file can be:

 On systems that support very large files (exceeding 2 Gbytes), such as
 IRIX 6.x or SunOS 5.6 with the large file compilation environment, it
 is possible to create and access very large netCDF files with version
 3.4.  The remaining size constraints are that the file offset to the
 beginning of the record variables (if any) must be less than 2 Gbytes,
 and the relative offset to the start of each fixed length variable or
 each record variable within a record must be less than 2 Gbytes.
 Hence, a very large netCDF file might have

   * no record variables, some ordinary fixed-length variables, and a
     very large (exceeding 2 Gbytes) fixed-length variable; or
   * some ordinary fixed-length and record variables, and a very large
     record variable; or
   * some ordinary fixed-length and record variables and a huge number
     of records.

 If you create very large netCDF files, remember that they will only be
 usable on other systems that support very large files.  The netCDF file
 format has not changed, so files less than 2 Gbytes in size are still
 writable and readable on all systems on which netCDF is supported.

To eliminate the above weaker file size constraints would require a
new netCDF format.  So far the original format (version 1, since 1987)
has been sufficient for all versions of the software through the
latest netCDF 3.5 release.  Implementing software that would support a
new format (based on HDF-5) but that would also continue to permit
access to files in the previous format has been in our long-term plans
for netCDF, but requires more resources than we have available
currently.  So there will be no new release in the near future that
doesn't have the above restrictions on variable size or number of
records in a netCDF file.

--Russ

_____________________________________________________________________

Russ Rew                                         UCAR Unidata Program
address@hidden                     http://www.unidata.ucar.edu