Hi Alan,
> I'm working with a team at NCAR and at NCSA to run and visualize WRF at
> very high resolution, and we are running into a NetCDF variable size
> limit of 4 G (floats, not bytes). I have seen a lot of comments about
> this limit on the Web, but I can't find whether there is a workaround.
Different netCDF versions have different limits, and as people use larger
files, we keep running into and fixing bugs with large file support.
For example, this bug was fixed last summer in the netCDF-4.2.1 release:
Fixed bug in 64-bit offset files with large records,
when last record variable requires more than 2**32
bytes per record.
[ https://bugtracking.unidata.ucar.edu/browse/NCF-164 ]
> The web page
> http://www.unidata.ucar.edu/software/netcdf/docs/offset_format_limitations.html
> indicates that each fixed variable is limited to 4GB, but in fact we are
> able to get up to 16GB, since the limit seems to apply to the number of
> floats not the number of bytes.
Hmm, I thought the 4 GiB limit was for bytes, not values. But if you've
tested this, I'll assume that's right and look at fixing the documentation
later if we have the limits wrong. However, the authoritative documentation
for now is in these 3 FAQ's on variable sizes:
Have all netCDF size limits been eliminated?
Why are variables still limited in size?
How can I write variables larger than 4 GiB?
http://www.unidata.ucar.edu/software/netcdf/docs/faq.html#Large%20File%20Support10
The only limit I'm aware of on the number of values of a variable has to do
with how much space you can allocate in memory to hold all the values, if you
try to read them with one call.
> Can you tell me if there is a way of configuring NetCDF 4 to read and to
> write variables that exceed this limit?
Yes, when you create a netCDF-4 file with the creation flag that says it's
either netCDF-4 format or netCDF-4 Classic Model format (both of which use HDF5
underneath), then there are no practical limits to the size of a variable.
> If we eliminate the unlimited (time) dimension, can we read and write an
> arbitrarily large fixed-size variable as the last variable in the file?
Yes, in that case even if it's a netCDF-3 (classic format or 64-bit offset
format)
file. For an example, see:
http://www.unidata.ucar.edu/software/netcdf/docs/netcdf.html#Classic-Limitations
--Russ
Russ Rew UCAR Unidata Program
address@hidden http://www.unidata.ucar.edu
Ticket Details
===================
Ticket ID: MQD-873542
Department: Support netCDF
Priority: Normal
Status: Closed
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.