Hi Alan, > I'm working with a team at NCAR and at NCSA to run and visualize WRF at > very high resolution, and we are running into a NetCDF variable size > limit of 4 G (floats, not bytes). I have seen a lot of comments about > this limit on the Web, but I can't find whether there is a workaround. Different netCDF versions have different limits, and as people use larger files, we keep running into and fixing bugs with large file support. For example, this bug was fixed last summer in the netCDF-4.2.1 release: Fixed bug in 64-bit offset files with large records, when last record variable requires more than 2**32 bytes per record. [ https://bugtracking.unidata.ucar.edu/browse/NCF-164 ] > The web page > http://www.unidata.ucar.edu/software/netcdf/docs/offset_format_limitations.html > indicates that each fixed variable is limited to 4GB, but in fact we are > able to get up to 16GB, since the limit seems to apply to the number of > floats not the number of bytes. Hmm, I thought the 4 GiB limit was for bytes, not values. But if you've tested this, I'll assume that's right and look at fixing the documentation later if we have the limits wrong. However, the authoritative documentation for now is in these 3 FAQ's on variable sizes: Have all netCDF size limits been eliminated? Why are variables still limited in size? How can I write variables larger than 4 GiB? http://www.unidata.ucar.edu/software/netcdf/docs/faq.html#Large%20File%20Support10 The only limit I'm aware of on the number of values of a variable has to do with how much space you can allocate in memory to hold all the values, if you try to read them with one call. > Can you tell me if there is a way of configuring NetCDF 4 to read and to > write variables that exceed this limit? Yes, when you create a netCDF-4 file with the creation flag that says it's either netCDF-4 format or netCDF-4 Classic Model format (both of which use HDF5 underneath), then there are no practical limits to the size of a variable. > If we eliminate the unlimited (time) dimension, can we read and write an > arbitrarily large fixed-size variable as the last variable in the file? Yes, in that case even if it's a netCDF-3 (classic format or 64-bit offset format) file. For an example, see: http://www.unidata.ucar.edu/software/netcdf/docs/netcdf.html#Classic-Limitations --Russ Russ Rew UCAR Unidata Program address@hidden http://www.unidata.ucar.edu Ticket Details =================== Ticket ID: MQD-873542 Department: Support netCDF Priority: Normal Status: Closed
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.