Re: 4GigB variable size limit


I'll jump in too.

I work with global ocean modeling of biogeochemical tracers. In the 
past, because of computational cost, we have worked with low resolution 
models, which translated to horizontal dimensions of 100x116 and 25 
vertical levels. We are now moving to higher resolutions, with our 
current target have a horizontal dimension of 320x384 with 60 vertical 
levels. Each 3D single precision field at this resolution uses ~28Mb, so 
72 such fields fit into 2Gb and 145 fields fit into 4Gb. Or output 
module is designed to put all output fields into a single file that 
includes a record dimension, which is the time axis. Having the time 
axis be a record dimension eases later manipulation of the files with 
the NCO suite of tools. In classic netCDF format, we can only fit 72 3D 
fields into a record and this only doubles when moving to CDF2. Our 
biogeochemical module has 24 tracers and there are typically numerous 
diagnostic fields describing fluxes between each tracer field. When I 
started doing integrations with the higher resolution, I had to remove 
some diagnostic fields in order to fit within the 2Gb record limit. I 
resisted moving to CDF2 because of concern with typical tools being 
compatible with the new format.

I am very concerned that switched to CDF2 only gains us a factor of 2. 
Our grid has an approximate resolution of 1 degree. Colleagues of mine 
regularly use 0.4 degree grids and 0.1 degree grids, and are forced to 
use binary formatted output. Given our current output architecture of 
having all fields in a single file, which our analysis tools are 
constructed for, CDF2 just won't get us much beyond where we are. If 
NetCDF doesn't transition allow larger records, in the near future we 
will either have to redesign our output and analysis tools, which is 
time consuming, or we won't be able to use NetCDF. I'm not looking 
forward to either option.


Keith Lindsay      
email: klindsay@xxxxxxxx   phone: 303-497-1722   fax: 303-497-1700

To unsubscribe netcdfgroup, visit: