[netcdfgroup] NetCDF-3 file format changes
Mario Emmenlauer
mario at emmenlauer.de
Tue Feb 26 16:46:04 MST 2008
Hi,
Dave Allured wrote:
> If you don't mind me asking, have you worked with large Netcdf-3
> files with variables > 4GB, types NC_FLOAT and NC_INT only (i.e. 32
> bit types)?
Yes, I have several large files with > 4GB.
> Is the current Netcdf 3.6.2 software reliable with these in your experience?
It is reliable if you stick to very strict constraints. I would actually
rather say 'It works, but the constraints are not easy to enforce'.
- only one variable > 4GB per file, and this must be the last variable
in the file.
- no attributes stored for variables > 2GB.
- (does not apply to you if you only want 32bit types): use my fix from
the last mail then nc_byte and nc_short will work too.
(I do not recall precisely if the above limits are 2 or 4GB, have to look
it up tomorrow)
Cheers,
Mario
> It has seemed reliable for me, but I must confess my usage of files
> greater than 4GB has been very limited so far. Thank you for any
> feedback.
>
> Dave Allured
> CU/CIRES Climate Diagnostics Center (CDC)
> http://cires.colorado.edu/science/centers/cdc/
> NOAA/ESRL/PSD, Climate Analysis Branch (CAB)
> http://www.cdc.noaa.gov/
>
> Mario Emmenlauer wrote:
>> Hi,
>>
>> Roy Mendelssohn wrote:
>>> See http://www.unidata.ucar.edu/software/netcdf/docs/faq.html#Large%20File%20Support0.
>>> It has been possible since netcdf3.6
>> with severe bugs, as posted by me to this mailing list on 09/12/2007
>> (i.e. [netcdfgroup] possible bug in 3.6.2 with variables > 4GB).
>> For variable types NC_BYTE, NC_CHAR and NC_SHORT it crashes badly.
>>
>> I have actually never received an answer to my bug report, don't know
>> why nobody cared :-/
>>
>> BTW: below is an attached copy of the old mail
>>
>> Cheers,
>>
>> Mario
>>
>> ----
>> I hope this bug hasn't been filed before, I couldn't find it
>> on the list via a quick check.
>>
>> This concerns writing large (>4GB) variables in netcdf with
>> large-file-support. In netcdf-3.6.2/libsrc/var.c around line
>> 400 it reads:
>>
>> if( varp->xsz <= X_UINT_MAX / product )
>> /* if integer multiply will not overflow */
>> {
>> varp->len = product * varp->xsz;
>> } else {
>> /* OK for last var to be "too big", indicated by this special len */
>> varp->len = X_UINT_MAX;
>> }
>> switch(varp->type) {
>> case NC_BYTE :
>> case NC_CHAR :
>> case NC_SHORT :
>> if( varp->len%4 != 0 )
>> {
>> varp->len += 4 - varp->len%4; /* round up */
>> /* *dsp += 4 - *dsp%4; */
>> }
>> break;
>> default:
>> /* already aligned */
>> break;
>> }
>>
>> In the case of NC_BYTE, NC_CHAR and NC_SHORT, varp->len will end
>> up being X_UINT_MAX+1 instead of X_UINT_MAX. This in turn causes
>> an assertion when calling ncx_put_size_t later:
>> ncx.c:1812: ncx_put_size_t: Assertion `*ulp <= 4294967295U' failed.
>>
>> I could not think about a useful fix despite qualifying the
>> rounding with the product-overflow (just moved the else-part
>> further down):
>>
>> if( varp->xsz <= X_UINT_MAX / product )
>> /* if integer multiply will not overflow */
>> {
>> varp->len = product * varp->xsz;
>> switch(varp->type) {
>> case NC_BYTE :
>> case NC_CHAR :
>> case NC_SHORT :
>> if( varp->len%4 != 0 )
>> {
>> varp->len += 4 - varp->len%4; /* round up */
>> /* *dsp += 4 - *dsp%4; */
>> }
>> break;
>> default:
>> /* already aligned */
>> break;
>> }
>> } else {
>> /* OK for last var to be "too big", indicated by this special len */
>> varp->len = X_UINT_MAX;
>> }
>>
>> I hope this bug report is useful. If you can send me a better
>> patch against netcdf-3.6.2 I would highly appreciate it.
>>
>> Cheers,
>>
>> Mario Emmenlauer
>> ----
>>
>>
>>> On Feb 26, 2008, at 1:11 PM, Joe Sirott wrote:
>>>
>>>> Hi,
>>>>
>>>> In the "classic" netCDF file format (netCDF-3), a variable without a
>>>> record dimension cannot be larger than 2GB. This limitation has been
>>>> giving me a lot of headaches lately. I know that netCDF-4 is supposed to
>>>> solve this problem, but there are a number of reasons why netCDF-4 is
>>>> not a good option for me (no Java write support, for one).
>>>>
>>>> I could be missing something, but it seems like a very small change to
>>>> the netCDF-3 file format would solve this problem. The only requirement
>>>> would be changing the variable size info in the netCDF header from a 32
>>>> bit to a 64 bit int (and, of course, updating the version info in the
>>>> header).
>>>>
>>>> I'm guessing that I'm not the only netCDF user who has run into this
>>>> problem and who is also reluctant to move to netCDF-4. Any possibility
>>>> that Unidata could make these changes?
>>>>
>>>> Cheers,
>>>>
>>>> Joe S.
>>>>
>>>>
>>>> _______________________________________________
>>>> netcdfgroup mailing list
>>>> netcdfgroup at unidata.ucar.edu <mailto:netcdfgroup at unidata.ucar.edu>
>>>> For list information or to unsubscribe, visit:
>>>> http://www.unidata.ucar.edu/mailing_lists/
>>> **********************
>>> "The contents of this message do not reflect any position of the U.S.
>>> Government or NOAA."
>>> **********************
>>> Roy Mendelssohn
>>> Supervisory Operations Research Analyst
>>> NOAA/NMFS
>>> Environmental Research Division
>>> Southwest Fisheries Science Center
>>> 1352 Lighthouse Avenue
>>> Pacific Grove, CA 93950-2097
>>>
>>> e-mail: Roy.Mendelssohn at noaa.gov <mailto:Roy.Mendelssohn at noaa.gov> (Note
>>> new e-mail address)
>>> voice: (831)-648-9029
>>> fax: (831)-648-8440
>>> www: http://www.pfeg.noaa.gov/
>>>
>>> "Old age and treachery will overcome youth and skill."
>>>
>>>
>>>
>>>
>>> ------------------------------------------------------------------------
>>>
>>> _______________________________________________
>>> netcdfgroup mailing list
>>> netcdfgroup at unidata.ucar.edu
>>> For list information or to unsubscribe, visit: http://www.unidata.ucar.edu/mailing_lists/
>> _______________________________________________
>> netcdfgroup mailing list
>> netcdfgroup at unidata.ucar.edu
>> For list information or to unsubscribe, visit: http://www.unidata.ucar.edu/mailing_lists/
>
> _______________________________________________
> netcdfgroup mailing list
> netcdfgroup at unidata.ucar.edu
> For list information or to unsubscribe, visit: http://www.unidata.ucar.edu/mailing_lists/
>
More information about the netcdfgroup
mailing list