[netcdfgroup] NetCDF-3 file format changes

[Dave Allured]

Mario et al,

If you don't mind me asking, have you worked with large Netcdf-3 
files with variables > 4GB, types NC_FLOAT and NC_INT only (i.e. 32 
bit types)?  Is the current Netcdf 3.6.2 software reliable with 
these in your experience?

It has seemed reliable for me, but I must confess my usage of files 
greater than 4GB has been very limited so far.  Thank you for any 
feedback.

Dave Allured
CU/CIRES Climate Diagnostics Center (CDC)
http://cires.colorado.edu/science/centers/cdc/
NOAA/ESRL/PSD, Climate Analysis Branch (CAB)
http://www.cdc.noaa.gov/

Mario Emmenlauer wrote:
> Hi,
> 
> Roy Mendelssohn wrote:
>> See http://www.unidata.ucar.edu/software/netcdf/docs/faq.html#Large%20File%20Support0. 
>>  It has been possible since netcdf3.6
> 
> with severe bugs, as posted by me to this mailing list on 09/12/2007
> (i.e. [netcdfgroup] possible bug in 3.6.2 with variables > 4GB).
> For variable types NC_BYTE, NC_CHAR and NC_SHORT it crashes badly.
> 
> I have actually never received an answer to my bug report, don't know
> why nobody cared :-/
> 
> BTW: below is an attached copy of the old mail
> 
> Cheers,
> 
>     Mario
> 
> ----
> I hope this bug hasn't been filed before, I couldn't find it
> on the list via a quick check.
> 
> This concerns writing large (>4GB) variables in netcdf with
> large-file-support. In netcdf-3.6.2/libsrc/var.c around line
> 400 it reads:
> 
> if( varp->xsz <= X_UINT_MAX / product )
>    /* if integer multiply will not overflow */
> {
>    varp->len = product * varp->xsz;
> } else {
>    /* OK for last var to be "too big", indicated by this special len */
>    varp->len = X_UINT_MAX;
> }
> switch(varp->type) {
>    case NC_BYTE :
>    case NC_CHAR :
>    case NC_SHORT :
>      if( varp->len%4 != 0 )
>      {
>        varp->len += 4 - varp->len%4; /* round up */
>        /* *dsp += 4 - *dsp%4; */
>      }
>    break;
>    default:
>    /* already aligned */
>      break;
> }
> 
> In the case of NC_BYTE, NC_CHAR and NC_SHORT, varp->len will end
> up being X_UINT_MAX+1 instead of X_UINT_MAX. This in turn causes
> an assertion when calling ncx_put_size_t later:
> ncx.c:1812: ncx_put_size_t: Assertion `*ulp <= 4294967295U' failed.
> 
> I could not think about a useful fix despite qualifying the
> rounding with the product-overflow (just moved the else-part
> further down):
> 
> if( varp->xsz <= X_UINT_MAX / product )
>    /* if integer multiply will not overflow */
> {
>    varp->len = product * varp->xsz;
>    switch(varp->type) {
>      case NC_BYTE :
>      case NC_CHAR :
>      case NC_SHORT :
>        if( varp->len%4 != 0 )
>        {
>          varp->len += 4 - varp->len%4; /* round up */
>          /* *dsp += 4 - *dsp%4; */
>        }
>      break;
>      default:
>      /* already aligned */
>        break;
>    }
> } else {
>    /* OK for last var to be "too big", indicated by this special len */
>    varp->len = X_UINT_MAX;
> }
> 
> I hope this bug report is useful. If you can send me a better
> patch against netcdf-3.6.2 I would highly appreciate it.
> 
> Cheers,
> 
>    Mario Emmenlauer
> ----
> 
> 
>> On Feb 26, 2008, at 1:11 PM, Joe Sirott wrote:
>>
>>> Hi,
>>>
>>> In the "classic" netCDF file format (netCDF-3), a variable without a 
>>> record dimension cannot be larger than 2GB. This limitation has been 
>>> giving me a lot of headaches lately. I know that netCDF-4 is supposed to 
>>> solve this problem, but there are a number of reasons why netCDF-4 is 
>>> not a good option for me (no Java write support, for one).
>>>
>>> I could be missing something, but it seems like a very small change to 
>>> the netCDF-3 file format would solve this problem. The only requirement 
>>> would  be changing the variable size info in the netCDF header from a 32 
>>> bit to a 64 bit int (and, of course, updating the version info in the 
>>> header).
>>>
>>> I'm guessing that I'm not the only netCDF user who has run into this 
>>> problem and who is also reluctant to move to netCDF-4. Any possibility 
>>> that Unidata could make these changes?
>>>
>>> Cheers,
>>>
>>> Joe S.
>>>
>>>
>>> _______________________________________________
>>> netcdfgroup mailing list
>>> netcdfgroup at unidata.ucar.edu <mailto:netcdfgroup at unidata.ucar.edu>
>>> For list information or to unsubscribe,  visit: 
>>> http://www.unidata.ucar.edu/mailing_lists/ 
>> **********************
>> "The contents of this message do not reflect any position of the U.S. 
>> Government or NOAA."
>> **********************
>> Roy Mendelssohn
>> Supervisory Operations Research Analyst
>> NOAA/NMFS
>> Environmental Research Division
>> Southwest Fisheries Science Center
>> 1352 Lighthouse Avenue
>> Pacific Grove, CA 93950-2097
>>
>> e-mail: Roy.Mendelssohn at noaa.gov <mailto:Roy.Mendelssohn at noaa.gov> (Note 
>> new e-mail address)
>> voice: (831)-648-9029
>> fax: (831)-648-8440
>> www: http://www.pfeg.noaa.gov/
>>
>> "Old age and treachery will overcome youth and skill."
>>
>>
>>
>>
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> netcdfgroup mailing list
>> netcdfgroup at unidata.ucar.edu
>> For list information or to unsubscribe,  visit: http://www.unidata.ucar.edu/mailing_lists/ 
> _______________________________________________
> netcdfgroup mailing list
> netcdfgroup at unidata.ucar.edu
> For list information or to unsubscribe,  visit: http://www.unidata.ucar.edu/mailing_lists/