[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: 20050214:NetCDF 3.6.0 Bug

Ed,

I can better characterize the bug Kevin Thomas discovered now.  I'll
have to send out a note to netcdfgroup soon about the problem, to make
sure no one inadvertently corrupts valuable data files.

The simplest way to trigger the bug is to

 - create a netCDF file using the 64-bit-offset (CDF2) format variant
 - close the file
 - reopen it for writing
 - write an existing attribute or enter define mode to add a new
   dimension, variable, or attribute (it's not enough to just write data)
 - leave define mode or close the file

When leaving define mode or closing the file, the header gets
rewritten to be a classic (CDF1) header.  It's not just the 4-byte
magic number at the beginning of the file that gets rewritten, it's
the entire header, including the variable offsets, which get stuffed
into 32-bit slots.

This is OK if none of the variable offsets required more than 32 bits,
but otherwise the offsets get truncated and data for associated
variables and beyond cannot later be read correctly.

The reason this wasn't detected in our exhaustive tests was because
those files are too small, so in rewriting a CDF2 file as a CDF1 file,
everything works and there are no failures reported.  If we had a
function to detect the file format variant change, that could be used
to discover this problem, but we have no such function in the public
interface.

Kevin's 3-line fix 

> I added the following to function nc__open_mp():
>
>       if (fIsSet(ioflags, NC_64BIT_OFFSET)) {
>         fSet(ncp->flags, NC_64BIT_OFFSET);
>       }

is the best way to fix the problem, but I'm still trying to find a way
to detect the problem in the normal nc_test code before making a new
release.

Kevin, thanks again for the bug report and fix!  I've added you name
to the netCDF credits:

 http://my.unidata.ucar.edu/content/software/netcdf/credits.html

--Russ