[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[netCDF #DKW-240688]: nc_rename_var(), nc_rename_dim() bugs in netCDF4?

Charlie,

> Your suggested workaround with nc_redef() and nc_enddef() does not
> improve the outcome of renaming netCDF4 variables/dimensions with NCO.
> Not sure why. My guess is that netCDF internals get confused when
> number of variables and dimensions to juggle becomes "non-trivial".
> NCO test file in.nc has 10-100x more vars/dims than Unidata test files.

I'm surprised the workaround didn't work, because running without it exactly 
duplicates the symptoms you see (variable values change to default fill values) 
and commenting out the nc_redef() call before and nc_enddef() call after the 
renames eliminates the bug symptoms.

Since this is clearly a bug involving renaming (the use of nc_redef and 
nc_enddef should be gratuitous and not change values for netCDF-4 files), I 
will continue to work on fixing this bug first.

Nevertheless, I'd very much like to see a demo of the renaming bug that doesn't 
disappear when nc_redef/nc_enddef are commented out, when writing a netCDF-4 
(not netCDF-4 classic model) file. Maybe using variables with more than one 
dimension would demonstrate the bug.

> Will be happy to try out other similar workarounds that you suggest.
> Would be best if there were a workaround that did not require a
> library change or converting to netCDF3 first.

That's what I'll be working on, though I have a couple of other commitments 
with deadlines next week, and am only supposed to be working half-time ...

> > I'm embarrassed that we never tested data values in variables after
> renaming, but it
> > never occurred to me (or others who have worked on renaming bugs)
> that "just changing
> > a name" would lead to changes to data values.
> 
> Same here. Surprised we didn't catch this earlier. Clearly, most
> large scale data reprocessing still encounters only netCDF3 files.
> 
> (In the course of testing I encountered a possible bug with ncgen
> that I will submit separately.)

Yup, thanks, that one's easily reproducible.

--Russ

> c
> 
> ncrenam test 23: netCDF3: Valid values after renaming coordinate
> dimension in netCDF3 file.......... SVn ok
> ncrenam test 24: netCDF3: Valid values after renaming coordinate
> variable in netCDF3 file........... SVn ok
> ncrenam test 25: netCDF3: Valid values after renaming coordinate
> dimension and variable in netCDF... SVn ok
> ncrenam test 26: netCDF4-classic: Valid values after renaming coordinate
> dimension in netCDF4-cla... FAILED!
> ERR: FAILURE in ncrename failure: netCDF4-classic: Valid values
> after renaming coordinate dimension in netCDF4-classic file
> ERR::EXPLAIN: Result: [9.96921e+36] != Expected: [1]
> 
> ncrenam test 27: netCDF4-classic: Valid values after renaming coordinate
> variable in netCDF4-clas... FAILED!
> ERR: FAILURE in ncrename failure: netCDF4-classic: Valid values
> after renaming coordinate variable in netCDF4-classic file
> ERR::EXPLAIN: Result: [9.96921e+36] != Expected: [1]
> 
> ncrenam test 28: netCDF4-classic: Valid values after renaming coordinate
> dimension and variable i... FAILED!
> ERR: FAILURE in ncrename failure: netCDF4-classic: Valid values
> after renaming coordinate dimension and variable in netCDF4-classic file
> ERR::EXPLAIN: Result: [9.96921e+36] != Expected: [1]
> 
> ncrenam test 29: netCDF4: Valid values after renaming coordinate
> dimension in netCDF4 file.......... FAILED!
> ERR: FAILURE in ncrename failure: netCDF4: Valid values after
> renaming coordinate dimension in netCDF4 file
> ERR::EXPLAIN: Result: [9.96921e+36] != Expected: [1]
> 
> ncrenam test 30: netCDF4: Valid values after renaming coordinate
> variable in netCDF4 file........... FAILED!
> ERR: FAILURE in ncrename failure: netCDF4: Valid values after
> renaming coordinate variable in netCDF4 file
> ERR::EXPLAIN: Result: [9.96921e+36] != Expected: [1]
> 
> ncrenam test 31: netCDF4: Valid values after renaming coordinate
> dimension and variable in netCDF... FAILED!
> ERR: FAILURE in ncrename failure: netCDF4: Valid values after
> renaming coordinate dimension and variable in netCDF4 file
> ERR::EXPLAIN: Result: [9.96921e+36] != Expected: [1]
> 
> 
> 
> On 10/10/2014 05:44 AM, Unidata netCDF Support wrote:
> > Charlie,
> >
> > Here's more detail on the possible workaround for NCO until we
> > have a library fix.  I've tested more extensively with the current
> > snapshot netcdf-4.3.3-rc2, using both coord and non-coord variables
> > and renaming the coordinate dimension or the variable independently.
> >
> > It seems avoiding the nc_redef and nc_enddef calls around the nc_rename
> > calls makes everything work for netCDF-4 files, but not for netCDF-4
> > classic model files. It also seems that there is no bug in the case of
> > renaming a coordinate dimension used with a non-coordinate variable,
> > even for netCDF-4 classic model files, assuming my tests are correct.
> >
> > Here's my output from the attached test, run in the nc_test4 directory
> > of the build after adding it to Makefile.am:
> >
> > $ ./tst_rename_bug
> >
> > *** Testing netcdf rename bugs and fixes.
> > *** Test renaming coordinate variable and its dimension for classic 
> > file...ok.
> > *** Test renaming coordinate variable and its dimension for 64-bit offset 
> > file...ok.
> > *** Test renaming coordinate variable and its dimension for netCDF-4 
> > file...ok.
> > *** Test renaming coordinate variable and its dimension for netCDF-4 
> > classic model file...
> >     (lats_in[0] is -2147483647) but should be -90)
> >
> >     (lats_in[1] is -2147483647) but should be 90)
> > ok.
> > *** Test renaming just dimension of coordinate variable for classic 
> > file...ok.
> > *** Test renaming just dimension of coordinate variable for 64-bit offset 
> > file...ok.
> > *** Test renaming just dimension of coordinate variable for netcdf-4 
> > file...ok.
> > *** Test renaming just dimension of coordinate variable for netcdf-4 
> > classic file...
> >     (lats_in[0] is -2147483647) but should be -90)
> >
> >     (lats_in[1] is -2147483647) but should be 90)
> > ok.
> > *** Test renaming just coordinate variable for classic file...ok.
> > *** Test renaming just coordinate variable for 64-bit offset file...ok.
> > *** Test renaming just coordinate variable for netcdf-4 file...ok.
> > *** Test renaming just coordinate variable for netcdf-4 classic file...
> >     (lats_in[0] is -2147483647) but should be -90)
> >
> >     (lats_in[1] is -2147483647) but should be 90)
> > ok.
> > *** Test renaming just dimension of non-coord variable for classic 
> > file...ok.
> > *** Test renaming just dimension of non-coord variable for 64-bit offset 
> > file...ok.
> > *** Test renaming just dimension of non-coord variable for netcdf-4 
> > file...ok.
> > *** Test renaming just dimension of non-coord variable for netcdf-4 classic 
> > file...ok.
> >
> > Of course, all the "ok." lines after the netcdf-4 classic model tests are 
> > spurious,
> > just artifacts of forcing all the tests to run even after the first error.
> >
> > The above points to a possible library fix strategy that would ignore 
> > nc_redef and
> > nc_enddef calls used only for renaming in code for netCDF-4 and netCDF-4 
> > classic model
> > files.
> >
> > I'm not yet confident enough of this diagnosis to announce anything on the 
> > netcdfgroup
> > mailing list, and would appreciate if you could verify that taking out 
> > nc_redef/nc_endef
> > calls wrapping nc_rename calls for netCDF-4 files makes that part of the 
> > bug go away for
> > NCO.  I suspect your tests are more comprehensive than mine and may reveal 
> > problems with
> > my diagnosis.
> >
> > I'll run this set of tests on earlier netCDF versions to see if renaming 
> > things has ever
> > worked correctly in any netCDF-4 versions, but I have my doubts.
> >
> > I'm embarrassed that we never tested data values in variables after 
> > renaming, but it
> > never occurred to me (or others who have worked on renaming bugs) that 
> > "just changing
> > a name" would lead to changes to data values.
> >
> > --Russ
> >
> >> Here's a partial workaround for renaming dimensions and variables when
> >> the target is a netCDF-4 file, but not netCDF-4 classic model.  It seems
> >> that renaming a dimension and corresponding coordinate variable works
> >> OK in this case, as long as you do not call nc_redef before the rename
> >> and nc_enddef after (though that shouldn't matter):
> >>
> >> In other words, this works in my test:
> >>
> >> if (nc_open(FILE_NAME, NC_WRITE, &ncid)) ERR;
> >> if (nc_inq_dimid(ncid, DIM_NAME, &dimid)) ERR;
> >> if (nc_inq_varid(ncid, VAR_NAME, &varid)) ERR;
> >> /* if (nc_redef(ncid)) ERR; /\* not necessary and not allowed ? *\/ */
> >> if (nc_rename_dim(ncid, dimid, NDIM_NAME)) ERR;
> >> if (nc_rename_var(ncid, varid, NVAR_NAME)) ERR;
> >> /* if (nc_enddef(ncid)) ERR; /\* not necessary and not allowed ? *\/ */
> >> if (nc_get_var(ncid, varid, lats_in)) ERR;
> >>
> >> but uncommenting the nc_redef and nc_enddef calls corrupts the variable
> >> values with fill values.  The nc_redef and nc_enddef calls are of course
> >> required for the other 3 format variants.
> >>
> >> --Russ
> >>
> >>> On 10/09/2014 08:07 AM, Unidata netCDF Support wrote:
> >>>> Hi Charlie,
> >>>>
> >>>>> A user has pointed out a serious problem with ncrename.
> >>>>> As best as I can tell, this traces back to a netCDF4 library bug.
> >>>>> Please investigate and let me know if this is true or not.
> >>>>
> >>>> You're right, this looks fairly serious. We haven't finished
> >>>> determining the scope of the bug, but we'll let you know soon. A
> >>>> posting to netcdfgroup may be needed to warn users not to trust
> >>>> coordinate data in coordinate variables that have been renamed.
> >>>>
> >>>> Thanks for reporting the bug.
> >>>>
> >>>> --Russ
> >>>>
> >>>>> The bug apparently affects (at least) versions 4.3.1--4.3.2 of the
> >>>>> netCDF4 library (and current snapshots).
> >>>>> This bug corrupts values contained in netCDF4 arrays (not scalars)
> >>>>> that are renamed or that utilize dimensions that are renamed.
> >>>>> In other words, renaming variables and dimensions succeeds
> >>>>> yet it corrupts the values contained by the affected array variables.
> >>>>> This bug corrupts affected variables by replacing their values with the
> >>>>> default _FillValue for that variable's type.
> >>>>> Attached is a file whose values are corrupted when either the
> >>>>> dimension or the variables or both are renamed.
> >>>>>
> >>>>> ncks -O -4 -v lat -C -h -m -M ~/nco/data/in.nc ~/bug.nc # Create test
> >>>>> ncrename -O -v lat,tal ~/bug.nc ~/foo.nc # Broken
> >>>>> ncrename -O -d lat,tal ~/bug.nc ~/foo.nc # Broken
> >>>>> ncrename -O -d lat,tal -v lat,tal ~/bug.nc ~/foo.nc # Broken
> >>>>> ncks --cdl ~/foo.nc
> >>>>>
> >>>>> The only known workaround is to convert the file to netCDF3 first,
> >>>>> then rename as intended, then convert back. I checked renaming of
> >>>>> groups and attributes in netCDF4 files and these operations both
> >>>>> perform as intended. So the problem appears to be only with with
> >>>>> renaming dimensions and renaming array variables in netCDF4 files.
> >>>>>
> >>>>> Thanks!
> >>>>> Charlie
> >>>>>
> >>>>> zender@givre:~$ ncks --cdl ~/bug.nc
> >>>>> netcdf bug {
> >>>>> dimensions:
> >>>>> lat = 2 ;
> >>>>>
> >>>>> variables:
> >>>>> float lat(lat) ;
> >>>>>
> >>>>> data:
> >>>>> lat = -90, 90 ;
> >>>>>
> >>>>> } // group /
> >>>>> zender@givre:~$ ncrename -O -v lat,tal ~/bug.nc ~/foo.nc
> >>>>> ncrename: In total renamed 0 attributes, 0 dimensions, 0 groups, and 1
> >>>>> variable
> >>>>> zender@givre:~$ ncks --cdl ~/foo.nc
> >>>>> netcdf foo {
> >>>>> dimensions:
> >>>>> lat = 2 ;
> >>>>>
> >>>>> variables:
> >>>>> float tal(lat) ;
> >>>>>
> >>>>> // global attributes:
> >>>>> :history = "Wed Oct  8 10:24:15 2014: ncrename -O -v lat,tal
> >>>>> /home/zender/bug.nc /home/zender/foo.nc" ;
> >>>>>
> >>>>> data:
> >>>>> tal = 9.96921e+36, 9.96921e+36 ;
> >>>>>
> >>>>> } // group /
> >>>>> zender@givre:~$
> >>>>>
> >>>>> --
> >>>>> Charlie Zender, Earth System Sci. & Computer Sci.
> >>>>> University of California, Irvine 949-891-2429 )'(
> >>>>>
> >>>>>
> >>>>
> >>>> Russ Rew                                         UCAR Unidata Program
> >>>> address@hidden                      http://www.unidata.ucar.edu
> >>>>
> >>>>
> >>>>
> >>>> Ticket Details
> >>>> ===================
> >>>> Ticket ID: DKW-240688
> >>>> Department: Support netCDF
> >>>> Priority: Normal
> >>>> Status: Closed
> >>>>
> >>>
> >>> --
> >>> Charlie Zender, Earth System Sci. & Computer Sci.
> >>> University of California, Irvine 949-891-2429 )'(
> >>>
> >>>
> >>
> >> Russ Rew                                         UCAR Unidata Program
> >> address@hidden                      http://www.unidata.ucar.edu
> >>
> >>
> >
> > Russ Rew                                         UCAR Unidata Program
> > address@hidden                      http://www.unidata.ucar.edu
> >
> >
> >
> > Ticket Details
> > ===================
> > Ticket ID: DKW-240688
> > Department: Support netCDF
> > Priority: Normal
> > Status: Closed
> >
> 
> --
> Charlie Zender, Earth System Sci. & Computer Sci.
> University of California, Irvine 949-891-2429 )'(
> 
> 

Russ Rew                                         UCAR Unidata Program
address@hidden                      http://www.unidata.ucar.edu



Ticket Details
===================
Ticket ID: DKW-240688
Department: Support netCDF
Priority: Normal
Status: Closed