[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[netCDF #PFU-753378]: Error in closing netCDF file (due to presence of user-defined type)



Lynton,
> many thanks for this. I am following progress....
> However, is it possible to have an indication of timescales (or work effort)
> This is a a serious bug for me and I would prefer to wait for its
> resolution before
> continuing with the current software development work.

It's hard to estimate how much work it will take to fix this.  My
latest efforts make it appear as if the problem is in nc_enddef(), but
a quick look at that didn't result in seeing the bug.  This problem is
in the area of trying to model netCDF's shared dimensions using HDF5's
dimension scales, but HDF5 dimension scales aren't adequate by
themselves, so Ed had to "bolt on" extra artifacts, consisting of
lists and attributes in the HDF5 representation that aren't visible in
the netCDF-4 files, to try to fill the gap.  There have been several
bugs in this part of the netCDF-4 implementation, all involving
something breaking depending on someone invoking netCDF functions in
an order that we don't test or didn't anticipate.

Ed's no longer available for consulting on this, so I'm currently
trying to figure out what's going on by reading about the artifacts in
Appendix B of this document:

  
https://earthdata.nasa.gov/sites/default/files/esdswg/spg/rfc/esds-rfc-022/nasa_netcdf4_standard_v0.03.pdf

However, currently getting a blog finished and published on use of
chunking in netCDF-4 is higher priority, because it's overdue.  So
more progress in debugging the netCDF-4 bug, as my next highest
priority, will probably be delayed until later this week ...

--Russ


  

> However, this may be impractical.
> thanks
> Lynton
> 
> On 01/24/2013 10:54 PM, Unidata netCDF Support wrote:
> > Lynton,
> >
> > The Jira ticket for this bug, with two C example programs, is now available 
> > here:
> >
> >    https://bugtracking.unidata.ucar.edu/browse/NCF-217
> >
> > in case you want to follow the progress.
> >
> > --Russ
> >
> >> Lynton,
> >>
> >>> Thanks for the reply. In fact the "feature" you picked up was a genuine
> >>> mistake of mine
> >>> when translating from the C++ API to the C API. The real problem was
> >>> somewhat different
> >>> as I will explain. The programme I attach is the same as before but with
> >>> the user-type error corrected
> >>> and some data assigned to the variable "weightDDXXYY"
> >>>
> >>> I can compile the code fine and run it fine.
> >>>
> >>> However, when I run ncdump I get problems. In this case the output is
> >>> wrong, but in other cases ncdump can actually crash.
> >>> The error appears to be associated with assigning values to the variable
> >>> "ironBoundaries" on line 44 of efit++.cpp.
> >>> This causes the dimensioning of weightDDXXYY to be screwed up, at least
> >>> according to ncdump.
> >>> However h5dump appears not to have the same problem suggesting that
> >>> there is a problem in ncdump !!
> >>>
> >>> To see this for yourself, compare the files efitOut.txt (ncdump output)
> >>> and efitOut.hdf5.txt (h5Dump output).
> >>> You will see that the dimensioning of weightDDXXYY is apparently 
> >>> different.
> >>>
> >>> Note as I said before, this is using netCDF version 4.2
> >> OK, now I can reproduce the bug.  It appears to be an example of the bug 
> >> that depends
> >> on the order in which netCDF functions are called, but the results should 
> >> not depend on
> >> the order.
> >>
> >> I'm attaching a version of your program that works when I reorder the 
> >> function calls to
> >> appear in the following groups of calls:
> >>
> >> create file and groups
> >> define types
> >> define dimensions
> >> define variables
> >> write data
> >>
> >> and it works as expected.  I don't know if there's a simpler permutation 
> >> of statement orders
> >> that would also work.
> >>
> >> The fact that it doesn't work in the order you used is definitely a major 
> >> bug.
> >> I'm also creating a Jira ticket for this and will consider it a priority 
> >> to try
> >> to diagnose the underlying problem and fix it.
> >>
> >> --Russ
> >>
> >>> On 01/24/2013 01:46 PM, Unidata netCDF Support wrote:
> >>>> Hi Lynton,
> >>>>
> >>>>> I have a short programme that throws up an HDF5 error: NC_EHDFERR  when 
> >>>>> closing. It appears to be connected with defining a user-defined type:
> >>>>> Have  you got any idea what the problem is?
> >>>>>
> >>>>>
> >>>>> The output of the code is:
> >>>>> 0 1
> >>>>> 0 2
> >>>>> 0 3
> >>>>> 0 4
> >>>>> 0 5
> >>>>> 0 6
> >>>>> 0 7
> >>>>> 0 9
> >>>>> 0 10
> >>>>> -101 11
> >>>> It looks to me as if you started to define a netCDF user-defined type
> >>>> named "ironBoundaryType", but didn't finish that definition.  Then you
> >>>> tried to define netCDF variables of the incompletely defined type.
> >>>> It's a bug that the netCDF API lets you do this without returning an
> >>>> error until you close the file.  I'm not sure whether there's also a
> >>>> corresponding bug in HDF5 that allows this.
> >>>>
> >>>> To complete the definition of the user-defined type, you need to fill
> >>>> out the type with repeated calls to nc_insert_compound(). Call the
> >>>> nc_insert_compound function once for each field (member) you wish to
> >>>> insert into the compound type.  Don't define variables using a type
> >>>> until you finish defining the type.
> >>>>
> >>>> I'll enter a Jira ticket for this later and try to determine where the 
> >>>> bug
> >>>> is, but it may have to wait until after we get the 4.3 release for the C
> >>>> library out ...
> >>>>
> >>>> --Russ
> >> Russ Rew                                         UCAR Unidata Program
> >> address@hidden                      http://www.unidata.ucar.edu
> >>
> >>
> > Russ Rew                                         UCAR Unidata Program
> > address@hidden                      http://www.unidata.ucar.edu
> >
> >
> >
> > Ticket Details
> > ===================
> > Ticket ID: PFU-753378
> > Department: Support netCDF
> > Priority: Normal
> > Status: Closed
> >
> 
> 
Russ Rew                                         UCAR Unidata Program
address@hidden                      http://www.unidata.ucar.edu



Ticket Details
===================
Ticket ID: PFU-753378
Department: Support netCDF
Priority: Normal
Status: Closed