Re: [netcdfgroup] HFD errors when updating netCDF4 files

• To: netcdfgroup@xxxxxxxxxxxxxxxx
• Subject: Re: [netcdfgroup] HFD errors when updating netCDF4 files
• From: Dave Allured - NOAA Affiliate <dave.allured@xxxxxxxx>
• Date: Thu, 26 Jan 2017 15:46:18 -0700

All,

We traced this problem to a known bug.  In NetCDF-4, the number of times an
attribute can be modified over the life of a file is currently limited by
the per-variable HDF5 attribute creation index.  This is a 16-bit counter
with maximum value 65535.

This becomes a problem for data sets with attributes that are updated
frequently.  At the NetCDF user level, the problem shows as return status
-107, or "NetCDF: Can't open HDF5 attribute", as reported below by Cathy
Smith.

Please see this Github issue for more details.  Thanks to Constantine
Khroulev for a great analysis, as well as previous reporters Heiko Klein
and Andrey Paramonov (HDF forum).

    https://github.com/Unidata/netcdf-c/issues/350

We look forward to some kind of fix at the NetCDF or HDF5 level.

--Dave A.
NOAA/ESRL/PSD/CIRES


On Tue, Nov 15, 2016 at 8:45 AM, Cathy Smith (NOAA Affiliate) <
cathy.smith@xxxxxxxx> wrote:

> Hi all
> We are trying to figure out why we have been getting a particular HDF5
> error when trying to change an attribute. I run update code for datasets
> via cron. At random times over the las year for at least 5 separate files
> in different datasets with different update code, we discovered we suddenly
> couldn't update the actual range attribute for a variable as the netCDF API
> did not see the attribute anymore. There was no  obvious cause . Once the
> file was "broken" it stayed that way. Trying to update or fix the attribute
> gives the errors below.
>
> To fix the file we either restore the file from backup or regenerate it
> using nccopy on the broken file.
>
> Someone here has done some extensive testing. They found that Fortran,
> NCO, and NCL can't work on the attribute. Also, h5edit can't. But, some
> HDF5  applications that view the file do not see an error. And the file
> gives no error with ncdump though it doesn't show the actual_range
> attribute.
>
> Has anyone seen this error before? Any ideas on the cause come to mind? We
> are debating NFS issues or library as the cause but are leaning towards the
> former, at least partially.
>
> I have a broken and working file in ncftp ftp.cdc.noaa.gov
> cd Public/csmith/netcdf
>
> Thanks for any insight
>
> Cathy Smith
> ESRL/PSD
>
> Commands we ran to show error:
> ncdump -h  precip.V1.0.2016.nc <http://precip.V1.0.2016.nc>
> <http://precip.V1.0.2016.nc> |grep act
>                     lat:actual_range = 20.125f, 49.875f ;
>                     lon:actual_range = 230.125f, 304.875f ;
>                     time:actual_range = 1893408., 1900824. ;
>
>     When I try to add it back, I get
>
>     ncatted -h -O -a actual_range,precip,c,f,"100000.,-100000."
>     precip.V1.0.2016.nc <http://precip.V1.0.2016.nc>
> <http://precip.V1.0.2016.nc>
>     nco_err_exit(): ERROR Short NCO-generated message (usually name of
>     function that triggered error): nco_enddef()
>     nco_err_exit(): ERROR Error code is -107. Translation into English
>     with nc_strerror(-107) is "NetCDF: Can't open HDF5 attribute"
>     nco_err_exit(): ERROR NCO will now exit with system call
>     exit(EXIT_FAILURE)
>
> and
>  ncatted -h -O -a actual_range,precip,o,f,"10000
>
>     0.,-100000." precip.V1.0.2016.nc <http://precip.V1.0.2016.nc>
> <http://precip.V1.0.2016.nc>
>     nco_err_exit(): ERROR Short NCO-generated message (usually name of
>     function that triggered error): nco_enddef()
>     nco_err_exit(): ERROR Error code is -107. Translation into English
> with
>     nc_strerror(-107) is "NetCDF: Can't open HDF5 attribute"
>     nco_err_exit(): ERROR NCO will now exit with system call
> exit(EXIT_FAILURE)
>
> ----------------------------------------------
>
>

• Follow-Ups:
  • Re: [netcdfgroup] HFD errors when updating netCDF4 files
    • From: dmh@xxxxxxxx