[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[netCDF #ETC-878093]: segmentation fault of ncdump in netCDF-4

Hi Kent,

These bugs are finally fixed and committed, and the fixes will be included in 
netCDF release 4.2.1, as well as a snapshot release that should be created
tonight.

There were actually no bugs discovered in nccopy, all the problems were in 
the netCDF libsrc4 directory.  I've tested the fixes on all the .h5 files you
supplied, and ncdump now appears to display the fixed- and variable-length
strings correctly.

At your convenience, we would appreciate it if you could verify that the code 
now
works as intended, either when we announce a 4.2.1-beta release or in one of the
snapshot releases available earlier.

--Russ

> How is the file_spaceid defined?I don't understand why it fail when closing
> the file space handle.
> I still think the problem is due to the mismatched string size, though
> since h5dump can dump the data successfully.
> ncdump must malloc memory for the string beforehand? Is that size correct?
> 
> Kent
> 
> 
> >
> > Oops, sorry, the segmentation violation actually occurs after the call to
> > H5DRead() in the subsequent call to H5SClose:
> >
> > /**   if (var->xtype == NC_CHAR && mem_typeid > 0 && H5Tclose(mem_typeid)
> > < 0)
> >      BAIL2(NC_EHDFERR);*/
> >   if (file_spaceid > 0)
> >   {
> >      if (H5Sclose(file_spaceid) < 0)
> >
> > which gets a segfault in calling malloc:
> >
> > Program received signal SIGSEGV, Segmentation fault.
> > 0x000000313a677c0e in _int_malloc () from /lib64/libc.so.6
> >
> 
> 
> 
> >
> > --Russ
> > >
> > > > address@hidden> wrote:
> > > >
> > > > > Kent,
> > > > >
> > > > > I tried running ncdump on this file using valgrind, and got some more
> > > > > information that might be of use in debugging the problem.
> > > > >
> > > > > --Russ
> > > > >
> > > > > $ valgrind ./ncdump grid_1_3d_xyz_aug.h5
> > > > > group: HDFEOS\ INFORMATION {
> > > > >  variables:
> > > > >        char StructMetadata.0 ;
> > > > >
> > > > >  // group attributes:
> > > > >                :HDFEOSVersion = "HDFEOS_5.1.13" ;
> > > > >  data:
> > > > >
> > > > > ==19314== Source and destination overlap in memcpy(0x5c97b90,
> > 0x5c99550,
> > > > > 32000)
> > > > > ==19314==    at 0x4A075E6: memcpy (mc_replace_strmem.c:635)
> > > > > ==19314==    by 0x4EB5ED2: H5D_contig_readvv_sieve_cb
> > (H5Dcontig.c:796)
> > > > > ==19314==    by 0x50780A2: H5V_opvv (H5V.c:1453)
> > > > > ==19314==    by 0x4EB5B1F: H5D_contig_readvv (H5Dcontig.c:887)
> > > > > ==19314==    by 0x4EC57FC: H5D_select_io (H5Dselect.c:149)
> > > > > ==19314==    by 0x4EC5C3C: H5D_select_read (H5Dselect.c:273)
> > > > > ==19314==    by 0x4EB57CC: H5D_contig_read (H5Dcontig.c:559)
> > > > > ==19314==    by 0x4EC13E8: H5D_read (H5Dio.c:447)
> > > > > ==19314==    by 0x4EC1848: H5Dread (H5Dio.c:173)
> > > > > ==19314==    by 0x44F888: nc4_get_vara (nc4hdf.c:1051)
> > > > > ==19314==    by 0x45D4C5: nc4_get_vara_tc (nc4var.c:1467)
> > > > > ==19314==    by 0x45D56B: NC4_get_vara (nc4var.c:1484)
> > > > > ==19314==
> > > > > ==19314== Invalid write of size 8
> > > > > ==19314==    at 0x4A0765B: memcpy (mc_replace_strmem.c:635)
> > > > > ==19314==    by 0x4EB5ED2: H5D_contig_readvv_sieve_cb
> > (H5Dcontig.c:796)
> > > > > ==19314==    by 0x50780A2: H5V_opvv (H5V.c:1453)
> > > > > ==19314==    by 0x4EB5B1F: H5D_contig_readvv (H5Dcontig.c:887)
> > > > > ==19314==    by 0x4EC57FC: H5D_select_io (H5Dselect.c:149)
> > > > > ==19314==    by 0x4EC5C3C: H5D_select_read (H5Dselect.c:273)
> > > > > ==19314==    by 0x4EB57CC: H5D_contig_read (H5Dcontig.c:559)
> > > > > ==19314==    by 0x4EC13E8: H5D_read (H5Dio.c:447)
> > > > > ==19314==    by 0x4EC1848: H5Dread (H5Dio.c:173)
> > > > > ==19314==    by 0x44F888: nc4_get_vara (nc4hdf.c:1051)
> > > > > ==19314==    by 0x45D4C5: nc4_get_vara_tc (nc4var.c:1467)
> > > > > ==19314==    by 0x45D56B: NC4_get_vara (nc4var.c:1484)
> > > > > ==19314==  Address 0x5c97b90 is 0 bytes inside a block of size 1
> > alloc'd
> > > > > ==19314==    at 0x4A06031: malloc (vg_replace_malloc.c:236)
> > > > > ==19314==    by 0x41055F: emalloc (utils.c:41)
> > > > > ==19314==    by 0x40AB4D: vardata (vardata.c:484)
> > > > > ==19314==    by 0x408896: do_ncdump_rec (ncdump.c:1579)
> > > > > ==19314==    by 0x408A56: do_ncdump_rec (ncdump.c:1618)
> > > > > ==19314==    by 0x408BC7: do_ncdump (ncdump.c:1655)
> > > > > ==19314==    by 0x409E1E: main (ncdump.c:2432)
> > > > > ==19314==
> > > > > ==19314== Invalid read of size 4
> > > > > ==19314==    at 0x4F481B8: H5I_object_verify (H5I.c:2130)
> > > > > ==19314==    by 0x4FA5B47: H5Sclose (H5S.c:364)
> > > > > ==19314==    by 0x44FB87: nc4_get_vara (nc4hdf.c:1134)
> > > > > ==19314==    by 0x45D4C5: nc4_get_vara_tc (nc4var.c:1467)
> > > > > ==19314==    by 0x45D56B: NC4_get_vara (nc4var.c:1484)
> > > > > ==19314==    by 0x414D37: NC_get_vara (dvarget.c:34)
> > > > > ==19314==    by 0x415963: nc_get_vara (dvarget.c:460)
> > > > > ==19314==    by 0x40ADD4: vardata (vardata.c:537)
> > > > > ==19314==    by 0x408896: do_ncdump_rec (ncdump.c:1579)
> > > > > ==19314==    by 0x408A56: do_ncdump_rec (ncdump.c:1618)
> > > > > ==19314==    by 0x408BC7: do_ncdump (ncdump.c:1655)
> > > > > ==19314==    by 0x409E1E: main (ncdump.c:2432)
> > > > > ==19314==  Address 0x4a424f5f444e4509 is not stack'd, malloc'd or
> > > > > (recently) free'd
> > > > > ==19314==
> > > > > ==19314==
> > > > > ==19314== Process terminating with default action of signal 11
> > (SIGSEGV)
> > > > > ==19314==  General Protection Fault
> > > > > ==19314==    at 0x4F481B8: H5I_object_verify (H5I.c:2130)
> > > > > ==19314==    by 0x4FA5B47: H5Sclose (H5S.c:364)
> > > > > ==19314==    by 0x44FB87: nc4_get_vara (nc4hdf.c:1134)
> > > > > ==19314==    by 0x45D4C5: nc4_get_vara_tc (nc4var.c:1467)
> > > > > ==19314==    by 0x45D56B: NC4_get_vara (nc4var.c:1484)
> > > > > ==19314==    by 0x414D37: NC_get_vara (dvarget.c:34)
> > > > > ==19314==    by 0x415963: nc_get_vara (dvarget.c:460)
> > > > > ==19314==    by 0x40ADD4: vardata (vardata.c:537)
> > > > > ==19314==    by 0x408896: do_ncdump_rec (ncdump.c:1579)
> > > > > ==19314==    by 0x408A56: do_ncdump_rec (ncdump.c:1618)
> > > > > ==19314==    by 0x408BC7: do_ncdump (ncdump.c:1655)
> > > > > ==19314==    by 0x409E1E: main (ncdump.c:2432)
> > > > >   StructMetadata.0 = ==19314==
> > > > > ==19314== HEAP SUMMARY:
> > > > > ==19314==     in use at exit: 1,833,234 bytes in 2,909 blocks
> > > > > ==19314==   total heap usage: 10,658 allocs, 7,749 frees, 2,383,459
> > bytes
> > > > > allocated
> > > > > ==19314==
> > > > > ==19314== LEAK SUMMARY:
> > > > > ==19314==    definitely lost: 976 bytes in 9 blocks
> > > > > ==19314==    indirectly lost: 0 bytes in 0 blocks
> > > > > ==19314==      possibly lost: 0 bytes in 0 blocks
> > > > > ==19314==    still reachable: 1,832,258 bytes in 2,900 blocks
> > > > > ==19314==         suppressed: 0 bytes in 0 blocks
> > > > > ==19314== Rerun with --leak-check=full to see details of leaked
> > memory
> > > > > ==19314==
> > > > > ==19314== For counts of detected and suppressed errors, rerun with:
> > -v
> > > > > ==19314== ERROR SUMMARY: 692 errors from 3 contexts (suppressed: 4
> > from 4)
> > > > > Memory fault(coredump)
> > > > >
> > > > >
> > > > > > New Ticket: segmentation fault of ncdump in netCDF-4
> > > > > >
> > > > > > Hi Russ,
> > > > > >
> > > > > > For some time we've observed that ncdump cannot dump some HDF5
> > string
> > > > > data
> > > > > > properly. It caused segmentation fault. However, ncdump -H can
> > dump the
> > > > > > header successfully.
> > > > > > The files  are augmented HDF-EOS5 files that are  netCDF-4
> > compliant.
> > > > > > We observed this problem in version netCDF-4.1.1. With the current
> > > > > > netCDF-4.1.3, we still saw the same problem.
> > > > > >
> > > > > > You can find two example files:
> > > > > >
> > > > > >
> > > > >
> > ftp://ftp.hdfgroup.uiuc.edu/pub/outgoing/donglm/netcdf_seg_fault/grid_1_3d_xy
> > > > > > z_aug.h5
> > > > > >
> > > > >
> > ftp://ftp.hdfgroup.uiuc.edu/pub/outgoing/donglm/netcdf_seg_fault/za_1_3d_yztd
> > > > > > _aug.h5
> > > > > >
> > > > > > A detailed report can be found:
> > > > > >
> > > > >
> > ftp://ftp.hdfgroup.uiuc.edu/pub/outgoing/donglm/netcdf_seg_fault/error_report
> > > > > > .txt
> > > > > >
> > > > > > The h5dump output of one file can be found:
> > > > > >
> > > > >
> > ftp://ftp.hdfgroup.uiuc.edu/pub/outgoing/donglm/netcdf_seg_fault/output_Struc
> > > > > > tMetadata.0_h5dump
> > > > > >
> > > > > > We tested it on a linux 32-bit machine. I think it happens on all
> > > > > platforms.
> > > > > >
> > > > > > This string  is just a simple fixed size string. Although the size
> > is
> > > > > > relatively large(32000), it should be handled.
> > > > > > Let me know if you need more information.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Kent
> > > > > > --
> > > > > > Kent Yang The HDF Group
> > > > > > 1800 So. Oak St., Suite 203, Champaign, IL 61820
> > > > > > 217.531.6107
> > > > > > http://hdfgroup.org http://hdfeos.org
> > > > > >
> > > > > >
> > > > > >
> > > > > > Ticket Details
> > > > > > ===================
> > > > > > Ticket ID: ETC-878093
> > > > > > Department: Support netCDF
> > > > > > Priority: Normal
> > > > > > Status: Open
> > > > > > Link:
> > > > >
> > https://www.unidata.ucar.edu/esupport/staff/index.php?_m=tickets&_a=vi
> > > > > > ewticket&ticketid=19510
> > > > >
> > > > >
> > > > >
> > > > > Ticket Details
> > > > > ===================
> > > > > Ticket ID: ETC-878093
> > > > > Department: Support netCDF
> > > > > Priority: Normal
> > > > > Status: Closed
> > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > Kent Yang The HDF Group
> > > > 1800 So. Oak St., Suite 203, Champaign, IL 61820
> > > > 217.531.6107
> > > > http://hdfgroup.org http://hdfeos.org
> > > >
> > > >
> > > Russ Rew                                         UCAR Unidata Program
> > > address@hidden                      http://www.unidata.ucar.edu
> > >
> > >
> > Russ Rew                                         UCAR Unidata Program
> > address@hidden                      http://www.unidata.ucar.edu
> >
> >
> >
> > Ticket Details
> > ===================
> > Ticket ID: ETC-878093
> > Department: Support netCDF
> > Priority: Normal
> > Status: Closed
> >
> >
> 
> 
> --
> Kent Yang The HDF Group
> 1800 So. Oak St., Suite 203, Champaign, IL 61820
> 217.531.6107
> http://hdfgroup.org http://hdfeos.org
> 
> 

Russ Rew                                         UCAR Unidata Program
address@hidden                      http://www.unidata.ucar.edu



Ticket Details
===================
Ticket ID: ETC-878093
Department: Support netCDF
Priority: Critical
Status: Closed