[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[netCDF #AIX-376820]: NetCDF

Hi Charles,

> We're using NetCDF 4.0.1
> 
> I think I found a memeory leak when using compression.
> 
> Please see the attached sample file
> 
> When I run it in "docompression" mode, I can see the size of my
> process grow
> If I use the nocompression mode, then it is stable.
> 
> Can you see anything obvious I'm doing wrong when implementing the
> shuffle/deflate ?

No, it looks like your use of shuffle/deflate is OK.  The use of
stride[1] = 0 and stride[2] = 0 is actually an error that is caught
and would be returned when you call nc_put_vars, but this is
incidental.  I changed your code to

    ret = 
nc_def_var_deflate(fid,var,cdms_shuffle,cdms_deflate,cdms_deflate_level);
    if (ret!=0) {
       fprintf(stderr, "nc_def_var_deflate %s\n", nc_strerror(ret)); 
       fflush(stderr); 
       fflush(stdout); 
       exit(1);}
    }

which produced the output

  $ ./test_my_netcdf
  nc_def_var_deflate NetCDF: Illegal stride

then changed all the strides to 1 (which means put every element), and
it ran OK.  I verified that the process memory doesn't seem to grow
when docompression == 0 and it does seem to grow when docompression ==
1, but I don't think there is really a memory leak.  Running your
program under the Solaris "bcheck" memory leak detection utility
showed no leaks.  The way HDF5 uses memory can fool some programs,
such as "top" or "prstat" into showing what appears to be a memory
leak.  As one of the HDF5 developers said:

  ... I'll mention a possible HDF5 library-related one: we use
  internal "free list stores" to recycle commonly used data
  structures, etc.  Generally, this is a performance win, at the
  expense of using more memory.  However, it does tend to fool some
  memory leak tracking tools into thinking the memory is gone forever
  (we do release it all when our atexit() callback is called), or at
  least increasing indefinitely (it's not).

  You can compile the HDF5 library to avoid using the free list stores
  completely by defining the "H5_NO_FREE_LISTS" macro, or you can call
  H5garbage_collect() at runtime to release all the memory in the free
  lists back to the OS.  Some memory checking tools also have other
  false positive "errors" that can be suppressed by defining the
  "H5_USING_MEMCHECKER" macro during compile time (which also disables
  the free list stores).  For a UNIX build, this can be enabled with the
  "--enable-using-memchecker" configure option.

I tested your program using the current snapshot netCDF software
(4.1-beta2) and HDF5 version 1.8.3-snap2 (from July 1).  There were
some memory leaks fixed in both HDF5 and netCDF-4 in some of the
earlier releases.  If you find a leak that can be traced to the HDF5
layer, all we can do is report it to the HDF5 developers.  If your
program eventually crashes from running out of memory, it's probably a
real leak, but if you are just seeing memory use grow in a process
monitoring tool, it may be an illusion.

Please let us know if the program eventually runs out of memory or if
there is evidence of an actual memory leak other than reports from a
process monitoring utility.

--Russ

Russ Rew                                         UCAR Unidata Program
address@hidden                     http://www.unidata.ucar.edu



Ticket Details
===================
Ticket ID: AIX-376820
Department: Support netCDF
Priority: Normal
Status: Closed