[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[netCDF #ZBS-432143]: NcFile memory leak (?)

> Greg,
>
> The memory leak is in the underlying C library, not in the C++
> layer.  It only occurs when opening netCDF-4 files, not
> equivalent netCDF-3 files, so it might even be in the underlying
> HDF5 layer, but that seems unlikely.  We'll continue to
> investigate ...
>
> --Russ
>
> Russ Rew                                         UCAR Unidata Program
> address@hidden                     http://www.unidata.ucar.edu
>
>

Howdy Russ and Greg!

I believe that there is no memory leak. I think the problem is that we are
misinterpreting the fields of the mallinfo struct.

I found some documentation here:
http://www.gnu.org/software/libtool/manual/libc/Statistics-of-Malloc.html

Here's some relevant parts:
int arena
    This is the total size of memory allocated with sbrk by malloc, in bytes.
int uordblks
    This is the total size of memory occupied by chunks handed out by malloc.
int fordblks
    This is the total size of memory occupied by free (not in use) chunks.

When reduced to the simplest test case, your test program came down to reading
the same test file twice, which resulted in different arena values after file
closes. This seems to imply that memory leaked.

But a carefull reading of the fordblks documentation leads me to think that the
GNU malloc is doing some last-minute harvesting of memory (i.e. when the
program exits).

The mallinfo structure shows the following after the two file closes:

{arena = 679936, ordblks = 258, smblks = 472, hblks = 0, hblkhd = 0, usmblks =
0, fsmblks = 21232, uordblks = 227344, fordblks = 452592, keepcost = 78960}

{arena = 1261568, ordblks = 141, smblks = 28, hblks = 0, hblkhd = 0, usmblks =
0, fsmblks = 1776, uordblks = 243872, fordblks = 1017696, keepcost = 117568}

Note that in the second case, the arena is higher, leading us to think there is
a leak, but the fordblks is also higher. So the program may have freed the
memory properly, but that's not yet reflected in the mallinfo.arena field.
Since the program is still running at this point, we have not given the program
time to clean up.

On the other hand, when I run the program with valgrind I get it's assurance
that "no memory leaks are possible." I like that kind of confidence!

 ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 4 from 1)
==20527== malloc/free: in use at exit: 0 bytes in 0 blocks.
==20527== malloc/free: 339,048 allocs, 339,048 frees, 79,875,278 bytes
allocated.
==20527== For counts of detected errors, rerun with: -v
==20527== All heap blocks were freed -- no leaks are possible.

(I also think this is a crazy amount of mallocs and frees. Is the HDF5 library
doing all this? I don't know.)

Also, using valgrind seems to turn off the fancy memory management too, because
the arena number remains unchanged no matter how many times I open and close
the file. That is, valgrind makes this entire problem go away! ;-)

So my conclusion is that there is no memory leak here, and what we are seeing
with the mallinfo.arena field is an artifact of memory management.

Please let me know if this doesn't seem right.

Thanks,

Ed

Ticket Details
===================
Ticket ID: ZBS-432143
Department: Support netCDF
Priority: Urgent
Status: Closed