[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[netCDF #PEB-847323]: Re: [netcdf-hdf] NF90_GET_VAR memory leak?

Hi Ted,

In trying to set up an example to see if I can reproduce the problem
you are seeing and isolate it to either the C interface or the HDF5
layer, a question about how you are processing the data arises.

Currently your outer loop is on variables, with a loop on times inside
that and a loop on files inside that.  Do you open all 100 files first
and keep them all open during the reading of a subset of the data to
write the output?  Or do you open each input file before reading its data
and close it before opening the next input file?

The reason I ask is that in the former case, a chunk cache is allocated
for each open file as a place to keep uncompressed chunks so that the
uncompression only needs to be done once and not many times for each
variable.  But this means you have 100 chunk caches.  If you either
had files as the outer loop variable while processing the files sequentially
or at least closed each file before opening the next one so that you only
had one input file open at a time, you would only need one chunk cache
allocated at a time.  What you think is a memory leak may instead be merely
the many large chunk caches that must be kept in memory for each open
file.

If you run "ncdump -h -s" on one of the input files and send the output, we
can tell what the size of chunk caches are.  The default chunk cache sizes
changed between releases 4.0 and 4.0.1.

--Russ


>  Ted Mansell <address@hidden> writes:
> 
> ... the files being read have compression (deflate)
> turned on, and are netCDF-4 classic model (no fancy stuff here). I
> have not tried with reading netcdf3 files, either.
> 
> I am seeing a memory leak in NF90_GET_VAR (hdf5-1.8.2/netcdf-4.0.1
> and previous release version netcdf-4.0). I am reading multiple (say
> 100) netcdf-4 files to combine the data into a single file. (Each
> file contains a rectangular portion of a contiguous grid). The data
> variables are 3D plus an unlimited time dimension. It appears that
> the memory increase occurs on the first read of a given variable
> from a given file, and reading successive times of that variable
> doesn't cause any further increase, that is for successive values of
> 'k' in
>
> DO ivar = 1,nvariables
>
> ....
>
> DO k = 1,ntimes
> DO jj = 1,nfiles
> ...
> ! read portion of 3d data from each file:
> status = NF90_GET_VAR
> (ncid(jj),varid1,var3D(istart:istop,jstart:jstop,:),start=(/
> 1,1,1,k/))
>
> ENDDO
>
> ! write big combined 3d data
> status = NF90_PUT_VAR(ncid1,varid2,var3D,(/1,1,1,k/))
>
> ENDDO
>
> ENDDO
> 
> I'm guessing that the memory leak is about the size of the 3d array
> that is read, but it's hard to say. After reading 43 3d variables
> from 100 files, the memory usage (shown by 'top') has grown by
> almost 2GB, which is about the total data size of the first reads
> (1.8GB).  That could be coincidence but consistent with a second,
> different dataset. In 'top', I see the memory jump at first 'k'
> iteration.  If I comment out only the NF90_GET_VAR call then the
> memory usage doesn't change at all. When another program reads in
> data from the single big netcdf file, I don't think I see any undue
> memory usage -- just the total data size for one time,
> approximately.
> 
> I have no idea if this is a netcdf or hdf5 issue. Since I'm calling
> from fortran I wonder if there is some temporary memory used to
> switch the data from C to Fortran array ordering that doesn't get
> released?
>
> Thanks for any help, and I'm willing to test fixes.

Russ Rew                                         UCAR Unidata Program
address@hidden                     http://www.unidata.ucar.edu



Ticket Details
===================
Ticket ID: PEB-847323
Department: Support netCDF
Priority: Normal
Status: Closed