NOTE: The netcdf-hdf mailing list is no longer active. The list archives are made available for historical reasons.
Dear developers,I am seeing a memory leak in NF90_GET_VAR (hdf5-1.8.2/netcdf-4.0.1 and previous release version netcdf-4.0). I am reading multiple (say 100) netcdf-4 files to combine the data into a single file. (Each file contains a rectangular portion of a contiguous grid). The data variables are 3D plus an unlimited time dimension. It appears that the memory increase occurs on the first read of a given variable from a given file, and reading successive times of that variable doesn't cause any further increase, that is for successive values of 'k' in
DO ivar = 1,nvariables .... DO k = 1,ntimes DO jj = 1,nfiles ... ! read portion of 3d data from each file:status = NF90_GET_VAR (ncid(jj),varid1,var3D(istart:istop,jstart:jstop,:),start=(/1,1,1,k/))
ENDDO ! write big combined 3d data status = NF90_PUT_VAR(ncid1,varid2,var3D,(/1,1,1,k/)) ENDDO ENDDOI'm guessing that the memory leak is about the size of the 3d array that is read, but it's hard to say. After reading 43 3d variables from 100 files, the memory usage (shown by 'top') has grown by almost 2GB, which is about the total data size of the first reads (1.8GB). That could be coincidence but consistent with a second, different dataset. In 'top', I see the memory jump at first 'k' iteration. If I comment out only the NF90_GET_VAR call then the memory usage doesn't change at all. When another program reads in data from the single big netcdf file, I don't think I see any undue memory usage -- just the total data size for one time, approximately.
I have no idea if this is a netcdf or hdf5 issue. Since I'm calling from fortran I wonder if there is some temporary memory used to switch the data from C to Fortran array ordering that doesn't get released?
Thanks for any help, and I'm willing to test fixes. Best, -- Ted
netcdf-hdf archives: