Hi,
We have a storage shortage and we'd like to leverage the compression to
save space.
The idea works fine for some time.
However, I was testing different NetCDF versions and happens to find that
the nccopy to compressed NetCDF-4 file is magnitudes slower on version
4.6.0+.
The test method follows:
OS: CentOS 6.10
HDF5: 1.10.5
NetCDF: 4.4.1.1, 4.5.0, 4.6.2, 4.7.0, HPC-NetCDF 4.6.2.4 from NOAA
All building uses system-provided GCC suite 4.4.1.
The test is simple: nccopy a ~400MB classic NetCDF file to NetCDF-4 with:
nccopy -4 -d 1 -s input.nc output.nc4
With version v4.5.0 and lower, the process takes ~7 seconds.
HPC-NetCDF also gives similar speed.
(gzip-ing the same file takes similar time.)
But with version v4.6.2+, the copy can take ~25 minutes!
Our production system still uses 4.4.0, so the speed should be okay.
But I'd like to know more so I can plan the updates accordingly.
Mike Chen
Research Assistant / System Engineer
Dept. of Atmospheric Science
National Taiwan University
