All,
A simple question. We've recently been encountering some rather slow
compression speeds with netCDF files and we are trying to figure out what might
be the cause. We aren't sure if it's our disks, or maybe we are chunking the
files incorrectly, or...whatever, so I looked around and found this page:
https://www.unidata.ucar.edu/software/netcdf/docs/netcdf_perf_chunking.html
and the bm_file program. I whipped up a netCDF-C with --enable-benchmarks and
built it and tried a test with a "not sure how to use it" "-c" option (still
puzzling over what it means) on a file of mine and:
$ bm_file -v -h -d -f 3 -o test.nc4 -c 0:-1:0:4:4:4
stock-gcm-2020Oct15-1day-c180.geosgcm_prog.20150415_1800z.nc4
copying stock-gcm-2020Oct15-1day-c180.geosgcm_prog.20150415_1800z.nc4 to
test.nc4 on 1 processors with endianness 0 and...
slow_count=10, doublecheck=1
0: reading metadata took 122104 micro-seconds.
Sorry! Unexpected result, bm_file.c, line: 506 - NetCDF: Attempting netcdf-3
operation on netcdf-4 file
And, yeah, if you look at the source:
/* Only classic model files may be used as input. */
if ((ret = nc_inq_format(ncid_in, in_format)))
ERR1(ret);
if (*in_format == NC_FORMAT_NETCDF4)
ERR1(NC_ENOTNC3);
but the top also says:
This program only works on classic model netCDF files. That is,
groups, user-defined types, and other new netCDF-4 features are not
handled by this program. (Input files may be in netCDF-4 format, but
they must conform to the classic model for this program to work.)
Now the file I'm trying this on is pretty boring. We don't do much exciting in
like 99% of our netCDF output unless maybe we have some weird metadata:
// global attributes:
:_NCProperties = "version=2,netcdf=4.7.4,hdf5=1.10.6," ;
:_SuperblockVersion = 0 ;
:_IsNetcdf4 = 1 ;
:_Format = "netCDF-4" ;
Any ideas what I might be able to do to make this file work with bm_file? Or,
perhaps, is there a netCDF-4 equivalent of it?
Matt
PS: Or, I suppose, any hints/thoughts on how to make deflation faster in netCDF
(netCDF-Fortran, technically)?
--
Matt Thompson, SSAI, Ld Scientific Programmer/Analyst
NASA GSFC, Global Modeling and Assimilation Office
Code 610.1, 8800 Greenbelt Rd, Greenbelt, MD 20771
Phone: 301-614-6712 Fax: 301-614-6246
http://science.gsfc.nasa.gov/sed/bio/matthew.thompson
