[netcdfgroup] bm_file...but for netCDF-4?

• To: "netcdfgroup@xxxxxxxxxxxxxxxx" <netcdfgroup@xxxxxxxxxxxxxxxx>
• Subject: [netcdfgroup] bm_file...but for netCDF-4?
• From: "Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC]" <matthew.thompson@xxxxxxxx>
• Date: Thu, 22 Oct 2020 19:17:57 +0000
• Arc-authentication-results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nasa.gov; dmarc=pass action=none header.from=nasa.gov; dkim=pass header.d=nasa.gov; arc=none
• Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Y1i6W796XK4vcMb77S11VxTCacvY+YsbYXjLJCCm5tk=; b=Hft4/Tcd2IxJmMsW6RYtgOmJyCyQNNtEpA6XUVwE98/ZLWqVHT2M38qSRsHc8fx9jlsS6dHMFXZ7zdB5UBwCUkrhi5jnNUUqwpLDmIeVS+6iceN3Sjpzb9L3FrLREKYbot5k9o2QFD2ZgQJ02vgeG/sWQ6mkBaBaMt0EOU1q8Y2RrutGiWLmSKpvRtwbPK3mpXF4vCUs1E4rHlSER1vTJWpI0Wf8nwCSxP7Lrn2pbgNIsMy09ngqcjtijemjc9ApNq0xNj7quhKR4aUojXEOu0v+fRsrYARJXZXEX0OpZocrjDqGcLclda+Lsu5mhDnoSzm2l0jKMJdR9FvssoNw8Q==
• Arc-seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=k56g7WpZTQV8HU+Mh3HL6GTdMmb51InJR3tqugOWKluSPQaG5VAPY5MdK3FRiiebkDVQ9yEuuLn+9haK84LCJBf+BdDVA218vCwYv9bldNUFjS7ZP50+W9EFC40eB6GCW4TR4aaJ+C3ZSYwH1/docZ5EK5bOz985y8nSqKx79L6CvBQnC6TTcoMbbyHECqZ5koppTq96YBKW+hZGDstZPwRukkW5n62mnR8CFCe8RfBoJf8pERqBVq9B0sw0yN0/VWV+0TO78r5OyD/SFbXlqe2qlOpVzbQw8rPDIPVlODaX9/PGTlEh8chJGzNcfyQ3SaLytN349jmDz3oVxg/ByQ==
• Authentication-results: unidata.ucar.edu; dkim=none (message not signed) header.d=none; unidata.ucar.edu; dmarc=none action=none header.from=nasa.gov;

All,

A simple question. We've recently been encountering some rather slow 
compression speeds with netCDF files and we are trying to figure out what might 
be the cause. We aren't sure if it's our disks, or maybe we are chunking the 
files incorrectly, or...whatever, so I looked around and found this page:

https://www.unidata.ucar.edu/software/netcdf/docs/netcdf_perf_chunking.html

and the bm_file program. I whipped up a netCDF-C with --enable-benchmarks and 
built it and tried a test with a "not sure how to use it" "-c" option (still 
puzzling over what it means) on a file of mine and:

$ bm_file -v -h -d -f 3 -o test.nc4 -c 0:-1:0:4:4:4  
stock-gcm-2020Oct15-1day-c180.geosgcm_prog.20150415_1800z.nc4
copying stock-gcm-2020Oct15-1day-c180.geosgcm_prog.20150415_1800z.nc4 to 
test.nc4 on 1 processors with endianness 0 and...
slow_count=10, doublecheck=1
0: reading metadata took 122104 micro-seconds.
Sorry! Unexpected result, bm_file.c, line: 506 - NetCDF: Attempting netcdf-3 
operation on netcdf-4 file

And, yeah, if you look at the source:

    /* Only classic model files may be used as input. */
    if ((ret = nc_inq_format(ncid_in, in_format)))
        ERR1(ret);
    if (*in_format == NC_FORMAT_NETCDF4)
        ERR1(NC_ENOTNC3);

but the top also says:

  This program only works on classic model netCDF files. That is,
  groups, user-defined types, and other new netCDF-4 features are not
  handled by this program. (Input files may be in netCDF-4 format, but
  they must conform to the classic model for this program to work.)

Now the file I'm trying this on is pretty boring. We don't do much exciting in 
like 99% of our netCDF output unless maybe we have some weird metadata:

// global attributes:
           :_NCProperties = "version=2,netcdf=4.7.4,hdf5=1.10.6," ;
           :_SuperblockVersion = 0 ;
           :_IsNetcdf4 = 1 ;
           :_Format = "netCDF-4" ;

Any ideas what I might be able to do to make this file work with bm_file? Or, 
perhaps, is there a netCDF-4 equivalent of it?

Matt

PS: Or, I suppose, any hints/thoughts on how to make deflation faster in netCDF 
(netCDF-Fortran, technically)?
--
Matt Thompson, SSAI, Ld Scientific Programmer/Analyst
NASA GSFC,    Global Modeling and Assimilation Office
Code 610.1,  8800 Greenbelt Rd,  Greenbelt,  MD 20771
Phone: 301-614-6712                 Fax: 301-614-6246
http://science.gsfc.nasa.gov/sed/bio/matthew.thompson

• Follow-Ups:
  • Re: [netcdfgroup] bm_file...but for netCDF-4?
    • From: Ed Hartnett