« AWIPS Tips: AWIPS... | Main | NSF Unidata Update:... »

Version 5.2.1 of the netCDF Operators (NCO) has been released. NCO is an Open Source package that consists of a dozen standalone, command-line programs that take netCDF files as input, then operate (e.g., derive new data, average, print, hyperslab, manipulate metadata) and output the results to screen or files in text, binary, or netCDF formats.

The NCO project is coordinated by Professor Charlie Zender of the Department of Earth System Science, University of California, Irvine. More information about the project, along with binary and source downloads, are available on the SourceForge project page.

From the release message:

Version 5.2.1 fixes an issue with ncremap and ncclimo in MPI mode. Another small fix to enables GCC compilation in pedantic mode. No new features are implemented, but it was too late to recall 5.2.0.

Version 5.2.0 includes four major new features and various fixes. The features: 1) All operators append draft CF Convention behavior for metadata to encode lossy compression. 2) ncclimo timeseries mode now supports all input methods (including automatic filename generation) long-supported by climo mode. 3) ncremap Make-Weight-File (MWF) mode has been revamped and now support specifiable lists of algorithms. Last but not least, 4) ncks --s1d now converts CLM/ELM restart files from their native, inscrutable sparse 1-D (S1D) format to normal-looking gridded files, without loss of information.

New Features

1. ncks can now help analyze initial condition and restart datasets produced by the E3SM ELM and CESM CLM/CTSM land-surface models. Whereas gridded history datasets from these ESMs use a standard gridded data format, these land-surface "restart files" employ a custom packing format that unwinds multi-dimensional data into sparse, 1-D (S1D) arrays that are not easily visualized. ncks can now convert these S1D files into gridded datasets where all dimensions are explicitly declared (rather than unrolled or "packed"). Invoke this conversion feature with the --s1d option and point ncks to a file that contains the horizontal coordinates (which restart files do not explicitly contain) and the restart file. The output file is the fully gridded input file, with no loss of information:

   ncks --s1d --hrz=elmv3_history.nc elmv3_restart.nc out.nc

   The output file contains all input variables placed on a lat-lon or unstructured grid, with new dimensions for Plant Funtional Type (PFT) and multiple elevation class (MEC).
   http://nco.sf.net/nco.html#s1d
2. ncclimo timeseries mode now supports all input methods (including automatic filename generation) long-supported by climo mode. Previously ncclimo (in timeseries mode) had to receive explicit lists of input files, either from stdin or from the command line. Now ncclimo will automatically generate the input file list for files that adhere to common CESM/E3SM naming conventions (usually for monthly average files). The syntax is identical to that long used in climo mode:

   % ncclimo --split -c $caseid -s 2000 -e 2024 -i $drc_in -o $drc_out

   http://nco.sf.net/nco.html#ncclimo
3. ncremap supports --alg_lst=alg_lst, a comma-separated list of the algorithms that MWF-mode uses to create map-files. This option can be used to shorten or alter the default list, which is 'esmfaave,esmfbilin,ncoaave,ncoidw,traave,trbilin,trfv2,trintbilin'. Each name in the list should be the primary name of an algorithm, not a synonym. For example, use 'esmfaave,traave' not 'aave,fv2fv_flx' (the latter are backward-compatible synonyms for the former). The algorithm list must be consistent with grid-types supplied: ESMF algorithms work with meshes in ESMF, SCRIP, or UGRID formats. NCO algorithms only work with meshes in SCRIP format. TempestRemap algorithms work with meshes in ESMF, Exodus, SCRIP, or UGRID formats. On output, ncremap inserts each algorithm name into the output map-file name in this format: map_src_to_dst_alg.date.nc. For example,

   % ncremap -P mwf --alg_lst=esmfnstod,ncoaave,ncoidw,traave,trbilin \
   -s ocean.QU.240km.scrip.181106.nc -g ne11pg2.nc --nm_src=QU240 \
   --nm_dst=ne11pg2 --dt_sng=20240201
   ...
   % ls map*
   map_QU240_to_ne11pg2_esmfnstod.20240201.nc
   map_QU240_to_ne11pg2_ncoaave.20240201.nc
   map_QU240_to_ne11pg2_ncoidw.20240201.nc
   map_QU240_to_ne11pg2_traave.20240201.nc
   map_QU240_to_ne11pg2_trbilin.20240201.nc
   map_ne11pg2_to_QU240_esmfnstod.20240201.nc
   map_ne11pg2_to_QU240_ncoaave.20240201.nc
   map_ne11pg2_to_QU240_ncoidw.20240201.nc
   map_ne11pg2_to_QU240_traave.20240201.nc
   map_ne11pg2_to_QU240_trbilin.20240201.nc
   

   http://nco.sf.net/nco.html#alg_lst
   http://nco.sf.net/nco.html#ncremap
4. All NCO operators now support the draft CF Convention on encoding metadata that describes lossy compression applied to the dataset. See https://github.com/cf-convention/cf-conventions/issues/403. For example, all variables quantized by NCO now receive attributes that contain the level of quantization and that point to a container variable that describes the algorithm:

   % ncks -O -7 --cmp='btr|shf|zst' in.nc foo.nc
   % ncks -m -v ts foo.nc
   char compression_info ;
   char compression_info ;
   compression_info:family = "quantize" ;
   compression_info:algorithm = "BitRound" ;
   compression_info:implementation = "libnetcdf version 4.9.3-development" ;
   float ts(time,lat,lon) ;
   ts:standard_name = "surface_temperature" ;
   ts:lossy_compression = "compression_info" ;
   ts:lossy_compression_nsb = 9 ;
   

   http://nco.sf.net/nco.html#qnt
5. ncks supports a new flag, --chk_bnd, that reports whether all coordinate variables in a file contain associated "bounds" variables. This check complies with CF Conventions and with NASA's Dataset Interoperability Working Group (DIWG) recommendations:

   $ ncks --chk_bnd ~/nco/data/in.nc
   ncks: WARNING nco_chk_bnd() reports coordinate Lat does not contain
   "bounds" attribute
   ncks: WARNING nco_chk_bnd() reports coordinate Lon does not contain
   "bounds" attribute
   ncks: INFO nco_chk_bnd() reports total number of coordinates without
   "bounds" attribute is 2
   

   http://nco.sf.net/nco.htlm/chk_bnd
6. ncremap supports the TempestRemap trfv2 algorithm, a 2nd order FV reconstruction, that is cell-integrated on the target grid.

   ncremap --alg_typ=trfv2 -s grd_src.nc -g grd_dst.nc --map=map.nc

   http://nco.sf.net/nco.htlm/trfv2

Additional details are available in the ChangeLog.