Version 5.4.0 of the netCDF Operators (NCO) has been released. NCO is an Open Source package that consists of a dozen standalone, command-line programs that take netCDF files as input, then operate (e.g., derive new data, average, print, hyperslab, manipulate metadata) and output the results to screen or files in text, binary, or netCDF formats.

The NCO project is coordinated by Professor Charlie Zender of the Department of Earth System Science, University of California, Irvine. More information about the project, along with binary and source downloads, are available on the SourceForge project page.

New Features

  1. ncclimo now generates ensemble monthly climatologies. Simply feed the files from all ensemble members to ncclimo using any supported input method. Redirecting standard input is often most concise. Consider three ensemble members:

    # Example: Three Member Decadal-mean Ensemble Climatology
drc_top=/big/doe/computer/E3SMv3
caseid1=v3.LR.historical_0051
caseid2=v3.LR.historical_0101
caseid3=v3.LR.historical_0151
fl_nsm1=${drc_top}/${caseid1}/run/${caseid1}.elm.h0.185?-??.nc
fl_nsm2=${drc_top}/${caseid2}/run/${caseid2}.elm.h0.185?-??.nc
fl_nsm3=${drc_top}/${caseid3}/run/${caseid3}.elm.h0.185?-??.nc
ls ${fl_nsm1} ${fl_nsm2} ${fl_nsm3} | ncclimo --fml_nm=ensemble ...

    This approach scales to any number of ensemble members. All the usual climo-mode options, such as regridding, work. For N-member ensembles, ncclimo averages N-times as many months as a single member climo to get the mean January, February, etc. This takes about N-times longer. Seasonal and annual means are computed from monthly means so their speed is unchanged, as is regridding speed. For now, the output statistics are climatological means. Future work could support outputting standard deviations. Let us know what you would like to see in future versions.

  2. ncclimo --mth_end now defaults to 11 months after --mth_srt. Formerly, users specifying a non-January start month (--mth_srt) were required to also explicitly provide --mth_end, even though in climatology mode the end month must always be 11 months after the start month. Now --mth_end is automatically set to mth_srt+11 (mod 12) when it is not specified by the user. For example, specifying --mth_srt=5 now automatically sets --mth_end=4 (May through April annual cycle). Explicitly-specified --mth_end values are still accepted and take precedence. In timeseries mode, --mth_end may still be explicitly specified when the range of months spans a non-integral number of years. Alternatively, one can now avoid the --mth_srt and --mth_end options entirely even with non-January start months. Do this by specifying the years in YYYYMM format:

    ncclimo -s 2013 -e 2014 --mth_srt=5 ... # mth_end automatically set to 4
ncclimo -s 2013 -e 2014 --mth_srt=5 --mth_end=4 ... # same, explicit
ncclimo -s 201305 -e 201404 ... # Omit --mth_srt/end options

  3. The --rad (retain_all_dimensions) option is now supported by many data operators: ncap2, ncecat, nces, ncflint, ncpdq, and ncwa. Previously only ncks supported --rad. When --rad is specified, dimensions that are not referenced by any extracted variable (orphaned dimensions) are retained in the output file rather than silently dropped. The E3SM ELM model, for example, uses orphan dimensions to trigger different physics treatments. When ncks is invoked in printing mode, --rad also causes orphaned dimensions to be printed (they are not printed by default). Synonyms --retain_all_dimensions, --orphan_dimensions, and --rph_dmn are also accepted. Thanks to Jon Wolfe (LANL) for prompting this feature.

    ncap2 --rad -s 'one=1' in.nc out.nc # Keep all dimensions
nces --rad in.nc out.nc # Keep all dimensions
ncks --rad -m -v T in.nc # Print all dimensions

  4. All data operators now support the newish netCDF ability to set the metadata block size in netCDF4 files. Option --blk_sz allows users to specify sizes other than the default 2048 bytes. Large files chunked files often require much more space because the metadata block is where HDF5 stores the location of every chunk. If this list grows too long it must point to another metadata block and HDF5 must follow the trail, possibly through thousands of metadata blocks, just to find a single chunk. CMIP7 submissions will need to take advantage of larger sizes. One megabyte is a good size for large, chunked, netCDF4 files. Setting blk_sz=0 will reset the block size to the default value of the active netCDF4/HDF5 library (which may increase beyond 2048 B in the future). This option only affects HDF5 files.

    ncks -4 -O ~/nco/data/in.nc ~/foo.nc # Default 2048 B
ncks -4 -O --blk=0 ~/nco/data/in.nc ~/foo.nc # Default
ncks -4 -O --blk=1048576 ~/nco/data/in.nc ~/foo.nc # 1 MB
  5. ncremap now gracefully tolerates the absence of the reference surface pressure variable in hybrid sigma-pressure coordinate datasets. Previouly these variables (usually named P0 or P0 had to be contained in the the vertical grid of the input data. However, the de facto convention is that p0=P0=100000 Pa. Now when the reference pressure is missing (as is the case with older versions of EAMxx), ncremap supplies this value internally and completes without failing.
  6. ncremap now supports regridding MODIS MOD10CM datasets. The dimensions in those dataset have been added to ncremap's internal database so explicitly specifying which is latitude and which is longitude is unnecessary.
  7. ncclimo and ncremap now use the srun --exclusive option to prevent compute nodes from sharing their resources with concurrently running others jobs. This been shown to substantially increase throughput of some NCO-based analysis workflows such as LIVVkit. Thanks to Mike Kelleher (ORNL) for suggesting this.
  8. In ncremap the --alg option is now a synonym for --alg_typ. Just a small change to reduce typing.
  9. ncremap and ncclimo have been significantly refactored. They now run in "set -eou" mode and use modernized expression evaluation and test constructs that fail on any Bash-reported errors. This makes debugging easier. Both codes have now been tested with the most common use-cases. However, we might have missed some "unbound variables". Please send us an error report if you notice this. The workaround is to run with an earlier version of NCO.
Bug Fixes
  1. Previous versions of the BitGroom algorithm could produce incorrect answers when single-precision variables were quantized with NSD=7 significant digits. The maximum value now accepted is NSD=6. Thanks to Ed Hartnett for catching this.
  2. Certain combinations of climo-mode and input-mode could cause the CASEID to be prepended to filenames twice. The workaround is to use a different input method. The solution is to upgrade.

 

Full release statement at http://nco.sf.net/ANNOUNCE