Version 4.9.4 of the netCDF Operators (NCO) has been released. NCO is an Open Source package that consists of a dozen standalone, command-line programs that take netCDF files as input, then operate (e.g., derive new data, average, print, hyperslab, manipulate metadata) and output the results to screen or files in text, binary, or netCDF formats.
The NCO project is coordinated by Professor Charlie Zender of the Department of Earth System Science, University of California, Irvine. More information about the project, along with binary and source downloads, are available on the SourceForge project page.
From the release message:
Version 4.9.4 contains new features focused on de-interleaving time
coordinates and per-record weighting for
ncra, high-freqency (e.g.,
diurnally-resolved) climatologies and new defaults for
new distance-weight extrapolation algorithm for
nces. New NCO-wide features include: support for
unbuffered I/O that can speed-up I/O for large record variables,
more precise quantization than Bit Grooming, and faster arithmetic
that takes advantage of SIMD directives on OpenMP-enabled builds.
Overall release is packed with interesting new features...read on.
NCO may now be configured with
--enable-gpuat build-time to offload certain arithmetically intensive computations to the GPUs with select architectures and compilers. This feature currently has no speed benefits, and needs a volunteer to lead development. Please contact me if interested.
All operators now support unbuffered I/O with netCDF3 files when
invoked with the flag
--uioor longer synonyms
--share_all. This flag invokes the netCDF library NC_SHARE flag which enables unbuffered (non-cached) I/O. Unbufferend I/O may significantly reduce throughput time when large record variables are written or read. Performance improvements may depend on netCDF version. Thanks to Barron Henderson for this suggestion.
ncks -v T in.nc out.nc # Default, buffered I/O ncks --uio -v T in.nc out.nc # Unbuffered I/O ncra --uio -v T,Q,U,V in*.nc out.nchttp://nco.sf.net/nco.html#uio
The default quantization method has changed from Bit Grooming to
Bit Rounding, contributed by Rostislav Kouznetsov with suggestions
from Milan Klower. As implemented, Bit Rounding will substantially
improve the precision for the specified number of significant
digits to retain. A future version of NCO will re-tune the number
of bits per retained digit to turn this precision advantage into
a compression advantage. An article submitted by R. Kouznetsov to
GMD describes Bit Rounding more...precisely.
The multi-file, multi-record operators,
ncrcatnow support interleaved time-coordinates in groups of records. Interleaving (or de-interleaving, depending on one's perspective) means altering the order of records in a group to be processed. Specifically, the interleaving feature causes the operator to treat as sequential records those that are separated by multiples of the specified interleave parameter within a group of records. Specify the interleave parameter as the fifth hyperslab argument. The interleave feature sequences records with respect to their position relative to the beginning of each sub-cycle. Records a multiple of interleave from sub-cycle beginning are first extracted (by
ncrcat) or reduced (by
ncra), then records offset from these by one, two, et cetera up to interleave-1. Thus interleaving allows deconvolution of periodic phenomena within a time-series. Some examples to reify the abstract:
Let in1.nc = [1..10], in2.nc = [11..20], and in12.nc = [1..20].
ncra -d time,,,,10,5 in1.nc ~/foo.nc # 3.5, 4.5, 5.5, 6.5, 7.5 ncrcat -d time,0,4,,6,2 in1.nc ~/foo.nc 1, 3, 5, 2, 4, 6 (+WARNING) ncrcat -d time,2,,10,4,2 in12.nc ~/foo.nc # 3, 5, 4, 6, 13, 15, 14, 16 ncra -d time,2,,10,4,2 in12.nc ~/foo.nc # 4, 5, 14, 15 ncra -d time,,,,10,2 in1.nc in2.nc ~/foo.nc # 5, 6, 15, 16 ncra -d time,,,,10,2 in12.nc ~/foo.nc # 5, 6, 15, 16http://nco.sf.net/nco.html#interleave
ncesnow supports the
--wgt) option for per-file weights. This option is similar to the
ncra --wgtoption. The
ncesversion also accepts a variable name that contains a scalar per-file-weight. Per-file weights are useful when computing statistics of ensembles whose members should be weighted unevenly. Hence these three commands produce the same answers, though the second and third are much more flexible and can have non-integral weights:
nces 1.nc 2.nc 2.nc out.nc nces -w 1,2 1.nc 2.nc out.nc nces -w var 1.nc 2.nc out.nchttp://nco.sf.net/nco.html#nces
ncranow supports the
--prw) option to utilize command-line weights specified by
--wgt) for per-record weights instead of per-file-weights. This is useful when computing weighted averages with cyclically varying weights, since the weight given on the command line will be repeated for the length of the timeseries. Consider, for example, a CMIP6 timeseries of historical monthly mean emissions that one wishes to convert to an timeseries of annual-mean emissions. One can weight each month by its number of days via:
ncra --per_record_weights --mro -d time,,,12,12 --wgt \ 31,28,31,30,31,30,31,31,30,31,30,31 ~/monthly.nc ~/annual.ncThanks to Philip Cameron-Smith of LLNL for this suggestion.
ncraaccepts the new flag
--prm_ints) to output statistics of integer-valued input variables in floating-point precision in the output file. By default, NCO arithmetic operators such as
ncraauto-promote integers to double-precision prior to arithmetic, then conduct the arithmetic, then demote the values back to integers for final output. This default behavior quantizes the mantissa of the values and prevents, e.g., turning statistical means of boolean (0 or 1-valued) input data into floating point probabilities. The
--promote_intsflag causes the statistical means of integer (including NC_BYTE) inputs to be output as single-precision floating point (NC_FLOAT) variables. This allows use arithmetic to be performed on Boolean values stored in the space-conserving NC_BYTE (single byte) format in input files.
ncra --prm_ints in*.nc out.ncThanks to Paul Ullrich of UC Davis for this suggestion.
ncremapunderstands new dimensions used in DOE E3SM MPAS BGC simulations.
ncremapalso supports the new
--pdq_optto override internal presets and to future-proof itself against unexpected new dimensions from any model input.
ncremap -P mpasseaice --map=map.nc in.nc out.nc ncremap --pdq='-a Time,new_dim,nCells' --map=map.nc in.nc out.nc ncremap --pdq='-a time,new_dim,lat,lon' --map=map.nc in.nc out.ncThanks to Ahmed Elshall for reporting the new dimensions.
ncremapallows access to a new regridding algorithm based on distance-weighted extrapolation (DWE). DWE is similar to the ESMF nearestidavg extrapolation alorithm, and accepts the same two parameters as input:
--xtr_xpnsets the (absolute value of) the exponent used in distance weighting (default is 2.0), and
--xtr_nspsets the number of source points used in the extrapolation (default is 8).
ncremapcan apply DWE to the entire destination grid, or just to points with missing/masked values.
ncremap --alg_typ=nco_dwe -s src.nc -d dst.nc -m map.nc ncremap -a nco_dwe --xtr_xpn=1.0 -s src.nc -d dst.nc -m map.nc ncremap -a nco_dwe --xtr_nsp=1 -s src.nc -d dst.nc -m map.ncThanks to Henry Butowsky for implementing the new method.
ncks --dt_fmtoption now applies equally well to JSON and XML output as to CDL output:
% ncks -d time,0 -v time --cdl --dt_fmt=3 ~/nco/data/in.nc ... time = "1964-03-13T21:09:0.000000" ; ... % ncks -d time,0 -v time --json --dt_fmt=3 ~/nco/data/in.nc ... "data": ["1964-03-13T21:09:0.000000"] ... % ncks -d time,0 -v time --xml --dt_fmt=3 ~/nco/data/in.nc ... <ncml:values separator="*">1964-03-13T21:09:0.000000</ncml:values> ...Thanks to Troy Mare for this suggestion.
--cb) option to produce CF-conformant climatological times and bounds. This option takes a comma-separated argument list of five relevant input parameters:
yr_srtis the climatology start-year,
yr_endis the climatology end-year,
mth_srtis the climatology start-month (in [1..12] format),
mth_endis the climatology end-month (in [1..12] format), and
tpdis the number of timestpes per day (with the special exception that
tpd=0indicates monthly data, not diurnally-resolved data. A seasonal summer climatology created from monthly mean input data spanning June, 2000 to August, 2020 should call
--clm_bnd=2000,2020,6,8,0, whereas a diurnally resolved climatology of the same period with 6-hourly input data resolution would use
ncra --cb=2014,2016,1,1,0 2014_01.nc 2015_01.nc 2016_01.nc clm_JAN.nchttp://nco.sf.net/nco.html#cb
ncclimohas changed default settings for two parameters. As of this version,
ncclimosets the options "
-a sdd --no_amwg_links" by default. For seasonally contiguous DJF climos one must now explicitly set "
-a scd". To create symbolic links to climatology files with AMWG names, one must now explicitly request
ncclimonow supports the high-frequency climos and splitting. Access these capabilities by specifying the climatology-mode options
hfs, respectively. In both cases (climos and splitting) the input file(s) name will not be constructed automatically and must be provided via stdin, positional command-line arguments, or a director to glob. For climos,
ncclimowill detect the number of timesteps per day (
tpd) in the input data, and compute the climatological mean diurnal cycle from the input data. The output is similar to monthly climos, except each climatological monthly, seasonal, or annual output file will contain tpd timesteps to represent the diurnal cycle.
# Split high-frequency timeseries into CMIP-like timeseries cd ;ls *.h4.nc | ncclimo --clm_md=hfs -v=T \ --ypf=1 --yr_srt=56 --yr_end=76 --drc_out= # Generate diurnal climos from high-frequency CMIP6 timeseries cd ;ls *.h4.nc | ncclimo --clm_md=hfc -c --yr_srt=2001 --yr_end=2002 --drc_out=http://nco.sf.net/nco.html#ncclimo
ncclimonow outputs more CF-conformant climatological times and bounds for all climatologies. Previously,
ncclimooutput a time-centered value for climatological bounds, now it outputs an initial
YYYYMMDDformat, as recommended by CF examples such as http://cfconventions.org/Data/cf-conventions/cf-conventions-1.8/cf-conventions.html#climatological-statistics Example 7.13.
Additional details are available in the ChangeLog.