The netCDF Operators NCO version 4.9.4 have landed.
http://nco.sf.net (Homepage, Mailing lists, Help)
http://github.com/nco (Source Code, Issues, Releases, Developers)
What's new?
Version 4.9.4 contains new features focused on de-interleaving time
coordinates and per-record weighting for ncra, high-freqency (e.g.,
diurnally-resolved) climatologies and new defaults for ncclimo, a
new distance-weight extrapolation algorithm for ncremap, per-file
weights for nces. New NCO-wide features include: support for
unbuffered I/O that can speed-up I/O for large record variables,
more precise quantization than Bit Grooming, and faster arithmetic
that takes advantage of SIMD directives on OpenMP-enabled builds.
Overall release is packed with interesting new features...read on.
Work on NCO 4.9.5 has commenced and will improve high-freqency
splitting and climo capabilities, more SIMD acceleration, and
more GPU offloading support.
Enjoy,
Charlie
NEW FEATURES (full details always in ChangeLog):
A. NCO may now be configured with --enable-gpu at build-time to
offload certain arithmetically intensive computations to the GPUs
with select architectures and compilers. This feature currently
has no speed benefits, and needs a volunteer to lead development.
Please contact me if interested.
B. All operators now support unbuffered I/O with netCDF3 files when
invoked with the flag --uio or longer synonyms --unbuffered_io or
--share_all. This flag invokes the netCDF library NC_SHARE flag
which enables unbuffered (non-cached) I/O. Unbufferend I/O may
significantly reduce throughput time when large record variables
are written or read. Performance improvements may depend on netCDF
version. Thanks to Barron Henderson for this suggestion.
ncks -v T in.nc out.nc # Default, buffered I/O
ncks --uio -v T in.nc out.nc # Unbuffered I/O
ncra --uio -v T,Q,U,V in*.nc out.nc
http://nco.sf.net/nco.html#uio
C. The default quantization method has changed from Bit Grooming to
Bit Rounding, contributed by Rostislav Kouznetsov with suggestions
from Milan Klower. As implemented, Bit Rounding will substantially
improve the precision for the specified number of significant
digits to retain. A future version of NCO will re-tune the number
of bits per retained digit to turn this precision advantage into
a compression advantage. An article submitted by R. Kouznetsov to
GMD describes Bit Rounding more...precisely.
http://nco.sf.net/nco.html#bg
D. The multi-file, multi-record operators, ncra and ncrcat now
support interleaved time-coordinates in groups of records.
Interleaving (or de-interleaving, depending on one's perspective)
means altering the order of records in a group to be processed.
Specifically, the interleaving feature causes the operator to treat
as sequential records those that are separated by multiples of the
specified interleave parameter within a group of records.
Specify the interleave parameter as the fifth hyperslab argument.
The interleave feature sequences records with respect to their
position relative to the beginning of each sub-cycle.
Records a multiple of interleave from sub-cycle beginning
are first extracted (by ncrcat) or reduced (by ncra), then records
offset from these by one, two, et cetera up to interleave-1.
Thus interleaving allows deconvolution of periodic phenomena within
a time-series. Some examples to reify the abstract:
Let in1.nc = [1..10], in2.nc = [11..20], and in12.nc = [1..20].
ncra -d time,,,,10,5 in1.nc ~/foo.nc # 3.5, 4.5, 5.5, 6.5, 7.5
ncrcat -d time,0,4,,6,2 in1.nc ~/foo.nc 1, 3, 5, 2, 4, 6 (+WARNING)
ncrcat -d time,2,,10,4,2 in12.nc ~/foo.nc # 3, 5, 4, 6, 13, 15, 14, 16
ncra -d time,2,,10,4,2 in12.nc ~/foo.nc # 4, 5, 14, 15
ncra -d time,,,,10,2 in1.nc in2.nc ~/foo.nc # 5, 6, 15, 16
ncra -d time,,,,10,2 in12.nc ~/foo.nc # 5, 6, 15, 16
http://nco.sf.net/nco.html#interleave
http://nco.sf.net/nco.html#ilv
E. nces now supports the -w (or --wgt) option for per-file weights.
This option is similar to the ncra --wgt option. The nces version
also accepts a variable name that contains a scalar per-file-weight.
Per-file weights are useful when computing statistics of ensembles
whose members should be weighted unevenly. Hence these three
commands produce the same answers, though the second and third
are much more flexible and can have non-integral weights:
nces 1.nc 2.nc 2.nc out.nc
nces -w 1,2 1.nc 2.nc out.nc
nces -w var 1.nc 2.nc out.nc
http://nco.sf.net/nco.html#nces
http://nco.sf.net/nco.html#xmp_nces
F. ncra now supports the --per_record_weights (or --prw) option to
utilize command-line weights specified by -w (or --wgt) for
per-record weights instead of per-file-weights. This is useful when
computing weighted averages with cyclically varying weights, since
the weight given on the command line will be repeated for the
length of the timeseries. Consider, for example, a CMIP6 timeseries
of historical monthly mean emissions that one wishes to convert to
an timeseries of annual-mean emissions. One can weight each month
by its number of days via:
ncra --per_record_weights --mro -d time,,,12,12 --wgt \
31,28,31,30,31,30,31,31,30,31,30,31 ~/monthly.nc ~/annual.nc
Thanks to Philip Cameron-Smith of LLNL for this suggestion.
http://nco.sf.net/nco.html#ncra
http://nco.sf.net/nco.html#xmp_ncra
http://nco.sf.net/nco.html#prw
G. ncra accepts the new flag --promote_ints (or --prm_ints) to output
statistics of integer-valued input variables in floating-point
precision in the output file. By default, NCO arithmetic operators
such as ncra auto-promote integers to double-precision prior to
arithmetic, then conduct the arithmetic, then demote the values
back to integers for final output. This default behavior quantizes
the mantissa of the values and prevents, e.g., turning statisitical
means of boolean (0 or 1-valued) input data into floating point
probabilities. The --promote_ints flag causes the statistical means
of integer (including NC_BYTE) inputs to be output as
single-precision floating point (NC_FLOAT) variables. This allows
use arithmetic to be performed on Boolean values stored in the
space-conserving NC_BYTE (single byte) format in input files.
ncra --prm_ints in*.nc out.nc
Thanks to Paul Ullrich of UC Davis for this suggestion.
http://nco.sf.net/nco.html#prm_ints
http://nco.sf.net/nco.html#promote_ints
H. ncremap understands new dimensions used in DOE E3SM MPAS BGC
simulations. ncremap also supports the new --pdq_opt to
override internal presets and to future-proof itself against
unexpected new dimensions from any model input.
ncremap -P mpasseaice --map=map.nc in.nc out.nc
ncremap --pdq='-a Time,new_dim,nCells' --map=map.nc in.nc out.nc
ncremap --pdq='-a time,new_dim,lat,lon' --map=map.nc in.nc out.nc
Thanks to Ahmed Elshall for reporting the new dimensions.
http://nco.sf.net/nco.html#pdq_opt
I. ncremap allows access to a new regridding algorithm based on
distance-weighted extrapolation (DWE). DWE is similar to the ESMF
nearestidavg extrapolation alorithm, and accepts the same two
parameters as input: --xtr_xpn sets the (absolute value of) the
exponent used in distance weighting (default is 2.0), and --xtr_nsp
sets the number of source points used in the extrapolation (default
is 8). ncremap can apply DWE to the entire destination grid, or
just to points with missing/masked values.
ncremap --alg_typ=nco_dwe -s src.nc -d dst.nc -m map.nc
ncremap -a nco_dwe --xtr_xpn=1.0 -s src.nc -d dst.nc -m map.nc
ncremap -a nco_dwe --xtr_nsp=1 -s src.nc -d dst.nc -m map.nc
Thanks to Henry Butowsky for implementing the new method.
http://nco.sf.net/nco.html#dwe
J. The ncks --dt_fmt option now applies equally well to JSON and XML
output as to CDL output:
% ncks -d time,0 -v time --cdl --dt_fmt=3 ~/nco/data/in.nc
...
time = "1964-03-13T21:09:0.000000" ;
...
% ncks -d time,0 -v time --json --dt_fmt=3 ~/nco/data/in.nc
...
"data": ["1964-03-13T21:09:0.000000"]
...
% ncks -d time,0 -v time --xml --dt_fmt=3 ~/nco/data/in.nc
...
<ncml:values separator="*">1964-03-13T21:09:0.000000</ncml:values>
...
Thanks to Troy Mare for this suggestion.
http://nco.sf.net/nco.html#dt_fmt
http://nco.sf.net/nco.html#json
http://nco.sf.net/nco.html#xml
K. ncra, nces, and ncrcat introduce the --clm_nfo (or --cb) option to
produce CF-conformant climatological times and bounds.
This option takes a comma-separated argument list of five relevant
input parameters: --cb=yr_srt,yr_end,mth_srt,mth_end,tpd,
where yr_srt is the climatology start-year, yr_end is the
climatology end-year, mth_srt is the climatology start-month (in
[1..12] format), mth_end is the climatology end-month (in [1..12]
format), and tpd is the number of timestpes per day (with the
special exception that tpd=0 indicates monthly data, not
diurnally-resolved data. A seasonal summer climatology created from
monthly mean input data spanning June, 2000 to August, 2020 should
call ncra with --clm_bnd=2000,2020,6,8,0, whereas a diurnally
resolved climatology of the same period with 6-hourly input data
resolution would use --clm_bnd=2000,2020,6,8,4.
ncra --cb=2014,2016,1,1,0 2014_01.nc 2015_01.nc 2016_01.nc clm_JAN.nc
http://nco.sf.net/nco.html#cb
http://nco.sf.net/nco.html#ncra
L. ncclimo has changed default settings for two parameters.
As of this verion, ncclimo sets the options "-a sdd
--no_amwg_links" by default. For seasonally contiguous DJF climos
one must now explicitly set "-a scd". To create symbolic links
to climatology files with AMWG names, one must now explicitly
request --amwg_links.
http://nco.sf.net/nco.html#dec_md
http://nco.sf.net/nco.html#lnk_flg
M. ncclimo now supports the high-frequency climos and splitting.
Access these capabilities by specifying the climatology-mode
options hfc and hfs, respectively. In both cases (climos and
splitting) the input file(s) name will not be constructed
automatically and must be provided via stdin, positional
command-line arguments, or a director to glob. For climos,
ncclimo will detect the number of timesteps per day (tpd) in the
input data, and compute the climatological mean diurnal cycle
from the input data. The output is similar to monthly climos,
except each climatological monthly, seasonal, or annual output
file will contain tpd timesteps to represent the diurnal cycle.
# Split high-frequency timeseries into CMIP-like timeseries
cd ${drc_in};ls ${caseid}*.h4.nc | ncclimo --clm_md=hfs -v=T \
--ypf=1 --yr_srt=56 --yr_end=76 --drc_out=${HOME}
# Generate diurnal climos from high-frequency CMIP6 timeseries
cd ${drc_in};ls ${caseid}*.h4.nc | ncclimo --clm_md=hfc
-c ${caseid} --yr_srt=2001 --yr_end=2002 --drc_out=${HOME}
http://nco.sf.net/nco.html#ncclimo
http://nco.sf.net/nco.html#clm_md
N. ncclimo now outputs more CF-conformant climatological times and
bounds for all climatologies. Previously, ncclimo output a
time-centered value for climatological bounds, now it outputs
an initial YYYYMMDD format, as recommended by CF examples such as
http://cfconventions.org/Data/cf-conventions/cf-conventions-1.8/cf-conventions.html#climatological-statistics
Example 7.13
BUG FIXES:
A. ncremap/ncks: Fix vertical interpolation from hybrid-to-hybrid
files when surface pressure is in vertical grid file.
This capability worked up to 4.9.1, and was inadvertently broken in
4.9.2 and 4.9.3. The workaround is to use 4.9.1, or move
the desired PS field from the gridfile to the input file.
The solution is to upgrade. Thanks to Wuyin Lin for reporting.
Full release statement at http://nco.sf.net/ANNOUNCE
--
Charlie Zender, Earth System Sci. & Computer Sci.
University of California, Irvine 949-891-2429 )'(
