The netCDF Operators NCO version 4.5.5 are ready.
http://nco.sf.net (Homepage, Mailing lists)
http://github.com/nco (Source Code, Releases, Developers)
What's new?
NCO fully supports CDF5-format datasets, ncremap continues to
accrue useful features, and some corner case bugs were fixed.
Work on NCO 4.5.6 has commenced and will better support eliciting
latitude/longitude coordinates using the CF "coordinates" convention
and regridding variables whose horizontal dimensions are not the
most-rapidly-varying.
Enjoy,
Charlie
NEW FEATURES (full details always in ChangeLog):
A. NCO supports the CDF5 file type first introduced by pnetCDF and now
supported in Unidata netCDF as of version 4.4.0.
CDF5 uses 64-bit offsets for a large address space (like the netCDF
64BIT type) AND arrays may have more than 2^32 elements.
Use '-5', '--5', '--fl_fmt=[cdf5|64bit_data|pnetcdf]'.
Conversion to and between all netCDF filetypes is supported:
ncks -5 in_3.nc out_5.nc
ncks -5 in_7.nc out_5.nc
ncks -7 in_5.nc out_7.nc
http://nco.sf.net/nco.html#cdf5
B. ncremap: -P option triggers filetype-specific workflow procedures
(such as automatic permutation). Currently filetype-specific
handling is defined for AIRS, HIRDLS, MLS, and MPAS files:
ncremap -P airs -i airs.nc -m map.nc -o out.nc
ncremap -P hirdls -i airs.nc -m map.nc -o out.nc
ncremap -P mls -i airs.nc -m map.nc -o out.nc
ncremap -P mpas -i mpas.nc -m map.nc -o out.nc
One aspect of this handling is to rearrange dimension ordering
from the way NASA/MPAS distributes generates these datasets so
that horizontal spatial dimensions (lat, lon) come last.
This is friendlier for regridding applications.
C. ncremap: -m map_fl supplies names to generated mapfiles and ncremap
annotates map-files it creates with full history. This helps retain
provenance in generated mapfiles. And ncremap has a friendlier
mode, "map-only" mode, to generate mapfiles for later use.
To use this mode, call ncremap without any input files to remap.
ncremap will then generate the mapfile then exit.
ncremap -i in.nc -d dst.nc -m map.nc -o out.nc # Named mapfile
ncremap -s grd_in.nc -g grd_out.nc -m map.nc # Map-only mode
http://nco.sf.net/nco.html#ncremap
D. ncremap -j job_nbr option for MPI parallelism
http://nco.sf.net/nco.html#ncremap
E. ncra/nces/ncwa: New operation tabs()=total_absolute_value.
The the -y tabs option is like -y ttl except the sum is over
absolute values. tabs(), mibs(), mebs(), and mabs() are analogous
to ttl(), min(), mean(), and max().
nces -y tabs in.nc in.nc out.nc
ncra -y tabs in.nc in.nc out.nc
ncwa -y tabs in.nc out.nc
http://nco.sf.net/nco.html#op_typ
F. Improve support for Cygwin builds
BUG FIXES:
A. ncremap: Inferred curvilinear grids always return points now in
counterclockwise order. Version 4.5.4 sometimes did not.
This caused bad things to happen in the weight-generators.
B. ncremap: Inferring grids from curvilinear coordinates no longer
confused by "branch cuts" (i.e., the date-line). ncremap version
4.5.4 would sometimes infer concave gridcells that then confused
mapping software. Thanks to Feng Ding for reporting the problem.
C. ncpdq: Fix problem that could cause '--gaa' option to fail in
earlier versions.
D. nces: Copy attributes of coordinate variables in nces --nsm_grp.
Previous versions omitted the attributes of coordinates (not
regular variables) in the ensemble average group. Thanks to Hannah
Nissan for reporting and Pedro Vicente for fixing this.
E. ncra/nces: Correctly normalize "mebs" arithmetic. Under certain
conditions previous versions did not correctly normalize sums
computed with the mebs (mean absolute value) operator. Problem
reported by Will Koeppen.
F. ncap2: Fix incorrect handling of negative dimension indices.
Problem reported by Maksym Petrenko.
G. ncatted/ncrename: Fully support --glb_att_add=--gaa.
This corrects an earlier oversight.
http://nco.sf.net/nco.html#gaa
KNOWN PROBLEMS DUE TO NCO:
This section of ANNOUNCE reports and reminds users of the
existence and severity of known, not yet fixed, problems.
These problems occur with NCO 4.5.5 built/tested under
MacOS with netCDF 4.3.3.1 on HDF5 1.8.16 and with
Linux with netCDF 4.4.1-development (20160212) on HDF5 1.8.13.
A. NOT YET FIXED (NCO problem)
Correctly read arrays of NC_STRING with embedded delimiters in
ncatted arguments
Demonstration:
ncatted -D 5 -O -a
new_string_att,att_var,c,sng,"list","of","str,ings" ~/nco/data/in_4.nc
~/foo.nc
ncks -m -C -v att_var ~/foo.nc
20130724: Verified problem still exists
TODO nco1102
Cause: NCO parsing of ncatted arguments is not sophisticated
enough to handle arrays of NC_STRINGS with embedded delimiters.
B. NOT YET FIXED (NCO problem?)
ncra/ncrcat (not ncks) hyperslabbing can fail on variables with
multiple record dimensions
Demonstration:
ncrcat -O -d time,0 ~/nco/data/mrd.nc ~/foo.nc
20140826: Verified problem still exists
20140619: Problem reported by rmla
Cause: Unsure. Maybe ncra.c loop structure not amenable to MRD?
Workaround: Convert to fixed dimensions then hyperslab
KNOWN PROBLEMS DUE TO BASE LIBRARIES/PROTOCOLS:
A. NOT YET FIXED (netCDF4 or HDF5 problem?)
Specifying strided hyperslab on large netCDF4 datasets leads
to slowdown or failure with recent netCDF versions.
Demonstration with NCO <= 4.4.5:
time ncks -O -d time,0,,12 ~/ET_2000-01_2001-12.nc ~/foo.nc
Demonstration with NCL:
time ncl < ~/nco/data/ncl.ncl
20140718: Problem reported by Parker Norton
20140826: Verified problem still exists
20140930: Finish NCO workaround for problem
Cause: Slow algorithm in nc_var_gets()?
Workaround #1: Use NCO 4.4.6 or later (avoids nc_var_gets())
Workaround #2: Convert file to netCDF3 first, then use stride
B. NOT YET FIXED (netCDF4 library bug)
Simultaneously renaming multiple dimensions in netCDF4 file can
corrupt output
Demonstration:
ncrename -O -d lev,z -d lat,y -d lon,x ~/nco/data/in_grp.nc ~/foo.nc
# Completes but file is unreadable
ncks -v one ~/foo.nc
20150922: Confirmed problem reported by Isabelle Dast, reported to
Unidata
20150924: Unidata confirmed problem
20160212: Verified problem still exists in netCDF library
Bug tracking: https://www.unidata.ucar.edu/jira/browse/fxm
More details: http://nco.sf.net/nco.html#ncrename_crd
C. NOT YET FIXED (would require DAP protocol change?)
Unable to retrieve contents of variables including period '.' in name
Periods are legal characters in netCDF variable names.
Metadata are returned successfully, data are not.
DAP non-transparency: Works locally, fails through DAP server.
Demonstration:
ncks -O -C -D 3 -v var_nm.dot -p
http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc # Fails to
find variable
20130724: Verified problem still exists.
Stopped testing because inclusion of var_nm.dot broke all test scripts.
NB: Hard to fix since DAP interprets '.' as structure delimiter in
HTTP query string.
Bug tracking: https://www.unidata.ucar.edu/jira/browse/NCF-47
D. NOT YET FIXED (would require DAP protocol change)
Correctly read scalar characters over DAP.
DAP non-transparency: Works locally, fails through DAP server.
Problem, IMHO, is with DAP definition/protocol
Demonstration:
ncks -O -D 1 -H -C -m --md5_dgs -v md5_a -p
http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc
20120801: Verified problem still exists
Bug report not filed
Cause: DAP translates scalar characters into 64-element (this
dimension is user-configurable, but still...), NUL-terminated
strings so MD5 agreement fails
"Sticky" reminders:
A. Reminder that NCO works on most HDF4 and HDF5 datasets, e.g.,
HDF4: AMSR MERRA MODIS ...
HDF5: GLAS ICESat Mabel SBUV ...
HDF-EOS5: AURA HIRDLS OMI ...
B. Pre-built executables for many OS's at:
http://nco.sf.net#bnr
--
Charlie Zender, Earth System Sci. & Computer Sci.
University of California, Irvine 949-891-2429 )'(
