[netcdfgroup] netCDF Operators NCO version 4.6.0 are ready

• To: NCO Announce Mail List <nco-announce@xxxxxxxxxxxxxxxxxxxxx>, netCDF Mail List <netcdfgroup@xxxxxxxxxxxxxxxx>
• Subject: [netcdfgroup] netCDF Operators NCO version 4.6.0 are ready
• From: Charlie Zender <zender@xxxxxxx>
• Date: Thu, 12 May 2016 12:48:43 -0700

The netCDF Operators NCO version 4.6.0 are ready.

http://nco.sf.net (Homepage, Mailing lists)
http://github.com/nco (Source Code, Releases, Developers)

What's new?
4.6.0 may be most notable for the debut of ncclimo, a new operator
that generates climatologies from monthly-mean input.
Perhaps it's a tie with ncap2, which has a singularly useful new
feature: variable lists/pointers. ncap2 also has a reduced memory
footprint and a function to simplify adding CF-bounds variables
(Thanks Henry!). As usual, ncremap continues to accrue useful
features, the most notable of which is learning grid information
from the CF "coordinates" attribute, if any.

Work on NCO 4.6.1 has commenced and will better support regridding
variables whose horizontal dimensions are not the
most-rapidly-varying.

Enjoy,
Charlie

NEW FEATURES (full details always in ChangeLog):

A. ncclimo produces climatological monthly means, seasonal means,
   annual mean, and optionally regrids all these files. Like ncremap,
   ncclimo is actually a front-end script to manage the complexity
   of invoking the underlying operators (ncks for ncremap and ncra for
   ncclimo). ncclimo produces climatologies starting from monthly-mean
   files of most CESM-like model and observational data formats.
   Invoke ncclimo with, e.g.,
   ncclimo -s start_yr -e end_yr -c run_id -i drc_in -o drc_out
   ncclimo -s 5 -e 10 -c hist -m ocn -i drc_in -o drc_out
   ncclimo -s 1980 -e 2016 -c merra2_198001.nc -i drc_in -o drc_out
   ncclimo has sophisticated parallelism and regridding options.
   Datasets tested include ALM, CAM, CLM, CICE, CISM, CLM, MPAS-I,
   MPAS-O, and POP. ncclimo resulted from work with the DOE ACME
   project, and we are grateful for their support.
   http://nco.sf.net/nco.html#ncclimo
   http://nco.sf.net/nco.html#merra2

B. ncap2 now works with pointers to variables and attributes.
   This allows, e.g., loops over arbitrary sets of variables, and
   make creating aggregations of variables much simpler. An ncap2
   script that converts all float variables to double precision:
   @all=get_vars_in();
   *sz=@all.size();
   for(idx=0;idx<sz;idx++){
     @var_nm=sprint(@all(idx));
     if(*@var_nm.type() == NC_FLOAT) *@var_nm=*@var_nm.double();
   }
   Congratulations Henry for finishing this impressive functionality!
   More documentation will be added in the coming weeks. For now,
   http://nco.sf.net/nco.html#ncap2

C. ncap2 has reduced memory usage. Scalars are no longer stretched
   to conform to arrays prior to arithmetic.

D. ncap2 has a new function to create CF-compliant bounds:
   defdim("bnd_dmn",2);
   bounds_var=make_bounds(crd_var,$bnd_dmn,"bounds_var_nm");
   The "bounds_var_nm" is optional. Passing it causes make_bounds() to
   add the attribute crd_var@bounds="bounds_var_nm".
   http://nco.sf.net/nco.html#make_bounds

E. ncap2 now propagates metadata when Left-Hand-Casting variables.
   Previous versions did not propagate metadata to LHC variables.
   This fix makes converting the _type_ of a variable easier.
   For example, to change one_dmn_rec_var to NC_SHORT, use
   ncap2 -s 'one_dmn_rec_var[time]=short(one_dmn_rec_var)' in.nc out.nc
   The output file will contain the original metadata.
   http://nco.sf.net/nco.html#ncap2

F. ncatted new mode "nappend" appends only to existing attributes.
   If the attribute does not yet exist, it is not created.
   Same as "append" mode, except "append" will create attributes.
   ncatted -a long_name,T,n,c,' nappended text' in.nc
   http://nco.sf.net/nco.html#ncatted

G. ncpdq now has a third unpacking mode.
   Access non-default modes with --upk=[0,1,2] where
   Mode 0: Applies netCDF convention. Default mode:
   upk=scale_factor*pck+add_offset
   Mode 1: HDF4 MODIS MOD08 convention:
   upk=scale_factor*(pck-add_offset)
   Mode 2: HDF4 MODIS MOD13 convention:
   upk=(pck-add_offset)/scale_factor
   Yes, be careful unpacking HDF4 data!
   http://nco.sf.net/nco.html#hdf_upk

H. ncra can now honor the CF convention for climatology bounds.
   Currently this is opt-in with the --cb or --c2b switches.
   --cb causes ncra to:
   1. Add a "climatology" attribute with value "climatology_bounds" to
   the time coordinate, if necessary
   2. Remove the "bounds" from the time coordinate, if necessary
   3. Output a variable named "climatology_bounds" with values that
   are minima/maxima of the input time coordinate bounds
   4. Omit any input time coordinate bounds attribute and variable
   5. Ensure cell_methods attribute for all variables is appropriate
   --c2b is like --cb except --c2b converts the input "climatology"
   bounds to a non-climatology "bounds" in the output.
   Use --c2b when averaging sub-sampled climatologies to produce a
   continuous (non-climatologically sub-sampled) mean.
   ncra --cb *_01.nc clm_JAN.nc
   ncra --cb clm_DEC.nc clm_JAN.nc clm_FEB.nc clm_DJF.nc
   ncra --c2b clm_DJF.nc clm_MAM.nc clm_JJA.nc clm_SON.nc clm_ANN.nc
   Depending on feedback, we may make --cb the default
   http://nco.sf.net/nco.html#cb

I. ncremap now defaults to $TMPDIR for storing intermediate files.
   This is user-configurable with the -U switch.
   http://nco.sf.net/nco.html#ncremap

J. ncks can now ingest and de-interleave ENVI images in BIL, BSQ, and
   BIP formats and store them as any netCDF type. ncks is a viable
   (and faster) replacement for that small subset of tasks normally
   done by the venerable GDAL toolkit (gdal_translate in particular).
   Thanks to David LeBauer of the University of Illinois/NCSA and DOE
   TERRAREF for supporting this feature.
   ncks --trr_wxy=926,1600,1 --trr typ_in=NC_USHORT --trr ntl_in=bil \
   --trr_in=in.nc foo.nc out.nc
   http://nco.sf.net/nco.html#terraref

K. NCO now treats as associated coordinates all variables listed in
   "grid_mapping" attributes. Variables such as
   "char albers_conical_equal_area" are automatically extracted along
   with variables that them. ncecat will not wrap mappings in a record
   dimension.
   http://nco.sf.net/nco.html#grid_mapping

L. ncremap and ncks now follow the CF "coordinates" convention when
   inferring grids and remapping. One specifies the variable that may
   have the "coordinates" attribute, and NCO will inspect and apply
   the "coordinates" convention for that variable for grid generation
   and remapping.
   ncremap -V var_nm -s src.nc -d dst.nc -m map.nc
   ncremap -V var_nm --map map.nc in.nc out.nc
   ncks --rgr_var=var_nm --rgr nfr=y --rgr grid=grd.nc in.nc foo.nc
   ncks --rgr_var=var_nm --map map.nc in.nc out.nc
   http://nco.sf.net/nco.html#ncremap
   http://nco.sf.net/nco.html#regrid

BUG FIXES:

A. Correct ncatted documentation for "append" mode.

B. Fix Append mode to follow same code path and defaults when
   requested with -A and with interactive responses to the
   exit/append/overwrite question. Thanks to Parker Norton for
   pointing out this inconsistency.

C. ncpdq fix complex reordering in group hierarchies.
   Previously ncpdq could misorder dimensions when multiple
   different dimensions shared the same short name (in different
   groups). Thanks to Pedro Vicente for tracking-down and fixing
   this bug! And finding and fixing another ncpdq bug that had
   not yet been reported in the wild.

D. ncap2 fix bug propagating attributes of variables with whitespace
   in their names.

KNOWN PROBLEMS DUE TO NCO:

   This section of ANNOUNCE reports and reminds users of the
   existence and severity of known, not yet fixed, problems.
   These problems occur with NCO 4.6.0 built/tested under
   MacOS with netCDF 4.3.3.1 on HDF5 1.8.16 and with
   Linux with netCDF 4.4.1-development (20160212) on HDF5 1.8.13.

A. NOT YET FIXED (NCO problem)

   Demonstration:

   ncks -m -C -v att_var ~/foo.nc

   20130724: Verified problem still exists
   TODO nco1102
   Cause: NCO parsing of ncatted arguments is not sophisticated
   enough to handle arrays of NC_STRINGS with embedded delimiters.

B. NOT YET FIXED (NCO problem?)

   Demonstration:
   ncrcat -O -d time,0 ~/nco/data/mrd.nc ~/foo.nc

   20140826: Verified problem still exists
   20140619: Problem reported by rmla
   Cause: Unsure. Maybe ncra.c loop structure not amenable to MRD?
   Workaround: Convert to fixed dimensions then hyperslab

KNOWN PROBLEMS DUE TO BASE LIBRARIES/PROTOCOLS:

A. NOT YET FIXED (netCDF4 or HDF5 problem?)
   Specifying strided hyperslab on large netCDF4 datasets leads
   to slowdown or failure with recent netCDF versions.

   Demonstration with NCO <= 4.4.5:
   time ncks -O -d time,0,,12 ~/ET_2000-01_2001-12.nc ~/foo.nc
   Demonstration with NCL:
   time ncl < ~/nco/data/ncl.ncl
   20140718: Problem reported by Parker Norton
   20140826: Verified problem still exists
   20140930: Finish NCO workaround for problem
   Cause: Slow algorithm in nc_var_gets()?
   Workaround #1: Use NCO 4.4.6 or later (avoids nc_var_gets())
   Workaround #2: Convert file to netCDF3 first, then use stride

B. NOT YET FIXED (netCDF4 library bug)

   Demonstration:

   ncks -v one ~/foo.nc

   20150924: Unidata confirmed problem
   20160212: Verified problem still exists in netCDF library
   20160512: Ditto

   Bug tracking: https://www.unidata.ucar.edu/jira/browse/fxm
   More details: http://nco.sf.net/nco.html#ncrename_crd

C. NOT YET FIXED (would require DAP protocol change?)
   Unable to retrieve contents of variables including period '.' in name
   Periods are legal characters in netCDF variable names.
   Metadata are returned successfully, data are not.
   DAP non-transparency: Works locally, fails through DAP server.

   Demonstration:

   20130724: Verified problem still exists.
   Stopped testing because inclusion of var_nm.dot broke all test scripts.

   Bug tracking: https://www.unidata.ucar.edu/jira/browse/NCF-47

D. NOT YET FIXED (would require DAP protocol change)
   Correctly read scalar characters over DAP.
   DAP non-transparency: Works locally, fails through DAP server.
   Problem, IMHO, is with DAP definition/protocol

   Demonstration:

   20120801: Verified problem still exists
   Bug report not filed
   Cause: DAP translates scalar characters into 64-element (this
   dimension is user-configurable, but still...), NUL-terminated
   strings so MD5 agreement fails

"Sticky" reminders:

A. Reminder that NCO works on most HDF4 and HDF5 datasets, e.g.,
   HDF4: AMSR MERRA MODIS ...
   HDF5: GLAS ICESat Mabel SBUV ...
   HDF-EOS5: AURA HIRDLS OMI ...

B. Pre-built executables for many OS's at:
   http://nco.sf.net#bnr

--
Charlie Zender, Earth System Sci. & Computer Sci.
University of California, Irvine 949-891-2429 )'(