[netcdfgroup] netCDF Operators NCO version 4.6.2 are ready

The netCDF Operators NCO version 4.6.2 are ready.

http://nco.sf.net (Homepage, Mailing lists)
http://github.com/nco (Source Code, Releases, Developers)

What's new?

4.6.2 is mainly a stability release to polish existing features and to
add minor new ones. The exception is the new JSON backend.
NCO now prints CDL, "Traditional NCO", XML, and JSON. Babel-icious eh?

We built the JSON backend to help a project (DOE Terraref).
Our first choice was to adopt an off-the-shelf netCDF->JSON tool.
However, no existing solution worked for us.
JSON is a loose syntax, and we made necessary design choices that
suited our application, and left some choices for later.
Are there syntactical variants you want us to add?

Some users of netCDF version 4.4.1 cannot build NCO from scratch
because a bug in the nc-config command kills NCO's 'configure;make'.
Unidata will ship a corrected nc-config in netCDF 4.4.2.
The 4.4.2-development branch already contains the necessary fix.
See (C) in KNOWN PROBLEMS DUE TO BASE LIBRARIES/PROTOCOLS below.

Work on NCO 4.6.3 has commenced. Planned improvements include more
flexibility in handling extensive variables during regridding, CMake
support, and brackets for multi-dimensional array values in JSON.

Enjoy,
Charlie

NEW FEATURES (full details always in ChangeLog):

A. JSON backend
   Similar the CDL and XML backends, ncks now supports JSON.
   To obtain JSON, use --jsn or --json instead of --cdl/--xml.
   ncks --json -v one ~/nco/data/in.nc
   {
     "variables": {
       "one": {
         "type": "float",
         "attributes": {
           "long_name": "one"
         },
         "data": 1.0
       }
     }
   }
   More sample output at:
   http://dust.ess.uci.edu/tmp/in.json and other *.json files.
   Other related options (-v, -g, -m, -M) have their usual meanings.
   The --jsn_att_fmt switch accesses formats of selectable verbosity.
   Thanks to Henry Butowsky for implementing the JSON backend,
   and to Pedro Vicente and Chris Barker for helpful discussions.
   http://nco.sf.net/nco.html#json

B. Multi-arguments refer to the format of command-line key-value pairs
   that NCO uses to support four different features that can have too
   many parameters to set via traditional command line arguments.
   The four features are regridding (rgr), global-attribute-adding
   (gaa), precision-preserving compression (ppc), and ENVI image
   processing for the DOE Terraref project (trr). Formerly, users were
   required to supply one value per command line option for these
   features, e.g.,
   "--rgr lat_nbr=64 --rgr lon_nbr=128 --rgr lat_grd=cap ..."
   The new multi-argument feature allows users to aggregate multiple
   key-value pairs per command line argument, e.g.,
   "--rgr lat_nbr=64#lon_nbr=128#lat_grd=cap# ..."
   Now all key-value pairs related to a single feature (like
   regridding, compression) can be provided as one arument!

   The string that delimits each pair is user-configurable and
   defaults to a single hash "#". Change it with, e.g., --dlm=":".
   Ensure your delimiters are not special shell characters, and are
   escaped (with backslashes) if present in key or value strings.
   Thanks to Jerome Mao for implementing multi-arguments.
   http://nco.sf.net/nco.html#mta

C. ncap2 now searches all paths in the NCO_PATH environment variable
   for any #include'd files that it cannot find in ${CWD}.

D. All builds tested on LLVM 8.0.0 and GCC 6.2.1 toolchains.

BUG FIXES:

A. ncclimo and ncremap could fail if ${BASH_SOURCE[0]} is unavailable.
   This could occur, e.g., with the dash shell and/or due to software
   modules that modify users' environments.
   This portability issue has been resolved.

B. ncatted failed to implement NaN on some MinGW systems with, e.g.,
   ncatted -O -a _FillValue,fll_val,m,f,nan ~/nco/data/in.nc ~/foo.nc
   We think this has been fixed. The workaround is to upgrade or use
   ncatted on *NIX or MacOS. Thanks to James Adams for reporting this.

C. ncks CDL now prints correct tokens for uint, int64, uint64 types.
   These are "ul", "ll", "ull", respectively. Previously NCO used
   different tokens because ncgen <= 4.3.0 choked on "ll" formats.
   So this was an intentional NCO "feature", not a bug.
   Use ncgen >= 4.3.1 to parse these CDL files.

KNOWN PROBLEMS DUE TO NCO:

   This section of ANNOUNCE reports and reminds users of the
   existence and severity of known, not yet fixed, problems.
   These problems occur with NCO 4.6.2 built/tested under
   MacOS 10.12.1 with netCDF 4.4.1 on HDF5 1.8.16 and with
   Linux with netCDF 4.4.2-development (20161116) on HDF5 1.8.16.

A. NOT YET FIXED (NCO problem)
Correctly read arrays of NC_STRING with embedded delimiters in ncatted arguments

   Demonstration:
ncatted -D 5 -O -a new_string_att,att_var,c,sng,"list","of","str,ings" ~/nco/data/in_4.nc ~/foo.nc
   ncks -m -C -v att_var ~/foo.nc

   20130724: Verified problem still exists
   TODO nco1102
   Cause: NCO parsing of ncatted arguments is not sophisticated
   enough to handle arrays of NC_STRINGS with embedded delimiters.

B. NOT YET FIXED (NCO problem?)
ncra/ncrcat (not ncks) hyperslabbing can fail on variables with multiple record dimensions

   Demonstration:
   ncrcat -O -d time,0 ~/nco/data/mrd.nc ~/foo.nc

   20140826: Verified problem still exists
   20140619: Problem reported by rmla
   Cause: Unsure. Maybe ncra.c loop structure not amenable to MRD?
   Workaround: Convert to fixed dimensions then hyperslab

KNOWN PROBLEMS DUE TO BASE LIBRARIES/PROTOCOLS:

A. NOT YET FIXED (netCDF4 or HDF5 problem?)
   Specifying strided hyperslab on large netCDF4 datasets leads
   to slowdown or failure with recent netCDF versions.

   Demonstration with NCO <= 4.4.5:
   time ncks -O -d time,0,,12 ~/ET_2000-01_2001-12.nc ~/foo.nc
   Demonstration with NCL:
   time ncl < ~/nco/data/ncl.ncl
   20140718: Problem reported by Parker Norton
   20140826: Verified problem still exists
   20140930: Finish NCO workaround for problem
   Cause: Slow algorithm in nc_var_gets()?
   Workaround #1: Use NCO 4.4.6 or later (avoids nc_var_gets())
   Workaround #2: Convert file to netCDF3 first, then use stride

B. NOT YET FIXED (netCDF4 library bug)
Simultaneously renaming multiple dimensions in netCDF4 file can corrupt output

   Demonstration:
ncrename -O -d lev,z -d lat,y -d lon,x ~/nco/data/in_grp.nc ~/foo.nc # Completes but file is unreadable
   ncks -v one ~/foo.nc

20150922: Confirmed problem reported by Isabelle Dast, reported to Unidata
   20150924: Unidata confirmed problem
   20160212: Verified problem still exists in netCDF library
   20160512: Ditto
   20161028: Verified problem still exists with netCDF 4.4.1

   Bug tracking: https://www.unidata.ucar.edu/jira/browse/fxm
   More details: http://nco.sf.net/nco.html#ncrename_crd

C. NOT YET FIXED (netCDF4 library bug) in main netCDF release
   FIXED in netCDF Development branch as of 20161116
nc-config/nf-config produce erroneous switches that cause NCO builds to fail
   This problem affects netCDF 4.4.1 on all operating systems.
   Some pre-compiled netCDF packages may have patched the problem.
   Hence it does affect my MacPorts install of netCDF 4.4.1.

   Demonstration:
   % nc-config --cflags # Produces extraneous text that confuses make
   Using nf-config: /usr/local/bin/nf-config
   -I/usr/local/include -I/usr/local/include -I/usr/include/hdf

   If your nc-config output contains the "Using ..." line, you are
   affected by this issue.

   20161029: Reported problem to Unidata
20161101: Unidata confirmed reproducibility, attributed to netCDF 4.4.1 changes
   20161116: Unidata patch is in tree for netCDF 4.4.2 release

D. NOT YET FIXED (would require DAP protocol change?)
   Unable to retrieve contents of variables including period '.' in name
   Periods are legal characters in netCDF variable names.
   Metadata are returned successfully, data are not.
   DAP non-transparency: Works locally, fails through DAP server.

   Demonstration:
ncks -O -C -D 3 -v var_nm.dot -p http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc # Fails to find variable

   20130724: Verified problem still exists.
   Stopped testing because inclusion of var_nm.dot broke all test scripts.
NB: Hard to fix since DAP interprets '.' as structure delimiter in HTTP query string.

   Bug tracking: https://www.unidata.ucar.edu/jira/browse/NCF-47

E. NOT YET FIXED (would require DAP protocol change)
   Correctly read scalar characters over DAP.
   DAP non-transparency: Works locally, fails through DAP server.
   Problem, IMHO, is with DAP definition/protocol

   Demonstration:
ncks -O -D 1 -H -C -m --md5_dgs -v md5_a -p http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc

   20120801: Verified problem still exists
   Bug report not filed
   Cause: DAP translates scalar characters into 64-element (this
   dimension is user-configurable, but still...), NUL-terminated
   strings so MD5 agreement fails

"Sticky" reminders:

A. Reminder that NCO works on most HDF4 and HDF5 datasets, e.g.,
   HDF4: AMSR MERRA MODIS ...
   HDF5: GLAS ICESat Mabel SBUV ...
   HDF-EOS5: AURA HIRDLS OMI ...

B. Pre-built executables for many OS's at:
   http://nco.sf.net#bnr

--
Charlie Zender, Earth System Sci. & Computer Sci.
University of California, Irvine 949-891-2429 )'(



  • 2016 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the netcdfgroup archives: