netCDF operators NCO version 2.9.9 are ready

• To: NCO Announce Mail List <nco-announce@xxxxxxxxxxxxxxxxxxxxx>,
• Subject: netCDF operators NCO version 2.9.9 are ready
• From: Charlie Zender <zender@xxxxxxx>
• Date: Tue, 7 Sep 2004 17:53:50 -0700

The netCDF operators NCO version 2.9.9 are ready. 

http://nco.sf.net (Homepage)
http://dust.ess.uci.edu/nco (Homepage "mirror")

First, the NCO->SDO proposal was funded by NSF. 
A new section on the NCO homepage describes the project and contains
pointers to the proposal and two job ads. For more details, see 

http://nco.sf.net#prp_sei

There are two significant feature improvements in version 2.9.9:

1. ncbo now supports threading.
Manually configure threading with -t (as with ncra, ncpdq, ncea), or
ncbo will pick a default for you.

2. ncpdq has powerful packing and unpacking capabilities
ncpdq stands for "Permute Dimensions Quickly" and now "Pack Data
Quietly" too. See http://nco.sf.net/nco.html#ncpdq for full info.
Prior to this release, ncap sported the only packing features in NCO.
However, ncap only packs one variable at a time.
ncpdq is more consistent with the NCO file-at-a-time paradigm.
To pack the entire file in.nc into output file out.nc use

ncpdq in.nc out.nc
ncpdq -P pck_plc -M pck_map in.nc out.nc

The main packing options to ncpdq are -P pck_plc and -M pck_map.
pck_plc is the "packing policy". Currently implemented policies are
  all_new: Pack unpacked variables, re-pack packed variables (default)
  all_xst: Pack unpacked variables, copy packed variables
  xst_new: Re-pack packed variables, copy unpacked variables
      upk: Unpack packed variables, copy unpacked variables

pck_map is the "packing map" which determines how variables of each
numeric type are packed. Currently implemented maps are

flt_sht: Pack floating precision types to NC_SHORT, pack nothing else (Default)
         [NC_DOUBLE,NC_FLOAT]->NC_SHORT, 
[NC_INT,NC_SHORT,NC_CHAR,NC_BYTE]->unaltered
flt_byt: Pack floating precision types to NC_BYTE: pack nothing else
         [NC_DOUBLE,NC_FLOAT]->NC_BYTE, 
[NC_INT,NC_SHORT,NC_CHAR,NC_BYTE]->unaltered
hgh_sht: Pack higher precision types to NC_SHORT, pack nothing else
         [NC_DOUBLE,NC_FLOAT,NC_INT]->NC_SHORT, 
[NC_SHORT,NC_CHAR,NC_BYTE]->unaltered
hgh_byt: Pack higher precision types to NC_BYTE, pack nothing else
         [NC_DOUBLE,NC_FLOAT,NC_INT,NC_SHORT]->NC_BYTE, 
[NC_CHAR,NC_BYTE]->unaltered
nxt_lsr: Convert each type to next lesser precision
         NC_DOUBLE->NC_INT, [NC_FLOAT,NC_INT]->NC_SHORT, 
[NC_CHAR,NC_BYTE]->unaltered

The default "all_new" packing policy with the default "flt_sht"
packing map reduces the typical NC_FLOAT-dominated file size by 50%.
"flt_byt" packing reduces an NC_DOUBLE-dominated file by 87%.
One nice feature of (lossy) scale_factor/add_offset packing is that
additional, loss-less packing algorithms perform well on top of it.

Caveats: 
1. The interaction of packing and missing_values is complex.
Test the missing_value behavior by performing a pack/unpack cycle
to make sure your data that are missing _stay_ missing and data
that are not misssing do not join the Alabama Air National Guard.
This may lead you to elect a new missing_value.
2. Packing into NC_CHAR (with, e.g., flt_chr) is actually allowed 
but is not recommended and so is not documented/advertised. 
Pack into NC_BYTE (with, e.g., flt_byt) instead.

FYI, the plan is to stabilize recent changes in NCO 3.0.0.
Then we will start development of the grid-enabled features of SDO.

Enjoy,
Charlie

Feature changes:
1. ncbo supports threading
1. ncpdq packs/unpacks

Bug-fixes:
1. Fix regex capability on Mac OS X

Build changes:
1. None

Other user-visible changes:
1. Documentated thread options http://nco.sf.net/nco.html#openmp
2. Yummier ncpdq documentation http://nco.sf.net/nco.html#ncpdq

User-invisible changes:
1. None
-- 
Charlie Zender, surname@xxxxxxx, (949) 824-2987, Department of Earth 
System Science, University of California, Irvine CA 92697-3100