Hello all,
The Unidata netCDF team is happy to announce the availability of netCDF
4.8.0, now with Zarr support (nczarr)!
This release extends the netcdf-c library to provide access to cloud
storage (e.g. Amazon S3 by providing a mapping from a subset of the full
netCDF Enhanced (aka netCDF–4) data model to a variant of the Zarr data
model. The NetCDF variant is called NCZarr. It is possible for other
implementations, such as Python Zarr, to read this variant. This
implementation can optionally read and write using the pure version 2 Zarr
storage format. It also can optionally support the XArray _ARRAY_DIMENSIONS
extension to Zarr.
In addition to this new functionality, the release notes appended to the
end of this announcement detail some of the other changes, improvements and
bug fixes included in this release.
Please see the following links for more information:
- Github release page:
https://github.com/Unidata/netcdf-c/releases/tag/v4.8.0
- v4.8.0 Documentation: https://www.unidata.ucar.edu/software/netcdf/docs
- NCZarr Documentation:
https://www.unidata.ucar.edu/software/netcdf/documentation/NUG/nczarr_head.html
Kudos to Dennis Heimbigner for the technical implementation of nczarr, and
our thanks to the members of the Zarr working group and our community who
have collaborated with us so patiently on this!
Our next priority is to get caught up on the backlog of PRs and issues
blocking the next releases of netCDF-Fortran and netCDF-CXX4.
Thank you all very much, have a great day!
-Ward
Ward Fisher
wfisher@xxxxxxxx
UCAR/Unidata - Software Engineer
NetCDF Team Lead
Full Release Notes4.8.0 - March 30, 2021
- [Enhancement] Bump the NC_DISPATCH_VERSION from 2 to 3, and as a side
effect, unify the definition of NC_DISPATCH_VERSION so it only needs to be
defined in CMakeLists.txt and configure.ac. See Github #1945
<https://github.com/Unidata/netcdf-c/pull/1945> for more information.
- [Enhancement] Provide better cross platform path name management. This
converts paths for various platforms (e.g. Windows, MSYS, etc.) so that
they are in the proper format for the executing platform. See Github
#1958 <https://github.com/Unidata/netcdf-c/pull/1958> for more
information.
- [Bug Fixes] The nccopy program was treating -d0 as turning deflation
on rather than interpreting it as “turn off deflation”. See Github #1944
<https://github.com/Unidata/netcdf-c/pull/1944> for more information.
- [Enhancement] Add support for storing NCZarr data in zip files. See Github
#1942 <https://github.com/Unidata/netcdf-c/pull/1942> for more
information.
- [Bug Fixes] Make fillmismatch the default for DAP2 and DAP4; too many
servers ignore this requirement.
- [Bug Fixes] Fix some memory leaks in NCZarr, fix a bug with long
strides in NCZarr. See Github #1913
<https://github.com/Unidata/netcdf-c/pull/1913> for more information.
- [Enhancement] Add some optimizations to NCZarr, dosome cleanup of code
cruft, add some NCZarr test cases, add a performance test to
NCZarr. See Github
#1908 <https://github.com/Unidata/netcdf-c/pull/1908> for more
information.
- [Bug Fix] Implement a better chunk cache system for NCZarr. The cache
now uses extendible hashing plus a linked list for provide a combination of
expandibility, fast access, and LRU behavior. See Github #1887
<https://github.com/Unidata/netcdf-c/pull/1887> for more information.
- [Enhancement] Provide .rc fields for S3 authentication:
HTTP.S3.ACCESSID and HTTP.S3.SECRETKEY.
- [Enhancement] Give the client control over what parts of a DAP2 URL
are URL encoded (i.e. %xx). This is to support the different decoding rules
that servers apply to incoming URLS. See Github #1884
<https://github.com/Unidata/netcdf-c/pull/1844> for more information.
- [Bug Fix] Fix incorrect time offsets from ncdump -t, in some cases
when the time units attribute contains both a *non-zero* time-of-day,
and a time zone suffix containing the letter “T”, such as “UTC”. See Github
#1866 <https://github.com/Unidata/netcdf-c/pull/1866> for more
information.
- [Bug Fix] Cleanup the NCZarr S3 build options. See Github #1869
<https://github.com/Unidata/netcdf-c/pull/1869> for more information.
- [Bug Fix] Support aligned access for selected ARM processors. See Github
#1871 <https://github.com/Unidata/netcdf-c/pull/1871> for more
information.
- [Documentation] Migrated the documents in the NUG/ directory to the
dedicated NUG repository found at https://github.com/Unidata/netcdf
- [Bug Fix] Revert the internal filter code to simplify it. From the
user’s point of view, the only visible change should be that (1) the
functions that convert text to filter specs have had their signature
reverted and renamed and have been moved to netcdf_aux.h, and (2) Some
filter API functions now return NC_ENOFILTER when inquiry is made about
some filter. Internally, the dispatch table has been modified to get rid of
the complex structures.
- [Bug Fix] If the HDF5 byte-range Virtual File Driver is available
)HDf5 1.10.6 or later) then use it because it has better performance than
the one currently built into the netcdf library.
- [Bug Fix] Fixed byte-range support with cURL > 7.69. See [
https://github.com/Unidata/netcdf-c/pull/1798].
- [Enhancement] Added new test for using compression with parallel I/O:
nc_test4/tst_h_par_compress.c. See [
https://github.com/Unidata/netcdf-c/pull/1784].
- [Bug Fix] Don’t return error for extra calls to nc_redef() for
netCDF/HDF5 files, unless classic model is in use. See [
https://github.com/Unidata/netcdf-c/issues/1779].
- [Enhancement] Added new parallel I/O benchmark program to mimic NOAA
UFS data writes, built when –enable-benchmarks is in configure. See [
https://github.com/Unidata/netcdf-c/pull/1777].
- [Bug Fix] Now allow szip to be used on variables with unlimited
dimension [https://github.com/Unidata/netcdf-c/issues/1774].
- [Enhancement] Add support for cloud storage using a variant of the
Zarr storage format. Warning: this feature is highly experimental and is
subject to rapid evolution [
https://www.unidata.ucar.edu/blogs/developer/en/entry/overview-of-zarr-support-in
].
- [Bug Fix] Fix nccopy to properly set default chunking parameters when
not otherwise specified. This can significantly improve performance in
selected cases. Note that if seeing slow performance with nccopy, then, as
a work-around, specifically set the chunking parameters. [
https://github.com/Unidata/netcdf-c/issues/1763].
- [Bug Fix] Fix some protocol bugs/differences between the netcdf-c
library and the OPeNDAP Hyrax server. Also cleanup checksum handling [
https://github.com/Unidata/netcdf-c/issues/1712].* [Bug Fix] IMPORTANT:
Ncgen was not properly handling large
data sections. The problem manifests as incorrect ordering of
data in the created file. Aside from examining the file with
ncdump, the error can be detected by running ncgen with the -lc
flag (to produce a C file). Examine the file to see if any
variable is written in pieces as opposed to a single call to
nc_put_vara. If multiple calls to nc_put_vara are used to write
a variable, then it is probable that the data order is
incorrect. Such multiple writes can occur for large variables
and especially when one of the dimensions is unlimited.
- [Bug Fix] Add necessary _
*declspec declarations to allow compilation of netcdf library without
causing errors or (*declspec related)
warnings [https://github.com/Unidata/netcdf-c/issues/1725].
- [Enhancement] When a filter is applied twice with different
parameters, then the second set is used for writing the dataset
[https://github.com/Unidata/netcdf-c/issues/1713].
- [Bug Fix] Now larger cache settings are used for sequential HDF5 file
creates/opens on parallel I/O capable builds; see Github #1716
<https://github.com/Unidata/netcdf-c/issues/1716> for more information.
- [Bug Fix] Add functions to libdispatch/dnotnc4.c to support
dispatch table operations that should work for any dispatch
table, even if they do not do anything; functions such as
nc_inq_var_filter [https://github.com/Unidata/netcdf-c/issues/1693].
- [Bug Fix] Fixed a scalar annotation error when scalar == 0; see Github
#1707 <https://github.com/Unidata/netcdf-c/issues/1707> for more
information.
- [Bug Fix] Use proper CURLOPT values for VERIFYHOST and VERIFYPEER; the
semantics for VERIFYHOST in particular changed. Documented in NUG/DAP2.md.
See [https://github.com/Unidata/netcdf-c/issues/1684].
- [Bug Fix][cmake] Correct an issue with parallel filter test logic in
CMake-based builds.
- [Bug Fix] Now allow nc_inq_var_deflate()/nc_inq_var_szip() to be
called for all formats, not just HDF5. Non-HDF5 files return NC_NOERR and
report no compression in use. This reverts behavior that was changed in the
4.7.4 release. See [https://github.com/Unidata/netcdf-c/issues/1691].
- [Bug Fix] Compiling on a big-endian machine exposes some missing
forward delcarations in dfilter.c.
