Maintaining netCDF: Updating Java Tutorial Code and Performance Testing in Python

Isabelle Pfander
Izzy (Isabelle) Pfander

I came into this summer internship with a goal of working on the Network Common Data Form (netCDF) libraries. NetCDF is a combination of software libraries and APIs describing a data model for scientific multidimensional arrays. I planned to improve the online user guide, write tutorial code, and learn about storage and efficiency.

[Read More]

NCZarr Support for Zarr Filters

[Note: See github issue 2006 for additional comments.]

To date, filters in the netcdf-c library referred to HDF5 style filters. The inclusion of Zarr support in the netcdf-c library (called NCZarr) creates the need to provide a new representation consistent with the way that Zarr files store filter information. For Zarr, filters are represented using the JSON notation. Each filter is defined by a JSON dictionary, and each such filter dictionary is guaranteed to have a key named "id" whose value is a unique string defining the filter algorithm: "lz4" or "bzip2", for example.

This document outlines the proposed process by which NCZarr will be able to utilize existing HDF5 filters. At the same time, it provides mechanisms to support storing filter metadata in the NCZarr container using the Zarr compliant Codec style representation of filters and their parameters.

[Read More]

Netcdf-4 Filter Support Changes

The netcdf-c library filters API in version 4.7.4 has been deprecated in favor of a modified version that unfortunately may cause incompatibilties for users.

The initial reason for the incompatible changes was to support the use of filters in the new NCZarr code. The changes were not completely thought out so it was decided to remove them and revert to previous behaviors. At some future point, the filter mechanism will be extended to support filters for NCZarr, but these will be proper extensions: the existing, reverted, filter API will continue to be supported with no user-visible modifications.

Unfortunately, some advanced users of netcdf filters may experience some compilation or execution problems for previously working code because of these reversions. In that case, please revise your code. Apologies are extended for any inconvenience. Note that it is possible to detect which mechanism is in place at build time.

In summary, the changes are of the following kinds:

  • Some functions were renamed for consistency.
  • Revert the way that the function nc_inq_var_filter was indicating no filters existed.
  • Some auxilliary functions for parsing textual filter specifications have been moved to netcdf_aux.h.
  • All of the "filterx" functions have been removed.
  • The undocumented function nc_filter_remove was deleted.

See the Github document https://github.com/Unidata/netcdf/blob/master/NUG/filters.md for details.

Highlights From My Summer Internship With Unidata

Lauren Prox
Lauren Prox

During the beginning of my internship, I devoted a great deal of time learning how to use Git and Github to collaborate on software development projects. After gaining this experience, I began improving documentation for a variety of Unidata remote repositories. I started with the netCDF-C repository and then moved on to the MetPy, Siphon, and Python Training remote repositories. This work was significant as it ensured that software users were able to locate resources, properly download software, and learn how to operate the software via informational materials.

[Read More]

Overview of Zarr Support in netCDF-C

Beginning with netCDF version 4.8.0, the Unidata NetCDF group has extended the netcdf-c library to provide access to cloud storage (e.g. Amazon S3) by providing a mapping from a subset of the full netCDF Enhanced (aka netCDF-4) data model to a variant of the Zarr data model that already has mappings to key-value pair cloud storage systems.

[Read More]
Unidata Developer's Blog
A weblog about software development by Unidata developers*
Unidata Developer's Blog
A weblog about software development by Unidata developers*

Welcome

FAQs

News@Unidata blog

Take a poll!

What if we had an ongoing user poll in here?

Browse By Topic
  • feed AWIPS (17)
Browse by Topic
« November 2022
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today