Chunking Algorithms for NetCDF-C

Unidata is in the process of developing a Zarr [] based variant of netcdf. As part of this effort, it was necessary to implement some support for chunking. Specifically, the problem to be solved was that of extracting a hyperslab of data from an n-dimensional variable (array in Zarr parlance) that has been divided into chunks (in the HDF5 sense). Each chunk is stored independently in the data storage -- Amazon S3, for example.

The algorithm takes a series of R slices of the form (first,stop,stride), where R is the rank of the variable. Note that a slice of the form (first, count, stride), as used by netcdf, is equivalent because stop = first + count*stride. These slices form a hyperslab.

The goal is to compute the set of chunks that intersect the hyperslab and to then extract the relevant data from that set of chunks to produce the hyperslab.

[Read More]

Adding Compression to NetCDF-3 (Classic)

This document proposes adding compression to the netcdf-3 (aka classic) file format. The proposal has numerous limitations because of the nature of the existing netcdf-3 format.

The algorithm and data format proposed here requires re-writing an existing netcdf-3 file to move it to the new format. In effect, the re-written file becomes archival (read-only).

[Read More]

Implementing Thread-safe Access to the netCDF-C Library

This document proposes an architecture for implementing thread-safe access to the netcdf-c library. Here, the term "thread-safe" means that multiple threads can access the netcdf-c library safely (i.e. without interference or deadlock or race conditions). This does not mean that the library is itself multi-threaded. Rather, access to the library is serialized so that only one thread at a time is executing the library code.

[Read More]

Unit Testing in the netCDF-c library using cmake

It appears to be the case that all current tests in the netcdf-c source tree are integration tests. That is they only access the library using calls to the externally visible API.

The DAP4 code testing is different in that it includes (for the first time) unit tests. These tests reference code that is considered internal to the library. For the library generated under autoconf, this is not a problem because those symbols are visible in the library: nothing is hidden.

[Read More]

Implementing Multi-threaded IO for the NetCDF-3 format

The issue of multi-threaded read and write of NetCDF files has repeatedly arisen in the netcdf mailing list (netcdfgroup@unidata.ucar.edu). Read on for a discussion of some options.

[Read More]
Unidata Developer's Blog
A weblog about software development by Unidata developers*
Unidata Developer's Blog
A weblog about software development by Unidata developers*

Welcome

FAQs

News@Unidata blog

Recent Entries:
Take a poll!

What if we had an ongoing user poll in here?

Browse By Topic
  • feed AWIPS (14)
Browse by Topic
« June 2019
SunMonTueWedThuFriSat
      
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
      
Today