Showing entries tagged [hdf5]

NetCDF4 use of dimension scales

Some thoughts on how shared dimensions could be done in a simpler way. [Read More]

Chunking Data: Choosing Shapes

In part 1, we explained what data chunking is about in the context of scientific data access libraries such as netCDF-4 and HDF5, presented a 38 GB 3-dimensional dataset as a motivating example, discussed benefits of chunking, and showed with some benchmarks what a huge difference chunk shapes can make in balancing read times for data that will be accessed in multiple ways.

In this post, I'll continue looking at that example dataset to see how we can derive good chunk shapes, generalize to other datasets, look at how long it can take to rechunk a multidimensional dataset, and look at the use of Solid State Disk (SSD) for both accessing and rechunking data.

[Read More]

Chunking Data: Why it Matters

What is data chunking? How can chunking help to organize large multidimensional datasets for both fast and flexible data access?  How should chunk shapes and sizes be chosen?  Can software such as netCDF-4 or HDF5 provide better defaults for chunking? If you're interested in those questions and some of the issues they raise, read on ...

[Read More]

NetCDF-4 Dimensions and HDF5 Dimension Scales

If you want to write HDF5 files directly without using the netCDF-4 library, or if you want to build a netCDF-4 compatible software layer on top of HDF5, read on.

[Read More]

HDF5 Dimension Scales - Part 3

The gritty details of what dimension scales look like at the file object level.

[Read More]
Unidata Developer's Blog
A weblog about software development by Unidata developers*
Unidata Developer's Blog
A weblog about software development by Unidata developers*

Welcome

FAQs

News@Unidata blog

Recent Entries:
Take a poll!

What if we had an ongoing user poll in here?

Browse By Topic
Browse by Topic
« July 2017
SunMonTueWedThuFriSat
      
1
2
4
5
6
7
8
9
11
12
13
14
15
16
18
19
20
21
22
23
25
26
27
28
29
30
31
     
Today