Showing entries tagged [netcdf-4]

Chunking Data: Choosing Shapes

In part 1, we explained what data chunking is about in the context of scientific data access libraries such as netCDF-4 and HDF5, presented a 38 GB 3-dimensional dataset as a motivating example, discussed benefits of chunking, and showed with some benchmarks what a huge difference chunk shapes can make in balancing read times for data that will be accessed in multiple ways.

In this post, I'll continue looking at that example dataset to see how we can derive good chunk shapes, generalize to other datasets, look at how long it can take to rechunk a multidimensional dataset, and look at the use of Solid State Disk (SSD) for both accessing and rechunking data.

[Read More]

Chunking Data: Why it Matters

What is data chunking? How can chunking help to organize large multidimensional datasets for both fast and flexible data access?  How should chunk shapes and sizes be chosen?  Can software such as netCDF-4 or HDF5 provide better defaults for chunking? If you're interested in those questions and some of the issues they raise, read on ...

[Read More]

netcdf-4 DLLs

A rampaging mob finally convinces me to spend some quality time with DLL development...

[Read More]

Data Format Summit Meeting

Last week, on Wednesday, the Unidata netCDF team spent the day with Quincey and Larry of the HDF5 team. This was great because we usually don't get to spend this much time with Quincey, and we worked out a lot of issues relating to netCDF/HDF5 interoperability.

I came away with the following action items:

  • switch to WEAK file close
  • enable write access for HDF5 files without creation ordering
  • deferred metadata read
  • show multi-dimensional atts as 1D, like Java
  • ignore reference types
  • try to allow attributes on user defined types
  • forget about stored property lists
  • throw away extra links to groups and objects (like Java does)
  • work with Kent/Elena on docs for NASA/GIP
  • hdf4 netCDF v2 API writes as well as reads HDF4. How should this be handled?
  • John suggests not using EOS libraries but just recoding that functionality.
  • HDF5 team will release tool for those in big-endian wasteland. It will rewrite the file.
  • should store software version in netcdf-4 file somewhere in hidden att.
  • use HDF5 function to find file type, this supports user block
  • read gip article
  • update netCDF wikipedia page with format compatibility info
  • data models document for GIP?

I have been assured that this blog is write-only, so I don't have to explain any of he above, because no one is reading this! ;-)

The tasks above, when complete, with together add up to a lot more interoperability between netCDF-4 and existing HDF5 data files, allowing netCDF tools to be used on HDF5 files.

NetCDF Presentation at HDF5 Workshop

This week I am attending the HDF5 workshop in Champaign, Illinois. I am learning a lot of interesting things about HDF5, and I gave a presentation on netCDF, which is now available on the netCDF web site for those that are interested:

Hartnett, E., 2010-09: NetCDF and HDF5 - HDF5 Workshop 2010.

It's great to see the HDF5 team again!

Unidata Developer's Blog
A weblog about software development by Unidata developers*
Unidata Developer's Blog
A weblog about software development by Unidata developers*

Welcome

FAQs

News@Unidata blog

Recent Entries:
Take a poll!

What if we had an ongoing user poll in here?

Browse By Topic
Browse by Topic
« February 2019
SunMonTueWedThuFriSat
     
1
2
3
5
6
7
8
9
10
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
  
       
Today