Netcdf-5 format (NOT REALLY) to be based on GRIB

NOTE:
This article was published back in 2012 as an April Fool's Day joke. Six years later, folks searching for information about netCDF see the part about version 5 and assume that the article is real, without reading to the end of the post where it says "P.S. April Fools." So here's your warning — this is a joke!

Most people know that the netCDF-4 format uses HDF-5 as the underlying file format. With chunking and compression, large datasets may be 2-10 times smaller than the same data stored in the netCDF-3 format. However, we have not been able to reach the compression efficiency of GRIB-2, which uses dynamic scale/offsets to turn floating point numbers into integers, and JPEG2000 wavelet compression to store the integers very efficiently. Carefully tuned GRIB-2 may be 40 times smaller than netCDF-3.

After a careful study of the options in how to compete with GRIB2 for storage size efficiency, working in collaboration with NCAR's Research Application Laboratory, we are glad to announce that the next version of netCDF, which we call netCDF-5, will be based on the GRIB-2 format. 

GRIB-2 also`has the tremendous advantage of not needing to store metadata directly in the files, instead storing just numeric references to controlled vocabulary in external tables. These external tables are controlled by appropriate governing authorities, so that uniform metadata and naming conventions  is always assured. NetCDF-5 will also take this superior approach.

Finally,  the GRIB-2 data model is an unordered collection of 2D data slices, instead of the much more complex multidimensional arrays from netCDF. One advantage of this is that data can be stored in any order, across different files. This allows users to store important information in the file name, so that applications know exactly what is in the file without having to open it. We expect netCDF-5 to follow this tried-and-true method, and we will be developing a set of translators to rewrite older netCDF formats into GRIB-2/netCDF-5, with the CF metadata stored directly in the filenames.

Stay tuned to this blog, where we will be releasing more details as we implement this important new advance in scientific data formats.

- John

(P.S. April Fools . . . here's a discussion of the realities of GRIB: GRIB and BUFR as Archival Data Formats?)

Comments:

Was checking the netcdf web site and I am happy to see that Unidata is finally coming back to reason. Have you also thought about supporting BUFR as a backend that is way superior to GRIB2 ? It is even more code table driven and it would be useful for our community to go towards these more powerful idea.

Great post John can't wait to see what I will get next year at that time ;-)

Posted by Guillaume Aubert on April 17, 2012 at 03:15 AM MDT #

... really sad that I missed this in real time! It makes such good sense that I've immediately tweeted it today, never too late to cause trouble :-)

Posted by Bryan Lawrence on April 19, 2012 at 09:31 AM MDT #

Post a Comment:
Comments are closed for this entry.
Unidata Developer's Blog
A weblog about software development by Unidata developers*
Unidata Developer's Blog
A weblog about software development by Unidata developers*

Welcome

FAQs

News@Unidata blog

Recent Entries:
Take a poll!

What if we had an ongoing user poll in here?

Browse By Topic
Browse by Topic
« July 2019
SunMonTueWedThuFriSat
 
3
4
5
6
7
9
10
11
12
13
14
17
18
19
20
21
23
24
25
26
27
28
29
30
31
   
       
Today