Part 2: Converting GRIB to NetCDF-4

Part 2: Converting GRIB to NetCDF-4

Last time I described the experiment of writing NCEP model GRIB files to netCDF-4. Here are the raw results of that experiment, using deflate level 3 and no shuffle for netCDF-4 compression:

 


NCEP Model Data: GRIB-1 and GRIB-2

 

There are a total of 51 NCEP model runs in this plot, each is one complete forecast run. Lets split the files out by GRIB-1 and GRIB-2:


 


 

As you can see, GRIB-2 has significantly better compression than GRIB-1, probably due to the JPEG-2000 wavelet compression. In case you are wondering about the file where netCDF-4 is smaller than GRIB 2 (the point between .5 and .75 ratio), that is RTMA_GUAM_2p5km_20140803_0600.grib2. It has only 17 records in it, and the netCDF-4 file is .561 smaller (440K vs 781K). This file does not use JPEG-2000 compression, and is a small grid (193 by 193) with most of its points over water, and so has more uniform data values.

There are 15 GRIB-1 files, and 36 GRIB-2 files, and the number of records in each file varies widely. If we use the number of records to find the weighted average, we get these results:

  1. Total over all files:
       Weighted average ratio = 2.18
       Total # grib records = 400,403
  2. Total over GRIB-1 files:
      Weighted average ratio = 1.32
      Total # grib records = 24,933
  3. Total over GRIB-2 files:
      Weighted average ratio = 2.24
      Total # grib records = 375,470
So, using out-of-the box netCDF4/HDF5 deflate compression, one could expect to get netCDF-4 files that are on average 1.32 larger than GRIB-1 and 2.24 larger than GRIB-2, on these kinds of precision-limited data.

Next time: results broken out by the number of bits stored for the variable.

 

 

Comments:

Post a Comment:
  • HTML Syntax: Allowed
Unidata Developer's Blog
A weblog about software development by Unidata developers*
Unidata Developer's Blog
A weblog about software development by Unidata developers*

Welcome

FAQs

News@Unidata blog

Recent Entries:
Take a poll!

What if we had an ongoing user poll in here?

Browse By Topic
Browse by Topic
« February 2019
SunMonTueWedThuFriSat
     
1
2
3
5
6
7
8
9
10
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
  
       
Today