Re: [netcdf-java] GRIB v2 files

On 6/20/2011 8:11 AM, Comiskey, Glenn wrote:
John,
While it is possible to present the data in this format having converted the GRIB v2 file to a NetCDF file, as you'll note from the quoted file sizes at the top of header.txt it results in an 11-fold increase in file size. If this was to be used for all GRIB v2 files it would require an enormous increase in storage capacity.
Regards,
Glenn

Hi Glenn:

Ive taken the liberty of cc'ing this to the netcdf-java list, as others may want to hear about this also.

1) The CDM reads the data in its native (GRIB2) format and does the conversion on the fly.

2) If you want to store the data in netCDF, you will get a factor of 10 or more increase in size for netCDF-3 format. The netCDF-4 library (built on HDF-5) allows one to store the data compressed. Our experiments indicate that GRIB2 compression still outperforms this by about a factor of 2. So currently we can reduce your factor of 11 to a factor of 2, if you switch to netCDF-4.

3) AFAIK, mostly this factor of 2 is due to GRIB JPEG-2000 wavelet compression. Eg this is what the data you sent me uses for encoding. We are working on adding this kind of compression to the netcdf-4 C library, and HDF-5 is interested in including this also. Our intention is to make the netCDF-4 format as space efficient as GRIB2. Im not sure if we will run into any roadblocks on this, but we are motivated to remove obstacles for netCDF adoption. I personally think that GRIB-2 should not be used as a long-term archive format, due to problems with tables, and also the kinds of problems that you have reported.

John
  • 2011 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the netcdf-java archives: