Decoder Packages Update - September 2006
Robb Kambic

The Decoders package's purpose is to put available raw data into usable formats. Since the data's resolution changes frequently, and the data parameters and fields use different times and levels, treating data as if it were on a regular grid may increase its size substantially. In addition, users want to aggregate files for temporal studies.

Because of growth in the size of decoded files, we have recently developed a new approach for making decoded data available to users.

Users typically want only a subset of the available parameters or fields. The new approach decodes the requested data subsets on-the-fly, creating file indexes of the raw data when they can be used multiple times for extraction. By using these indexes, fast data access is provided even for files over a gigabyte. The indexes contain other information as well, so programs do not have to access the files for metadata information, parameter names, and observation times, for example. This approach has proved successful for GRIB and BUFR data.

The GRIB Java Decoder was officially released last year, and currently it is used to supply the GRIB data to the IDV via the netcdf-Java library. This year the library has been modified to handle Ensemble data for GRIB2 type products and more parameter tables have been added for the different centers.

The BUFR Java library is still in beta release. It is currently in the process of being integrated into netcdf-Java library so it can supply the data to the IDV. The BUFR library can decode most BUFR files, but since the BUFR data packing scheme is so flexible, it is hard to test all the variations.

There has been little work on the old netCDF-Decoders package except for bug fixes because the GRIB Java Decoder can read the data and the netCDF Java library can be used to write netCDF files using the CF convention.

Converting the perl netCDF decoders into Java so they can use the netCDF-Java library instead of the netcdf-perl API is in the planning stages. This should improve the decoders' performance and make them more flexible.