Re: netCDF library

Nilesh Lahoti a écrit :
...
The netCDF library works great for us. However, I came across with one particular issue of netCDF and would like to discuss if there are any solution to this problem or something that can do to make its performance better. When we process emissions for our three dimensional grid of size (172 x 172 x 22) for 24 hours time period having hourly data, the file size is around 1 gigabyte(GB). There are several cells that have zero values and therefore the floating point value for pollutants in netCDF file has zero values. When we use gzip utility on unix to compress this files, the file size become almost 10 MB which saves us 99% of disk space. Now the question arise that if the netCDF is most compress scientific format, than is it possible to suppress this zero values of the floating point variable or is there any switch that can be used to handle zero values and reduce file size by any chance.
...
Hello,
You can also have a look at this page: http://snow.cit.cornell.edu/noon/z_netcdf.html This is a modification of the lowlevel file layer of netcdf library which allow compressing and uncompressing data on the fly (in fact the whole netCDF structure).

The compressed files produced are not readable by the standard library and it may be difficult to share those file. However there is a command line utility
(nczip) which allow converting the compressed file to a 'normal' netCDF
file and any classical netCDF file to a compressed one. The "magic
number" is not the same so there is no risk the library being confused
about the file type (CDF for standard netCDF files CDZ for compressed
ones).

The advantage is that there is no interface modification of the API (except
one attribute, NC_COMPRESS, on opening if you want to create
compressed files directly in your program) and this allow you to wait
until netCDF-4.

The annoyance is that all your programs must be relinked with the modified
library.
I have used this patch with success years ago but I am not sure that it will
work with recent versions of the netCDF library (specially the 64 bits
extension known as file format version 2).

--
Philippe Poilbarbe
CLS - Space Oceanography Division

==============================================================================
To unsubscribe netcdfgroup, visit:
http://www.unidata.ucar.edu/mailing-list-delete-form.html
==============================================================================


  • 2006 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the netcdfgroup archives: