netCDF library

[Rob Ross]

You could also write your data in a sparse format similar to something 
like CSR, using netCDF files as a container.

Rob

Russ Rew wrote:
> Hi,
> 
> Dave Allured wrote:
> 
>> ... Consider these options to reduce archival file size:
>>
>> 1.  Keep your current Netcdf format, but store your files gzip'ed.  Make 
>> uncompressing a standard part of opening the file.  Many application 
>> languages will allow you to call the shell to gunzip and delete a 
>> temporary file, so you can automate this.  gunzip is rather fast, as I 
>> recall.  As you stated, your file size is reduced by 99%.
>>
>> 2.  Netcdf 16-bit packed format.  Reduce file size by 50%.  You get 16 
>> bits for your combined precision and dynamic range.
>>
>> 3.  Netcdf 8-bit packed format.  Reduce file size by 75%.  You get 8 
>> bits for your combined precision and dynamic range.
> 
> That's good advice, and here's two additional options:
> 
> 4.  Switch to using HDF5, which supports a form of access called
>     "chunking", a multidimensional tiling of arrays that permits
>     accessing data a tile at a time, with each tile compressed
>     separately.  Thus if you only access a small subset of the data in
>     a large file, only a small part of the data is uncompressed, so
>     you don't pay the penalty of uncompressing the whole file on open.
>     With chunking, you can also choose which variables to compress
>     instead of compressing the entire file.
> 
> 5.  Wait for netCDF-4, which also supports chunking and compression by
>     delegating it to the HDF5 storage layer.  Currently the release of
>     netCDF-4 is being delayed until the corresponding HDF5 1.8
>     software on which it depends is ready (estimated December 2006),
>     but if you can wait, it may be easier to convert your code to use
>     netCDF-4 than HDF5.  You can also test the use of chunking in the
>     current netCDF-4 alpha release, but note that it uses artifacts
>     that will not be supported in the real netCDF-4 software.
> 
> --Russ
> 
> _____________________________________________________________________
> 
> Russ Rew                                         UCAR Unidata Program
> russ at unidata.ucar.edu                     http://www.unidata.ucar.edu
> 
> ==============================================================================
> To unsubscribe netcdfgroup, visit:
> http://www.unidata.ucar.edu/mailing-list-delete-form.html
> ==============================================================================
> 

==============================================================================
To unsubscribe netcdfgroup, visit:
http://www.unidata.ucar.edu/mailing-list-delete-form.html
==============================================================================