Due to the current gap in continued funding from the U.S. National Science Foundation (NSF), the NSF Unidata Program Center has temporarily paused most operations. See NSF Unidata Pause in Most Operations for details.

[netcdfgroup] ncgen should remove special attributes like _Shuffle

Hi,
newer versions of netcdf (tested with 4.2.1 and 4.1.1) allow the generation of netcdf4 files with shuffling, chunking and compression by using the attributes:
`_Format', `_Fletcher32, `_ChunkSizes', `_Endianness',_DeflateLevel', `_Shuffle', and `_Storage'.
Unfortunately, these attributes are not only used for creating the file, but are also written as normal attributes to the file. 

A simple example:

netcdf test {
dimensions:
    time = UNLIMITED ; // (2 currently)
variables:
    short time(time) ;
        time:_Storage = "chunked" ;
        time:_ChunkSizes = 1 ;
        time:_Shuffle = "true" ;
        time:_DeflateLevel = 3 ;
// global attributes:
        :_Format = "netCDF-4 classic model" ;
data:
 time = 12, 24 ;
}


looks after a 'ncgen -b', and 'ncdump -hs test.nc' like:

netcdf test {
dimensions:
        time = UNLIMITED ; // (2 currently)
variables:
        short time(time) ;
                time:_ChunkSizes = 1 ;
                time:_DeflateLevel = 3 ;
                time:_Shuffle = "true" ;
                time:_Storage = "chunked" ;
                time:_ChunkSizes = 1 ;
                time:_DeflateLevel = 3 ;
                time:_Shuffle = "true" ;
                time:_Endianness = "little" ;

// global attributes:
                :_Format = "netCDF-4 classic model" ;
}
When using some further processing, e.g. 'nccopy -k 3 -d 9 test.nc test9.nc', test9.nc will have ambiguos _DeflateLevel settings: 

$ ncdump -hs test9.nc
netcdf test9 {
dimensions:
        time = UNLIMITED ; // (2 currently)
variables:
        short time(time) ;
                time:_ChunkSizes = 1 ;
                time:_DeflateLevel = 3 ;
                time:_Shuffle = "true" ;
                time:_Storage = "chunked" ;
                time:_ChunkSizes = 1 ;
                time:_DeflateLevel = 9 ;
                time:_Endianness = "little" ;

// global attributes:
                :_Format = "netCDF-4" ;
}
I guess, all those _Attributes should never be written to the file by ncgen. (Tested with netcdf-4.2.1.1 on linux (ubuntu 12.04 64bit)). 

Regards,

Heiko
  • 2013 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the netcdfgroup archives: