NOTE: The netcdf-hdf mailing list is no longer active. The list archives are made available for historical reasons.
Quincey Koziol wrote:
i guess you are saying forget the bitfield type, this is a compression pipeline thing ? anyway, that sounds right to me.Hi John,the motivation for me would be to use bit-packing as a storage format, not a data type. we would add an option to pack wider types (usually float/double) using a scale/offset. this can get you a factor of 2-4 or so, whereas compression may not get you anything.however, this would only work if it remains a valid hdf5 file. It would be most useful if we can do arbitrary bit widths, but still useful if we are limited to multiples of 8.If we used a pipeline filter to implement this as a compression mechanism, the file would be fine and the higher levels would be unaware of any change in the storage format. It still would require some serious thought about how to handle bitfields in complicated datatypes. Also, you seem to be branching in two different directions - just extracting the bitfield from the type to make things smaller on disk and a different "compression" mechanism which performed scale/offset operations on data as it was written to disk. This first would just be a bit operation where the latter would be arithmetic operations. Quincey
i think it would be ok not to allow it on structure members.the scale/offset can be calculated easily from the data itself. often, people want to apply different scale/offset to different parts of the same array, eg vertical levels.
netcdf-hdf archives: