Re: HDF5 bitfields...

Quincey Koziol wrote:

Hi John,

the motivation for me would be to use bit-packing as a storage format, not a data type. we would add an option to pack wider types (usually float/double) using a scale/offset. this can get you a factor of 2-4 or so, whereas compression may not get you anything.

however, this would only work if it remains a valid hdf5 file. It would be most useful if we can do arbitrary bit widths, but still useful if we are limited to multiples of 8.
   If we used a pipeline filter to implement this as a compression mechanism,
the file would be fine and the higher levels would be unaware of any change
in the storage format.  It still would require some serious thought about how
to handle bitfields in complicated datatypes.
   Also, you seem to be branching in two different directions - just
extracting the bitfield from the type to make things smaller on disk and a
different "compression" mechanism which performed scale/offset operations on
data as it was written to disk.  This first would just be a bit operation where
the latter would be arithmetic operations.

   Quincey
i guess you are saying forget the bitfield type, this is a compression pipeline thing ? anyway, that sounds right to me.

i think it would be ok not to allow it on structure members.

the scale/offset can be calculated easily from the data itself. often, people want to apply different scale/offset to different parts of the same array, eg vertical levels.