NOTE: The netcdf-hdf mailing list is no longer active. The list archives are made available for historical reasons.
Hi John,
> >>the motivation for me would be to use bit-packing as a storage format,
> >>not a data type. we would add an option to pack wider types (usually
> >>float/double) using a scale/offset. this can get you a factor of 2-4 or
> >>so, whereas compression may not get you anything.
> >>
> >>however, this would only work if it remains a valid hdf5 file. It would
> >>be most useful if we can do arbitrary bit widths, but still useful if we
> >>are limited to multiples of 8.
> >>
> >>
> > If we used a pipeline filter to implement this as a compression
> > mechanism,
> >the file would be fine and the higher levels would be unaware of any change
> >in the storage format. It still would require some serious thought about how
> >to handle bitfields in complicated datatypes.
> > Also, you seem to be branching in two different directions - just
> >extracting the bitfield from the type to make things smaller on disk and a
> >different "compression" mechanism which performed scale/offset operations on
> >data as it was written to disk. This first would just be a bit operation
> >where
> >the latter would be arithmetic operations.
> >
> > Quincey
> >
> >
> i guess you are saying forget the bitfield type, this is a compression
> pipeline thing ? anyway, that sounds right to me.
Well, it would be somewhat easier to handle there, at least.
> i think it would be ok not to allow it on structure members.
Ok. Is this something that you think we need to develop now, or something
for your team, or just something to think about?
> the scale/offset can be calculated easily from the data itself. often,
> people want to apply different scale/offset to different parts of the
> same array, eg vertical levels.
Hmm, how would you parameterize this? Would a user select various parts
of the dataset's dataspace and specify scale/offset information for them?
Quincey
netcdf-hdf archives: