HDF group, please pass to Mike Folk...thanks
John, we are continuing to look at how we might implement the common data model i/o service provider layer on existing satellite data set formats. When one already has tens of thousands of files, it seems that you might want to write the format to the API using the i/o service provider, what we are calling bringing the API to the data (as opposed to converting all the data to the a new format). The old so-called 1b format satellite data sets are, unfortunately, in bit packed formats. I seem to recall that although HDF5 advertises bit manipulation, you indicated that it was not perhaps really there yet. If this is the case, then we could not bring the API to the data and we would only look at bringing the data to the API (i.e., rewrite all the data into a new format...doable but...).
So, does HDF5 support bit manipulation; within the framework of the CDM? Perhaps Mike knows the answer.
So there's 2 parts to this:
1) The IOSP that you have to write to "bring the API to the data" (I like it, can I steal it?) has to be able to read the file format, bit packing and all, and make it available for reading through the API. This is sufficient for giving access to the data.
If you want more details, I can ask Ed Hartnett, out Netcdf-4 expert, to provide them.
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.