Re: 1.8 file format

[Quincey Koziol <koziol@xxxxxxxxxxxx>]

Hi John,

On Jul 24, 2007, at 1:35 PM, John Caron wrote:

> Hi Quincey: comments are inline:
>
> Quincey Koziol wrote:
> > Hi John,
> > On Jul 24, 2007, at 12:22 PM, John Caron wrote:
> > > Meanwhile yet another question on a different topic:
> > >
> > > Im trying to figure out what the NPOESS data files are doing with  
> > > "Reference Types".
> > >
> > > Case 1 is a Reference with type = 0 (Object Reference)
> > >
> > > long VIIRS-AF-EDR_Aggr(5);
> > >    :AggregateBeginningDate = "2003125";
> > >    :AggregateBeginningGranuleID = "NPP001212126088";
> > >    :AggregateBeginningOrbitNumber = 9; // int
> > >    :AggregateBeginningTime = "10109.840960z";
> > >    :AggregateCreationDate = "2003125";
> > >    :AggregateEndingDate = "2003125";
> > >    :AggregateEndingGranuleID = "NPP001212126088";
> > >    :AggregateEndingOrbitNumber = 9; // int
> > >    :AggregateEndingTime = "101038.325248z";
> > >    :_LastModified = "2005-08-29T15:54:58Z";
> > >
> > > data:
> > >
> > > {2928, 3528, 3800, 4072, 4344}
> > >
> > > The 5 values are indeed object references to 5 other datasets in  
> > > the file. Any clues on how this is used, or is it an internal  
> > > structure that should be left alone?
> >     An object reference is simply the offset in the file of the  
> > object header for the object referenced.  Users shouldn't modify  
> > them directly, but use the H5R* API routines for working with them.
>
> I can understand a reference to another object as an alias.
> Bur what does an array of length 5 of references to 5 other objects  
> mean?

	The datatype is "object reference" and the datatype is "5", so that  
means the dataset's raw data is an array of 5 elements, each of which  
is an object reference (i.e. object ID) to another object in the file.

> > > h5dump is:
> > >
> > > DATASET "VIIRS-AF-EDR_Aggr" {
> > >            DATATYPE  H5T_REFERENCE
> > >            DATASPACE  SIMPLE { ( 5 ) / ( 5 ) }
> > >            ATTRIBUTE "AggregateBeginningDate" {
> > >               DATATYPE  H5T_STRING {
> > >                     STRSIZE 7;
> > >                     STRPAD H5T_STR_NULLTERM;
> > >                     CSET H5T_CSET_ASCII;
> > >                     CTYPE H5T_C_S1;
> > >                  }
> > >               DATASPACE  SIMPLE { ( 1, 1 ) / ( 1, 1 ) }
> > >            }
> > >            ATTRIBUTE "AggregateBeginningGranuleID" {
> > >               DATATYPE  H5T_STRING {
> > >                     STRSIZE 15;
> > >                     STRPAD H5T_STR_NULLTERM;
> > >                     CSET H5T_CSET_ASCII;
> > >                     CTYPE H5T_C_S1;
> > >                  }
> > >               DATASPACE  SIMPLE { ( 1, 1 ) / ( 1, 1 ) }
> > >            }
> > >            ....
> > >
> > >
> > >
> > > Case 2 is a Reference, with type = 1 (Dataset Region Reference).
> > >
> > > long VIIRS-AF-EDR_Gran_0(5);
> > >
> > > data:
> > >
> > >  {3299824, 14172636162555905, 8589934592, 3299824,  
> > > 14172636162555907}
> > >
> > > h5dump is
> > >
> > > DATASET "VIIRS-AF-EDR_Gran_0" {
> > >            DATATYPE  H5T_REFERENCE
> > >            DATASPACE  SIMPLE { ( 5 ) / ( 5 ) }
> > >
> > > The docs on this are pretty sketchy, i wonder if i could get an  
> > > expanded description of what the Dataset Region Reference  
> > > structure looks like? Heres whats there (last page of 1.6.5 doc):
> > >
> > > "Dataset region references are stored as a heap-ID which points  
> > > to the following information within the file-heap: an offset of  
> > > the object pointed to, number-type information (same format as  
> > > header message), dimensionality information (same format as  
> > > header message), sub-set start and end information (i.e. a  
> > > coordinate location for each), and field start and end names  
> > > (i.e. a [pointer to the] string indicating the first field  
> > > included and a [pointer to the] string name for the last field)."
> > >
> > > specifically:
> > >  "an offset of the object pointed to" = object id ?
> > >  "number-type information (same format as header message)"  
> > > datatype message?
> > >  "dimensionality information (same format as header message)" =  
> > > dataspace message?
> > >
> > > and then im even more lost as to what the fields are....
> >     Hmm, looks like the docs are out of date and/or wrong for  
> > region references.  Because region references are variable in  
> > length, the actual reference information is stored in the global  
> > heap and a (fixed length) heap ID is stored in the dataset's raw  
> > data - that part is right, although too briefly described.  The  
> > format of the data in the heap is wrong however.  The information  
> > actually stored in the heap is as follows:
> >     - The offset of the object header of the object (ie. dataset)  
> > pointed to (yes, an object ID)
> >     - A serialized form of a dataspace _selection_ of elements (in  
> > the dataset pointed to).  I don't have a formal description of  
> > this information now, but it's encoded in the H5S_<foo>_serialize 
> > () routines in src/H5S<foo>.c, where foo = {all, hyper, point, none}.
> >     There is _no_ datatype information stored for these sort of  
> > selections currently.
> >     I'll file a bug report in our bugzilla to make certain this  
> > description gets fixed in the file format spec.
>
> ok thanks, any further docs as you develop them would be appreciated.

	Definitely. :-)

> BTW, it doesnt seem like h5dump distinguishes between type 0 and  
> type 1 references.

	Hmm, what were your command line options to h5dump?  (And what  
version of the library were you using?)

	Quincey

Attachment: smime.p7s
Description: S/MIME cryptographic signature