[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[netCDF #XYO-599400]: A possble netCDF-4 bug?

Lynton,

> Thanks for the reply. There is evidently an issue with netCDF-4 in that
> being built upon HDF5, it is subject to the limitations imposed by HDF5.
> This is of  course unavoidable. Would you be able to update the netCDF4
> documentation to alert people using netCDF-4 to these limitations,
> 
> 
> Here is a  suggestion of some additional documentation to insert at the
> start of section 5.5:
> 
> 
> "
> Compound data types can be defined for netCDF-4/HDF5 format files. A
> compound datatype is similiar to a struct in C and contains a collection
> of one or more atomic or user-defined types. The netCDF-4 compound data
> must comply with the properties of the  HDF5 compound data type:
> http://www.hdfgroup.org/HDF5/doc/H5.intro.html#Intro-PMCreateCompound
> In summary these are:
> * It has a fixed total size, in bytes.
> * It consists of zero or more members (defined in any order) with
> unique names and which occupy non-overlapping regions within the datum.
> * Each member has its own datatype.
> * Each member is referenced by an index number between zero and
> N-1, where N is the number of members in the compound datatype.
> * Each member has a name which is unique among its siblings in a
> compound datatype.
> * Each member has a fixed byte offset, which is the first byte
> (smallest byte address) of that member in a compound datatype.
> * Each member can be a small array of up to four dimensions.
> 
> Note that currently there is a restriction that limits the use of fill
> (ref) to compound data types not larger than 2**16=65536bytes
> (equivalent to 8192 real*8 (doubles) or  16384 floats). Attempts to
> exceed this will result in  NC_EHDFERR error.
> "

Yes, thanks very much for the suggestion, I've added something similar 
to the snapshot C User's Guide which should appear in the snapshot
documentation on the web soon:

  http://www.unidata.ucar.edu/netcdf/docs/netcdf-c.html#Compound-Types

> =============================================================
> 
> Finally, one other matter regarding compound structures. I would like
> the user to specify the details of the compound data type at run time.
> For example I would like to specify a data type
> types:
> compound compoundType_3 {
> double member3(SIZE) ;
> }; // compoundType_3
> where SIZE is specified by the user at run time.
> This means that I cannot use a struct to represent the data type. A
> knock-on effect is that you can no-longer use sizeof and offsetof to
> determine the size of the data type and the offsets of the members (both
> are needed for defining compound data type).
> Would it be possible for  you to provide some easy-to-use utility to
> provide this information, in the absence of a struct?
> 
> I am aware that Dennis Heimbigner produced a blog note on how to do
> this, but it seemed rather complicated. I also note that HDF5 don't
> provide such a utility either, but I think this would be of great
> benefit to the netCDF community.

That's also an excellent suggestion, which I'll put on our "to do" list.

--Russ

> 
> Unidata netCDF Support wrote:
> > Lynton,
> >
> > Evidently the HDF5 limit that's being exceeded is H5O_MESG_MAX_SIZE, 
> > defined as
> > 65536       /*max obj header message size  */
> > in H5Oprivate.h.  You're right, this seems to have something to do with 
> > initializing
> > a header object with HDF5 fill values.  Your compound type has a double 
> > array member
> > with 8192 values, and sizeof(double)*8192 is 65536, so I think that's the 
> > problem.
> > I expect if you used float instead of double, you could have twice as many 
> > of them.
> >
> > I'm not sure why the HDF5 size limit for an array type is only 2**16, 
> > except that the
> > documentation says
> >
> >   A compound datatype is similar to a struct in C or a common block in 
> > Fortran.
> >   It is a collection of one or more atomic types or small arrays of such 
> > types.
> >     ...
> >   Each member can be a small array of up to four dimensions.
> >   http://www.hdfgroup.org/HDF5/doc/H5.intro.html#Intro-PMCreateCompound
> >
> > --Russ
> >
> >
> > Russ Rew                                         UCAR Unidata Program
> > address@hidden                     http://www.unidata.ucar.edu

Russ Rew                                         UCAR Unidata Program
address@hidden                     http://www.unidata.ucar.edu



Ticket Details
===================
Ticket ID: XYO-599400
Department: Support netCDF
Priority: Normal
Status: Closed