ncdigest V1 #1119

[Paul van Delst]

Jeff Whitaker wrote:
> Ed Hartnett wrote:
>> Jeff Whitaker <jswhit at fastmail.fm> writes:
>>
>>  
>>> By the way, while I'm ranting ...  Another potential problem is see is
>>> nested user-defined types.  Since this is allowed in netcdf-4, I can
>>> see the potential for creating horribly complicated files with
>>> compounds of vlens containing compounds containing enums etc.  Don't
>>> think many people would be crazy enough to use it, but it makes it
>>> very hard to create client code to read arbitrary netcdf-4 files,
>>> since you have to check for all those possibilities in general.  I
>>> wonder whether it might not be wise to sacrifice some generality for
>>> simplicity and just say that user-defined types can only be composed
>>> of primitive data types.
>>>
>>>     
>>
>> Howdy Jeff!
>>
>> Here at NetCDF World Headquarters, we also wonder how these new
>> features will be used by the netCDF community!
>>
>> Perhaps some of the HDF5 readers of this mailing list can provide some
>> real world examples of nested structures.
>>
>> To read them, the only general purpose answer I know of (as with
>> groups) is a recursive function. This is how the netCDF-4 file reading
>> code handles the problem, and, I believe, how ncdump handles the
>> problem (Russ will correct me if I'm wrong about ncdump).
>> Both of these are in C (for example libsrc4/nc4hdf5.c, function
>> rec_read_metadata().
>>
>> The problem, perhaps, is what to do with this information once you
>> have it.
>> For example, it's easy to plot a data file full of NC_INT, or
>> NC_FLOAT, but how do you plot a file full of data in compound types?
>>
>> It is a more complicated problem, but I'm sure the very clever
>> programmers working on the IDV will come up with something. ;-)
>>
>> Although these new features present many new complexities, the
>> CLASSIC_MODEL flag allows users to produce netCDF-4 files which don't
>> use any of the user-defined types, and thus can be read (and, with a
>> little modification on the nc_create call, written) by existing
>> software.
>>
>> Thanks!
>>
>> Ed
>>
>>   
> Ed:  I think it's relatively easy (and productive)  to deal with 
> compound types, where the elements are primitive (fixed size) data 
> types.  I think that allowing nested compound types (or compound types 
> containing vlens) increases the complexity exponentially though,   
> Unless there are some real use cases out there, I wonder if it is really 
> worth it.

This is an anecdotal datapoint, but I use nested structures now (only one level deep 
though) in my f95 code. I tend to write "atomic" I/O functions (wrappers around the 
required f90 API calls) for structures which translates to reading from separate files. It 
would be nice to be able to "layer" that sort of functionality. I guess it's a 
file-within-a-file type of thing.

Having said all that, I also value simplicity. netCDF is an *extremely* useful format from 
that perspective for what I do. Is there a hue and cry from the user community for 
multi-level nested compound type I/O in future netCDF versions? Otherwise, I think JeffW's 
initial idea,
   "I wonder whether it might not be wise to sacrifice some generality for
    simplicity and just say that user-defined types can only be composed
    of primitive data types."
(which I interpret as similar to my "one level deep" nesting) is a very good one.


cheers,

paulv

==============================================================================
To unsubscribe netcdfgroup, visit:
http://www.unidata.ucar.edu/mailing-list-delete-form.html
==============================================================================