[netcdfgroup] arrays of fixed-length strings

[John Caron]

Jeff Whitaker wrote:
> John Caron wrote:
>>
>>
>> Jeff Whitaker wrote:
>>> John Caron wrote:
>>>>
>>>>
>>>> Jeff Whitaker wrote:
>>>>> Ed Hartnett wrote:
>>>>>> Jeff Whitaker <jswhit at fastmail.fm> writes:
>>>>>>
>>>>>>  
>>>>>>> In netcdf-4.0, I don't see how to create variables which are 
>>>>>>> arrays of
>>>>>>> strings with length > 1.  I see how to create arrays of
>>>>>>> single-characters, and arrays of variable-length strings, but not
>>>>>>> strings of a specified length.
>>>>>>>
>>>>>>> Am I missing something, or is this not supported by HDF5?
>>>>>>>
>>>>>>>     
>>>>>>
>>>>>> Howdy Jeff!
>>>>>>
>>>>>> Strings are variable length by their nature.
>>>>>>
>>>>>> How about a two dimensional array of NC_CHAR?
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Ed
>>>>>>
>>>>>>
>>>>>>   
>>>>>
>>>>>
>>>>> Ed:  I use arrays of fixed length strings, padded with spaces, 
>>>>> quite a bit.  This simplifies the memory management issues 
>>>>> associated with arrays of variable length strings (which have no 
>>>>> counterpart in Fortran 90/95, although they are allowed in Fortran 
>>>>> 2003).  Below is an excerpt from my previous reply which explains 
>>>>> why I don't like using 2-D arrays of characters to represent 1-D 
>>>>> arrays of fixed-length strings:
>>>>>
>>>>> Russ: I realize you can use a array of shape ndim,8 to store an 
>>>>> array of ndim 8 character strings.  Thats the way I've done it with 
>>>>> netcdf-3 - it just feels clunky.  A typical use case for me is 
>>>>> station data, where you want to store the name of the station. I 
>>>>> end up the with an array of characters shaped (nstations,ncars) -  
>>>>> in fortran I read it into an  (nstations,nchars) character(len=1) 
>>>>> array (after first finding out what both nstations and nchars are), 
>>>>> then reshape it into a (nstations) character(len=nchars) array.  
>>>>> I'd rather just read it into a character(len=nchars) array straight 
>>>>> off.  Not a show stopper for sure, but it would be more 
>>>>> convenient.  I realize that specifying the data type would be 
>>>>> tricky,  instead of NC_CHAR, do you have a bunch of new types 
>>>>> NC_CHAR1, NC_CHAR2, ... NC_CHAR120?  Or a new function datatype = 
>>>>> nc_set_chartype(nchars)?  However, I bet it would get used a lot 
>>>>> more than the esoteric datatypes you have in netcdf-4 already 
>>>>> (enums and opaque for example).
>>>>
>>>> Hi Jeff:
>>>>
>>>> Suppose we stuck with fixed length char arrays for this case, but 
>>>> added a convenience method in the API that did the work for you. 
>>>> What would that convenience method look like?
>>>>
>>>> John
>>>
>>> John:  A convenience method won't really help much - the thing I'd 
>>> most like to avoid is defining another dimension to hold the number 
>>> of characters in each string.  Essentially, I'd like to have that 
>>> information transferred to the datatype.
>>
>> Hi Jeff:
>>
>> It seems unlikely that we'd want to add a multitude of datatypes for 
>> this purpose. The extra dimension seems the right thing if you really 
>> want to specify that all the Strings are of the same length.
>> John
> 
> John:  OK.  Since HDF5 defines a fixed-length string datatype like this:
> 
> 
>    tid = H5Tcopy(H5T_C_S1);
>    H5Tset_size(tid, <string length>);
> 
> I thought you might be able to create a datatype on the fly in netcdf 
> with something like
> 
> stringtype = nc_set_stringtype(<string length>)
> 
> and then use 'stringtype' instead of NC_CHAR when defining the variable.
> 
> I guess I'm just used to thinking in fortran, where the length of the 
> string is a property of the datatype (defined when the variable is 
> declared).  It seems like dimensions should hold information about the 
> underlying physical grid, not implementation details of how the variable 
> is stored in memory.  I guess it depends on whether you think of a 
> string as an array of characters, or as a datatype unto itself.
> 
> -Jeff

Hi Jeff:

Its possible that the C API could offer such a feature, that would be up to Russ and Ed. But in terms of the underlying data model, I think that Strings will be considered variable length. One could consider Fortran's handling of Strings to be, uh, clumsy. I dont know of any modern languages that follow that, and I vaguely remembering Pascal being criticized for making array lengths part of the type.

Its a good point about what info a Dimension should capture. They really start off as indicating storage size and layout. Shared use with coordinate variables give them a much deeper significance describing the "underlying physical grid", as you say. Surprisingly, HDF5 and OPeNDAP data models didnt make them seperate objects that could be shared, though both have added other constructs to get similar semantics (dimension scales and grid maps, respectively).

As always, these things are a tradeoff.

I appreciate hearing your thoughts about these kinds of things.

John