[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Netcdf4 Compound types and Examples

Roy,

> Okay guys, here is the "example" we are working on.  A long time ago
> we put the  World Ocean Database and the GTSPP data, all subsurface
> obs that we have or get, into the old HDF-EOS point structure. The
> way we had it structured was that each file was a 10-degree square
> with a name that contained the box number as in COADS.  The HDF-EOS
> point structure had levels that made what was a series of flat tables
> look like a hierarchy by having a link field.  So level 0 data would
> be all the box 2's  in the box 10, and the level 1 data would be the
> data with the box2 number as the link field.  This allowed you to
> search for all the data in a box2 efficiently
>
> A downside was that the number of rows in the field had to be equal
> to the sum total of depths at which there were observations.    This
> meant that there is a huge amount of empty space.

I understand the COADS box numbers for 10 degree or two degree boxes,
but I don't understand what you're saying about levels and link fields.
Is there a simpler or more detailed description of this data structure
somewhere that I could study?  I'd like to understand the requirements
and whether netCDF-4 could eliminate the wasted space while preserving
fast access.

>                                               ...  So how might we
> improve on this - why netcdf4!  We use vlen's to deal with the
> varying number of depths and a compound data type to combine together
> the info.  Groups and (sub)-Groups provide the same structure that
> was "emulated" in the point structure.  Or at least that was what we
> thought would be a good design given netcdf4's features, but that is
> what prompted my email about examples and design - there may well be
> better ways of doing this.
>
> However, unless I miraculously find both money and a good C
> programmer, or get Fortran wrappers to work, it will be sometime
> before it happens.
>
> BTW, at one point I used the hdf4tohdf5 program to convert these
> files to HDF5, so they can be read using that library.  I don't know
> that the conversion worked that cleanly to use with the HDF-EOS5
> library, but it should be possible to both read and write using the
> libs that are a part of Netcdf4.  I actually had a few days to begin
> to work on this, hence the series of emails.
>
> -Roy
>
>
>
>
> **********************
> "The contents of this message do not reflect any position of the U.S.
> Government or NOAA."
> **********************
> Roy Mendelssohn
> Supervisory Operations Research Analyst
> NOAA/NMFS
> Environmental Research Division
> Southwest Fisheries Science Center
> 1352 Lighthouse Avenue
> Pacific Grove, CA 93950-2097
>
> e-mail: address@hidden (Note new e-mail address)
> voice: (831)-648-9029
> fax: (831)-648-8440
> www: http://www.pfeg.noaa.gov/
>
> "Old age and treachery will overcome youth and skill."
>
>