Re: [netcdfgroup] Files with compound types ARE portable!

• To: Ed Hartnett <ed@xxxxxxxxxxxxxxxx>
• Subject: Re: [netcdfgroup] Files with compound types ARE portable!
• From: Jeff Whitaker <jswhit@xxxxxxxxxxx>
• Date: Tue, 12 May 2009 05:21:37 -0600

Ed Hartnett wrote:

Jeff Whitaker <jswhit@xxxxxxxxxxx> writes:

Ed Hartnett wrote:

Howdy Jeff,

Just an update: we have confirmed this bug and I am working on it
now. I have also added this to our automatic testing, so that we can
see it break here on our test machines, and ensure that once it's
fixed, it doesn't break again.

I will let you know when this fix is available in the snapshot -
hopefully not more than a few days from now...

Thanks again for find this!

Ed

Ed:  Any progress on this one?

Howdy Jeff!

Yes, there has been much progress, and it you get the snapshot I believe
your case will now work. (And if you look in the ncdump and libsrc4
directories you can see where I use your CDL and the files generated
from it for a new series of cross-platform testing that's now part of
the build.)

phony_var = {20000, 4} ;

without the __packed__ attribute set, the right data is read back

phony_var = {20000, 300000} ;

Interesting, h5dump also now displays the same wrong data as ncdump

     DATA {
     (0): {
           20000,
           4
        }
     }

-Jeff

However I am not done with this problem, there are some other cases that
need to be checked (like compound inside of vlen, and vice-versa). So I
am still working on the code and adding more cross-platform tests. I
have also taken this opportunity for some core libsrc4 refactoring with
respect to the handling of user-defined types.

I don't think I have yet solved all the compound types problems on all
platforms, as my new tests are still causing failures on the IRIX and
AIX platforms. I am looking into this now.

Also, to answer your original concern, this does not mean that the data
files are not portable. They are. The bug was in the way that the
netCDF-4 gave out information on the user-defined types in the
file. This is why generic read programs (like ncdump) can demonstrate
the problem, while cross-platform testing of the compound type do not
show the problem.

But the files that you created are perfectly fine, and when this bug fix
is complete they will be understood properly by ncdump and other
programs that generically read a netCDF-4 file with compounds inside
compound types.

As always, a good way to confirm this is to look at the h5dump
output. If h5dump shows the data properly, then it is stored properly in
the HDF5 file. Also, since netCDF-4 creates perfectly normal-looking
HDF5 files, any HDF5 visualization package can be used on a netCDF-4
file to look at the data. (I don't know if any visualization package can
display a data from a compound within a compound type!)

I will send out an update when this cross-platform work is complete.

Thanks!

Ed

• References:
  • [netcdfgroup] files with compound types not portable?
    • From: Jeff Whitaker
  • Re: [netcdfgroup] files with compound types not portable?
    • From: Ed Hartnett
  • Re: [netcdfgroup] files with compound types not portable?
    • From: Jeff Whitaker
  • [netcdfgroup] Files with compound types ARE portable! (was: Re: files with compound types not portable?)
    • From: Ed Hartnett