Re: [netcdf-hdf] parallel IO in netcdf-beta-4

robl@xxxxxxxxxxx (Robert Latham) writes:

> On Thu, Nov 15, 2007 at 11:06:09AM -0600, Edward (Ted) Mansell wrote:
>> I did some compiling of netcdf-4 (both beta 1 and the Oct. 22
>> snapshot with hdf5-1.8.0-of20070814), and while data compression is
>> working fine, parallel IO doesn't work (make check fails on the
>> parallel tests).  Is it safe to say that parallel IO just doesn't
>> work yet in netcdf-4? 
>
> Parallel I/O definitely works in netcdf-4, but did need a little bit
> of love.  
>
> Here's more or less how to get parallel MPI-IO out of netcdf-4
> - start with hdf5-1.8.0-beta3 for now.  
>
> - configure hdf5 for parallel I/O:  (basically build hdf5 with mpicc)
>
> - get ahold of something newer than the oct 22 netcdf-4 snapshot. Ed
>   committed some fixes to the configure process which make
>   parallel-netcdf4 work even better
>
> ==rob

Howdy Rob!

Thanks for the kinds words.

Some important API changes were made in HDF5 after beta3. The netCDF-4
code has already been changed to handle these HDF5 changes, so the
current netCDF-4 snapshot will not build with HDF5-1.8.0 beta3 any
longer.

There was a minor issue in HDF5-1.8.0-beta4, so that won't work
either. Instead, netCDF-4 users need to use the HDF5 snapshot made
available on the netCDF-4 download page:
http://www.unidata.ucar.edu/software/netcdf/builds/snapshot/netcdf-4/

(Whatever HDF5 version is needed for the netCDF-4 build, it will be
found on the netCDF-4 download page.)

So, to modify Robert's instructions above:

1 - Get the HDF5 snapshot here:
http://www.unidata.ucar.edu/software/netcdf/builds/snapshot/netcdf-4/

(And get and install the zlib library from here too, if you don't
already have it.)

2 - Build HDF5 with --enable-parallel (and
--with-zlib=/somewhere). Install it somewhere.

3 - Get the netCDF-4 snapshot from:
http://www.unidata.ucar.edu/software/netcdf/builds/snapshot/netcdf-4/

4 - Build netCDF-4 with --with-hdf5=/wherever_parallel_hdf5_is. (If
you want netCDF to run parallel tests, and you can run programs from
the command line with mpiexec, then add the --enable-parallel-tests
option to configure.)

For complete instructions on building the netCDF-4 snapshot, you can
look at the snapshot Installation and Porting Guide, available on the
snapshot documentation page:
http://www.unidata.ucar.edu/software/netcdf/netcdf-4/newdocs/

Special thanks to Robert for helping get netCDF-4.0 building and
working properly with parallel I/O options. ;-)

Any users of netCDF-4 parallel I/O should email me with any problems
they encounter. (Or, even better, if they don't encounter any!) I am
currently working on benchmarking and further testing of the parallel
I/O features of netCDF-4.0, and would be very interested in any user
experiences, good or bad.

Thanks!

Ed
-- 
Ed Hartnett  -- ed@xxxxxxxxxxxxxxxx