Building netCDF for parallel I/O requires some effort, but it's worth it!
I used the mpich2 library. The Fedora repository distribution of mpich2 is broken with respect to shared libraries (apparently a known bug). So I installed from source, getting the mpic2 tarball here: http://www.mcs.anl.gov/research/projects/mpich2/downloads/tarballs/1.3.2p1/mpich2-1.3.2p1.tar.gz.
Mpich2 does not, apparently, use libtool, because it would not build shared libraries for me, until I set CFLAGS=-fPIC myself.
It was also required to set -fPIC for the zlib build to get shared zlib libraries. Then I built HDF5 with --enable-shared --enable-parallel and got a shared build of the HDF5 libraries.
With all that done, the netCDF snapshot release builds and tests cleanly. No special configure options are required - it detects that HDF5 was build with --enable-parallel and builds in parallel I/O. I do have to set CC=mpicc to call the MPI compiler wrapping script.
All this was needed because a recent switch to shared builds by default was breaking parallel builds on my main test machine, which was built with static-only parallel HDF5 and mpich2 libraries.
could I use this
Posted by Flint Minki Jeong on June 28, 2011 at 07:14 AM MDT #
Howdy Flint Minki Jeong!
Yes, of course you can use this! NetCDF is free software, and is available on the Unidata web site.
There are two ways to use parallel I/O, on netCDF classic and 64-bit offset files using the parallel-netcdf library, and on netCDF-4/HDF5 files using HDF5.
Both parallel-netcdf and HDF5 depend on the MPI libraries, which must be installed first. (MPICH2 is a free software implementation).
Starting from scratch:
1 - install MPICH2 (from this point onward, build everything with mpicc)
2 - install zlib
3 - configure hdf5 with --enable-parallel, build and install.
4 - build parallel-netcdf (optional)
5 - configure netcdf with --enable-parallel-tests and build netcdf (also use configure option --enable-pnetcdf if you're using the parallel-netcdf library)
I run this on a 16-core linux box and it works great. Programs that use more than 16 processors are just emulated on however many processors you have. Very handy for testing before moving on to the expensive supercomputer!
Good luck!
Ed
Posted by Ed Hartnett on June 29, 2011 at 10:17 AM MDT #