Re: parallel netcdf?

Hi John,

You wrote:

> Greetings,
> 
> I am sending this e-mail to the group of people who I think might
> be able to give me some insight into the current status of
> parallel netcdf (or parallel hdf/netcdf).  I currently have a couple
> of months to investigate implementing some sort of parallel netcdf
> capability in our code.  Note that this is not meant to be a research
> endeavor, but to actually introduce some new technology /
> improved capabilities.
> 
> A couple of notes on the code itself.  It is a parallel global
> Atmospheric Chemical Transport Model (ACTM) that uses MPI to
> communicate between the processors.  It is written in F77/F90 and
> uses netCDF for much of its I/O.  It is a mature code, highly portable,
> and has been run on a variety of MP platforms over the years.  Of
> course like most models, it is continues to be "enhanced" regularly.
> We do parallel reads of large amounts of netCDF data, but currently all 
> significant netCDF output is funneled through a single Master processor. 
> This output can be many gigabytes as well.
> 
> After looking into this for a short time and trying to figure out if
> there was a particular direction I should go in, the following choices /
> issues seemed to emerge:
> 
> * Wait a couple of years for the newly funded proposal to merge netCDF
> with HDF to be completed?  I wish I could, but we really would like
> something now, and I also have much of my time funded to do this
> between now and 9/30/03.

As the PI of this effort, I agree that you should try one of the other
parallel netCDF implementations, with your time constraints.  From
what I've seen, the Northwestern Univ./ANL effort
<http://www-unix.mcs.anl.gov/parallel-netcdf/> has achieved some
impressive performance results and is closest to ready to try at this
point, although I haven't kept up with recent progress of the
NERSC/LBL effort.

> * Work with NERSC/LBL and try to use the parallel netCDF capability
> that they have developed.  Not sure if this just runs on a single
> platform, if it has a Fortran interface, what state it is in, etc.?
> 
> * Ditto for Northwestern Univ./ANL.  It was actually a tutorial I took
> at SuperComputing from Rob Ross, et.al. that got me interested in this.
> 
> * Somehow use HDF to provide the parallel netCDF capability.  Convert
> the code to use HDF (probably a significant amount of work) and convert
> the netCDF input files to HDF?  Or maybe HDF can read netCDF files?  Would
> probably want to convert the HDF output files as well, but not sure.  If
> this sort of format conversion is possible, how much time would it take?
> We have lots and lots of netCDF input files.
> 
> * Other possibilities?
> 
> Any guidance any of you can give me would be much appreciated.
> 
> Regards,
> John
> 
> -- 
> ===========================
> John R. Tannahill
> Lawrence Livermore Nat. Lab.
> P.O. Box 808, M/S L-103
> Livermore, CA  94551
> 925-423-3514
> Fax:  925-423-4908
> ===========================

--Russ

_____________________________________________________________________

Russ Rew                                         UCAR Unidata Program
russ@xxxxxxxxxxxxxxxx                      http://my.unidata.ucar.edu