NOTE: The netcdf-hdf mailing list is no longer active. The list archives are made available for historical reasons.
------- Forwarded Message
cc: Woo-Sun Yang <wyang@xxxxxxx>, Russ Rew <russ@xxxxxxxxxxxxxxxx>,
Rob Ross <rross@xxxxxxxxxxx>, "R.K. Owen" <rkowen@xxxxxxxxx>,
Mike Folk <mfolk@xxxxxxxxxxxxx>, Linnea Cook <cook13@xxxxxxxx>,
Jianwei Li <jianwei@xxxxxxxxxxxxxxxxxxxx>, Harsh Anand
<harsh@xxxxxxxxx>,
Chris Ding <cding@xxxxxxxxx>, Doug Rotman <rotman1@xxxxxxxx>,
Richard Hedges <richard-hedges@xxxxxxxx>, Karl Taylor
<taylor13@xxxxxxxx>
John,
I certainly would encourage you to try out the
NERSC version of parallel netCDF for the following reasons:
1. The NERSC version has an fortran interface which is
quite close to the sequential netCDF interface. Woo-sun has
done some systematic tests. Th rates are about 150MB/s
from 8 processors.
2. Even after the U. Northwest/ANL codes or the HDF netCDF
has complete parallel interface, that would not deviate too much from
the NERSC version / standard netCDF interface. So the
necessary recoding efforts are minimal, I believe.
There could be more comprehensive interface routines in the new
parallel netCDF, e.g., something similar to gather/scatter in MPI.
But these can be specified in the standard netCDF interface as well...
3. The main point for using parallel netCDF is to ensure
that data segments written from different processors are non-overlapping.
4. I/O rates are only one important factor in parallel I/O.
The convenience of parallel I/O are also important. Carefully
study how the flexibility and convenience of parallel netCDF would
offer on your codes (instead of funnel everying through master processor)
is the most important step you have to go through. From this study
you may actually suggest what the interface should be or
most convenient for your codes.
I'm not trying to sell the NERSC version of parallel netCDF.
I'm just point out you can use it for initial study and the
results/codes will be useful for more efficient/powerful versions of
parallel netCDF.
Chris
John Tannahill wrote:
> Greetings,
>
> I am sending this e-mail to the group of people who I think might
> be able to give me some insight into the current status of
> parallel netcdf (or parallel hdf/netcdf). I currently have a couple
> of months to investigate implementing some sort of parallel netcdf
> capability in our code. Note that this is not meant to be a research
> endeavor, but to actually introduce some new technology /
> improved capabilities.
>
> A couple of notes on the code itself. It is a parallel global
> Atmospheric Chemical Transport Model (ACTM) that uses MPI to
> communicate between the processors. It is written in F77/F90 and
> uses netCDF for much of its I/O. It is a mature code, highly portable,
> and has been run on a variety of MP platforms over the years. Of
> course like most models, it is continues to be "enhanced" regularly.
> We do parallel reads of large amounts of netCDF data, but currently all
> significant netCDF output is funneled through a single Master processor.
> This output can be many gigabytes as well.
>
> After looking into this for a short time and trying to figure out if
> there was a particular direction I should go in, the following choices /
> issues seemed to emerge:
>
> * Wait a couple of years for the newly funded proposal to merge netCDF
> with HDF to be completed? I wish I could, but we really would like
> something now, and I also have much of my time funded to do this
> between now and 9/30/03.
>
> * Work with NERSC/LBL and try to use the parallel netCDF capability
> that they have developed. Not sure if this just runs on a single
> platform, if it has a Fortran interface, what state it is in, etc.?
>
> * Ditto for Northwestern Univ./ANL. It was actually a tutorial I took
> at SuperComputing from Rob Ross, et.al. that got me interested in this.
>
> * Somehow use HDF to provide the parallel netCDF capability. Convert
> the code to use HDF (probably a significant amount of work) and convert
> the netCDF input files to HDF? Or maybe HDF can read netCDF files? Would
> probably want to convert the HDF output files as well, but not sure. If
> this sort of format conversion is possible, how much time would it take?
> We have lots and lots of netCDF input files.
>
> * Other possibilities?
>
> Any guidance any of you can give me would be much appreciated.
>
> Regards,
> John
>
> --
> ===========================
> John R. Tannahill
> Lawrence Livermore Nat. Lab.
> P.O. Box 808, M/S L-103
> Livermore, CA 94551
> 925-423-3514
> Fax: 925-423-4908
> ===========================
--
Chris Ding, Staff Computer Scientist Phone: 510-486-6901
NERSC Division, Building 50F, Rm 1608 FAX: 510-486-5812
Lawrence Berkeley National Lab Email: CHQDing@xxxxxxx
1 Cyclotron Rd, Berkeley, CA 94720 http://www.nersc.gov/~cding
------- End of Forwarded Message
netcdf-hdf archives: