[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: NetCDF, shadow files, parallel I/O on T3D

Hi Rick,

> I'm about to begin working with some folks who are
> using a special version of NetCDF in which a parallel I/O
> library has been integrated into it for use on the T3D as
> a nodal I/O library.  Tim Williams is the one who
> filled me in on all of these details, says in response
> to my question as to which version of NetCDF he's using:
> 
> "... 2.4b4, I believe. The modifications to the source to make it work
> with par_io are going to be adopted by the Unidata people in future versions,
> if they haven't already.  Steve Luzmoor of CRI (sits, or at least used to sit,
> at NERSC) developed par_io and made the modifications to the netcdf source
> (very minor) to make it work."
> 
> 
> So my questions to you:
> 
>       1. do you have a "very latest" version of NetCDF that
>          has these mods in it for a Cray T3D?

The current version, 2.4.2, available as

    ftp://ftp.unidata.ucar.edu/pub/netcdf/netcdf.tar.Z

contains the mods for the T3D that we got from Steve Luzmoor.  These
reference functions from a prototype Cray par_io library that we can't
distribute with netCDF, so you may have to obtain the current version of
that library from Cray.  While we were integrating these changes, it seemed
that the par_io interfaces were changing as well and more changes were
planned, so it's possible that the current netCDF release may need some
tweaking to work with the current par_io release.  If you have problems
building the current release for T3D with the par_io library that CRI can't
solve, please let us know.

>       2. did you ever include in your master release the
>          concept we discussed of "shadow files" in which the
>          metadata is copied to a tiny local file so when
>          the completed terabyte file is written to archival
>          storage the metadata is still easily accessable?

No, although it seemed like a good idea and it's come up again recently, it
didn't make it to the top of our rather long list of what should be added
to the library, especially since we couldn't convince ourselves that it
couldn't be done above the level of the library by someone interested in
that functionality.  Currently, our top priority is adding type-conversion
interfaces that eliminate the need for void* pointers, and incorporating
packing into the external types, so users can store 11-bit data
efficiently, for example.

>          And, if so, do you have the "very latest" version
>          mentioned in 1 above with this shadowing capability built in?
> 
> If the answer to these is Yes, which I do hope is the case,
> please fill me in on how I can obtain this library.  Thanks
> very much.  I hope all is well with you.

Thanks.  Details about the current version and what's changed are in the
"what's New with netCDF" link at

    http://www.unidata.ucar.edu/packages/netcdf/whatsnew.html

______________________________________________________________________________

Russ Rew                                           UCAR Unidata Program
address@hidden                              http://www.unidata.ucar.edu