[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: 19991015: Fortran 90 modules and 2 Gig limit

From: Nicholas Yue <address@hidden>
To: address@hidden
Subject: Fortran 90 modules and 2 Gig limit

Hi Nicholas,

>       Now that there is 3.5 beta, is there some location where the following
> topics are discussed ?
> 
>       (1) Fortran 90 module for NetCDF

We're close to finishing an experimental version of the Fortran 90
module for testing.  It was mostly developed by Robert Pincus of SSEC.
It provides a much simpler interface than Fortran 77, from 76
functions to 31.  I would like to get it into the final release of
version 3.5, if possible.  What's holding it up now is documentation
and testing.  If you would like to be an early tester of this, I'd be
happy to make it available to you.

>       (2) Overcoming the 2Gig limit; and

There's nothing new in netCDF 3.5 for overcoming this limit, except
what was already in 3.4, because 3.5 still uses the same file format.
Here's what was in the 3.4 announcement about this:

   On systems that support very large files (exceeding 2 Gbytes), such as
   IRIX 6.x or SunOS 5.6 with the large file compilation environment, it
   is possible to create and access very large netCDF files with version
   3.4.  The remaining size constraints are that the file offset to the
   beginning of the record variables (if any) must be less than 2 Gbytes,
   and the relative offset to the start of each fixed length variable or
   each record variable within a record must be less than 2 Gbytes.
   Hence, a very large netCDF file might have

     * no record variables, some ordinary fixed-length variables, and a
       very large (exceeding 2 Gbytes) fixed-length variable; or
     * some ordinary fixed-length and record variables, and a very large
       record variable; or
     * some ordinary fixed-length and record variables and a huge number
       of records.

   If you create very large netCDF files, remember that they will only be
   usable on other systems that support very large files.  The netCDF file
   format has not changed, so files less than 2 Gbytes in size are still
   writable and readable on all systems on which netCDF is supported.

To eliminate the above weaker file size constraints would require a
new netCDF format.  So far the original format (version 1, since 1987)
has been sufficient for all versions of the software through the
latest netCDF 3.5 release.  Implementing software that would support a
new format (based on HDF-5) but that would also continue to permit
access to files in the previous format has been in our long-term plans
for netCDF.

>       (3) HDF-5 and NetCDF format merger ?

There's been some progress on this recently, but NCSA has the only
about it: a prototype of the netCDF 3 C interface was successfully
implemented on top of the HDF-5 API by a programmer at NCSA last
summer.  This prototype has limitations, since it was written more as
a proof-of-concept than an implementation that could be distributed
and supported.  Nevertheless, it proved the practicality of a netCDF
interface on the HDF-5 format, paving the way for a netCDF-4 that
would have no 2Gbyte limits, could use MPI/IO, could support chunking
and compression, etc.  The main problems now are getting resources to
implement this ...

--Russ

_____________________________________________________________________

Russ Rew                                         UCAR Unidata Program
address@hidden                     http://www.unidata.ucar.edu