[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: 971112: netCDF 3.3 Buffersizes

> >From: Olaf Heudecker <address@hidden>
> >Subject: Buffersizes
> >Organization: Alfred-Wegener-Institute for Polar and Marine Research
> >Keywords: 199711120900.CAA08843 netCDF 3.3 Cray
>
> Dear netCDF team
>
> I have successfully installed netcdf-3.3.1 on the local Cray T3E and
> J90. Now there are several users asking about the memory usage of
> netCDF. In a series of tests to investigate the I/O performance
> of the T3E we found that netCDF 3.3.1 seems to use a 4MB internal
> asynchronous library buffer (besides the bufa:336:2 assign default).
>
> Could you please tell me whether there is a way to tune this library
> buffer size (for specific useres) and how to do it? Do you know the
> default sizes on the different architectures?
>
> Thanks a lot
> Olaf R. Heudecker
> Alfred-Wegener-Institute for Polar and Marine Research

For netcdf I/O on the CRAY, we can identify a couple of levels of buffering.

There is a "contract" between the higher layers of
netcdf and the "ncio" layer which says "we won't request more than
'chunksize' bytes per request. This controls the maximum size of
the buffer used internal to the ncio implementation for system
read and write.

On the Cray, the ncio implementation is usually 'ffio.c', which uses the
cray specific ffio library. This is in contrast to the implementation
used on most other systems, 'posixio.c', which used POSIX read(), write(),
lseek() calls. The ffio library has all sorts of controls (which I don't
pretend
to understand), including the bufa directive you cite above.
If you are using some sort of assign statement external to netcdf, it
is probably being overridden by the following code sequence from
ffio.c:
        ControlString = getenv("NETCDF_FFIOSPEC");
        if(ControlString == NULL)
        {
                 ControlString="bufa:336:2";
        }
        fd = ffopens(path, oflags, 0, 0, &stat, ControlString);

EG, if the NETCDF_FFIOSPEC environment variable is not set, use the
default shown. Otherwise, use the environment variable.

Currently for ffio, the "chunksize" contract  is set to the st_oblksize
member of struct ffc_stat_s obtained from a fffcntl(). You can check
this by putting a breakpoint in ffio.c:blksize().

As you may be able to tell from ffio.c, we have stubbed out interlayer
communication to allow this to be set as the *sizehintp argument to
ncio_open() and ncio_create(). In netcdf-3.4, we intend to fully implement this
and expose this parameter to the user. This is almost complete. If you are
interested in early access to this, let me know.

-glenn