4.7 UNICOS Optimization
As was mentioned in the previous section, it is possible to replace
the I/O layer in order to increase I/O efficiency. This has been done
for UNICOS, the operating system of Cray computers similar to the Cray
Y-MP.
Additionally, it is possible for the user to obtain even greater I/O
efficiency through appropriate setting of the NETCDF_FFIOSPEC
environment variable. This variable specifies the Flexible File I/O
buffers for netCDF I/O when executing under the UNICOS operating
system (the variable is ignored on other operating systems). An
appropriate specification can greatly increase the efficiency of
netCDF I/O–to the extent that it can surpass default FORTRAN binary
I/O. Possible specifications include the following:
bufa:336:2- 2, asynchronous, I/O buffers of 336 blocks each (i.e., double
buffering). This is the default specification and favors sequential
I/O.
cache:256:8- 8, synchronous, 256-block buffers. This favors larger random accesses.
cachea:256:8:2- 8, asynchronous, 256-block buffers with a 2 block
read-ahead/write-behind factor. This also favors larger random
accesses.
cachea:8:256:0- 256, asynchronous, 8-block buffers without
read-ahead/write-behind. This favors many smaller pages without
read-ahead for more random accesses as typified by slicing netCDF
arrays.
cache:8:256,cachea.sds:1024:4:1- This is a two layer cache. The first (synchronous) layer is composed
of 256 8-block buffers in memory, the second (asynchronous) layer is
composed of 4 1024-block buffers on the SSD. This scheme works well
when accesses proceed through the dataset in random waves roughly
2x1024-blocks wide.
All of the options/configurations supported in CRI's FFIO library are
available through this mechanism. We recommend that you look at CRI's
I/O optimization guide for information on using FFIO to it's
fullest. This mechanism is also compatible with CRI's EIE I/O library.
Tuning the NETCDF_FFIOSPEC variable to a program's I/O pattern can
dramatically improve performance. Speedups of two orders of magnitude
have been seen.
