7.3 Using Less Time for Data Access
If netCDF access time is a bottleneck, these
techniques may help.
- Change dimension order for variables so that you
read data in the same order it was written, with most frequently varying dimension
last. In particular, avoid writing or reading one value per record
(per index of the unlimited dimension), as this will usually access
a new disk block for every data value.
- Each seek to a new location in a file causes buffer
flushes, so reading randomly negates most of the advantages of buffering.
- If data must be written in a different order than it is read,
consider transposing the data to the most efficient read order once
if it will be read many times.
- If you will always write all data values for each variable,
consider turning off "fill mode" that pre-fills data on the disk with
fill values. This can save 50% of write time, but it eliminates the
possibility of later
detecting unwritten values.
- For writing a netCDF file from multiple processors when a
parallel file system is available, use parallel
netCDF or netCDF-4 parallel I/O.
- If too much time is spent byte swapping data between disk and
memory on little-endian architectures, consider using netCDF-4,
HDF5, or a big-endian architecture for I/O.