Hi Jon:
Jon Blower wrote:
Thanks John and Joe. Yes, I do know that disk I/O is the limiting
factor, but optimising it isn’t easy due to all the buffers and disk
caches (as you and Joe have pointed out). Interestingly, I can “see”
these caches. When I read random chunks of data from a file, sometimes
a read takes ~1ms, sometimes ~5ms and sometimes ~10ms, with not much in
between these values (a trimodal distribution). I think these must be
three levels of caching. Also, if I run the same test multiple times on
the same file, the number of 10ms reads drops off, and the number of 1ms
reads increases. (I’m on a Windows XP laptop with a 5400 rpm hard drive.)
I guess the only way to bypass the caches would be to cycle between a
large set of data files, which are in total bigger than the disk caches.
(I’m trying to simulate a busy server environment.)
If your server is running linux or solaris instead of windows XP, your will have different I/O results.
By the way, I’ve been digging in the IOSPs and the ucar RandomAccessFile
class. The ucar RAF seems to be the same as java.io.RAF except that it
implements an 8k buffer which is supposed to increase performance. But
the code of N3raf (which extends N3iosp and I assume is the default
class used for data reading) uses raf.readToByteChannel(), which
bypasses the 8k buffer. So could a java.io.RAF have been used in this case?
ucar.RAF forked java.RAF to add buffering back in Java 1.0 days. It has
accumulated various other conveniences since then, i think byte ordering is one
(?). also, java.RAF is a final class so cant be sublassed by
HTTPRandomAccessFile. For these and various reasons, one could not revert to
use java.RAF except by forking the CDM code.
raf.readToByteChannel() is an experiment to try to allow "streamed reading", ie
direct file to network transfer. It can be used in such a restricted manner that its not
very useful generally. Its not used by the Grid/WMS code.
To expand a little on my use case: in general, to create a
low-resolution map of data for a WMS, one has to read only a small
fraction of the available data in the file. So I’m looking for an
efficient way to read sparse clouds of data (not evenly-spaced).
Reading point-by-point is not efficient, but nor is reading lots of
data, converting it to new types, then throwing most of it away.
What about writing low-resolution versions of the data and using that when
possible?
Cheers, Jon
