Re: HDF5 chunking questions...

Hi Ed,

> Quincey et. al.,
> 
> Given an n-dimensional dataspace, with only one unlimited
> (i.e. extendable) dimension, tell me how to select the chunk size for
> each dimension to get a good read performance for large data files.
> 
> Would you care to suggest any smart algorithms to yeild better
> performance for various situations?
    Unfortunately there aren't generic instructions for this sort of thing,
it's very application-I/O-pattern dependent.  A general heuristic is to pick
lower and upper bounds on the size of a chunk (in bytes) and try to make the
chunks "squarish" (in n-D).  One thing to keep in mind is that the default
chunk cache in HDF5 is 1MB, so it's probably worthwhile to keep chunks under
half of that.  A reasonable lower limit is a small multiple of the block size
of a disk (usually 4KB).
    Generally, you are trying to avoid the situation below:

        Dataset with 10 chunks (dimension sizes don't really matter):
        +----+----+----+----+----+
        |    |    |    |    |    |
        |    |    |    |    |    |
        | A  | B  | C  | D  | E  |
        +----+----+----+----+----+
        |    |    |    |    |    |
        |    |    |    |    |    |
        | F  | G  | H  | I  | J  |
        +----+----+----+----+----+

        If you are writing hyperslabs to part of each chunk like this:
        (hyperslab 1 is in chunk A, hyperslab 2 is in chunk B, etc.)
        +----+----+----+----+----+
        |1111|2222|3333|4444|5555|
        |6666|7777|8888|9999|0000|
        | A  | B  | C  | D  | E  |
        +----+----+----+----+----+
        |    |    |    |    |    |
        |    |    |    |    |    |
        | F  | G  | H  | I  | J  |
        +----+----+----+----+----+

        If the chunk cache is only large enough to hold 4 chunks, then chunk
    A will be preempted from the cache for chunk E (when hyperslab 5 is
    written), but will immediately be re-loaded to write hyperslab 6 out.

    Unfortunately, our general purpose software can't predict the I/O pattern
that users will access the data in, so it is a tough problem.  One the one hand,
you want to keep the chunks small enough that they will stick around in the
cache until they are finished being written/read, but you want the chunks to
be larger so that the I/O on them is more efficient. :-/

    Quincey