Re: FileFlatField

• To: "Peter X. Cao" <xcao@xxxxxxxxxxxxx>
• Subject: Re: FileFlatField
• From: Bill Hibbard <hibbard@xxxxxxxxxxxxxxxxx>
• Date: Tue, 21 Dec 1999 10:41:43 -0600

Hi Peter,

> Suppose I want to construct a FlatField for a 5D double dataset.
> Here is some example code.
> 
> ====================================================================
> 
> RealType dimension_types[] = {new RealType("dim1"), new RealType("dim1"),
>     new RealType("dim2"), new RealType("dim3"), new RealType("dim4")};
> RealTupleType domain = new RealTupleType(dimension_types);
> RealTupleType range = new RealTupleType(new RealType("Data range"));
> FunctionType field_type = new FunctionType(domain, range);
> 
> int[] d = {1000, 500, 100, 50, 20};
> IntegerNDSet domain_set = new IntegerNDSet(domain, d);
> DoubleSet[] range_set = {new DoubleSet(range)};
> FlatField ff = new FlatField(field_type, domain_set, null, null, range_set,
> null);
> ======================================================================
> 
> Obviously, it will not work because the dataset has 400GB
> (1000*500*100*50*20*8).
> So I want to use FileFlatField instead of FlatField. There is only one
> constructor
> for FileFlatField, FileFlatField(FileAccessor accessor, CacheStrategy
> strategy).
> I don't know how to set the FileAccessor and CacheStrategy. I couldn't find
> any
> such example in the Developers Guide. Could tell me how to construct such a
> FileFlatField ?

First, the current disk caching mechanism assumes that each
FlatField fits in memory - there is no current class that
supports caching parts of FlatFields into memory as needed.

So you'll have to use a different MathType for such a large
data object, something like:

  (time -> (var -> ((x, y, z) -> value)))

and hope that the FlatFields, with MathType ((x, y, z) -> value),
fit in memory.  If they won't fit, the only current alternative
would be to further factor the MathType to something like:

  (time -> (var -> (z -> (x, y) -> value))))

One of these dasy we hope to come up with a new class that
supports caching portions of FlatFields.

The only current example of a file format adapter that uses
disk caching is HDF-EOS.  The visad.data.hdfeos.HdfeosFlatField
class invokes the visad.data.FileFlatField constructor and
passes an HdfeosAccessor and an CacheStrategy to them.  The
CacheStrategy is just the default class (least recently used
strategy), but HdfeosAccessor knows where to find data in the
file.

Tom Rink (rink@xxxxxxxxxxxxx) wrote the HDF-EOS adapter, so you
might want to address detailed questions about this to him.

Cheers,
Bill
----------------------------------------------------------
Bill Hibbard, SSEC, 1225 W. Dayton St., Madison, WI  53706
hibbard@xxxxxxxxxxxxxxxxx  608-263-4427  fax: 608-263-6738
http://www.ssec.wisc.edu/~billh/vis.html

• References:
  • FileFlatField
    • From: Peter X. Cao