[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: 20001101: Transforming coordinates in Java netCDF


Don Murray wrote:
> 
> I'm working on a test program to read in some of ATD's radar
> sweepfiles in netCDF format.  One of the things I need to
> do is transpose the two coordinate indices for a Variable
> (not sure if that is the right terminology).  In trying to
> do this, I've found two problems with the Variable.toArray()
> method in ucar.netcdf:
> 
> 1) The transpose takes a long time
> 2) There toArray() method does not support converting the
>    component type.
> 
> Here's a snippet of code where I'm trying to do the transform:
> 
>         float[][] rangeValues = new float[vars2D.size()][numSamples*numGates];
>         for (int i = 0; i < vars2D.size(); i++)
>         {
>             Variable v2D = ncFile.get((String) vars2D.get(i));
>             // transpose from time, maxCells to maxCells, time
>             MultiArray maTransposed =
>                 new MultiArrayProxy(v2D, new TransposeMap(0,1));
>             System.out.println("reading data for " + v2D.getName());
>             shorts = (short[]) maTransposed.toArray();
>             System.out.println("read data for " + v2D.getName());
>             for (int j = 0; j < shorts.length; j++)
>                 rangeValues[i][j] = (float) shorts[j];
>             System.out.println("converted data for " + v2D.getName());
>         }
> 
> In this code, the time between "reading data" and "read data" is
> long (at least 2-3 seconds per variable for arrays sizes of 41 and
> 1008).  This compares to a straight v2D.toArray() that takes msecs.

yes, TransposeMap() is unacceptably slow, one of the motivations for
ma2.

i'm not positive how MultiArrayProxy works; its possible you are reading
from the file one element at a time.
try:
                MultiArray ma = v2D.copyout(...);

then do the transpose on ma. If this speeds things up by, say a factor
of 100, then the MultiArrayProxy cost may be acceptable.


> 
> Am I doing this the right way?  Could I use the ucar.ma2/ucar.nc2
> package to do the same thing? (I didn't see a toArray() method
> anywhere)  Could the algorithm in the toArray() method be improved?

you can use ma2:

  public Array transpose(int dim1, int dim2);
 // Create a new Array using same backing store as this Array, by
transposing two of the indices.

this should have no performance hit.

the toArray() equivalent is "hidden" in ArrayAbstract:
  public java.lang.Object copyTo1DJavaArray();



> 
> As for the type conversion, the Javadocs for
> Variable.toArray(dst, origin, shape) state that this had to wait
> for JDK 1.2.  However, Steve does essentially this same conversion in
> visad.data.netcdf.in.Util, so perhaps that code could be migrated
> into the ucar.netcdf.Variable package.  I can't use that because
> it doesn't support the transposing of indices.

I'm not sure about why you want to do the type conversion. Both
multiarray and ma2 take the philosophy of more or less hiding the
"backing store" array, and providing type conversion during the data
access, eg getDouble(), getFloat() ... I assume its for storage
efficiency?

An obvious thing you can do is to do the type conversion yourself. You
could then wrap the array in ma2.Array to get the transpose, but you
could also do the transpose during type conversion since you have to
touch each data element anyway, so it would be most efficient if thats
what you need.  

If you really need java arrays for VisAD, then efficiency would require
a one-pass convert and transpose operation. Perhaps we should consider
adding an operation to Array such as:

          public java.lang.Object copyTo1DJavaArray(Class dataType);

This could then be done is such a way as to combine the data reordering
and type conversion. So then you would get the ma2.Array, transpose it
and call the above routine.

Anyone have any thoughts on the general value of adding this method?