Hello all,
In NJ 4.2.20100409.0054 the method ucar.nc2.FileWriter.copySome() is
supposed to copy data for a large variable in a series of small
chunks. As written, however, it actually attempts to copy everything
at once. (Here's hoping that Thunderbird preserves the whitespace in
my preformatted code samples, or else this post will be very tough to
follow.)
private static void copySome(NetcdfFileWriteable ncfile, Variable
oldVar, int nelems) throws IOException {
String newName = N3iosp.makeValidNetcdfObjectName( oldVar.getName());
int[] shape = oldVar.getShape();
int[] origin = new int[oldVar.getRank()];
int size = shape[0];
for (int i = 0; i< size; i += nelems) {
origin[0] = i;
int left = size - i;
shape[0] = Math.min(nelems, left);
Array data;
try {
data = oldVar.read(origin, shape);
...
I'm not exactly sure what the intended logic was, but it's clear that
in the first iteration of the loop, origin will be all zeroes (e.g.
{0, 0, 0} if the rank of oldVar is 3) and that shape will be identical
to oldVar's shape. Therefore, the code will attempt to read all of
oldVar's data at once and an OutOfMemoryError will result if oldVar is
large.
On 03/29/2010, Robert Bridle proposed some new code for
FileWriter.copyVarData() (which calls copySome()) that would split the
write job into chunks:
///////////// ORIGINAL CODE //////////////////////
int nelems = (int) (size / maxSize);
if (nelems<= 1)
copyAll(ncfile, oldVar);
else
copySome(ncfile, oldVar, nelems);
////////////////////////////////////////////////////
////////////// PROPOSED CODE //////////////////////
/* if(size> maxSize)
{
int[] shape = oldVar.getShape();
// determine the size of all the dimensions, other than the
first.
long sizeOfOtherDimensions = 1;
for (int i = 1; i< shape.length; i++) {
if (shape[i]>= 0)
sizeOfOtherDimensions *= shape[i];
}
// determine number of bytes in all the dimensions, other than
the first.
long bytesInOtherDimensions = sizeOfOtherDimensions *
oldVar.getElementSize();
// first dimension chunk-size that will fit within maxSize of
memory.
int firstDimensionChunkSize = (int)
(maxSize/bytesInOtherDimensions);
//System.out.println("We can fit: " + firstDimensionChunkSize
+ " chunks in: " + maxSize + " bytes of memory.");
copySome(ncfile, oldVar, firstDimensionChunkSize);
}
else
{
copyAll(ncfile, oldVar);
} */
////////////////////////////////////////////////////
This will write the data in N chunks where N is the size of the
outer-most dimension. But what about when the stride of the outer
dimension is very large? For example, there's a variable from a
massive aggregated dataset I'm working with that has the CDL:
float pr(ensemble=8, time=1560, lat=128, lon=256);
which means an outer-most dimension stride of 1560*128*256 =
51,118,080. Using 32-bit floats, that would require 195 MB to
store--quite a bit larger than the maxSize of 1 MB.
So, I propose a different algorithm:
/**
* An index that computes chunk shapes. It is intended to be used
to compute the origins and shapes for a series
* of contiguous writes to a multidimensional array.
*/
public static class ChunkingIndex extends Index {
public ChunkingIndex(int[] shape) {
super(shape);
}
/**
* Computes the shape of the largest
possible<b>contiguous</b> chunk, starting at {@link
#getCurrentCounter()}
* and with {@code size<= maxChunkSize}.
*
* @param maxChunkSize the maximum size of the chunk shape.
The actual size of the shape returned is likely
* to be different, and can be found with
{@link Index#computeSize}.
* @return the shape of the largest possible contiguous chunk.
*/
public int[] computeChunkShape(int maxChunkSize) {
int[] chunkShape = new int[rank];
for (int iDim = 0; iDim< rank; ++iDim) {
chunkShape[iDim] = maxChunkSize / stride[iDim];
chunkShape[iDim] = (chunkShape[iDim] == 0) ? 1 :
chunkShape[iDim];
chunkShape[iDim] = Math.min(chunkShape[iDim],
shape[iDim] - current[iDim]);
}
return chunkShape;
}
}
private static void copySome(NetcdfFileWriteable ncfile, Variable
oldVar, int nelems) throws IOException {
String newName =
N3iosp.makeValidNetcdfObjectName(oldVar.getName());
ChunkingIndex index = new ChunkingIndex(oldVar.getShape());
while (index.currentElement()< index.getSize()) {
try {
int[] chunkOrigin = index.getCurrentCounter();
int[] chunkShape = index.computeChunkShape(nelems);
Array data = oldVar.read(chunkOrigin, chunkShape);
if (oldVar.getDataType() == DataType.STRING) {
data = convertToChar(ncfile.findVariable(newName),
data);
}
if (data.getSize()> 0) {// zero when record dimension
= 0
ncfile.write(newName, chunkOrigin, data);
if (debugWrite) {
System.out.println("write " + data.getSize() +
" bytes");
}
}
index.setCurrentCounter(index.currentElement() + (int)
Index.computeSize(chunkShape));
} catch (InvalidRangeException e) {
e.printStackTrace();
throw new IOException(e.getMessage());
}
}
}
This will result in chunks that are *always* smaller than nelems,
regardless of oldVar's size or shape. For example, if
oldVar.getShape() == { 5, 16, 8 } and nelems = 100, the origins and
shapes of the chunk read/writes will be:
origin shape size
r/w: [0, 0, 0] , [1, 12, 8], 96
r/w: [0, 12, 0], [1, 4, 8] , 32
r/w: [1, 0, 0] , [1, 12, 8], 96
r/w: [1, 12, 0], [1, 4, 8] , 32
r/w: [2, 0, 0] , [1, 12, 8], 96
r/w: [2, 12, 0], [1, 4, 8] , 32
r/w: [3, 0, 0] , [1, 12, 8], 96
r/w: [3, 12, 0], [1, 4, 8] , 32
r/w: [4, 0, 0] , [1, 12, 8], 96
r/w: [4, 12, 0], [1, 4, 8] , 32
As you can see, none of the chunks is actually 100 elements in size,
but given the constraints of the Netcdf API, I don't think it can be
helped. We'd need to be able to read and write 1D Arrays of values
from/to a specific offset in the 1D backing array.
If you're interested, I've attached a patch containing the changes.
Regards,
Christian Ward-Garrison
_______________________________________________
netcdf-java mailing list
netcdf-java@xxxxxxxxxxxxxxxx
For list information or to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/
