Re: 20050329: netCDF Java - Out Of Memory Error

• To: Peyush Jain <peyush.jain@xxxxxxxx>
• Subject: Re: 20050329: netCDF Java - Out Of Memory Error
• From: John Caron <caron@xxxxxxxxxxxxxxxx>
• Date: Wed, 30 Mar 2005 09:53:41 -0700

Peyush Jain wrote:

Hi John,

I am still having memory problems (java.lang.OutOfMemoryError). I have 480
(to be precise) 1-D arrays with data and an additional 1-D array with time
data. I get this information every few seconds. By moving the offset (for
all 481 arrays) and not writing to the file every time data is received, I
run out of memory. So, I am wondering if there is a way to write to the disk
after each time data is received.

you want to write the data out to disk which is what ncfile.write() does.
do this each time step instead of accumulating data.

in your case, you probably can  reuse your data arrays.

some comments below

I am not sure how to do this: "not store more than one time step at a time".

So, here is what I do... I create 481 variables (with unlimited length) and
every time I get new data I move the offset by 200 (length of each incoming
array). The following is a very simple version of what I am trying to do:

----------------------------------------------------------------------------
Dimension[] dims;
Dimension timeDim;

private static boolean createFile = true;
private static int offset = 0;

public void processData(Data time, Data[] d)
{
   if(createFile)
   {
       timeDim = ncfile.addDimension("time", -1);
       ncfile.addVariable("time", double.class, new Dimension[]{timeDim});

dims = new Dimension[d.length];

       for(int i = 0; i < d.length; i++)
       {
           dims[i] = ncfile.addDimension(d[i].getName(), -1);

           ncfile.addVariable(d[i].getName(), int.class, new
Dimension[]{dims[i]});

       }

       try
       {
           ncfile.create();
       }
       catch(IOException e)
       {
           System.err.println("ERROR creating file");
       }

       createFile = false;
   }

   int origin[] = new int[1];
   origin[0] = offset;

   try
   {
       ncfile.write("time", origin,
ArrayAbstract.factory(time.getArray()));

for(int i = 0; i < d.length; i++)

       {
           ncfile.write(d[i].getName(), origin,
ArrayAbstract.factory(d[i].getArray()));
       }

       offset = offset + 200;
       ncfile.flush();
   }
   catch(IOException e)
   {
       System.err.println("ERROR writing file");
   }
}

----------------------------------------------------------------------------

Thanks a lot,

Peyush