[
Date Prev][
Date Next][
Thread Prev][
Thread Next][
Date Index][
Thread Index]
Re: java netCDF issue
- Subject: Re: java netCDF issue
- Date: Fri, 13 Oct 2000 16:08:25 -0600
Emory,
> This is my issue. I am dealing with very large netCDF files. I would
> like to create multiple instances of these files in one JVM. Creating
> multiple instances is not a problem, but to access the data, I have to
> create array objects, which uses a very large amount of memory.
> this is the problem. When the dimensions and variables are translated
> to arrays so I can read the data, there is no room left in the JVM to
> open another netCDF file and do the same thing.
> What I want to do is somehow be able to access the data without holding
> an entire index array in memory. Can I do this with only a partial
> array that moves like a window?
>
> I included the part of my code below that creates the index arrays from
> which the data is extracted. Though I don't expect you to look at it.
> That would be asking a bit much.
>
> If it is all helpful, basically, what I do is create an array for
> dimensions, attributes derived
> from dimensions, and attributes respectively.
I haven't read your code carefully enough to understand exactly what
you are doing, but you can certainly access only a partial array,
using the "origin" and "shape" parameters of the copyout method,
rather than reading the whole variable into memory, as you are doing
now:
m_uDimMA[iDimIndex] =var.copyout(new int[var.getRank()],var.getLengths());
Above you are using an int array of (0, 0, ...0) as the "origin" and all
the dimension lengths as the "shape", so this copies the whole array.
To just use a slice of the array at a time, for example, you could use
different origins and corresponding shapes that incremented the first
index for origin and used 1 for the first element of the shapes array,
something like:
origin = new int[var.getRank()];
shape = var.getLengths();
...
// for each slice
origin[0]++;
shape[0] = 1;
m_uDimMA[iDimIndex] =var.copyout(origin, shape);
--Russ
--------------------------------------------------------------------
> /**
> * This class open a netcdf file and creates .ds and .dstp files
> necessary for
> * the dstp server to support netcdf format
> */
> class NetCDFFileObject
> {
>
> private int m_iNumDims;
> private int m_iNumAtts;
> private int m_iNumVars;
> private int m_iNumDVRs;
> private int[] m_iDim;
> private MultiArray[] m_uAttMA;
> private MultiArray[] m_uDimMA;
> private Variable[] m_uAttVR;
> private Variable[] m_uDVRVR;
> private Variable[] m_uDimVR;
> private Netcdf m_ncNetCDF;
>
>
>
> public NetCDFFileObject()
> {;}
>
>
> void initialize(String a_sDataDir, String a_sFileName)
> {
> try {
> //create the netCDF File Object
> m_ncNetCDF = new NetcdfFile(a_sDataDir+a_sFileName,
> true); // open it
> readonly
>
> //There are a number of issues here in building the row.
> //There may be variables of differring dimensions
>
> //First, the dimensions need to be collected and put
> into multiarray
> arrays
> //Then the variables need to be collected, but some
> variables may
> "pair" with
> //dimensions, so they can't be treated as normal
> dependent variables.
> //Once the dependency in the variables is sorted out,
> create double
> arrays
> //of multiarrays to hold 1) the dimension at the 0th
> level, 2) the
> dependent
> //variables above it
>
> DimensionSet uDimSet = m_ncNetCDF.getDimensions();
>
> VariableIterator vi = m_ncNetCDF.iterator();
>
> m_iNumDims = uDimSet.size();
> m_iNumVars = m_ncNetCDF.size();
> m_iNumDVRs = 0;
> m_iNumAtts = 0;//m_iNumVars-m_iNumDims;
>
> Variable var;
> //get the number of non-dimensional variables
> while(vi.hasNext())
> {
> var = vi.next();
> // //System.out.println(var.getName());
> if((var.getLengths()).length!=1)
> ++m_iNumAtts;
> }
>
> m_iNumDVRs = m_iNumVars - m_iNumDims - m_iNumAtts;
>
> // //System.out.println(m_iNumVars +" = "+m_iNumDims+" +
> "+m_iNumAtts);
>
>
> //get all of the non-dimensions as variables and store
> them into an
> array
> //then the dimensional variables
> //Since the logical order of the variables is not
> obvious, the only
> way
> //is to get an array
> m_uAttMA = new MultiArray[m_iNumAtts];
> m_uAttVR = new Variable[m_iNumAtts];
> m_uDVRVR = new Variable[m_iNumDVRs];
> m_uDimMA = new MultiArray[m_iNumDims];
> m_uDimVR = new Variable[m_iNumDims];
> //now, get all of the dimensions into multiArrays
> DimensionIterator uDI;
> String sNames[] = new String[m_iNumDims];
> int iDimIndex=0;
> int iVarIndex=0;
> int iDVRIndex=0;
>
> vi = m_ncNetCDF.iterator();
> boolean bDimAdded=false;
> for (int i=0; i < m_iNumVars; i++)
> {
> var = vi.next();
> uDI = uDimSet.iterator();
> for(int j=0; j < m_iNumDims; j++)
> {
> Dimension uD = uDI.next();
> if( var.getName().equals(uD.getName()))
> { m_uDimVR[iDimIndex] = var;
> sNames[iDimIndex] =
> var.getName();
> m_uDimMA[iDimIndex]
> =var.copyout(new
> int[var.getRank()],var.getLengths());
> ++iDimIndex;
> bDimAdded=true;
> }
> }
> if(!bDimAdded)
> {
> if((var.getLengths()).length!=1)
> {
> m_uAttVR[iVarIndex] = var;
> // m_uAttMA[iVarIndex] =
> var.copyout(new
> int[var.getRank()],var.getLengths());
> ++iVarIndex;
> }
> else
> {
> // uDVRMA[iDVRIndex] =
> var.copyout(new
> int[var.getRank()],var.getLengths());
> m_uDVRVR[iDVRIndex] = var;
> ++iDVRIndex;
> }
> }
> bDimAdded=false;
> }
