Re: Negative seek offset error message while writing output - follow up

  • To: Gabriella Turek <g.turek@xxxxxxxxxx>
  • Subject: Re: Negative seek offset error message while writing output - follow up
  • From: John Caron <caron@xxxxxxxxxxxxxxxx>
  • Date: Wed, 04 Apr 2007 20:38:31 -0600
If im calculating correctly, this file would be about 1.8 TBytes.

We do have a "large file format" that could theoretically handle this, but the 
nj22 does not yet support large file writing. That feature will be added soon, but I'm 
not sure exactly when. The C library does support this now.

But I wonder if your OS/ disk system can handle that large of a file, and if 
you've thought about breakint it up into smaller files? If you wrote one file 
for each time step, you would still have 8Gb size files.

Also with a dimension structure of (time,east,north,ens), you are optimizing to 
look at all the ensemble results at a single point, then looping over points, 
then over time. With files this large, you will have performance differences of 
1000 or more depending on your read acccess pattern.



Gabriella Turek wrote:
Hi John, I can provide the code (attached), it's a single class, but the
input files are huge. The final size of the array is: (time,
east,north,ens) = (241, 3668,3551,50), but I am trying to write it in
chucks of arrays of size (241, 1,3551,50) at starting index  (0,
ieast,0,0). Time is always defined as unlimited in all our files.
You will see that in the real code I am actually constructing 4 arrays
of this shape, but the code always fails at the very first attempt of
writing the first array when ens >= 14. If you think running the code
would be really crucial I can put the the input files on my ftp server.
Many thanx!
Gaby

John Caron wrote:

Hi Gabriella:

These kinds of errors are usually due to a misunderstanding of what
shape the array has to be.
Can you send a small test program that reproduces the problem?

Gabriella Turek wrote:

I have investigated this more and it turns out that I can run the code
for nens =< 13. I wonder if I am running into some kind of array size
limitation? I would not think so since in javadocs it says that the
arrays can be as big as 2*10^9. But perhaps there's a cumulative effect?
Thanx
gaby





------------------------------------------------------------------------

import java.io.IOException;
import java.io.FileNotFoundException;
import java.io.InputStream;
import java.io.FileInputStream;
import java.io.File;
import java.io.FileWriter;
import java.util.List;
import java.util.ArrayList;
import java.util.Properties;
import java.awt.Point;
import java.text.SimpleDateFormat;
import java.util.Calendar;
import java.util.GregorianCalendar;
import java.util.Date;

import ucar.ma2.InvalidRangeException;
import ucar.nc2.NetcdfFile;
import ucar.nc2.NetcdfFileWriteable;
import ucar.nc2.Dimension;
import ucar.nc2.Attribute;
import ucar.nc2.Variable;
import ucar.ma2.ArrayInt;
import ucar.ma2.ArrayDouble;
import ucar.ma2.DataType;

/**
 * @author g.turek
 */
public class LandslideModel
{
        private static String landslideSpatial;
        private static String topnetOutput;
        private static String topnetSpatial;
        private static String landslideAncillary;
        private static String outputFile;
        private static String logFile;
        private static String configFile = "LandslideModel.config";
  private static final double DTOR = Math.PI/180.0;
  private static final double UW = 9810.0;
  private static final String OS = System.getProperty("os.name");
  //Input data from ancillary file
  private static ArrayInt.D2 nrchm;         // Model NRCH's
  private static int nrchm_fv;
  private static ArrayDouble.D2 awi;        // Averageѕ wetness index for each 
point on the grid
  private static double awi_fv;
  private static double aw;
  //Input data from landslide spatial file
  private static ArrayDouble.D1 nzmge;      // easting
  private static String eunit;
  private static String eaxis;
  private static ArrayDouble.D1 nzmgn;      // northing
  private static String nunit;
  private static String naxis;
  private static ArrayDouble.D2 weti;       // wetness index
  private static double weti_fv;
  private static double wetin;
  private static ArrayInt.D2 dtsp;          // depth to shear plane (cm!)
  private static int dtsp_fv;
  private static int dt;
  private static ArrayInt.D2 duw;           // dry soil unit weight
  private static int duw_fv;
  private static int du;
  private static ArrayInt.D2 slope_angle;   // slope angle at each grid point 
(degrees!)
  private static int slope_angle_fv;
  private static int sa;
  private static ArrayInt.D2 sat_soil;      // percent soil saturated water 
content
  private static int sat_soil_fv;
  private static int ss;
  private static ArrayInt.D3 cohesion;      // cohesion
  private static int cohesion_fv;
  private static int co0,co1;
  private static ArrayInt.D3 frangle;       // friction angle (degrees!)
  private static int frangle_fv;
  private static int fr0,fr1;
  //Input data from topnet files
  private static ArrayDouble.D1 time;       // time
        private static String   sname;
  private static String unit;
  private static String axis;
  private static String calendar;
  private static double time_fv;
  private static ArrayDouble.D3 zbarh2o;    // depth of saturated zone (meters)
  private static double zbarh2o_fv;
  private static double zb;
  private static ArrayDouble.D3 soilh2o;    // water in soil (meters)
  private static double soilh2o_fv;
  private static double sl;
  private static ArrayDouble.D1 fparam;     // topmodel f parameter
  private static double fparam_fv;
  private static double fp;
  //Output data
  private static ArrayInt.D4 z;          // depth
  private static ArrayInt.D4 smin;       // min strength
  private static ArrayInt.D4 smax;       // min strength
  private static ArrayInt.D4 strn;       // strain
  private static ArrayInt.D4 fail_prob;  //failure probability
  private static int fpb0;
  private static FileWriter fw;

  private static double d, w, strain, smn, smx;
        private static int nnzmge;
        private static int nnzmgn;
        private static int nv;
        private static int nrch;
        private static int nens;
        private static int nt;
        private static int tstart;
        private static int tend;
  private static double minSlope;

  private static NetcdfFileWriteable outFile;

        private static boolean debug = false;
        private static boolean interactive = false;

        public static void main(String[] args)
        {
                parseArgs(args);
    readConfiguration();
if (interactive) {
      calculate_ls();
      System.exit(0);
    }
fw = openOutput(logFile); log("Beginning time step = " + tstart);
    log("Ending time step = " + tend);
                long start = System.currentTimeMillis();
    readAncillaryFile();
    readTopnetSpatial();
    readTopnetOutput();
    readLandslideSpatial();
    writeOutputMetadata();

    log("Start loop");

    int rchindx;
    try
    {
      outFile.write("time", time);
      time = null;
      outFile.write("nzmge", nzmge);
nzmge = null; outFile.write("nzmgn", nzmgn); nzmgn = null; for (int east = 0; east < nnzmge; east++) // loop over nzmge
      {
        //z = new ArrayInt.D4(tend-tstart+1,1, nnzmgn, nens);
        smin = new ArrayInt.D4(tend-tstart+1,1, nnzmgn, nens);
        smax = new ArrayInt.D4(tend-tstart+1,1, nnzmgn, nens);
        strn = new ArrayInt.D4(tend-tstart+1,1, nnzmgn, nens);
        fail_prob = new ArrayInt.D4(tend-tstart+1,1, nnzmgn,nens);
        for (int north = 0; north < nnzmgn; north++) // loop over nzmgn
        {
          sa = slope_angle.get(east,north);
          aw = awi.get(east,north);
          du = duw.get(east,north);
          dt = dtsp.get(east,north);
          ss = sat_soil.get(east,north);
          wetin = weti.get(east,north);
          co0 = 0;
          //co0 = cohesion.get(east,north,0);
          co1 = cohesion.get(east,north,1);
          fr0 = frangle.get(east,north,0);
          fr1 = frangle.get(east,north,1);
          rchindx = nrchm.get(east,north);
          if (rchindx == nrchm_fv || aw == awi_fv || wetin == weti_fv ||
              sa == slope_angle_fv || du == duw_fv || dt == dtsp_fv ||
              ss == sat_soil_fv || co0 == cohesion_fv || co1 == cohesion_fv ||
              fr0 == frangle_fv || fr1 == frangle_fv )
          { // reach or a constant parameter is nodata - set all output to 
nodata
            int tcntr = 0;
            for (int t = tstart; t <= tend; t++) // loop over time
            {
              for  (int ens = 0; ens < nens; ens++) // loop over ensambles
              {
                //z.set(tcntr,0,north,ens,-9999);
                smin.set(tcntr,0,north,ens,-9999);
                smax.set(tcntr,0,north,ens,-9999);
                strn.set(tcntr,0,north,ens,-9999);
                fail_prob.set(tcntr,0,north,ens,-9999);
              }
              tcntr++;
            }
          } else if (sa < minSlope) { // slope to low - fp = 0
            int tcntr = 0;
            for (int t = tstart; t <= tend; t++) // loop over time
            {
              for  (int ens = 0; ens < nens; ens++) // loop over ensambles
              {
                //z.set(tcntr,0,north,ens,-9999);
                smin.set(tcntr,0,north,ens,-9999);
                smax.set(tcntr,0,north,ens,-9999);
                strn.set(tcntr,0,north,ens,-9999);
                fail_prob.set(tcntr,0,north,ens,0);
              }
              tcntr++;
            }
          } else { // all spatial inputs have valid values - go thru time/ens 
loops
            fp = fparam.get(rchindx);
            int tcntr = 0;
            for (int t = tstart; t <= tend; t++) // loop over time
            {
              for  (int ens = 0; ens < nens; ens++) // loop over ensambles
              {
                zb = zbarh2o.get(t,rchindx,ens);
                sl = soilh2o.get(t,rchindx,ens);

                if (zb == zbarh2o_fv || sl == soilh2o_fv)
{ // a TopNet output is nodata - cannot calculate depth to GW and weight! //z.set(tcntr,0,north,ens,-9999); smin.set(tcntr,0,north,ens,-9999); // no GW - so cannot calculate strength smax.set(tcntr,0,north,ens,-9999); strn.set(tcntr,0,north,ens,-9999); // no GW - so cannot calculate strain fail_prob.set(tcntr,0,north,ens,-9999);
                } else { // we have TopNet output - calculate!
                  calculate_ls();
                  //z.set(tcntr,0,north,ens,(int)d);
                  strn.set(tcntr,0,north,ens,(int)strain);
                  smin.set(tcntr,0,north,ens,(int)smn);
                  smax.set(tcntr,0,north,ens,(int)smx);
                  fail_prob.set(tcntr,0,north,ens,fpb0);
                }
              }
              tcntr++;
            }
          }
        }
        //outFile.write("z", new int[] {0,east,0,0}, z);
        outFile.write("smin", new int[] {0,east,0,0}, smin);
        outFile.write("smax", new int[] {0,east,0,0}, smax);
        outFile.write("strain", new int[] {0,east,0,0}, strn);
        outFile.write("failure_probability", new int[] {0,east,0,0}, fail_prob);
      }
      //To disk
      outFile.flush();
      long end = System.currentTimeMillis();
      double elapsed = (double)(end - start)/1000.;
      log("Time elapsed : " + elapsed + " seconds");
      fw.close();
      System.exit(0);
    } catch (InvalidRangeException ire) {
      log("Invalid range exception: " + ire.getMessage());
    } catch (IOException ioe) {
      log("Unable to write to output file: " + ioe.getMessage());
    } finally {
      try
      {
        fw.close();
        System.exit(1);
      } catch (IOException ioe) {}
    }
        }

  private static void calculate_ls()
  {
    if (interactive)
    {
      writeTTYLine("Enter slope angle [deg] (sa)");
      sa = (int)readTTYLine();
      writeTTYLine("Enter average wetness (aw)");
      aw = readTTYLine();
      writeTTYLine("Enter average wetness (wetin)");
      wetin = readTTYLine();
      writeTTYLine("Enter dry soil unit weight [N/m3] (du)");
      du = (int)readTTYLine();
      writeTTYLine("Enter depth to shear plane [cm](dt)");
      dt = (int)readTTYLine();
      writeTTYLine("Enter saturated soilwater content [%] (ss)");
      ss = (int)readTTYLine();
      writeTTYLine("Enter cohesion 0 [N/m**2](co0)");
      co0 = (int)readTTYLine();
      writeTTYLine("Enter cohesion 1 [N/m**2] (co1)");
      co1 = (int)readTTYLine();
      writeTTYLine("Enter friction angle 0 (fr0)");
      fr0 = (int)readTTYLine();
      writeTTYLine("Enter friction angle 1 (fr1)");
      fr1 = (int)readTTYLine();
      writeTTYLine("Enter fparam (fp)");
      fp = readTTYLine();
      writeTTYLine("Enter depth of saturated zone [m] (zb)");
      zb = readTTYLine();
      writeTTYLine("Enter water in soil [m] (sl)");
      sl = readTTYLine();
    }
if (fr0< 0.0 || fr1<0) log("Negative friction angle!!! " + w); // depth to saturated zone [m]
    d = zb +(aw-wetin)/fp;
    writeTTYLine("Depth to saturated zone : " + d);
    // corrected depth to saturated zone
    d = Math.max(Math.min(dt/100.0,d),0.0); // limit to [0,dt] for later calc
    writeTTYLine("Corrected depth to saturated zone : " + d);
    // weight of soil + water above shear plain (must convert DTSP in m)
    w = du*(dt/100.0) + UW*Math.min(sl,d) + UW*ss/100.0*((dt/100.0)-d);
    writeTTYLine("Weight of soil + water above shear plain : " + w);
    if (w < 0.0)  log("Negative weight!!! " + w);
    // shear strain in downslope direction
    strain = w*Math.sin(sa*DTOR);
    writeTTYLine("Shear strain in downslope direction : " + strain);
    if (strain < 0.0) log("Negative strain!!! " + strain);
    // shear strength in downslope direction
    smn = co0+Math.max((w-UW*((dt/100.0)-d)),0.0)* 
Math.cos(sa*DTOR)*Math.tan(fr0*DTOR);
    smx = co1+Math.max((w-UW*((dt/100.0)-d)),0.0)* 
Math.cos(sa*DTOR)*Math.tan(fr1*DTOR);
    if (smn < 0.0) log("Negative strength!!! " + smn);
    //if (smn<strain)  log("Failure " + (strain - smn)/(smx - smn) );
    writeTTYLine("Min shear strength in downslope direction : " + smn);
    writeTTYLine("Max shear strength in downslope direction : " + smx);
    // failure probability
    fpb0 = (int)(100.0*((strain - smn)/(smx - smn)));
    fpb0 = Math.max(Math.min(100,fpb0),0); // limit to [0,100]
    //if (fpb0>0) System.out.println("Calc Failure " + fpb0);
    //log("calculate_ls failure probability : " + fpb0);
    writeTTYLine("Failure probability : " + fpb0);
  }

  private static void writeOutputMetadata()
  {
    try
    {
      outFile = NetcdfFileWriteable.createNew(outputFile, false);
      outFile.addGlobalAttribute("title", "landslide model output file");
      outFile.addGlobalAttribute("file_creation_time", getDate());
      //define mode
      Dimension timedim = outFile.addUnlimitedDimension("nt");
      Dimension edim = outFile.addDimension("nzmge", nnzmge);
      Dimension ndim = outFile.addDimension("nzmgn", nnzmgn);
      Dimension ensdim = outFile.addDimension("nens", nens);

      //addVariable(java.lang.String varName, DataType dataType, Dimension[] 
dims)
      outFile.addVariable("time", DataType.DOUBLE, new Dimension[] {timedim});
      outFile.addVariableAttribute("time", "standard_name", sname);
      outFile.addVariableAttribute("time", "unit", unit);
      outFile.addVariableAttribute("time", "axis", axis);
      outFile.addVariableAttribute("time", "calendar", calendar);
      outFile.addVariableAttribute("time", "_FillValue", time_fv);
      outFile.addVariable("nzmge", DataType.DOUBLE, new Dimension[] {edim});
      outFile.addVariableAttribute("nzmge", "unit", eunit);
      outFile.addVariableAttribute("nzmge", "axis", eaxis);
      outFile.addVariable("nzmgn", DataType.DOUBLE, new Dimension[] {ndim});
      outFile.addVariableAttribute("nzmge", "unit", nunit);
      outFile.addVariableAttribute("nzmge", "axis", naxis);
      /*
      outFile.addVariable("z", DataType.INT, new Dimension[] {timedim, edim, 
ndim, ensdim});
      outFile.addVariableAttribute("z", "long_name", "depth of saturated zone");
      outFile.addVariableAttribute("z", "unit", "cm");
      outFile.addVariableAttribute("z", "_FillValue", -9999);
      */
      outFile.addVariable("smin", DataType.INT, new Dimension[] {timedim, edim, 
ndim, ensdim});
      outFile.addVariableAttribute("smin", "long_name", "minimum strength");
      outFile.addVariableAttribute("smin", "_FillValue", -9999);
      outFile.addVariable("smax", DataType.INT, new Dimension[] {timedim, edim, 
ndim, ensdim});
      outFile.addVariableAttribute("smax", "long_name", "maximum strength");
      outFile.addVariableAttribute("smax", "_FillValue", -9999);
      outFile.addVariable("strain", DataType.INT, new Dimension[] {timedim, 
edim, ndim, ensdim});
      outFile.addVariableAttribute("strain", "long_name", "strain");
      outFile.addVariableAttribute("strain", "_FillValue", -9999);
      outFile.addVariable("failure_probability", DataType.INT, new Dimension[] 
{timedim, edim, ndim, ensdim});
      outFile.addVariableAttribute("failure_probability", "long_name", "failure 
probability");
      outFile.addVariableAttribute("failure_probability", "unit", "percent");
      outFile.addVariableAttribute("failure_probability", "valid_min", 0);
      outFile.addVariableAttribute("failure_probability", "valid_max", 100);
      outFile.addVariableAttribute("failure_probability", "_FillValue", -9999);
      //finish define mode
      outFile.create();
    } catch (IOException ioe) {
      log("Unable to create output file: " + ioe.getMessage());
    }
  }

  /**
* Read topnet output file */ private static void readTopnetOutput()
  {
log("****** Opening " + topnetOutput); try { NetcdfFile f = NetcdfFile.open(topnetOutput);
                        //listDimensions(f);
                        //listVariables(f);

      Dimension d = (Dimension)f.findDimension("nens");
      nens = d.getLength();
                        log("nens = " + nens);
      nens = 13;

      d = (Dimension)f.findDimension("nrch");
      nrch = d.getLength();
                        log("nrch = " + nrch);

      d = (Dimension)f.getUnlimitedDimension();
      nt = d.getLength();
                        log("nt = " + nt);
Variable v = (Variable)f.findVariable("time");
      time = (ArrayDouble.D1)v.read(new int[] {tstart}, new int[] 
{tend-tstart+1});
      int[] dims = time.getShape();
log("time rank = " + v.getRank() + ", shape = " + dims[0]); Attribute a = v.findAttribute("standard_name");
                        sname= (String)a.getStringValue();
      a = v.findAttribute("units");
                        unit = (String)a.getStringValue();
      a = v.findAttribute("axis");
                        axis = (String)a.getStringValue();
      a = v.findAttribute("calendar");
                        calendar = (String)a.getStringValue();
      a = v.findAttribute("_FillValue");
time_fv = ((Double)a.getNumericValue()).doubleValue();
                        v = (Variable)f.findVariable("zbarh2o");
                        zbarh2o = (ArrayDouble.D3)v.read();
      dims = zbarh2o.getShape();
log("zbarh2o rank = " + v.getRank() + ", shape = " + dims[0] + "," + dims[1] + "," + dims[2]); a = v.findAttribute("_FillValue"); zbarh2o_fv = ((Double)a.getNumericValue()).doubleValue();
                        v = ((Variable)f.findVariable("soilh2o"));
                        soilh2o = (ArrayDouble.D3)v.read();
      dims = soilh2o.getShape();
log("soilh2o rank = " + v.getRank() + ", shape = " + dims[0] + "," + dims[1] + "," + dims[2]); a = v.findAttribute("_FillValue"); soilh2o_fv = ((Double)a.getNumericValue()).doubleValue(); f.close();
    } catch (InvalidRangeException ire) {
      log("Error reading time vector: " + ire.getMessage());
      System.exit(1);
    } catch (IOException ioe) {
      log("Unable to open topnet output file: " + ioe.getMessage());
      System.exit(1);
    }
  }
/** * Read topnet spatial file */ private static void readTopnetSpatial()
  {
log("****** Opening " + topnetSpatial); try { NetcdfFile f = NetcdfFile.open(topnetSpatial);
                        //listDimensions(f);
                        //listVariables(f);

                        Variable v = ((Variable)f.findVariable("topmodf"));
log("fparam rank = " + v.getRank()); fparam = (ArrayDouble.D1)v.read();
      Attribute a = v.findAttribute("_FillValue");
fparam_fv = ((Double)a.getNumericValue()).doubleValue(); f.close();
    } catch (IOException ioe) {
      log("Unable to open topnet spatial file: " + ioe.getMessage());
      System.exit(1);
    }
  }

  /**
* Read ancillary file */ private static void readAncillaryFile()
  {
log("****** Opening " + landslideAncillary); try { NetcdfFile f = NetcdfFile.open(landslideAncillary);
                        //listDimensions(f);
                        //listVariables(f);

                        Dimension d = (Dimension)f.findDimension("nzmge");
                        nnzmge = d.getLength();
log("nnzmge = " + nnzmge);
                        d = (Dimension)f.findDimension("nzmgn");
                        nnzmgn = d.getLength();
log("nnzmgn = " + nnzmgn);
                        Variable v = ((Variable)f.findVariable("nrch_model"));
                        nrchm = (ArrayInt.D2)v.read();
      int[] dims = nrchm.getShape();
                        log("nrchm rank = " + v.getRank() + ", shape = " + dims[0] + 
"," + dims[1]);
      Attribute a = v.findAttribute("_FillValue");
nrchm_fv = ((Integer)a.getNumericValue()).intValue();
                        v = ((Variable)f.findVariable("avg_wetness_index"));
                        awi = (ArrayDouble.D2)v.read();
      dims = awi.getShape();
                        log("awi rank = " + v.getRank() + ", shape = " + dims[0] + 
"," + dims[1]);
      a = v.findAttribute("_FillValue");
awi_fv = ((Double)a.getNumericValue()).doubleValue(); f.close();
    } catch (IOException ioe) {
      log("Unable to open ancillary file: " + ioe.getMessage());
      System.exit(1);
    }
  }

  /**
* Read landslide spatial file */ private static void readLandslideSpatial()
  {
log("****** Opening " + landslideSpatial); try { NetcdfFile f = NetcdfFile.open(landslideSpatial);
                        //listDimensions(f);
                        //listVariables(f);

                        Dimension d = (Dimension)f.findDimension("nzmge");
                        int nne = d.getLength();

                        d = (Dimension)f.findDimension("nzmgn");
                        int nnn = d.getLength();

      if (nnzmge != nne ||  nnzmgn != nnn)
      {
        log("Mismatched grid sizes, something is wrong!");
        System.exit(1);
      }

                        d = (Dimension)f.findDimension("nv");
                        nv = d.getLength();
                        log("nv = " + nv);

                        Variable v = ((Variable)f.findVariable("nzmge"));
                        nzmge = (ArrayDouble.D1)v.read();
      int[] dims = nzmge.getShape();
                        log("nzgme rank = " + v.getRank() + ", shape = " + 
dims[0]);
      Attribute a = v.findAttribute("unit");
      eunit = (String)a.getStringValue();
      a = v.findAttribute("axis");
      eaxis = (String)a.getStringValue();

                        v = ((Variable)f.findVariable("nzmgn"));
                        nzmgn = (ArrayDouble.D1)v.read();
      dims = nzmgn.getShape();
                        log("nzgmn rank = " + v.getRank() + ", shape = " + 
dims[0]);
      a = v.findAttribute("unit");
      nunit = (String)a.getStringValue();
      a = v.findAttribute("axis");
      naxis = (String)a.getStringValue();

                        v = ((Variable)f.findVariable("depth_to_shearplane"));
                        dtsp = (ArrayInt.D2)v.read();
      dims = dtsp.getShape();
                        log("dtsp rank = " + v.getRank() + ", shape = " + dims[0] + 
"," + dims[1]);
      a = v.findAttribute("_FillValue");
dtsp_fv = ((Integer)a.getNumericValue()).intValue();
                        v = ((Variable)f.findVariable("wetness_index"));
                        weti = (ArrayDouble.D2)v.read();
      dims = weti.getShape();
                        log("weti rank = " + v.getRank() + ", shape = " + dims[0] + 
"," + dims[1]);
      a = v.findAttribute("_FillValue");
weti_fv = ((Double)a.getNumericValue()).doubleValue();
                        v = ((Variable)f.findVariable("dry_soil_unit_weight"));
                        duw = (ArrayInt.D2)v.read();
      dims = duw.getShape();
                        log("duw rank = " + v.getRank() + ", shape = " + dims[0] + 
"," + dims[1]);
      a = v.findAttribute("_FillValue");
duw_fv = ((Integer)a.getNumericValue()).intValue();
                        v = ((Variable)f.findVariable("slope_angle"));
                        slope_angle = (ArrayInt.D2)v.read();
      dims = slope_angle.getShape();
                        log("slope_angle rank = " + v.getRank() + ", shape = " + dims[0] 
+ "," + dims[1]);
      a = v.findAttribute("_FillValue");
slope_angle_fv = ((Integer)a.getNumericValue()).intValue();
                        v = 
((Variable)f.findVariable("saturated_soilwater_content"));
                        sat_soil = (ArrayInt.D2)v.read();
      dims = sat_soil.getShape();
                        log("sat_soil rank = " + v.getRank() + ", shape = " + dims[0] + 
"," + dims[1]);
      a = v.findAttribute("_FillValue");
sat_soil_fv = ((Integer)a.getNumericValue()).intValue();
                        v = ((Variable)f.findVariable("cohesion"));
                        cohesion = (ArrayInt.D3)v.read();
      dims = cohesion.getShape();
                        log("cohesion rank = " + v.getRank() + ", shape = " + dims[0] + 
"," + dims[1] + "," + dims[2]);
      a = v.findAttribute("_FillValue");
cohesion_fv = ((Integer)a.getNumericValue()).intValue();
                        
                        v = ((Variable)f.findVariable("friction_angle"));
log("friction_angle rank = " + v.getRank()); frangle = (ArrayInt.D3)v.read();
      dims = frangle.getShape();
                        log("frangle rank = " + v.getRank() + ", shape = " + dims[0] + 
"," + dims[1] + "," + dims[2]);
      a = v.findAttribute("_FillValue");
frangle_fv = ((Integer)a.getNumericValue()).intValue();
                        f.close();      
    } catch (IOException ioe) {
      log("Unable to open landslide spatial file: " + ioe.getMessage());
      System.exit(1);
    }
  }

        /**
         * Dimension information
         */
        private static void listDimensions(NetcdfFile nf)
        {
                        Dimension d;
                        List l = nf.getDimensions();
                        int nd = l.size();
                        log("Found " + nd + " dimensions:");
                        for (int i = 0; i < nd; i++)
                        {
                                d = (Dimension)l.get(i);
                                log("   " + d.getName() + " = " + 
d.getLength());
                        }
        }

        /**
         * Variables information
         */
        private static void listVariables(NetcdfFile nf)
        {
                List l = nf.getVariables();
                int     nd = l.size();
                log("Found " + nd + " variables:");
                Dimension d;    
                Variable v;
                String s;
                List l1;
                int nl1;
                for (int i = 0; i < nd; i++)
                {
                        v = (Variable)l.get(i);
                        l1 = v.getDimensions();
                        nl1 = l1.size();
                        s = "   " + v.getName() + ": size = " + v.getSize() + ", " + nl1 
+ " dimensions  = ";
                        for (int j = 0; j < nl1; j++)
                        {
                                d = (Dimension)l1.get(j);
                                s += v.getDimension(j).getLength() + " ";
                        }

                        s += ", data type = " + v.getDataType().toString() + ", 
description = " + v.getDescription();
                        log(s);
                }
        }

  /**
   *  Read a configuration file into a Properties object from either the local 
disc or a jar file
   */
  public static void readConfiguration()
  {
    Properties p = new Properties();
    try
    {
      File f = new File(configFile);
if (f.exists()) {
        //Try reading from local disc
        FileInputStream fs = new FileInputStream(f.getAbsolutePath());
        p.load(fs);
        if (fs != null) fs.close();
      } else {
        //Reading from a jar file
        if (ClassLoader.getSystemResource(f.getAbsolutePath()) != null)
        {
          InputStream is = 
ClassLoader.getSystemResource(f.getAbsolutePath()).openStream();
          p.load(is);
          is.close();
        } else {
          //File really does not exist
          throw new FileNotFoundException("Unable to find configuration file " 
+ f.getAbsolutePath());
        }
      }
            tstart = Integer.parseInt((String)p.get("begTimestep"));
            tend = Integer.parseInt((String)p.get("endTimestep"));
            minSlope = Double.parseDouble((String)p.get("minSlope"));
            landslideSpatial = (String)p.get("landslideSpatial");
            topnetOutput = (String)p.get("topnetOutput") ;
            topnetSpatial = (String)p.get("topnetSpatial");
            landslideAncillary = (String)p.get("landslideAncillary");
            outputFile = (String)p.get("outputFile");
            logFile = (String)p.get("logFile");

    } catch (FileNotFoundException fnfe) {
      log("Unable to find file: " + fnfe.getMessage());
      System.exit(1);
    } catch (IOException ioe) {
      log("Unable to open file: " + ioe.getMessage());
      System.exit(1);
    }
        }

  private static void log(String s)
  {
    try
    {
      fw.write(s + "\n");
      if (debug) System.out.println(s);
    } catch (IOException e) {
      System.out.println("Couldn't write to log file: " + e.getMessage());
    }
  }

  private static final void writeTTYLine(String s)
  {
    if (interactive) System.out.println(s);
  }

  /**
   *  Gets input from the command line
   *  Note: must look for '\r' for it to work properly in DOS
   *  @return tty input
   */
  private static final double readTTYLine()
  {
    StringBuffer buf = new StringBuffer(80);
    int c = 0;
    try
    {
      while ((c = System.in.read()) != -1)
      {
        char ch = (char)c;
        if (ch == '\r')
        {
          // This is a temporary fix.  XXXX We really need lookahead.
          if (OS.equalsIgnoreCase ("macos")) break;
          else
          {
            System.in.read();
            break;
          }
        } else if (ch == '\n') break;       // Unix flavors.
        buf.append(ch);
      }
    } catch (IOException e) {
      System.err.println(e.getMessage());
    }
    return Double.parseDouble(buf.toString());
  }

  /**
   * Open output file
   * @param fileName the output file
   */
  private static FileWriter openOutput(String fileName)
  {
    FileWriter file = null;
    try
    {
      file = new FileWriter(fileName);

    } catch (IOException e) {
      System.out.println("Couldn't write log file: " + e.getMessage());
      System.exit(1);
    }
    return file;
  }

        /**
* Parse arguments */
        private static void parseArgs(String[] args)
        {
                int numArgs = args.length;
                int index = 0;
                while (index < numArgs)
                {
                        if (args[index].equals("-h"))
                        {
                                usage();

                        } else if (args[index].equals("-i")) {
                                interactive = true;
                                index++;
                                
                        } else if (args[index].equals("-d")) {
                                debug = true;
                                index++;
                                
                        } else if (args[index].equals("-c")) {
                                configFile = args[++index];
                                index++;

                        } else {
        usage();
      }
                }
        }

        /**
         * Command line switches
         */
        private static void usage()
        {
                System.out.println("**** LandslideModel optional invocation flags 
****");
                System.out.println("    -c <configfile>  : config file");
                System.out.println("    -h                 : this listing");
                System.out.println("    -i                 : interactive mode");
                System.out.println("    -d                 : turn debug on 
listing");
                System.exit(0);
        }

  /**
   *  Get current date in dd/mm/yyyy hh:mm:ss.SSS format
   *  @return date string
   */
  private static String getDate()
  {
    SimpleDateFormat format = new SimpleDateFormat("dd'/'MM'/'yyyy 
HH:mm:ss.SSS");
    format.setLenient(false);
    Calendar cal = new GregorianCalendar();
    Date d = cal.getTime();
    return format.format(d);
  }
}


  • 2007 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the netcdf-java archives: