Re: [netcdf-java] Writing a 2 Gb NetCDF File

Hi Valerio:

1. 2.2 does not support writing large file.
2. 4.0 had various bugs which should now be fixed. I will release 4.0.17 later today. you must add

    ncFile.setLargeFile(true);

to enable files > 2 Gb.

3. for performance, call

    ncFile.setFill(false);
    ncFile.setLength(approxSize);

before the create() call. with this, it took 267 secs to run on my windows 
machine.

the most performance is to write the data in physical order. when i did that, the program completed in 72 secs. below is modified version of your program.

thanks for helping to debug!

-----

  public void testBig() throws IOException, InvalidRangeException {

    long start = System.nanoTime();
    System.out.println("Begin <=");

    String varName = "example";

    int timeSize = 8;
    int latSize = 8022;
    int lonSize = 10627;

System.out.println("File size (B) = " + (long) timeSize * latSize * lonSize * 4); System.out.println("File size~ (MB) = " + Math.round((long) timeSize * latSize * lonSize * 4 / Math.pow(2, 20)));

NetcdfFileWriteable ncFile = NetcdfFileWriteable.createNew("D:/temp/bigFile2.nc");
    ncFile.setFill(false);
    ncFile.setLargeFile(true);

    long approxSize = (long) timeSize * latSize * lonSize * 4 + 4000;
    ncFile.setLength(approxSize);

    String timeUnits = "hours since 2008-06-06 12:00:0.0";
    String coordUnits = "degrees";

    Dimension[] dim = new Dimension[3];

    dim[0] = setDimension(ncFile, "time", timeUnits, timeSize);
    dim[1] = setDimension(ncFile, "lat", coordUnits, latSize);
    dim[2] = setDimension(ncFile, "lon", coordUnits, lonSize);

    ncFile.addVariable(varName, DataType.FLOAT, dim);

    ncFile.addVariableAttribute(varName, "_FillValue", -9999);
    ncFile.addVariableAttribute(varName, "missing_value", -9999);

    System.out.println("Creating netcdf <=");
    ncFile.create();
    long stop = System.nanoTime();
    double took = (stop - start) * .001 * .001 * .001;
    System.out.println("That took "+took+" secs");
    start = stop;

    System.out.println("Writing netcdf <=");

    int[] shape = new int[]{1, 1, lonSize};
    float[] floatStorage = new float[lonSize];
    Array floatArray = Array.factory(float.class, shape, floatStorage);
    for (int t = 0; t < timeSize; t++) {
      for (int i = 0; i < latSize; i++) {
        int[] origin = new int[]{t, i, 0};
        ncFile.write(varName, origin, floatArray);
      }
    }

    ncFile.close();

    System.out.println("Done <=");
    stop = System.nanoTime();
    took = (stop - start) * .001 * .001 * .001;
    System.out.println("That took "+took+" secs");
    start = stop;
  }

private static Dimension setDimension(NetcdfFileWriteable ncFile, String name, String units, int length) {

    Dimension dimension = ncFile.addDimension(name, length);
    ncFile.addVariable(name, DataType.FLOAT, new Dimension[]{dimension});
    ncFile.addVariableAttribute(name, "units", units);

    return dimension;
  }


  • 2008 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the netcdf-java archives: