[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Java netCDF 2.2 / Structures





Norman Fomferra wrote:
Dear John,

Thanks to the very intuitive (read-only) Java netCDF API we have now been able to quickly provide a reader for netCDF formatted satellite raster data into the open-source ESA / BEAM software (http://www.brockmann-consult.de/beam/).

You're welcome.


Unfortunately I have my problems with the writable API. It simply won't let me create unlimited arrays of Structure records. I would very much appreciate your comments to the (very, very simple!) test code I have attached.

Theres a very simple answer. Netcdf (version 3) file format does not support 
writing structures, so the API does not have that option.

You can "write structures" using the record dimension, but you are doing it at a lower level, namely writing the variable data for all record variables for record 0, then 1, then 2, etc.
I have made some comments in your file below. Below that is example code.


I am pretty sure that other Java netCDF newbies will run into similar problems as well, so maybe you can use the code as a demonstration of how the API shall NOT be used. Especially, because I was not able to find any unit-test or example code which demonstrates the use of Structures with the Java API.

I have used netcdf-2.2.14.jar + nlog4j-1.2.22.jar with JDK 1.5.0_06 on a Windows XP PC. By the way, I think it was not a good idea to force clients to use the nlog4j library now. Most professional applications that use the netCDF API already use some kind of logging. E.g. in BEAM we use the Java Logging API which is even capable of interacting with logging services that already exist on the host operating system. We are happy about every JAR we do not need in our classpath.

The slf4j is a facade that can use various loggers, including the jdk 1.4 
logger. nlog4j is the implementation that uses log4j. See the slf4j web site to 
download the jdk 1.4 implementation, which you can substitute for nlog4j.




Tell me, if more feedback on the netCDF Java API is welcome and if so, in which form.

Yes, feedback is good, though we can not always respond quickly. Send to the 
netcdf-java email list.




Best regards and thanks in advance for your help

   Norman



____________________________________________

Norman Fomferra (Brockmann Consult)

GITZ / GKSS
Max-Planck-Str. 2 21502 Geesthacht
Germany

Tel:  +49 (0)4152 889 303
Fax:  +49 (0)4152 889 333
Mail: address@hidden
Web: http://www.brockmann-consult.de ___________________________________________


------------------------------------------------------------------------


import ucar.nc2.*;
import ucar.ma2.DataType;

import java.io.IOException;
import java.util.Arrays;


/**
 * Provides 2 tests which create an unlimited array of structure records.
 * {@link #testAddRecordStructure} has no effect and
 * {@link #testAddMyOwnStructure} throws an exception:
 * <pre>
 * Exception in thread "main" java.lang.IllegalStateException: unknown DataType 
== Structure
 *     at ucar.nc2.N3header.getType(N3header.java:434)
 *     at ucar.nc2.N3header.writeVars(N3header.java:627)
 *     at ucar.nc2.N3header.create(N3header.java:487)
 *     at ucar.nc2.N3iosp.create(N3iosp.java:299)
 *     at ucar.nc2.NetcdfFileWriteable.create(NetcdfFileWriteable.java:320)
 *     at 
TestUnlimitedStructureArray.testAddMyOwnStructure(TestUnlimitedStructureArray.java:107)
 *     at TestUnlimitedStructureArray.main(TestUnlimitedStructureArray.java:24)
 * </pre>
 */
public class TestUnlimitedStructureArray {

    public static final String LOCATION_1 = "test1.nc";
    public static final String LOCATION_2 = "test2.nc";

    /**
     * Performs both tests.
     * @param args ignored
     */
    public static void main(String[] args) {
        try {
            testAddRecordStructure(LOCATION_1);
            testAddMyOwnStructure(LOCATION_2);
        } catch (IOException e) {
            e.printStackTrace();
        }
    }

    /**
     * Test 1: The method tries to create the unlimited array of records using 
the
     * NetcdfFileWriteable.addRecordStructure() from all unlimited
     * 1-D arrays in the file.
     */
    public static void testAddRecordStructure(String location) throws 
IOException {
        // Obtain file instance in 'define mode'
        final NetcdfFileWriteable file = 
NetcdfFileWriteable.createNew(location, false);

        // Define the unlimited dimension
        final Dimension dimN = file.addDimension("n", 0, true, true, false);

        // Define four unlimited arrays
        file.addVariable("bin_number", DataType.INT, new Dimension[]{dimN});
        file.addVariable("counts", DataType.SHORT, new Dimension[]{dimN});
        file.addVariable("sum", DataType.FLOAT, new Dimension[]{dimN});
        file.addVariable("sum_sqr", DataType.FLOAT, new Dimension[]{dimN});
        // create the file which is then no longer in 'define mode'
        file.create();

        // addRecordStructure() only works after file.create(),
        // because otherwise the file has no 'spi'.
        // Note that this breaks with the API convention, because 'file'
        // is now not in 'define mode' anymore.
        file.addRecordStructure();

Remove addRecordStructure, it doesnt help you with writing


        dumpVariable(file, "record"); // --> record is correctly dumped out
        file.close();

        dumpFile(location); // --> 'record' is gone, arrays are the same! ERROR?
    }

You dont want to do any of testAddMyOwnStructure(). You can only use the 
methods in NetcdfFileWriteable.


    /**
     * Test 2: The method tries to create the unlimited array of Structure 
record from all unlimited
     * 1-D arrays in the file by utilizing the 
NetcdfFileWriteable.addRecordStructure() .
     */
    public static void testAddMyOwnStructure(String location) throws 
IOException {
        // Obtain file instance in 'define mode'
        final NetcdfFileWriteable file = 
NetcdfFileWriteable.createNew(LOCATION_2, false);

        // Define the unlimited dimension
        final Dimension dimN = file.addDimension("n", 0, true, true, false);

        // Create the unlimited array of structure type 'bin_list'
        final Structure binList = new Structure(file, null, null, "bin_list");
        binList.setDimensions(Arrays.asList(new Dimension[]{dimN}));

        // Define four structure members
        Variable v1 = new Variable(file, null, binList, "bin_number");
        v1.setDataType(DataType.INT);
        v1.setDimensions((String) null); // = scalar
        binList.addMemberVariable(v1);

        Variable v2 = new Variable(file, null, binList, "counts");
        v2.setDataType(DataType.SHORT);
        v2.setDimensions((String) null); // = scalar
        binList.addMemberVariable(v2);

        Variable v3 = new Variable(file, null, binList, "sum");
        v3.setDataType(DataType.FLOAT);
        v3.setDimensions((String) null); // = scalar
        binList.addMemberVariable(v3);

        Variable v4 = new Variable(file, null, binList, "sum_sqr");
        v4.setDataType(DataType.FLOAT);
        v4.setDimensions((String) null); // = scalar
        binList.addMemberVariable(v4);

        // add structure array to root group
        file.addVariable(null, binList);

        // Note: Strange to say, at this point, file.findVariable("bin_list")
        // yields 'null'. So I had to call file.finish() first.
        file.finish();
        // Ok, now we find 'bin_list', but file is still in 'define mode'
        dumpVariable(file, "bin_list"); // --> 'bin_list' looks good to me

        // create the file which is then no longer in 'define mode'
        file.create(); // throws java.lang.IllegalStateException: unknown 
DataType == Structure

        // Never comes here

        file.close();

        dumpFile(location);
    }

    private static void dumpFile(final String location) throws IOException {
        final NetcdfFile file = NetcdfFile.open(location);
        System.out.println("file '" + location + "' = \n" + file);
        file.close();
    }

    private static void dumpVariable(final NetcdfFile file, final String 
fullName) {
        final Variable variable = file.findVariable(fullName);
        System.out.println("variable '" + fullName + "' = \n" + variable);
    }
}

package ucar.nc2;

import junit.framework.*;

import ucar.ma2.*;
import java.io.IOException;

/**
* Simple example to create a new netCDF file with a "record structure"
* @author : John Caron
*/
public class TestWriteRecordStructure extends TestCase  {

 static String fileName = TestNC2.topDir+"testWriteRecordStructure.nc"; // 
default name of file created
 static boolean dumpAfterCreate = false;

 public TestWriteRecordStructure( String name) {
   super(name);
 }

 public void testWriteRecordStructure() throws IOException, 
InvalidRangeException {
   NetcdfFileWriteable writeableFile = new NetcdfFileWriteable(fileName, false);

   // define dimensions, including unlimited
   Dimension latDim = writeableFile.addDimension("lat", 3);
   Dimension lonDim = writeableFile.addDimension("lon", 4);
   Dimension timeDim = writeableFile.addDimension("time", -1, true, true, 
false);

   // define Variables
   Dimension[] dim3 = new Dimension[3];
   dim3[0] = timeDim;
   dim3[1] = latDim;
   dim3[2] = lonDim;

   writeableFile.addVariable("lat", DataType.FLOAT, new Dimension[] {latDim});
   writeableFile.addVariableAttribute("lat", "units", "degrees_north");

   writeableFile.addVariable("lon", DataType.FLOAT, new Dimension[] {lonDim});
   writeableFile.addVariableAttribute("lon", "units", "degrees_east");

   writeableFile.addVariable("rh", DataType.INT, dim3);
   writeableFile.addVariableAttribute("rh", "long_name", "relative humidity");
   writeableFile.addVariableAttribute("rh", "units", "percent");

   writeableFile.addVariable("T", DataType.DOUBLE, dim3);
   writeableFile.addVariableAttribute("T", "long_name", "surface temperature");
   writeableFile.addVariableAttribute("T", "units", "degC");

   writeableFile.addVariable("time", DataType.INT, new Dimension[] {timeDim});
   writeableFile.addVariableAttribute("time", "units", "hours since 
1990-01-01");

   // create the file
   writeableFile.create();

   // write out the non-record variables
   writeableFile.write("lat", Array.factory(new float[] {41, 40, 39}));
   writeableFile.write("lon", Array.factory(new float[] {-109, -107, -105, 
-103}));

   //// heres where we write the record variables

   // different ways to create the data arrays. Note the outer dimension has 
shape 1.
   ArrayInt rhData = new ArrayInt.D3(1, latDim.getLength(), lonDim.getLength());
   ArrayDouble.D3 tempData = new ArrayDouble.D3(1, latDim.getLength(), 
lonDim.getLength());
   Array timeData = Array.factory( DataType.INT, new int[] {1});

   int[] origin = new int[] {0, 0, 0};
   int[] time_origin = new int[] {0};

   // loop over each record
   for (int time=0; time<10; time++) {
     // make up some data for this record, using different ways to fill the 
data arrays.
     timeData.setInt(timeData.getIndex(), time * 12);

     Index ima = rhData.getIndex();
     for (int lat=0; lat<latDim.getLength(); lat++)
       for (int lon=0; lon<lonDim.getLength(); lon++) {
         rhData.setInt(ima.set(0, lat, lon), time * lat * lon);
         tempData.set(0, lat, lon, time * lat * lon / 3.14159);
       }

     // write the data out for this record
     time_origin[0] = time;
     origin[0] = time;

     writeableFile.write("rh", origin, rhData);
     writeableFile.write("T", origin, tempData);
     writeableFile.write("time", time_origin, timeData);
   }

   // all done
   writeableFile.close();
 }
}