[netcdf-java] Improper writing of bytes to unlimited variables in 4.2.18

I've encountered a problem writing values of type byte to a variable with a single unlimited dimension to a netcdf-3 format file, in revision 4.2.18 (and probably earlier versions as well). It appears to improperly pad single-byte values out to 4-byte words. If I don't make the dimension unlimited, I don't have this problem.

Writing a simple array of consecutive byte values gives me this:

netcdf test-byte-unlimited {
dimensions:
        X = 5 ;
        D = UNLIMITED ; // (18 currently)
variables:
        double X(X) ;
        byte V(D) ;
data:

 X = _, _, _, _, _ ;

 V = 1, 0, 0, 0, 2, 0, 0, 0, 3, 0, 0, 0, 4, 0, 0, 0, 5, 0 ;
}

I'm expecting:
 V = 1, 2, 3, 4, 5, 6, 7, 8, 9, -1, -2, -3, -4, -5, -6, -7, -8, -9 ;


Following is a patch that appears to solve the problem for me, followed by my test-case code. It seems this issue has come up; there was a commented-out fix for padding of byte values, but it only applied for files with a single variable; I really need to have this working for files with other fixed-dimension variables in them as well.

Note that the test case below does indeed pass; re-reading the written variable using NetCDF-Java returns the entire array without the padding. With the padding, though, the file size is larger (186 vs 240 bytes on my system), and the C API shows incorrect values.

This is the first time I've looked at NetCDF library code of any kind - it will probably be obvious that I'm not familiar with the file format internals.

Here's the patch:

diff -rupN ncSrc-4.2.18-base/cdm/src/main/java/ucar/nc2/iosp/netcdf3/N3header.java ncSrc-4.2.18/cdm/src/main/java/ucar/nc2/iosp/netcdf3/N3header.java --- ncSrc-4.2.18-base/cdm/src/main/java/ucar/nc2/iosp/netcdf3/N3header.java 2010-12-17 10:08:16.000000000 -0800 +++ ncSrc-4.2.18/cdm/src/main/java/ucar/nc2/iosp/netcdf3/N3header.java 2011-01-14 16:54:17.000000000 -0800
@@ -883,18 +883,10 @@ public class N3header {
       raf.writeInt(n);
     }

- // Note on padding: In the special case of only a single record variable of character, byte, or short
-    // type, no padding is used between data values.
-    boolean usePadding = true;
-    /* if (n == 1) {
-      Variable var = vars.get(0);
-      DataType dtype = var.getDataType();
- if ((dtype == DataType.CHAR) || (dtype == DataType.BYTE) || (dtype == DataType.SHORT))
-        usePadding = false;
-    } */
-
     for (int i = 0; i < n; i++) {
       Variable var = vars.get(i);
+      DataType dtype = var.getDataType();
+
       writeString(var.getName());

       // dimensions
@@ -908,7 +900,8 @@ public class N3header {
         if (!dim.isUnlimited())
           vsize *= dim.getLength();
       }
-      if (usePadding)
+      if (!var.isUnlimited() ||
+ ((dtype != DataType.CHAR) && (dtype != DataType.BYTE) && (dtype != DataType.SHORT)))
         vsize += padding(vsize);

       // variable attributes



------
Test case:

    @Test
    public void byteReadWriteUnlimitedDim() throws Exception {
        File f = new File("/tmp/test-byte-unlimited.nc");
byte[] data = {1, 2, 3, 4, 5, 6, 7, 8, 9, -1, -2, -3, -4, -5, -6, -7, -8, -9};
        if (f.exists()) {
            f.delete();
        }
NetcdfFileWriteable outf = NetcdfFileWriteable.createNew(f.getPath());
        //Dimension d = outf.addDimension("D", data.length);
        Dimension d0 = outf.addDimension("X", 5);
        Dimension d = outf.addUnlimitedDimension("D");
Variable v0 = outf.addVariable("X", DataType.DOUBLE, new Dimension[]{d0}); Variable v = outf.addVariable("V", DataType.BYTE, new Dimension[]{d});
        assertEquals(1, v.getElementSize());
        outf.create();

Array arr = Array.factory(DataType.BYTE, new int[]{data.length}, data);
        outf.write(v.getName(), arr);
        outf.close();

        NetcdfFile ncf = NetcdfFile.open(f.getPath());
        Variable inv = ncf.findVariable("V");
        assertEquals(inv.getDataType(), DataType.BYTE);
        int[] org = {0};
byte[] readdata = (byte[]) inv.read(org, inv.getShape()).copyTo1DJavaArray();

        assertEquals(1, inv.getElementSize());
        assertArrayEquals(data, readdata);
        // this test passes, but ncdump shows improper zero-padding
    }


--------



--
Chris Chamberlin  <chris.chamberlin@xxxxxxxx>
NOAA Center for Tsunami Research, NOAA PMEL/UW JISAO
7600 Sand Point Way NE Bldg 3, Seattle, WA 98115 USA
+1 206-526-6809          http://nctr.pmel.noaa.gov/



  • 2011 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the netcdf-java archives: