[netcdf-java] Variable shape problem & EOFException

[John Caron]

Hi Nick:

You're right its a bug that happens when interleaving read and writes 
without fill. I will have a fix soon, and you will need to call flush() 
after writing to ensure correct read.

Thanks for your careful example, I never would have found it without it.

John

Nick Bower wrote:
> Adding an ncFile.flush() to the submitted test case as suggested does 
> not resolve the problem.  I think it needs to be marked as a bug until a 
> documented way around it surfaces.
>
>
> Ethan Davis wrote:
>   
>> Hi Nick,
>>
>> I'm not sure if this is a recommended way but you might try to flush() 
>> the NetcdfFileWriteable before reading.
>>
>> John might have another answer. He is out of the office this week so 
>> his email may be spotty.
>>
>> Hope that helps,
>>
>> Ethan
>>
>> Nick Bower wrote:
>>     
>>> Hello.  Is the lack of response here because this is not considered a 
>>> bug?  Is there a another recommended way perhaps to safely increase a 
>>> dimension and invalidate the memory cache in the process?
>>>
>>> Nick
>>>
>>>
>>> Nick Bower wrote:
>>>  
>>>       
>>>> I've found a problem in which cache/memory and disk shape 
>>>> information about variables will disagree with v2.2.22 of Java 
>>>> Netcdf library.
>>>>
>>>> When you add a new value to a variable, automatically increasing the 
>>>> length of a dimension, subsequent reads can throw EOFException 
>>>> because RandomAccessFile is instructed to read more values than the 
>>>> file contains - the cached and actual shapes disagree.
>>>>
>>>> I've created a runnable test case below to explain and demonstrate 
>>>> success and failure conditions.
>>>>
>>>> I am getting around this now by not interleaving read/write 
>>>> operations on variables, but instead reading all variables' data to 
>>>> memory, then performing any writes I need to after.
>>>>
>>>> TestInsertRecord.java:
>>>>
>>>>
>>>> package com.metoceanengineers.datafeeds.netcdf.test;
>>>>
>>>> import java.io.File;
>>>> import java.io.IOException;
>>>> import java.text.DateFormat;
>>>> import java.text.SimpleDateFormat;
>>>>
>>>> import junit.framework.TestCase;
>>>> import ucar.ma2.Array;
>>>> import ucar.ma2.ArrayInt;
>>>> import ucar.ma2.DataType;
>>>> import ucar.nc2.Dimension;
>>>> import ucar.nc2.NetcdfFileWriteable;
>>>>
>>>> public class TestInsertRecord extends TestCase {
>>>>       DateFormat dateFormat = new SimpleDateFormat("yyyyMMdd HHMM");
>>>>       protected NetcdfFileWriteable createNc(String prefix) throws 
>>>> IOException {
>>>>               File mainline = File.createTempFile(prefix+"-", ".nc");
>>>>        NetcdfFileWriteable mainlineNc = 
>>>> NetcdfFileWriteable.createNew(mainline.getAbsolutePath(), false);
>>>>
>>>>        Dimension recordsDim = 
>>>> mainlineNc.addUnlimitedDimension("records");
>>>>               Dimension timeDims[] = {recordsDim};
>>>>        Dimension var1Dims[] = {recordsDim}; // 1D
>>>>               mainlineNc.addVariable("time", DataType.INT, timeDims);
>>>>        mainlineNc.addVariable("var1", DataType.INT, var1Dims);
>>>>
>>>>        mainlineNc.create();
>>>>        return mainlineNc;
>>>>    }
>>>>
>>>>
>>>>    protected String getNcInstance() throws Exception {
>>>>
>>>>        NetcdfFileWriteable mainlineNc = createNc("testfile");
>>>>               int[] origin = {0};
>>>>               ArrayInt.D1 timeArr = new ArrayInt.D1(2);
>>>>        timeArr.set(0, (int)dateFormat.parse("20071130 
>>>> 0924").getTime());
>>>>        timeArr.set(1, (int)dateFormat.parse("20071130 
>>>> 0926").getTime());
>>>>        mainlineNc.write("time", origin, timeArr);
>>>>               ArrayInt.D1 var1Arr = new ArrayInt.D1(2);
>>>>        var1Arr.set(0, 10);
>>>>        var1Arr.set(1, 12);
>>>>        mainlineNc.write("var1", origin, var1Arr);
>>>>
>>>>        mainlineNc.close();
>>>>
>>>>        return mainlineNc.getLocation();
>>>>    }
>>>>      /**
>>>>     * Append new data to end of existing variables.
>>>>     *
>>>>     * @throws Exception
>>>>     */
>>>>    public void testAppendWorksOk() throws Exception {
>>>>               String ncFilename = getNcInstance();
>>>>        NetcdfFileWriteable ncFile = 
>>>> NetcdfFileWriteable.openExisting(ncFilename, false);
>>>>
>>>>        /*
>>>>         * Append value (20071130 0924, 11) into (time, var1)
>>>>         */
>>>>               ArrayInt.D1 newTimeValue = new ArrayInt.D1(1);
>>>>        newTimeValue.set(0, (int)dateFormat.parse("20071130 
>>>> 0925").getTime());
>>>>
>>>>        ArrayInt.D1 newVarValue = new ArrayInt.D1(1);
>>>>        newVarValue.set(0, 11);
>>>>               int[] origin = {2};
>>>>
>>>>        /* The first write will expand the variables,
>>>>         * but second write ok as we're just writing
>>>>         * and not reading */
>>>>               ncFile.write("time", origin, newTimeValue);
>>>>        ncFile.write("var1", origin, newVarValue);
>>>>               assertEquals(3, 
>>>> ncFile.findDimension("records").getLength());
>>>>           }
>>>>
>>>>    /**
>>>>     * Test insertion of a record in between the 2 existing
>>>>     * records by reading the existing tail, inserting new data
>>>>     * and re-appending.
>>>>     *
>>>>     * Triggers EOFException through interleaved read/writes
>>>>     *
>>>>     * @throws Exception
>>>>     */
>>>>    public void testInsertFails() throws Exception {
>>>>               String ncFilename = getNcInstance();
>>>>        NetcdfFileWriteable ncFile = 
>>>> NetcdfFileWriteable.openExisting(ncFilename, false);
>>>>
>>>>        ArrayInt.D1 newTimeValue = new ArrayInt.D1(1);
>>>>        newTimeValue.set(0, (int)dateFormat.parse("20071130 
>>>> 0925").getTime());
>>>>
>>>>        ArrayInt.D1 newVarValue = new ArrayInt.D1(1);
>>>>        newVarValue.set(0, 11);
>>>>               /* Going to insert at 1, so read existing value,
>>>>         * write down new one, and re-append old tail.
>>>>         */
>>>>               int[] insertPointOrigin = {1};
>>>>        int[] appendOrigin = {2};
>>>>        int[] shape = {1};
>>>>               Array tailTime = 
>>>> ncFile.findVariable("time").read(insertPointOrigin, shape);
>>>>        ncFile.write("time", insertPointOrigin, newTimeValue);
>>>>        ncFile.write("time", appendOrigin, tailTime);
>>>>               /* Next line excepts - why?  Because the last write 
>>>> above at
>>>>         * records index 2 triggers an increase in the CACHED/MEMORY
>>>>         * length of all variables to 3, but on disk it's still the
>>>>         * original length 2.
>>>>         *
>>>>         *  Therefore we get EOFException.
>>>>         */
>>>>               Array tailVar1 = 
>>>> ncFile.findVariable("var1").read(insertPointOrigin, shape);
>>>>        ncFile.write("var1", insertPointOrigin, newVarValue);
>>>>        ncFile.write("var1", appendOrigin, tailVar1);
>>>>               assertEquals(3, 
>>>> ncFile.findDimension("records").getLength());
>>>>           }
>>>>
>>>> }
>>>>
>>>>
>>>>     
>>>>         
>>> _______________________________________________
>>> netcdf-java mailing list
>>> netcdf-java at unidata.ucar.edu
>>> For list information or to unsubscribe, visit: 
>>> http://www.unidata.ucar.edu/mailing_lists/   
>>>       
>
>
> _______________________________________________
> netcdf-java mailing list
> netcdf-java at unidata.ucar.edu
> For list information or to unsubscribe, visit: http://www.unidata.ucar.edu/mailing_lists/ 
>