Re: [netcdf-java] Variable shape problem & EOFException

sorry, the versioning is rather baroque. the bug is fixed in version >=
2.2.22.14

theres a readme file inside each jar that has this number in it, in case
theres any doubt

Nick Bower wrote:
> Ok.  Could you clarify the version number?  I'm on 2.2.22 now - you mentioned 
> 2.22.14 just now.  Is there that much difference between the two?
> 
> 
> John Caron wrote:
>> release 2.22.14 should fix this problem. be sure to call flush() after 
>> writing and before reading
>>
>> Nick Bower wrote:
>>> Adding an ncFile.flush() to the submitted test case as suggested does 
>>> not resolve the problem.  I think it needs to be marked as a bug 
>>> until a documented way around it surfaces.
>>>
>>>
>>> Ethan Davis wrote:
>>>  
>>>> Hi Nick,
>>>>
>>>> I'm not sure if this is a recommended way but you might try to 
>>>> flush() the NetcdfFileWriteable before reading.
>>>>
>>>> John might have another answer. He is out of the office this week so 
>>>> his email may be spotty.
>>>>
>>>> Hope that helps,
>>>>
>>>> Ethan
>>>>
>>>> Nick Bower wrote:
>>>>    
>>>>> Hello.  Is the lack of response here because this is not considered 
>>>>> a bug?  Is there a another recommended way perhaps to safely 
>>>>> increase a dimension and invalidate the memory cache in the process?
>>>>>
>>>>> Nick
>>>>>
>>>>>
>>>>> Nick Bower wrote:
>>>>>  
>>>>>      
>>>>>> I've found a problem in which cache/memory and disk shape 
>>>>>> information about variables will disagree with v2.2.22 of Java 
>>>>>> Netcdf library.
>>>>>>
>>>>>> When you add a new value to a variable, automatically increasing 
>>>>>> the length of a dimension, subsequent reads can throw EOFException 
>>>>>> because RandomAccessFile is instructed to read more values than 
>>>>>> the file contains - the cached and actual shapes disagree.
>>>>>>
>>>>>> I've created a runnable test case below to explain and demonstrate 
>>>>>> success and failure conditions.
>>>>>>
>>>>>> I am getting around this now by not interleaving read/write 
>>>>>> operations on variables, but instead reading all variables' data 
>>>>>> to memory, then performing any writes I need to after.
>>>>>>
>>>>>> TestInsertRecord.java:
>>>>>>
>>>>>>
>>>>>> package com.metoceanengineers.datafeeds.netcdf.test;
>>>>>>
>>>>>> import java.io.File;
>>>>>> import java.io.IOException;
>>>>>> import java.text.DateFormat;
>>>>>> import java.text.SimpleDateFormat;
>>>>>>
>>>>>> import junit.framework.TestCase;
>>>>>> import ucar.ma2.Array;
>>>>>> import ucar.ma2.ArrayInt;
>>>>>> import ucar.ma2.DataType;
>>>>>> import ucar.nc2.Dimension;
>>>>>> import ucar.nc2.NetcdfFileWriteable;
>>>>>>
>>>>>> public class TestInsertRecord extends TestCase {
>>>>>>       DateFormat dateFormat = new SimpleDateFormat("yyyyMMdd HHMM");
>>>>>>       protected NetcdfFileWriteable createNc(String prefix) throws 
>>>>>> IOException {
>>>>>>               File mainline = File.createTempFile(prefix+"-", ".nc");
>>>>>>        NetcdfFileWriteable mainlineNc = 
>>>>>> NetcdfFileWriteable.createNew(mainline.getAbsolutePath(), false);
>>>>>>
>>>>>>        Dimension recordsDim = 
>>>>>> mainlineNc.addUnlimitedDimension("records");
>>>>>>               Dimension timeDims[] = {recordsDim};
>>>>>>        Dimension var1Dims[] = {recordsDim}; // 1D
>>>>>>               mainlineNc.addVariable("time", DataType.INT, timeDims);
>>>>>>        mainlineNc.addVariable("var1", DataType.INT, var1Dims);
>>>>>>
>>>>>>        mainlineNc.create();
>>>>>>        return mainlineNc;
>>>>>>    }
>>>>>>
>>>>>>
>>>>>>    protected String getNcInstance() throws Exception {
>>>>>>
>>>>>>        NetcdfFileWriteable mainlineNc = createNc("testfile");
>>>>>>               int[] origin = {0};
>>>>>>               ArrayInt.D1 timeArr = new ArrayInt.D1(2);
>>>>>>        timeArr.set(0, (int)dateFormat.parse("20071130 
>>>>>> 0924").getTime());
>>>>>>        timeArr.set(1, (int)dateFormat.parse("20071130 
>>>>>> 0926").getTime());
>>>>>>        mainlineNc.write("time", origin, timeArr);
>>>>>>               ArrayInt.D1 var1Arr = new ArrayInt.D1(2);
>>>>>>        var1Arr.set(0, 10);
>>>>>>        var1Arr.set(1, 12);
>>>>>>        mainlineNc.write("var1", origin, var1Arr);
>>>>>>
>>>>>>        mainlineNc.close();
>>>>>>
>>>>>>        return mainlineNc.getLocation();
>>>>>>    }
>>>>>>      /**
>>>>>>     * Append new data to end of existing variables.
>>>>>>     *
>>>>>>     * @throws Exception
>>>>>>     */
>>>>>>    public void testAppendWorksOk() throws Exception {
>>>>>>               String ncFilename = getNcInstance();
>>>>>>        NetcdfFileWriteable ncFile = 
>>>>>> NetcdfFileWriteable.openExisting(ncFilename, false);
>>>>>>
>>>>>>        /*
>>>>>>         * Append value (20071130 0924, 11) into (time, var1)
>>>>>>         */
>>>>>>               ArrayInt.D1 newTimeValue = new ArrayInt.D1(1);
>>>>>>        newTimeValue.set(0, (int)dateFormat.parse("20071130 
>>>>>> 0925").getTime());
>>>>>>
>>>>>>        ArrayInt.D1 newVarValue = new ArrayInt.D1(1);
>>>>>>        newVarValue.set(0, 11);
>>>>>>               int[] origin = {2};
>>>>>>
>>>>>>        /* The first write will expand the variables,
>>>>>>         * but second write ok as we're just writing
>>>>>>         * and not reading */
>>>>>>               ncFile.write("time", origin, newTimeValue);
>>>>>>        ncFile.write("var1", origin, newVarValue);
>>>>>>               assertEquals(3, 
>>>>>> ncFile.findDimension("records").getLength());
>>>>>>           }
>>>>>>
>>>>>>    /**
>>>>>>     * Test insertion of a record in between the 2 existing
>>>>>>     * records by reading the existing tail, inserting new data
>>>>>>     * and re-appending.
>>>>>>     *
>>>>>>     * Triggers EOFException through interleaved read/writes
>>>>>>     *
>>>>>>     * @throws Exception
>>>>>>     */
>>>>>>    public void testInsertFails() throws Exception {
>>>>>>               String ncFilename = getNcInstance();
>>>>>>        NetcdfFileWriteable ncFile = 
>>>>>> NetcdfFileWriteable.openExisting(ncFilename, false);
>>>>>>
>>>>>>        ArrayInt.D1 newTimeValue = new ArrayInt.D1(1);
>>>>>>        newTimeValue.set(0, (int)dateFormat.parse("20071130 
>>>>>> 0925").getTime());
>>>>>>
>>>>>>        ArrayInt.D1 newVarValue = new ArrayInt.D1(1);
>>>>>>        newVarValue.set(0, 11);
>>>>>>               /* Going to insert at 1, so read existing value,
>>>>>>         * write down new one, and re-append old tail.
>>>>>>         */
>>>>>>               int[] insertPointOrigin = {1};
>>>>>>        int[] appendOrigin = {2};
>>>>>>        int[] shape = {1};
>>>>>>               Array tailTime = 
>>>>>> ncFile.findVariable("time").read(insertPointOrigin, shape);
>>>>>>        ncFile.write("time", insertPointOrigin, newTimeValue);
>>>>>>        ncFile.write("time", appendOrigin, tailTime);
>>>>>>               /* Next line excepts - why?  Because the last write 
>>>>>> above at
>>>>>>         * records index 2 triggers an increase in the CACHED/MEMORY
>>>>>>         * length of all variables to 3, but on disk it's still the
>>>>>>         * original length 2.
>>>>>>         *
>>>>>>         *  Therefore we get EOFException.
>>>>>>         */
>>>>>>               Array tailVar1 = 
>>>>>> ncFile.findVariable("var1").read(insertPointOrigin, shape);
>>>>>>        ncFile.write("var1", insertPointOrigin, newVarValue);
>>>>>>        ncFile.write("var1", appendOrigin, tailVar1);
>>>>>>               assertEquals(3, 
>>>>>> ncFile.findDimension("records").getLength());
>>>>>>           }
>>>>>>
>>>>>> }
>>>>>>
>>>>>>
>>>>>>             
>>>>> _______________________________________________
>>>>> netcdf-java mailing list
>>>>> netcdf-java@xxxxxxxxxxxxxxxx
>>>>> For list information or to unsubscribe, visit: 
>>>>> http://www.unidata.ucar.edu/mailing_lists/         
>>>
>>> _______________________________________________
>>> netcdf-java mailing list
>>> netcdf-java@xxxxxxxxxxxxxxxx
>>> For list information or to unsubscribe, visit: 
>>> http://www.unidata.ucar.edu/mailing_lists/   
> 
> _______________________________________________
> netcdf-java mailing list
> netcdf-java@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe, visit: 
> http://www.unidata.ucar.edu/mailing_lists/