[netcdf-java] Variable shape problem & EOFException
Nick Bower
nick.bower at metoceanengineers.com
Wed Dec 19 17:43:07 MST 2007
Ok. Could you clarify the version number? I'm on 2.2.22 now - you
mentioned 2.22.14 just now. Is there that much difference between the two?
John Caron wrote:
> release 2.22.14 should fix this problem. be sure to call flush() after
> writing and before reading
>
> Nick Bower wrote:
>> Adding an ncFile.flush() to the submitted test case as suggested does
>> not resolve the problem. I think it needs to be marked as a bug
>> until a documented way around it surfaces.
>>
>>
>> Ethan Davis wrote:
>>
>>> Hi Nick,
>>>
>>> I'm not sure if this is a recommended way but you might try to
>>> flush() the NetcdfFileWriteable before reading.
>>>
>>> John might have another answer. He is out of the office this week so
>>> his email may be spotty.
>>>
>>> Hope that helps,
>>>
>>> Ethan
>>>
>>> Nick Bower wrote:
>>>
>>>> Hello. Is the lack of response here because this is not considered
>>>> a bug? Is there a another recommended way perhaps to safely
>>>> increase a dimension and invalidate the memory cache in the process?
>>>>
>>>> Nick
>>>>
>>>>
>>>> Nick Bower wrote:
>>>>
>>>>
>>>>> I've found a problem in which cache/memory and disk shape
>>>>> information about variables will disagree with v2.2.22 of Java
>>>>> Netcdf library.
>>>>>
>>>>> When you add a new value to a variable, automatically increasing
>>>>> the length of a dimension, subsequent reads can throw EOFException
>>>>> because RandomAccessFile is instructed to read more values than
>>>>> the file contains - the cached and actual shapes disagree.
>>>>>
>>>>> I've created a runnable test case below to explain and demonstrate
>>>>> success and failure conditions.
>>>>>
>>>>> I am getting around this now by not interleaving read/write
>>>>> operations on variables, but instead reading all variables' data
>>>>> to memory, then performing any writes I need to after.
>>>>>
>>>>> TestInsertRecord.java:
>>>>>
>>>>>
>>>>> package com.metoceanengineers.datafeeds.netcdf.test;
>>>>>
>>>>> import java.io.File;
>>>>> import java.io.IOException;
>>>>> import java.text.DateFormat;
>>>>> import java.text.SimpleDateFormat;
>>>>>
>>>>> import junit.framework.TestCase;
>>>>> import ucar.ma2.Array;
>>>>> import ucar.ma2.ArrayInt;
>>>>> import ucar.ma2.DataType;
>>>>> import ucar.nc2.Dimension;
>>>>> import ucar.nc2.NetcdfFileWriteable;
>>>>>
>>>>> public class TestInsertRecord extends TestCase {
>>>>> DateFormat dateFormat = new SimpleDateFormat("yyyyMMdd HHMM");
>>>>> protected NetcdfFileWriteable createNc(String prefix) throws
>>>>> IOException {
>>>>> File mainline = File.createTempFile(prefix+"-", ".nc");
>>>>> NetcdfFileWriteable mainlineNc =
>>>>> NetcdfFileWriteable.createNew(mainline.getAbsolutePath(), false);
>>>>>
>>>>> Dimension recordsDim =
>>>>> mainlineNc.addUnlimitedDimension("records");
>>>>> Dimension timeDims[] = {recordsDim};
>>>>> Dimension var1Dims[] = {recordsDim}; // 1D
>>>>> mainlineNc.addVariable("time", DataType.INT, timeDims);
>>>>> mainlineNc.addVariable("var1", DataType.INT, var1Dims);
>>>>>
>>>>> mainlineNc.create();
>>>>> return mainlineNc;
>>>>> }
>>>>>
>>>>>
>>>>> protected String getNcInstance() throws Exception {
>>>>>
>>>>> NetcdfFileWriteable mainlineNc = createNc("testfile");
>>>>> int[] origin = {0};
>>>>> ArrayInt.D1 timeArr = new ArrayInt.D1(2);
>>>>> timeArr.set(0, (int)dateFormat.parse("20071130
>>>>> 0924").getTime());
>>>>> timeArr.set(1, (int)dateFormat.parse("20071130
>>>>> 0926").getTime());
>>>>> mainlineNc.write("time", origin, timeArr);
>>>>> ArrayInt.D1 var1Arr = new ArrayInt.D1(2);
>>>>> var1Arr.set(0, 10);
>>>>> var1Arr.set(1, 12);
>>>>> mainlineNc.write("var1", origin, var1Arr);
>>>>>
>>>>> mainlineNc.close();
>>>>>
>>>>> return mainlineNc.getLocation();
>>>>> }
>>>>> /**
>>>>> * Append new data to end of existing variables.
>>>>> *
>>>>> * @throws Exception
>>>>> */
>>>>> public void testAppendWorksOk() throws Exception {
>>>>> String ncFilename = getNcInstance();
>>>>> NetcdfFileWriteable ncFile =
>>>>> NetcdfFileWriteable.openExisting(ncFilename, false);
>>>>>
>>>>> /*
>>>>> * Append value (20071130 0924, 11) into (time, var1)
>>>>> */
>>>>> ArrayInt.D1 newTimeValue = new ArrayInt.D1(1);
>>>>> newTimeValue.set(0, (int)dateFormat.parse("20071130
>>>>> 0925").getTime());
>>>>>
>>>>> ArrayInt.D1 newVarValue = new ArrayInt.D1(1);
>>>>> newVarValue.set(0, 11);
>>>>> int[] origin = {2};
>>>>>
>>>>> /* The first write will expand the variables,
>>>>> * but second write ok as we're just writing
>>>>> * and not reading */
>>>>> ncFile.write("time", origin, newTimeValue);
>>>>> ncFile.write("var1", origin, newVarValue);
>>>>> assertEquals(3,
>>>>> ncFile.findDimension("records").getLength());
>>>>> }
>>>>>
>>>>> /**
>>>>> * Test insertion of a record in between the 2 existing
>>>>> * records by reading the existing tail, inserting new data
>>>>> * and re-appending.
>>>>> *
>>>>> * Triggers EOFException through interleaved read/writes
>>>>> *
>>>>> * @throws Exception
>>>>> */
>>>>> public void testInsertFails() throws Exception {
>>>>> String ncFilename = getNcInstance();
>>>>> NetcdfFileWriteable ncFile =
>>>>> NetcdfFileWriteable.openExisting(ncFilename, false);
>>>>>
>>>>> ArrayInt.D1 newTimeValue = new ArrayInt.D1(1);
>>>>> newTimeValue.set(0, (int)dateFormat.parse("20071130
>>>>> 0925").getTime());
>>>>>
>>>>> ArrayInt.D1 newVarValue = new ArrayInt.D1(1);
>>>>> newVarValue.set(0, 11);
>>>>> /* Going to insert at 1, so read existing value,
>>>>> * write down new one, and re-append old tail.
>>>>> */
>>>>> int[] insertPointOrigin = {1};
>>>>> int[] appendOrigin = {2};
>>>>> int[] shape = {1};
>>>>> Array tailTime =
>>>>> ncFile.findVariable("time").read(insertPointOrigin, shape);
>>>>> ncFile.write("time", insertPointOrigin, newTimeValue);
>>>>> ncFile.write("time", appendOrigin, tailTime);
>>>>> /* Next line excepts - why? Because the last write
>>>>> above at
>>>>> * records index 2 triggers an increase in the CACHED/MEMORY
>>>>> * length of all variables to 3, but on disk it's still the
>>>>> * original length 2.
>>>>> *
>>>>> * Therefore we get EOFException.
>>>>> */
>>>>> Array tailVar1 =
>>>>> ncFile.findVariable("var1").read(insertPointOrigin, shape);
>>>>> ncFile.write("var1", insertPointOrigin, newVarValue);
>>>>> ncFile.write("var1", appendOrigin, tailVar1);
>>>>> assertEquals(3,
>>>>> ncFile.findDimension("records").getLength());
>>>>> }
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>>
>>>> _______________________________________________
>>>> netcdf-java mailing list
>>>> netcdf-java at unidata.ucar.edu
>>>> For list information or to unsubscribe, visit:
>>>> http://www.unidata.ucar.edu/mailing_lists/
>>
>>
>> _______________________________________________
>> netcdf-java mailing list
>> netcdf-java at unidata.ucar.edu
>> For list information or to unsubscribe, visit:
>> http://www.unidata.ucar.edu/mailing_lists/
--
Regards, Nick
____________________________________
Dr. Nicholas Bower Ph.D.
Senior Software Developer/Integrator, IT
nick.bower at metoceanengineers.com <mailto:nick.bower at metoceanengineers.com>
MOELogo
*RPS Metocean Engineers*
31 Bishop Street, Jolimont WA 6014
*Phone:* +61 8 9387 7955 *Fax:* +61 8 9387 6686
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.unidata.ucar.edu/mailing_lists/archives/netcdf-java/attachments/20071220/b0c4f931/attachment-0001.html
-------------- next part --------------
A non-text attachment was scrubbed...
Name: metOceanHeader-small.jpg
Type: image/jpeg
Size: 4462 bytes
Desc: not available
Url : http://mailman.unidata.ucar.edu/mailing_lists/archives/netcdf-java/attachments/20071220/b0c4f931/attachment-0001.jpg
More information about the netcdf-java
mailing list