Re: [netcdf-java] Variable shape problem & EOFException

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
  <meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
Ok.&nbsp; Could you clarify the version number?&nbsp; I'm on 2.2.22 now - you
mentioned 2.22.14 just now.&nbsp; Is there that much difference between the
two?<br>
<br>
<br>
<br>
John Caron wrote:
<blockquote cite="mid476951BE.60208@xxxxxxxxxxxxxxxx" type="cite">release
2.22.14 should fix this problem. be sure to call flush() after writing
and before reading
  <br>
  <br>
Nick Bower wrote:
  <br>
  <blockquote type="cite">Adding an ncFile.flush() to the submitted
test case as suggested does not resolve the problem.&nbsp; I think it needs
to be marked as a bug until a documented way around it surfaces.
    <br>
    <br>
    <br>
Ethan Davis wrote:
    <br>
&nbsp;
    <blockquote type="cite">Hi Nick,
      <br>
      <br>
I'm not sure if this is a recommended way but you might try to flush()
the NetcdfFileWriteable before reading.
      <br>
      <br>
John might have another answer. He is out of the office this week so
his email may be spotty.
      <br>
      <br>
Hope that helps,
      <br>
      <br>
Ethan
      <br>
      <br>
Nick Bower wrote:
      <br>
&nbsp;&nbsp;&nbsp;
      <blockquote type="cite">Hello.&nbsp; Is the lack of response here
because this is not considered a bug?&nbsp; Is there a another recommended
way perhaps to safely increase a dimension and invalidate the memory
cache in the process?
        <br>
        <br>
Nick
        <br>
        <br>
        <br>
Nick Bower wrote:
        <br>
&nbsp;
        <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
        <blockquote type="cite">I've found a problem in which
cache/memory and disk shape information about variables will disagree
with v2.2.22 of Java Netcdf library.
          <br>
          <br>
When you add a new value to a variable, automatically increasing the
length of a dimension, subsequent reads can throw EOFException because
RandomAccessFile is instructed to read more values than the file
contains - the cached and actual shapes disagree.
          <br>
          <br>
I've created a runnable test case below to explain and demonstrate
success and failure conditions.
          <br>
          <br>
I am getting around this now by not interleaving read/write operations
on variables, but instead reading all variables' data to memory, then
performing any writes I need to after.
          <br>
          <br>
TestInsertRecord.java:
          <br>
          <br>
          <br>
package com.metoceanengineers.datafeeds.netcdf.test;
          <br>
          <br>
import java.io.File;
          <br>
import java.io.IOException;
          <br>
import java.text.DateFormat;
          <br>
import java.text.SimpleDateFormat;
          <br>
          <br>
import junit.framework.TestCase;
          <br>
import ucar.ma2.Array;
          <br>
import ucar.ma2.ArrayInt;
          <br>
import ucar.ma2.DataType;
          <br>
import ucar.nc2.Dimension;
          <br>
import ucar.nc2.NetcdfFileWriteable;
          <br>
          <br>
public class TestInsertRecord extends TestCase {
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; DateFormat dateFormat = new 
SimpleDateFormat("yyyyMMdd HHMM");
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; protected NetcdfFileWriteable createNc(String 
prefix) throws
IOException {
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
File mainline = File.createTempFile(prefix+"-", ".nc");
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; NetcdfFileWriteable mainlineNc =
NetcdfFileWriteable.createNew(mainline.getAbsolutePath(), false);
          <br>
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Dimension recordsDim =
mainlineNc.addUnlimitedDimension("records");
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
Dimension timeDims[] = {recordsDim};
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Dimension var1Dims[] = {recordsDim}; // 1D
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
mainlineNc.addVariable("time", DataType.INT, timeDims);
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; mainlineNc.addVariable("var1", 
DataType.INT, var1Dims);
          <br>
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; mainlineNc.create();
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; return mainlineNc;
          <br>
&nbsp;&nbsp; }
          <br>
          <br>
          <br>
&nbsp;&nbsp; protected String getNcInstance() throws Exception {
          <br>
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; NetcdfFileWriteable mainlineNc = 
createNc("testfile");
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
int[] origin = {0};
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
ArrayInt.D1 timeArr = new ArrayInt.D1(2);
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; timeArr.set(0, 
(int)dateFormat.parse("20071130
0924").getTime());
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; timeArr.set(1, 
(int)dateFormat.parse("20071130
0926").getTime());
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; mainlineNc.write("time", origin, timeArr);
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
ArrayInt.D1 var1Arr = new ArrayInt.D1(2);
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; var1Arr.set(0, 10);
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; var1Arr.set(1, 12);
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; mainlineNc.write("var1", origin, var1Arr);
          <br>
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; mainlineNc.close();
          <br>
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; return mainlineNc.getLocation();
          <br>
&nbsp;&nbsp; }
          <br>
&nbsp;&nbsp;&nbsp;&nbsp; /**
          <br>
&nbsp;&nbsp;&nbsp; * Append new data to end of existing variables.
          <br>
&nbsp;&nbsp;&nbsp; *
          <br>
&nbsp;&nbsp;&nbsp; * @throws Exception
          <br>
&nbsp;&nbsp;&nbsp; */
          <br>
&nbsp;&nbsp; public void testAppendWorksOk() throws Exception {
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
String ncFilename = getNcInstance();
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; NetcdfFileWriteable ncFile =
NetcdfFileWriteable.openExisting(ncFilename, false);
          <br>
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; /*
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; * Append value (20071130 0924, 11) 
into (time, var1)
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; */
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
ArrayInt.D1 newTimeValue = new ArrayInt.D1(1);
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; newTimeValue.set(0, 
(int)dateFormat.parse("20071130
0925").getTime());
          <br>
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ArrayInt.D1 newVarValue = new 
ArrayInt.D1(1);
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; newVarValue.set(0, 11);
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
int[] origin = {2};
          <br>
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; /* The first write will expand the 
variables,
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; * but second write ok as we're just 
writing
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; * and not reading */
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
ncFile.write("time", origin, newTimeValue);
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ncFile.write("var1", origin, newVarValue);
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
assertEquals(3,
ncFile.findDimension("records").getLength());
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; }
          <br>
          <br>
&nbsp;&nbsp; /**
          <br>
&nbsp;&nbsp;&nbsp; * Test insertion of a record in between the 2 existing
          <br>
&nbsp;&nbsp;&nbsp; * records by reading the existing tail, inserting new data
          <br>
&nbsp;&nbsp;&nbsp; * and re-appending.
          <br>
&nbsp;&nbsp;&nbsp; *
          <br>
&nbsp;&nbsp;&nbsp; * Triggers EOFException through interleaved read/writes
          <br>
&nbsp;&nbsp;&nbsp; *
          <br>
&nbsp;&nbsp;&nbsp; * @throws Exception
          <br>
&nbsp;&nbsp;&nbsp; */
          <br>
&nbsp;&nbsp; public void testInsertFails() throws Exception {
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
String ncFilename = getNcInstance();
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; NetcdfFileWriteable ncFile =
NetcdfFileWriteable.openExisting(ncFilename, false);
          <br>
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ArrayInt.D1 newTimeValue = new 
ArrayInt.D1(1);
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; newTimeValue.set(0, 
(int)dateFormat.parse("20071130
0925").getTime());
          <br>
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ArrayInt.D1 newVarValue = new 
ArrayInt.D1(1);
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; newVarValue.set(0, 11);
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
/* Going to insert at 1, so read existing value,
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; * write down new one, and re-append 
old tail.
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; */
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
int[] insertPointOrigin = {1};
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; int[] appendOrigin = {2};
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; int[] shape = {1};
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
Array tailTime =
ncFile.findVariable("time").read(insertPointOrigin, shape);
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ncFile.write("time", insertPointOrigin, 
newTimeValue);
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ncFile.write("time", appendOrigin, 
tailTime);
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
/* Next line excepts - why?&nbsp; Because the last write above
at
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; * records index 2 triggers an 
increase in the CACHED/MEMORY
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; * length of all variables to 3, but 
on disk it's still the
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; * original length 2.
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; *
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; *&nbsp; Therefore we get 
EOFException.
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; */
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
Array tailVar1 =
ncFile.findVariable("var1").read(insertPointOrigin, shape);
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ncFile.write("var1", insertPointOrigin, 
newVarValue);
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ncFile.write("var1", appendOrigin, 
tailVar1);
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
assertEquals(3,
ncFile.findDimension("records").getLength());
          <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; }
          <br>
          <br>
}
          <br>
          <br>
          <br>
&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </blockquote>
_______________________________________________
        <br>
netcdf-java mailing list
        <br>
<a class="moz-txt-link-abbreviated" 
href="mailto:netcdf-java@xxxxxxxxxxxxxxxx";>netcdf-java@xxxxxxxxxxxxxxxx</a>
        <br>
For list information or to unsubscribe, visit:
<a class="moz-txt-link-freetext" 
href="http://www.unidata.ucar.edu/mailing_lists/";>http://www.unidata.ucar.edu/mailing_lists/</a>&nbsp;&nbsp;
 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </blockquote>
    </blockquote>
    <br>
    <br>
_______________________________________________
    <br>
netcdf-java mailing list
    <br>
<a class="moz-txt-link-abbreviated" 
href="mailto:netcdf-java@xxxxxxxxxxxxxxxx";>netcdf-java@xxxxxxxxxxxxxxxx</a>
    <br>
For list information or to unsubscribe, visit:
<a class="moz-txt-link-freetext" 
href="http://www.unidata.ucar.edu/mailing_lists/";>http://www.unidata.ucar.edu/mailing_lists/</a>
 &nbsp; </blockquote>
</blockquote>
<br>
<br>
<div class="moz-signature">-- <br>
<p><span style="font-size: 10pt; font-family: Verdana,sans-serif;">Regards,
Nick</span><br>
<span style="color: rgb(204, 204, 
204);">____________________________________</span></p>
<p><span style="font-size: 10pt; font-family: Verdana,sans-serif;">Dr.
Nicholas Bower</span>
<span style="font-size: 8pt; font-family: Verdana,sans-serif;">Ph.D.<br>
<span
 style="font-size: 8pt; font-family: Verdana,sans-serif; font-style: 
italic;">Senior
Software Developer/Integrator, IT</span><br>
<br>
<a 
href="mailto:nick.bower@xxxxxxxxxxxxxxxxxxxxx";>nick.bower@xxxxxxxxxxxxxxxxxxxxx</a></span></p>
<img style="border: 1px solid black;"
 src="cid:part1.07080401.04090103@metoceanengineers.com" alt="MOELogo"><br>
<p><span style="font-size: 10pt; font-family: Verdana,sans-serif;"><b>RPS
Metocean Engineers</b><br>
31 Bishop Street, Jolimont WA 6014<br>
<b>Phone:</b> +61 8 9387 7955 &nbsp;&nbsp;&nbsp;<b>Fax:</b> +61 8 9387 
6686</span></p>
</div>
</body>
</html>

JPEG image