<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
Ok. Could you clarify the version number? I'm on 2.2.22 now - you
mentioned 2.22.14 just now. Is there that much difference between the
two?<br>
<br>
<br>
<br>
John Caron wrote:
<blockquote cite="mid476951BE.60208@xxxxxxxxxxxxxxxx" type="cite">release
2.22.14 should fix this problem. be sure to call flush() after writing
and before reading
<br>
<br>
Nick Bower wrote:
<br>
<blockquote type="cite">Adding an ncFile.flush() to the submitted
test case as suggested does not resolve the problem. I think it needs
to be marked as a bug until a documented way around it surfaces.
<br>
<br>
<br>
Ethan Davis wrote:
<br>
<blockquote type="cite">Hi Nick,
<br>
<br>
I'm not sure if this is a recommended way but you might try to flush()
the NetcdfFileWriteable before reading.
<br>
<br>
John might have another answer. He is out of the office this week so
his email may be spotty.
<br>
<br>
Hope that helps,
<br>
<br>
Ethan
<br>
<br>
Nick Bower wrote:
<br>
<blockquote type="cite">Hello. Is the lack of response here
because this is not considered a bug? Is there a another recommended
way perhaps to safely increase a dimension and invalidate the memory
cache in the process?
<br>
<br>
Nick
<br>
<br>
<br>
Nick Bower wrote:
<br>
<br>
<blockquote type="cite">I've found a problem in which
cache/memory and disk shape information about variables will disagree
with v2.2.22 of Java Netcdf library.
<br>
<br>
When you add a new value to a variable, automatically increasing the
length of a dimension, subsequent reads can throw EOFException because
RandomAccessFile is instructed to read more values than the file
contains - the cached and actual shapes disagree.
<br>
<br>
I've created a runnable test case below to explain and demonstrate
success and failure conditions.
<br>
<br>
I am getting around this now by not interleaving read/write operations
on variables, but instead reading all variables' data to memory, then
performing any writes I need to after.
<br>
<br>
TestInsertRecord.java:
<br>
<br>
<br>
package com.metoceanengineers.datafeeds.netcdf.test;
<br>
<br>
import java.io.File;
<br>
import java.io.IOException;
<br>
import java.text.DateFormat;
<br>
import java.text.SimpleDateFormat;
<br>
<br>
import junit.framework.TestCase;
<br>
import ucar.ma2.Array;
<br>
import ucar.ma2.ArrayInt;
<br>
import ucar.ma2.DataType;
<br>
import ucar.nc2.Dimension;
<br>
import ucar.nc2.NetcdfFileWriteable;
<br>
<br>
public class TestInsertRecord extends TestCase {
<br>
DateFormat dateFormat = new
SimpleDateFormat("yyyyMMdd HHMM");
<br>
protected NetcdfFileWriteable createNc(String
prefix) throws
IOException {
<br>
File mainline = File.createTempFile(prefix+"-", ".nc");
<br>
NetcdfFileWriteable mainlineNc =
NetcdfFileWriteable.createNew(mainline.getAbsolutePath(), false);
<br>
<br>
Dimension recordsDim =
mainlineNc.addUnlimitedDimension("records");
<br>
Dimension timeDims[] = {recordsDim};
<br>
Dimension var1Dims[] = {recordsDim}; // 1D
<br>
mainlineNc.addVariable("time", DataType.INT, timeDims);
<br>
mainlineNc.addVariable("var1",
DataType.INT, var1Dims);
<br>
<br>
mainlineNc.create();
<br>
return mainlineNc;
<br>
}
<br>
<br>
<br>
protected String getNcInstance() throws Exception {
<br>
<br>
NetcdfFileWriteable mainlineNc =
createNc("testfile");
<br>
int[] origin = {0};
<br>
ArrayInt.D1 timeArr = new ArrayInt.D1(2);
<br>
timeArr.set(0,
(int)dateFormat.parse("20071130
0924").getTime());
<br>
timeArr.set(1,
(int)dateFormat.parse("20071130
0926").getTime());
<br>
mainlineNc.write("time", origin, timeArr);
<br>
ArrayInt.D1 var1Arr = new ArrayInt.D1(2);
<br>
var1Arr.set(0, 10);
<br>
var1Arr.set(1, 12);
<br>
mainlineNc.write("var1", origin, var1Arr);
<br>
<br>
mainlineNc.close();
<br>
<br>
return mainlineNc.getLocation();
<br>
}
<br>
/**
<br>
* Append new data to end of existing variables.
<br>
*
<br>
* @throws Exception
<br>
*/
<br>
public void testAppendWorksOk() throws Exception {
<br>
String ncFilename = getNcInstance();
<br>
NetcdfFileWriteable ncFile =
NetcdfFileWriteable.openExisting(ncFilename, false);
<br>
<br>
/*
<br>
* Append value (20071130 0924, 11)
into (time, var1)
<br>
*/
<br>
ArrayInt.D1 newTimeValue = new ArrayInt.D1(1);
<br>
newTimeValue.set(0,
(int)dateFormat.parse("20071130
0925").getTime());
<br>
<br>
ArrayInt.D1 newVarValue = new
ArrayInt.D1(1);
<br>
newVarValue.set(0, 11);
<br>
int[] origin = {2};
<br>
<br>
/* The first write will expand the
variables,
<br>
* but second write ok as we're just
writing
<br>
* and not reading */
<br>
ncFile.write("time", origin, newTimeValue);
<br>
ncFile.write("var1", origin, newVarValue);
<br>
assertEquals(3,
ncFile.findDimension("records").getLength());
<br>
}
<br>
<br>
/**
<br>
* Test insertion of a record in between the 2 existing
<br>
* records by reading the existing tail, inserting new data
<br>
* and re-appending.
<br>
*
<br>
* Triggers EOFException through interleaved read/writes
<br>
*
<br>
* @throws Exception
<br>
*/
<br>
public void testInsertFails() throws Exception {
<br>
String ncFilename = getNcInstance();
<br>
NetcdfFileWriteable ncFile =
NetcdfFileWriteable.openExisting(ncFilename, false);
<br>
<br>
ArrayInt.D1 newTimeValue = new
ArrayInt.D1(1);
<br>
newTimeValue.set(0,
(int)dateFormat.parse("20071130
0925").getTime());
<br>
<br>
ArrayInt.D1 newVarValue = new
ArrayInt.D1(1);
<br>
newVarValue.set(0, 11);
<br>
/* Going to insert at 1, so read existing value,
<br>
* write down new one, and re-append
old tail.
<br>
*/
<br>
int[] insertPointOrigin = {1};
<br>
int[] appendOrigin = {2};
<br>
int[] shape = {1};
<br>
Array tailTime =
ncFile.findVariable("time").read(insertPointOrigin, shape);
<br>
ncFile.write("time", insertPointOrigin,
newTimeValue);
<br>
ncFile.write("time", appendOrigin,
tailTime);
<br>
/* Next line excepts - why? Because the last write above
at
<br>
* records index 2 triggers an
increase in the CACHED/MEMORY
<br>
* length of all variables to 3, but
on disk it's still the
<br>
* original length 2.
<br>
*
<br>
* Therefore we get
EOFException.
<br>
*/
<br>
Array tailVar1 =
ncFile.findVariable("var1").read(insertPointOrigin, shape);
<br>
ncFile.write("var1", insertPointOrigin,
newVarValue);
<br>
ncFile.write("var1", appendOrigin,
tailVar1);
<br>
assertEquals(3,
ncFile.findDimension("records").getLength());
<br>
}
<br>
<br>
}
<br>
<br>
<br>
</blockquote>
_______________________________________________
<br>
netcdf-java mailing list
<br>
<a class="moz-txt-link-abbreviated"
href="mailto:netcdf-java@xxxxxxxxxxxxxxxx";>netcdf-java@xxxxxxxxxxxxxxxx</a>
<br>
For list information or to unsubscribe, visit:
<a class="moz-txt-link-freetext"
href="http://www.unidata.ucar.edu/mailing_lists/";>http://www.unidata.ucar.edu/mailing_lists/</a>
</blockquote>
</blockquote>
<br>
<br>
_______________________________________________
<br>
netcdf-java mailing list
<br>
<a class="moz-txt-link-abbreviated"
href="mailto:netcdf-java@xxxxxxxxxxxxxxxx";>netcdf-java@xxxxxxxxxxxxxxxx</a>
<br>
For list information or to unsubscribe, visit:
<a class="moz-txt-link-freetext"
href="http://www.unidata.ucar.edu/mailing_lists/";>http://www.unidata.ucar.edu/mailing_lists/</a>
</blockquote>
</blockquote>
<br>
<br>
<div class="moz-signature">-- <br>
<p><span style="font-size: 10pt; font-family: Verdana,sans-serif;">Regards,
Nick</span><br>
<span style="color: rgb(204, 204,
204);">____________________________________</span></p>
<p><span style="font-size: 10pt; font-family: Verdana,sans-serif;">Dr.
Nicholas Bower</span>
<span style="font-size: 8pt; font-family: Verdana,sans-serif;">Ph.D.<br>
<span
style="font-size: 8pt; font-family: Verdana,sans-serif; font-style:
italic;">Senior
Software Developer/Integrator, IT</span><br>
<br>
<a
href="mailto:nick.bower@xxxxxxxxxxxxxxxxxxxxx";>nick.bower@xxxxxxxxxxxxxxxxxxxxx</a></span></p>
<img style="border: 1px solid black;"
src="cid:part1.07080401.04090103@metoceanengineers.com" alt="MOELogo"><br>
<p><span style="font-size: 10pt; font-family: Verdana,sans-serif;"><b>RPS
Metocean Engineers</b><br>
31 Bishop Street, Jolimont WA 6014<br>
<b>Phone:</b> +61 8 9387 7955 <b>Fax:</b> +61 8 9387
6686</span></p>
</div>
</body>
</html>
