Re: [netcdf-java] Thread safe read access

  • To: Hanns Holger Rutz <rutz@xxxxxx>
  • Subject: Re: [netcdf-java] Thread safe read access
  • From: Hanns Holger Rutz <rutz@xxxxxx>
  • Date: Tue, 23 Apr 2013 18:38:40 +0200
ah.... actually I was synchronising on the _variable_ and that is obviously not 
sufficient. I corrected it now to synchronise on the _nc2.NetcdfFile_, and the 
exception is gone at least now, so hopefully that does the isolation.

i'm still interested in some information regarding recommendations in 
concurrent setups.

best, .h.h.



On 23 Apr 2013, at 18:32, Hanns Holger Rutz wrote:

> hi there,
> 
> I'm having a question regarding thread safety and concurrency with the netcdf 
> java library.
> 
> basically i'm running a thread that scans a large file to calculate overall 
> statistics, and then when i try to plot that file in the meantime, i get 
> spurious errors like this:
> 
> Exception in thread "AWT-EventQueue-0" 
> java.lang.ArrayIndexOutOfBoundsException: -41846
>       at ucar.unidata.io.RandomAccessFile.read(RandomAccessFile.java:538)
>       at ucar.unidata.io.RandomAccessFile.readInt(RandomAccessFile.java:1134)
>       at ucar.unidata.io.RandomAccessFile.readLong(RandomAccessFile.java:1218)
>       at 
> ucar.unidata.io.RandomAccessFile.readDouble(RandomAccessFile.java:1325)
>       at ucar.nc2.iosp.IospHelper.readData(IospHelper.java:153)
>       at ucar.nc2.iosp.IospHelper.readDataFill(IospHelper.java:79)
>       at ucar.nc2.iosp.netcdf3.N3raf.readData(N3raf.java:63)
>       at ucar.nc2.iosp.netcdf3.N3iosp.readData(N3iosp.java:453)
>       at ucar.nc2.NetcdfFile.readData(NetcdfFile.java:1941)
>       at ucar.nc2.Variable.reallyRead(Variable.java:859)
>       at ucar.nc2.Variable._read(Variable.java:831)
>       at ucar.nc2.Variable.read(Variable.java:709)
>       at at.iem.sysson.Implicits$SyRichVariable.readSafe(Implicits.scala:111)
> 
> The `readSafe` in the last line actually uses a `synchronized` lock on the 
> file, so at least I'm making sure that no two threads are calling 
> `Variable.read` at the same time. But still it seems that something is messed 
> up.
> 
> So are there any pitfalls in this approach that could result in mutable state 
> being mixed up when using the file from multiple threads? Is there a 
> technique to ensure consistency (other than locking calls to Variable.read)?
> 
> I have a vague feeling that the problem might be that read doesn't actually 
> read the whole data, but ma2.Array uses some cache and runs actual I/O after 
> Variable.read has completed. I'm using iterators on the array with 
> `getNextFloat` and `getNextDouble`. Do I need to isolate them as well somehow?
> 
> Thanks a lot!
> 
> Best regards,
> 
> 
> Hanns Holger Rutz
> 
> 
> 
> 
> ----
> University of Music and Performing Arts – KUG
> Institute of Electronic Music and Acoustics – IEM
> Inffeldgasse 10/III
> 8010 Graz
> Austria
> 
> 
> 
> _______________________________________________
> netcdf-java mailing list
> netcdf-java@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe, visit: 
> http://www.unidata.ucar.edu/mailing_lists/ 

----
University of Music and Performing Arts – KUG
Institute of Electronic Music and Acoustics – IEM
Inffeldgasse 10/III
8010 Graz
Austria





  • 2013 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the netcdf-java archives: