Hello,
I'm hoping I can get some pointers to improve the way im using the NetCDF
library.
At the moment the processing I'm doing is writing to several different NetCDF
files, multiple variables a row at a time. These are not currently
multi-threaded.
When the processed data is small I don't see any issues (100's of rows),
however when I start running a bigger chain (10's of thousands of rows) I see
the performance of NetCDF Java plummet, a quick look at whats happening with
VisualVM shows that most of my application times (~60%) is spent in
`Nc4Iosp.writeData()`.
Which leads me to believe I'm using the library wrong :), my initial thought
having worked with the C Library directly before was to adjust the write
buffer, but I don't see any support for that in the Java lib and considering it
would likely affect the C Lib I'm not sure it would help with the write data
call.
I had briefly looked into just buffering my rows so I write every 10-100 rows
to see what effect that would have on performance and memory usage, however I
hit a bit of an issue with the variables that have an unlimited dimension of
columns (most variables I have are row x column), in that I was unable to
figure out how to create an Array that supported unlimited dimensions.
We currently use the NetcdfFileWriter to writer data to the underlying NetCDF 4
files, I know the API suggests using the FileWriter2, but I couldn't see a way
to use that, that also allowed us to 'stream' data into the underlying files.
Any suggestions would be greatly appreciated.
Thanks,
Robin
WARNING: This message contains confidential and/or proprietary information
which may be subject to privilege or immunity and which is intended for the use
of its addressee only. Should you receive this message in error, you are kindly
requested to inform the sender and to definitively remove it from any paper or
electronic format. Any other use of this e-mail is strictly forbidden. Thank
you in advance for your cooperation.
Please consider the environment before printing this email.
