"Charlie Zender" <zender@xxxxxxx> writes:
> Hi Quincey,
>
>>> Ideas/suggestions as to what is going wrong or how to debug this
>>> problem would be much appreciated.
>>
>> Are you using a threadsafe version of HDF5? (i.e. one that is
>> configured with the "--enable-threadsafe" option) It's also possible
>> that netCDF-4 needs some locking mechanisms also, but that's a
>> question for Ed or Russ.
>
> Happy to report that your suggestion appears to solve our problem.
> ncbo performs as expected if, and, on SMP systems, only if, we
> build HDF5 with --enable-threadsafe. We are still testing to
> make sure there are no regressions on our other netCDF operators.
> But things look much better, and no re-coding necessary!
>
> This begs some questions:
>
> 1. Are there benefits to building HDF without --enable-threadsafe?
>
> If not, can you make it the HDF/netCDF4 default? at least
> on multi-core systems?
>
> We have learned, I think, to disable our progams' (NCO's)
> threading unless it is linked to threadsafe HDF.
> Otherwise users will experience unpredictable NCO failure.
>
> 2. How should we test, at NCO compile time, whether the
> underlying netCDF4/HDF install is threadsafe?
>
> Thanks,
> Charlie
Howdy Charlie!
You are scaring me!
Neither netcdf-3 nor netcdf-4 are known to be threadsafe. But the
reason you are not seeing this is that you are probably only doing
reads (and writes?) of data, and not any metadata operations.
(For example, if a thread tried to add a variable, the other threads
wouldn't see it until there some kind of sync step.)
However, when reading data, all that is required is the calculation of
the correct offsets from the metadata, so multiple threads can handle
this.
So just because threadsafe operation is possible with HDF5, does not
mean that netcdf-4 is threadsafe!
(We have discussed making it threadsafe, but we haven't done it yet.)
So proceed with caution in using netCDF in a threaded environment.
I think anything that is done in define mode should be pretty safe,
but I'm not sure what happens with record variables. If you add a
record, that changes the metadata, and the other threads won't see
that...
Thanks,
Ed
--
Ed Hartnett -- ed@xxxxxxxxxxxxxxxx
