[netcdf-hdf] netcdf4 and OpenMP

[Quincey Koziol]

Hi Charlie,

On May 23, 2008, at 8:42 AM, Charlie Zender wrote:

> Hi Quincey,
>
>>> 1. Are there benefits to building HDF without --enable-threadsafe?
>>    As Orion mentioned, the C++ and FORTRAN wrappers aren't  
>> currently compatible with the threadsafe option.  I don't think it  
>> would be too hard to address this, but it's probably enough work  
>> that we should look for some funding to do it.
>
> I hope HDF finds the resources to keep the C and Fortran API at  
> equivalent levels of functionality.

	We do try, yes.  For instance the upcoming 1.8.1 release will have  
many new FORTRAN wrappers for new 1.8 API routines.

>>    The only other downside really is the increased overhead for  
>> each HDF5 API call (to perform the semaphore locking that allows  
>> threadsafety).
>
> Can you post or point me to numbers that quantify this?

	I don't have any, sorry.

>>> If not, can you make it the HDF/netCDF4 default? at least
>>> on multi-core systems?
>> That's probably difficult for us to detect at configure time.  :-/
>
> Yes, easier said than done.
> One might try imitating how packages like MPICH2 handle this.

	Hmm, I'm not familiar with how MPICH2 works for this.  Do you have a  
pointer?

>>> We have learned, I think, to disable our progams' (NCO's)
>>> threading unless it is linked to threadsafe HDF.
>>> Otherwise users will experience unpredictable NCO failure.
>>>
>>> 2. How should we test, at NCO compile time, whether the
>>> underlying netCDF4/HDF install is threadsafe?
>>    The "H5_HAVE_THREADSAFE" macro will be defined when the "hdf5.h"  
>> header is included and threadsafety is enabled.
>
> Good. This token will help us optimize NCO when built from source.
>
> I wanted our NCO programs to have threading available for general
> users but the configuration issues seem to dictate otherwise.
> For instance, we distribute NCO .debs and RPMs now with threading
> enabled by default because it "just works" with libnetcdf3.a.
> With netCDF4, however, pre-built binaries like debs and RPMs
> will depend on the distribution's libnetcdf4 .deb and RPM
> which likely will not be built on top of HDF --threadsafe.
>
> One outcome is that NCO users who rely on .debs and RPMs over custom
> installs will lose NCO threading on (at least) netCDF4 files.
> That is actually not
> a big deal since threading only speeds up math-heavy jobs (ncwa,
> ncap2) because I/O bottlenecks and threading overhead combine to
> slow down I/O intensive jobs (ncdiff, ncra,..). It's possible
> we may force OMP_NUM_THREADS=1 at run-time when input
> or output files are netCDF4 format, and only leave threading
> enabled for netCDF3 I/O in pre-built executables.

	Even when a threadsafe HDF5 is used, and NCO is only reading from  
netCDF-4 files, you could still have problems for multi-threaded  
access.  For instance, there are probably some global data structures  
in netCDF-4 that won't be protected by a semaphore and could get  
corrupted.  Maybe Ed can think about this and offer an opinion...

	Quincey


> Charlie
> -- 
> Charlie Zender, Department of Earth System Science, UC Irvine
> Sab. at CNRS/LGGE-Grenoble until 20080815 :) 011+33+476+824236
> Laboratoire de Glaciologie et Géophysique de l'Environnement
> 54 rue Molière BP 96, 38402 Saint Martin d'Hères Cedex, France
>