[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[netCDF #LTC-671596]: Multithreaded IO with libnetcdf4

Unfortunately, the netcdf library is not thread safe
because it depends on the HDF5 library, which is not thread safe.
It is unknown if accessing different variables in different
threads will work correctly.
The only possibility I can see is as follows.
1. create a thread pool
2. use large chunk sizes for your variables
3. inside your compression code, you can use threads from the pool
to compress pieces of each chunk that is passed to you.
In other words only use threading inside of your own compression code.


> 
> We have recently started uding compression features of NetCDF4.
> It gives substantial gain in space at expence of CPU resources
> on compression/decompression. For few other applications we
> use multithreaded compression/decompression to cope with compression
> costs. In some cases it turns out to be also faster than dealing with
> uncompressed data due to disk io.
> 
> Are NF90_PUT_VAR and NF90_GET_VAR functions thread-safe if called for
> different variables?
> 
> Or is there any way to distribute the compression/decompression overhead
> in Netcdf4 over few threads?
> 
> Thank you!
> 
> BR,
> Rostislav
> 
> 
> 
> --
> Rostislav Kouznetsov
> 
> **************************************
> Atmospheric composition unit
> Finnish Meteorological Institute
> Erik Palmenin aukio 1, 00560
> (PL-503 FI-00101)
> Helsinki Finland
> Phone: +358 29 539 4630
> **************************************
> 
> 

=Dennis Heimbigner
  Unidata


Ticket Details
===================
Ticket ID: LTC-671596
Department: Support netCDF
Priority: Normal
Status: Open
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.