Due to the current gap in continued funding from the U.S. National Science Foundation (NSF), the NSF Unidata Program Center has temporarily paused most operations. See NSF Unidata Pause in Most Operations for details.
Due to the current gap in continued funding from the U.S. National Science Foundation (NSF), the NSF Unidata Program Center has temporarily paused most operations. See NSF Unidata Pause in Most Operations for details.
Hi Russ, thanks for your answer and the explanation from the hdfgroup.For our purposes, it is not the parallel IO, which is important, but the bottleneck is in many cases zlib. Are there other ways to parallelize the 'zlib' portion of netcdf/hdf?
HeikoPS: Concerning unclear error messages: When linking a non-parallel netcdf-library with a program using nc_create_par, I got a error message like: "Parallel operation on file opened for non-parallel access." That message didn't help me to understand that I forgot to set LD_LIBRARY_PATH.
On 2015-01-06 15:46, Russ Rew wrote:
Heiko, HDF5 supports reading compressed data in parallel, but cannot write compressed data in parallel. The netCDF library should return a more specific error than "HDF5 error" in this case. Here's an explanation <http://www.hdfgroup.org/hdf5-quest.html#p5comp> for why HDF5 doesn't write compressed data in parallel. --Russ On Tue, Jan 6, 2015 at 5:26 AM, Heiko Klein <Heiko.Klein@xxxxxx <mailto:Heiko.Klein@xxxxxx>> wrote: Hi, we are usually using compressed netcdf4 files for our forecast meteorology files. This changes our bottleneck from IO to CPU, since compression (usually deflate level 3) takes some time. To speed this up, I intended to use the mpi-enabled parallel netcdf4/hdf5. I've been trying with netcdf 4.3.0 and netcdf 4.3.2 and hdf5-1.8.12 and hdf5-1.8.11 (I tried 1.8.13 and 1.8.14, too, but that didn't compile because of removed MPIPOSIX in hdf5). In all cases, I managed to get parallel netcdf working, but I never managed to create a deflated variable with parallel netcdf. As soon as I set something like nc_def_var_deflate(ncid, varid, 1, 1, 3) I get a 'HDF5 error' when I call nc_enddef the next (first) time. Removing the deflate command, everything works again. Please find attached your tst_parallel4 test-file with the added nc_def_var_deflate command. Best regards, Heiko -- Dr. Heiko Klein Tel. + 47 22 96 32 58 <tel:%2B%2047%2022%2096%2032%2058> Development Section / IT Department Fax. + 47 22 69 63 55 <tel:%2B%2047%2022%2069%2063%2055> Norwegian Meteorological Institute http://www.met.no P.O. Box 43 Blindern 0313 Oslo NORWAY _______________________________________________ netcdfgroup mailing list netcdfgroup@xxxxxxxxxxxxxxxx <mailto:netcdfgroup@xxxxxxxxxxxxxxxx> For list information or to unsubscribe, visit: http://www.unidata.ucar.edu/mailing_lists/
-- Dr. Heiko Klein Tel. + 47 22 96 32 58 Development Section / IT Department Fax. + 47 22 69 63 55 Norwegian Meteorological Institute http://www.met.no P.O. Box 43 Blindern 0313 Oslo NORWAY
netcdfgroup archives: