[netcdfgroup] 4.7.4 failing on multiple nodes

• To: netcdfgroup@xxxxxxxxxxxxxxxx
• Subject: [netcdfgroup] 4.7.4 failing on multiple nodes
• From: Ted Mansell <ted.mansell@xxxxxxxx>
• Date: Wed, 4 Nov 2020 19:07:52 -0600

Hi,

I thought I'd give netcdf 4.7.4 a try for the compression in parallel IO (using 
hdf5 1.10.7, pnetcdf 1.9.0, netcdf-fortran-4.5.3) on a NOAA cluster. I've been 
using intel 19 with mvapich2.3, which worked fine with earlier versions 
(4.3.something). So the problem I have is that it works fine on a single node, 
but get various failures when trying to run a job that uses 2 or more nodes. It 
also fails if the IO is not parallel (standard netcdf-4 where each process 
writes its data in turn).

I have also compiled everything (including cloud model code) using Intel MPI, 
which fails promptly with a seg fault when it tries to run on 2 nodes. (Here, I 
am comparing 4 or 9 threads on a single node or 16 threads split on 2 nodes. If 
I force the 16 thread version to run on a single node, it runs fine.)

The problem seems to be reproducible with a simple write/read test adapted from 
ftst_parallel.F, so it is seems not specific to my model code. Fails with both 
pnetcdf and mpiio

Any ideas what could be the issue here? I am stumped.

-- Ted

• Follow-Ups:
  • Re: [netcdfgroup] 4.7.4 failing on multiple nodes
    • From: Ed Hartnett