Regression of osc_rdma_acc_single_intrinsic

[devreal]

There seems to be a regression in Open MPI's implementation of atomic operations if the MCA parameter osc_rdma_acc_single_intrinsic is set to true: the attached test case results in what appears to be random results whereas without the MCA parameter the results meet the expectations.

Example runs:

$ mpirun -n 2 -N 1 ./mpi_fetch_op_local_remote
result:1000

$ mpirun -n 2 -N 1 -mca osc_rdma_acc_single_intrinsic true ./mpi_fetch_op_local_remote
result:1015
mpi_fetch_op_local_remote: mpi_fetch_op_local_remote.c:98: main: Assertion `sum == 1000*(comm_size-1)' failed.

mpi_fetch_op_local_remote.tar.gz

Built with:

$ mpicc mpi_fetch_op_local_remote.c -o mpi_fetch_op_local_remote

I just tested this with the 4.0.0 release. Setting the parameter with 3.1.2 works as expected.
This problem was observed on both a Cray XC40 and an IB-based cluster. Interestingly, the issue is only present if the local rank performs atomic updates (subtracting and readding a value), otherwise everything is fine.

I first reported this on the user ML but lost track of it: https://www.mail-archive.com/users@lists.open-mpi.org/msg32834.html

[ggouaillardet]

can you please give the just released 4.0.1 a try ?

[devreal]

Same problem with 4.0.1

[devreal]

I think I found the culprit: the check for !ompi_osc_rdma_peer_local_base (peer) in https://github.com/open-mpi/ompi/blob/master/ompi/mca/osc/rdma/osc_rdma_accumulate.c#L899 is not sufficient to determine whether or not the NIC can be used/avoided. Will post a PR soon.