UCX Fail in osu_put_bibw

[lappazos]

openucx/ucc#284

[gpaulsen]

Is this a dup of #8086? Or more specific to UCX?

[jsquyres]

Can you please report the full details of the bug here instead of just a link to a closed UCX bug?

[awlauria]

From the linked defect, it looks like a possibly related osc/ucx failure with MPI_Win_post()

Configuration

OMPI: v5.0.0a1
MOFED: MLNX_OFED_LINUX-5.4-1.0.3.0
Module: none
Test module: none
Nodes: jazz x12 (ppn=28(x12), nodelist=jazz[12-21,29-30])
 
MTT log:
http://hpcweb.lab.mtl.com/hpc/mtr_scrap/users/mtt/scratch/ucc/20210802_170250_189628_54877_jazz12.swx.labs.mlnx/html/test_stdout_Ccnq4w.txt
 
Cmd:
/hpc/mtr_scrap/users/mtt/scratch/ucc/20210802_170250_189628_54877_jazz12.swx.labs.mlnx/ompi_src/install/bin/mpirun -np 2 --display map --mca coll_ucc_enable 1 --mca coll_ucc_priority 100 -x UCC_TL_UCP_TUNE=allreduce:1 --map-by node --bind-to core /hpc/mtr_scrap/users/mtt/scratch/ucc/20210802_170250_189628_54877_jazz12.swx.labs.mlnx/installs/OSS4/tests/osu_micro_benchmark/osu-micro-benchmarks-5.6.2/mpi/one-sided/osu_put_bibw
 
Output:

# OSU MPI_Put Bi-directional Bandwidth Test v5.6.2
# Window creation: MPI_Win_allocate
# Synchronization: MPI_Win_post/start/complete/wait
# Size      Bandwidth (MB/s)[jazz13.swx.labs.mlnx:163928] ../../../../opal/mca/common/ucx/common_ucx_wpool.h:526  Error: ucp_atomic_cswap64 failed: -1
[jazz13:00000] *** An error occurred in MPI_Win_post
[jazz13:00000] *** reported by process [1362690049,1]
[jazz13:00000] *** on win ucx window 3
[jazz13:00000] *** MPI_ERR_OTHER: known error not in list
[jazz13:00000] *** MPI_ERRORS_ARE_FATAL (processes in this win will now abort,
[jazz13:00000] ***    and MPI will try to terminate your MPI job as well)+ rc=16

[jsquyres]

FYI @open-mpi/ucx

@awlauria @bwbarrett Is this part of the same bucket of one-sided issues that were discussed on the call this past Tuesday?

[gpaulsen]

@janjust Is this fixed yet?

[janjust]

@gpaulsen not yet - it's an issue with the MPI_Win_post/start/complete/wait synchronization.
@MamziB is on it

[janjust]

Fixed with: #10126