btl/smcuda: Add atomic_wmb() before sm_fifo_write

This change fixes #12270

Testing on c7g instance type (arm64) confirms this change elminates
hangs and crashes that were previously observed in 1 in 30 runs of
IMB alltoall benchmark.  Tested with over 300 runs and no failures.

The write memory barrier prevents other CPUs from observing the fifo
get updated before they observe the updated contents of the header
itself.  Without the barrier, uninitialized header contents caused
the crashes and invalid data.

Signed-off-by: Luke Robison <[email protected]>

Author: lrbison

opal/mca/btl/smcuda/btl_smcuda_fifo.h

@@ -85,6 +85,8 @@ static void add_pending(struct mca_btl_base_endpoint_t *ep, void *data, bool res
 #define MCA_BTL_SMCUDA_FIFO_WRITE(endpoint_peer, my_smp_rank, peer_smp_rank, hdr, resend,         \
                                   retry_pending_sends, rc)                                        \
     do {                                                                                          \
+        /* memory barrier: ensure writes to the hdr have completed */                             \
+        opal_atomic_wmb();                                                                        \
         sm_fifo_t *_fifo = &(mca_btl_smcuda_component.fifo[peer_smp_rank][FIFO_MAP(my_smp_rank)]);\
                                                                                                   \
         if (retry_pending_sends) {                                                                \