accelerator framework: cuda component assumes application initializes CUDA prior to calling MPI_init

[hppritcha]

I'm not sure if this is a bug or a feature, but in testing the accelerator framework work on a system with Nvidia GPUS, I noticed that the osu_mbw_mr test (the 5.8 version of OSU benchmarks) fails using cuda allocated buffers. The problem is that the cuda component. doesn't get initialized until after calling MPI_Init, so the address checker code is not employed. The other mpi/pt2pt tests initialize CUDA before calling MPI_Init and don't see this problem.

I've not tested the rocm component, so don't know if it would have the same issue.

[hppritcha]

@edgargabriel the system i use for rocm testing is down for an extended time. You may want to see if the osu_mbw_mr test works okay with the rocm component.

[edgargabriel]

@hppritcha the test passes for rocm memory. Not sure whether it makes a difference, I am using OSU benchmarks version 5.9 though.

[hppritcha]

my rocm machine came back on line and I confirmed what you @edgargabriel observed that osu_mbw_mr with D D args works nominally.

[janjust]

@Akshay-Venkatesh @open-mpi/cuda Guys, can you chime in on this issue.
Is it a requirement for cuda init init to be always initialized BEFORE MPI_Init? The new cuda framework changed the delayed initialization from delayed to after MPI_Init and that causes issues for some benchmarks, and certainly will cause issues for some users.

[Akshay-Venkatesh]

Is it a requirement for cuda init init to be always initialized BEFORE MPI_Init?

@janjust It shouldn't be a requirement.

I've tested mbw_mr test with ompi-5x and UCX and haven't encountered an issue.

@hppritcha Did you use UCX pml or something else? What command do you run and what exact cuda errors are you seeing?

FWIW, in UCX, we don't require cuda device initialization before MPI_Init but it is recommended as it helps with some topology decisions.

[janjust]

@edgargabriel

FWIW, in UCX, we don't require cuda device initialization before MPI_Init but it is recommended as it helps with some topology decisions.

I wonder if this is what you were referring to.

[wckzhang]

@Akshay-Venkatesh The reason for this is because some OMPI initialization code relies being within a cuda context. Previously there was a delayed initialization for some items like stream creation for asynchronous memcpy's. We can re-add that delayed initialization

[wckzhang]

PR to delay initialization: #11253

[hppritcha]

closed via #11253