[v5.0.x] ompi/dpm: make procs consistent before calling PMIx_Connect() #10564
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ompi_dpm_connect_accept() call PMIx_Connect() to establish connection, with "procs" as an argument. PMIx requires "procs" to be consistent accross clients. When it is used to set up inter-communicator communication, ompi_dpm_connect_accept() does not maintain the order of proc in "procs". This is because the function is called by both MPI_Comm_connect() and MPI_Comm_accept(), and it always put processes in local communicator in "procs" first, followed by processes in remote communicator. However, for caller of MPI_Comm_connect() and MPI_Comm_accept(), local communicator and remote communicator are different. This patch fixed the issue by sorting "procs" before it is used to call PMIx_Connect(), this ensures that "procs" are consistent accross processes. Signed-off-by: Wei Zhang <[email protected]> (cherry picked from commit 0294303)
|
back port #10557 to v5.0.x |
hppritcha
approved these changes
Jul 13, 2022
|
hmm looks like azure is unhappy |
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
looks like docker issues at mlnx: Error response from daemon: Get http://rdmz-harbor.rdmz.labs.mlnx/v2/hpcx/ompi_ci/manifests/latest: received unexpected HTTP status: 500 Internal Server Error |
|
bot:mellanox:retest |
|
The mellanox CI issue seems to be persistent. I tried a few times, and it always fail. |
jjhursey
approved these changes
Jul 13, 2022
|
Mellanox CI was also having trouble last week. @artemry-nv can you take a look? |
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
Some issues with the container registry - working on this. |
|
Mellanox CI fixed itself and passed. @awlauria Can you merge this PR? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
ompi_dpm_connect_accept() call PMIx_Connect() to establish
connection, with "procs" as an argument.
PMIx requires "procs" to be consistent accross clients.
When it is used to set up inter-communicator communication,
ompi_dpm_connect_accept() does not maintain the order of
proc in "procs". This is because the function is called
by both MPI_Comm_connect() and MPI_Comm_accept(), and
it always put processes in local communicator in "procs"
first, followed by processes in remote communicator.
However, for caller of MPI_Comm_connect() and MPI_Comm_accept(),
local communicator and remote communicator are different.
This patch fixed the issue by sorting "procs" before
it is used to call PMIx_Connect(), this ensures that "procs"
are consistent accross processes.
Signed-off-by: Wei Zhang [email protected]
(cherry picked from commit 0294303)