Fix incorrect TCP connections. #12259
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
This patch improves #12232 but does not solve it. |
bosilca
force-pushed
the
fix/tcp_connection_s
branch
from
f146bf9 to
5e5db44
Compare
|
Can I get a review on this please. |
devreal
reviewed
Mar 5, 2024
Comment on lines
+1507
to
+1509
| opal_show_help("help-mpi-btl-tcp.txt", "server cannot accept connection from self", true, | ||
| peer, OPAL_NAME_PRINT(guid), | ||
| opal_process_info.nodename, getpid()); |
There was a problem hiding this comment.
This seems like a recoverable case so do we always want to print this message?
We print a message for this when we handle the connection: https://github.com/open-mpi/ompi/pull/12259/files#diff-cc4089aecdcf0cb98dd15472b23f1a3f2af00357a180f986cdba0c89738472dcR462
There was a problem hiding this comment.
We can handle with this patch, but it is a misconfiguration and the user should fix in future runs. The other side is a verbose output, it will only show when the BTL verbosity is set to 20.
opal/mca/btl/tcp/btl_tcp_endpoint.c
Outdated
Comment on lines
794
to
801
| char tmp[2][16]; | ||
| inet_ntop(AF_INET, &((struct sockaddr_in *)&btl_endpoint->endpoint_btl->tcp_ifaddr)->sin_addr, tmp[0], 16); | ||
| inet_ntop(AF_INET, &((struct sockaddr_in *)&endpoint_addr)->sin_addr, tmp[1], 16); | ||
| opal_output(0, "proc %s bind socket to %s:%d before connecting to peer %s at %s:%d\n", | ||
| OPAL_NAME_PRINT(OPAL_PROC_MY_NAME), | ||
| tmp[0], htons(((struct sockaddr_in *) &btl_endpoint->endpoint_btl->tcp_ifaddr)->sin_port), | ||
| OPAL_NAME_PRINT(btl_endpoint->endpoint_proc->proc_opal->proc_name), | ||
| tmp[1], ntohs(((struct sockaddr_in *) &endpoint_addr)->sin_port)); |
There was a problem hiding this comment.
Probably lower the verbosity? (assuming 0 prints always?)
opal/mca/btl/tcp/btl_tcp_proc.c
Outdated
| #if OPAL_ENABLE_IPV6 | ||
| else if (MCA_BTL_TCP_AF_INET6 == remote_addrs[i].addr_family && | ||
| AF_INET6 == local_iter->af_family) { | ||
| if (!memcmp(&((struct sockaddr_in6 *) &local_iter->if_addr)->sin6_addr, |
bosilca
commented
Mar 5, 2024
If nodes have the same IP addresses (for containers or other purposes) and these addresses get published as part of the modex, a remote peer might try to use one of the addresses to connect. As both nodes have the same IP, there are several cases: - the "remote" port is not used by an OMPI process locally, the connection is refused or it timeouts. This is the "nicest" outcome, as a new IP will be used resulting in a successful connection and the continuation of the application. - the "remote" port is used by another OMPI process on the local node. A connection will be established but the incorrect guid will be exchanged leading to complaints, connection dropped and/or deadlocks. - the "remote" port is used by this process, basically resulting in a connection-to-self. Bad things happen, as we don't support TCP connections to self. Some output messages are generated, but the outcome is most likely a deadlock. Up to now, users were expected to exclude such interfaces from the accepted interfaces, but this patch removes this need. If we discover a local IP as part of the IP list of a remote peer, we drop it and never try to use it. This does not apply to local processes, so we can still use these interfaces for node level communications (which will work as we will connect to the correct port according to the destination process). Signed-off-by: George Bosilca <[email protected]>
bosilca
force-pushed
the
fix/tcp_connection_s
branch
from
934289b to
c50329f
Compare
devreal
approved these changes
Mar 6, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
If nodes have the same IP addresses (for containers or other purposes) and these addresses get published as part of the modex, a remote peer might try to use one of the addresses to connect. As both nodes have the same IP, there are several cases:
Up to now, users were expected to exclude such interfaces from the accepted interfaces, but this patch removes this need. If we discover a local IP as part of the IP list of a remote peer, we drop it and never try to use it. This does not apply to local processes, so we can still use these interfaces for node level communications (which will work as we will connect to the correct port according to the destination process).