Use one source of truth for NCCL environment variables #152
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
msimberg
requested review from
boeschf,
Madeeks,
bcumming and
RMeli
as code owners
|
preview available: https://docs.tds.cscs.ch/152 |
RMeli
reviewed
Jun 11, 2025
|
preview available: https://docs.tds.cscs.ch/152 |
msimberg
force-pushed
the
sync-nccl-pytorch-env-vars
branch
from
e3dd98c to
610f010
Compare
|
preview available: https://docs.tds.cscs.ch/152 |
bcumming
reviewed
Jun 12, 2025
| export FI_CXI_DEFAULT_TX_SIZE=32768 | ||
| export FI_CXI_RX_MATCH_MODE=software | ||
| # (8)! | ||
| --8<-- "docs/software/communication/nccl_env_vars" |
|
preview available: https://docs.tds.cscs.ch/152 |
RMeli
approved these changes
Jun 12, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Based on #146 (comment). This adds NCCL environment variables to a separate file and includes it in both the NCCL and pytorch sections.
@boeschf @RMeli could you let me know what you think of this? The bash comments are definitely uglier, but perhaps not too bad. They show up in both sections. I've removed the explicit
MPICH_GPU_SUPPORT_ENABLED=0from the NCCL side, but it's still mentioned in a warning (as before). The pytorch side still hasMPICH_GPU_SUPPORT_ENABLED=0explicitly.