Multi-GPU support

[iotamudelta]

Currently, we do not support multi-GPU on ROCm, nor do we assume it works right now. This issue is tracking insights and progress as we are trying to enable this.

[Jorghi12]

The development of RCCL appears partially functional. Exciting!

[sergey-serebryakov]

Hi!
Was trying to run PyTorch with eight MI25s (single node) and found this comment and then this thread. Do you have any estimates when PyTorch might get multi-GPU support on ROCm platform?
Thanks.

[iotamudelta]

@sergey-serebryakov We are actively working on adding multi-GPU support. We hope to have news on experimental support soon!

[sergey-serebryakov]

@iotamudelta Thank you! Looking forward to it.

[iotamudelta]

Closing, mGPU is supported now.