Skip to content

Enable ROCm CI support. #1260

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 17 commits into
base: main
Choose a base branch
from
Draft

Enable ROCm CI support. #1260

wants to merge 17 commits into from
+58 −14

Conversation

akashveramd
Copy link
Collaborator

@akashveramd akashveramd commented Jun 4, 2025 •
edited
Loading

In this PR-

  • Added support for creating ROCm docker image for torchtian and running torchtitan tests on ROCm.
  • Added ROCm CI support for integration_test unit test.

@facebook-github-bot
Copy link

Hi @akashveramd!

Thank you for your pull request and welcome to our community.

Action Required

In order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you.

Process

In order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA.

Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with CLA signed. The tagging process may take up to 1 hour after signing. Please give it that time before contacting us about it.

If you have received this in error or have any questions, please contact us at [email protected]. Thanks!

@akashveramd akashveramd changed the title Added support to run torchtitan tests on ROCm. Added support for creating ROCm docker image for torchtian & run torchtitan tests on ROCm. Jun 4, 2025
@akashveramd akashveramd marked this pull request as draft June 4, 2025 05:45
@facebook-github-bot
Copy link

Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Meta Open Source project. Thanks!

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Meta Open Source bot. label Jun 5, 2025
@akashveramd akashveramd changed the title Added support for creating ROCm docker image for torchtian & run torchtitan tests on ROCm. Added support for creating ROCm docker image for torchtitan & run torchtitan tests on ROCm. Jun 12, 2025
@akashveramd akashveramd changed the title Added support for creating ROCm docker image for torchtitan & run torchtitan tests on ROCm. Add support for creating ROCm docker image for torchtitan & run torchtitan tests on ROCm. Jun 12, 2025
@akashveramd akashveramd force-pushed the torchtitan_rocm_support_main branch from ebaafdc to ce60b15 Compare June 12, 2025 07:32
@akashveramd akashveramd changed the title Add support for creating ROCm docker image for torchtitan & run torchtitan tests on ROCm. Add support for creating ROCm docker image for torchtitan & enable ROCm CI support. Jun 13, 2025
jithunnair-amd
jithunnair-amd reviewed Jun 18, 2025
View reviewed changes
fegin
fegin reviewed Jun 23, 2025
View reviewed changes
@pytorch-bot PyTorch Bot
Copy link

pytorch-bot bot commented Jun 24, 2025

No ciflow labels are configured for this repo.
For information on how to enable CIFlow bot see this wiki

jithunnair-amd
jithunnair-amd suggested changes Jun 24, 2025
View reviewed changes
tests/integration_tests.py
TEST_WITH_ROCM
and test_flavor.test_name in skip_for_rocm_test_list
):
continue
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This logic makes sense to me, but if we really want to use the test setting in integration_tests_h100.py, we should move this logic to that file (and of course rename it to be more agnostic).

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jithunnair-amd : All tests in integration_tests_h100.py passes for rocm. Hence, we don't TEST_WITH_ROCM in integration_tests_h100.py. However, we need to talk about renaming integration_tests_h100.py filename as we also run it on rocm runners.
cc: @tianyu-l @fegin

@jithunnair-amd jithunnair-amd mentioned this pull request Jun 25, 2025
Draft
@akashveramd akashveramd changed the title Add support for creating ROCm docker image for torchtitan & enable ROCm CI support. Enable ROCm CI support. Jun 25, 2025
akashveramd added 12 commits July 1, 2025 16:34
@akashveramd
Added support to run torchtitan tests on ROCm.
4074d0c
@akashveramd
Added rocm ci support for integration_test_h100.
340478a
@akashveramd
Fixed a bug in build script. Removed ubuntu-cuda folder, instead usin…

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode
51427a7 
…g ubuntu folder for cuda Dockerfile.
@akashveramd
Included test in integration_tests.py after rebase.
2848d51
@akashveramd
Modified docker-builds.yml to build rocm docker image for torchtitan.
cb13ad4
@akashveramd
Fixed runner for cuda and rocm images in docker-builds.yml.
de9bdcc
@akashveramd
Added TEST_WITH_ROCM environment variable for running tests on rocm. …
f634f00 
…Fixed error in integration_tests.py. Fixed lint errors.
@akashveramd
Refactored integration_tests.py with skip tests for ROCm.
87a5a59
@akashveramd
Changed runner to i-0962598bd0e8298b3 for building ROCm docker image.
d748586
@akashveramd
Changed runner to linux.12xlarge for building ROCm docker image.
66eba9f
@akashveramd
Changed runner to linux.2xlarge for building ROCm docker image.
2d317c3
@akashveramd
Added support to use single Dockerfile for both cuda and rocm. Using …
Loading
Loading status checks…
18025ad 
…single workflow file to run integration tests including h100 for both cuda and rocm. Need different name for integration_test_8gpu_h100.yaml as we are also running it for rocm. Fixed file permission for integration_tests.py.
@akashveramd akashveramd force-pushed the torchtitan_rocm_support_main branch from d527f27 to 18025ad Compare July 3, 2025 05:15
@akashveramd
Changed rocm docker image name in docker-builds.yml.
Loading
Loading status checks…
724e202
@jithunnair-amd
Copy link

@huydhn Need your help in creating a new docker repo for torchtitan ROCm docker image: https://github.com/pytorch/torchtitan/actions/runs/16042425274/job/45266420732?pr=1260#step:7:1436

name unknown: The repository with name 'torchtitan/torchtitan-rocm-ubuntu-22.04-clang12' does not exist in the registry with id '308535385114'

@akashveramd
Reverted the changes to integration_test_8gpu_h100.yaml.
Loading
Loading status checks…
15a9554
@jithunnair-amd jithunnair-amd mentioned this pull request Jul 15, 2025
Open
@akashveramd
Empty dummy commit.
Loading
Loading status checks…
cb528bc
@akashveramd
Increased the timeout to 45 minutes to override timeout used in linux…
Loading
Loading status checks…
66e5c95 
…_job_v2.yml for integration_test_8gpu.yaml.
@akashveramd
Empty dummy commit.
Loading
Loading status checks…
efd11a8
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@fegin fegin

@tianyu-l tianyu-l

@jithunnair-amd jithunnair-amd

@wwwjn wwwjn

@wconstab wconstab

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
ciflow/rocm CLA Signed This label is managed by the Meta Open Source bot. module: rocm
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
5 participants
@akashveramd @facebook-github-bot @jithunnair-amd @fegin @tianyu-l