Skip to content

Add nightly build workflow #250

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 33 commits into from
May 20, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
d7e3176
Add nightly build workflow
huydhn May 16, 2024
bbb5da6
Fix wrong env script path
huydhn May 16, 2024
824a45c
Install pip deps
huydhn May 16, 2024
0f6baa5
Try to install gxx_linux-64 and gcc_linux-64
huydhn May 16, 2024
643d81f
Another attempt
huydhn May 16, 2024
2f60b9d
Set TORCH_CUDA_ARCH_LIST
huydhn May 17, 2024
c55b32d
Does this work?
huydhn May 17, 2024
e3beb78
Attempt to upload to pypi
huydhn May 17, 2024
9c74d3c
Pass the arch
huydhn May 17, 2024
3b244d7
Try pypa/gh-action-pypi-publish@release/v1
huydhn May 17, 2024
c3afbc8
Wrong workflow syntax
huydhn May 17, 2024
08ef87d
Try auditwheel
huydhn May 17, 2024
25f43d1
Repair wheel manylinux2014_x86_64
huydhn May 17, 2024
ed66ae3
Debug
huydhn May 17, 2024
d0459f4
Debug
huydhn May 17, 2024
740d917
Almost there
huydhn May 17, 2024
344f28b
Debug
huydhn May 17, 2024
b8848f9
Almost there
huydhn May 17, 2024
6f3c172
Pass -y to uninstall
huydhn May 17, 2024
057c4ca
Final testing
huydhn May 17, 2024
0bb6b6c
Upload other python versions
huydhn May 17, 2024
22a93c3
Remove debug
huydhn May 17, 2024
1ee576f
Remove build.yml
huydhn May 17, 2024
f91001e
Add more validations
huydhn May 18, 2024
61e8080
Merge branch 'main' into another-attempt-to-fix-nightly
huydhn May 18, 2024
ae2a904
Run all unit tests
huydhn May 18, 2024
5a7aad9
Fix typo
huydhn May 18, 2024
6dfe549
Run only test_ops
huydhn May 18, 2024
22a41ba
Passing the secrets
huydhn May 19, 2024
18bd9b1
Move pypi upload to test-infra
huydhn May 19, 2024
ea8ec8f
Enable CPU build
huydhn May 20, 2024
ed96ae3
Switch back to main after landing https://github.com/pytorch/test-inf…
huydhn May 20, 2024
8048bc7
Remove unrelated copy/paste comments
huydhn May 20, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
71 changes: 0 additions & 71 deletions .github/workflows/build.yml

This file was deleted.

45 changes: 45 additions & 0 deletions .github/workflows/build_wheels_linux.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
# From https://github.com/pytorch/test-infra/wiki/Using-Nova-Reusable-Build-Workflows
name: Build Linux Wheels

on:
pull_request:
paths:
- build/packaging/**
- .github/workflows/build_wheels_linux.yml
schedule:
- cron: '0 0 * * *' # Runs at midnight UTC every day
workflow_dispatch:

jobs:
generate-matrix:
uses: pytorch/test-infra/.github/workflows/generate_binary_build_matrix.yml@main
with:
package-type: wheel
os: linux
with-cpu: enable
with-cuda: enable
with-rocm: disable

build:
needs: generate-matrix
permissions:
id-token: write
contents: read
uses: pytorch/test-infra/.github/workflows/build_wheels_linux.yml@main
with:
# Set the ref to an empty string instead of the default nightly because
# torchao doesn't have nightly branch setup yet, instead the build is
# triggered daily from main with a schedule
repository: pytorch/ao
ref: ""
build-matrix: ${{ needs.generate-matrix.outputs.matrix }}
env-var-script: packaging/env_var_script_linux.sh
pre-script: packaging/pre_build_script.sh
post-script: packaging/post_build_script.sh
smoke-test-script: packaging/smoke_test.py
package-name: torchao
trigger-event: ${{ github.event_name }}
# This is the CUDA version to be uploaded to torchao-nightly pypi
upload-to-pypi: cu121
secrets:
PYPI_API_TOKEN: ${{ secrets.PYPI_API_TOKEN }}
19 changes: 19 additions & 0 deletions packaging/env_var_script_linux.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
# Copyright (c) Meta Platforms, Inc. and affiliates.
# All rights reserved.
#
# This source code is licensed under the BSD-style license found in the
# LICENSE file in the root directory of this source tree.

# This file is sourced into the environment before building a pip wheel. It
# should typically only contain shell variable assignments. Be sure to export
# any variables so that subprocesses will see them.
if [[ ${CHANNEL:-nightly} == "nightly" ]]; then
export TORCHAO_NIGHTLY=1
fi

# Set ARCH list so that we can build fp16 with SM75+, the logic is copied from
# pytorch/builder
TORCH_CUDA_ARCH_LIST="8.0;8.6"
if [[ ${CU_VERSION:-} == "cu124" ]]; then
TORCH_CUDA_ARCH_LIST="${TORCH_CUDA_ARCH_LIST};9.0"
fi
35 changes: 35 additions & 0 deletions packaging/post_build_script.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
#!/bin/bash
# Copyright (c) Meta Platforms, Inc. and affiliates.
# All rights reserved.
#
# This source code is licensed under the BSD-style license found in the
# LICENSE file in the root directory of this source tree.

set -eux

WHEEL_NAME=$(ls dist/)

pushd dist
# Prepare manywheel
auditwheel repair --plat manylinux2014_x86_64 -w . \
--exclude libtorch.so \
--exclude libtorch_python.so \
--exclude libtorch_cuda.so \
--exclude libtorch_cpu.so \
--exclude libc10.so \
--exclude libc10_cuda.so \
--exclude libcudart.so.12 \
--exclude libcudart.so.11.0 \
"${WHEEL_NAME}"

ls -lah .
# Clean up the linux_x86_64 wheel
rm "${WHEEL_NAME}"
popd

MANYWHEEL_NAME=$(ls dist/)
# Try to install the new wheel
pip install "dist/${MANYWHEEL_NAME}"
# and validating it by running the unit tests. Some tests are failing here and
# there, so let's add more of them later
pytest -v test/test_ops.py
14 changes: 14 additions & 0 deletions packaging/pre_build_script.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
#!/bin/bash
# Copyright (c) Meta Platforms, Inc. and affiliates.
# All rights reserved.
#
# This source code is licensed under the BSD-style license found in the
# LICENSE file in the root directory of this source tree.

set -eux

echo "This script is run before building torchao binaries"

pip install setuptools wheel twine auditwheel
pip install -r requirements.txt
pip install -r dev-requirements.txt
20 changes: 20 additions & 0 deletions packaging/smoke_test.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
#!/usr/bin/env python3
# Copyright (c) Meta Platforms, Inc. and affiliates.
# All rights reserved.
#
# This source code is licensed under the BSD-style license found in the
# LICENSE file in the root directory of this source tree.

import subprocess
import torchao.ops


def main():
"""
Run torchao binary smoke tests like importing and performing simple ops
"""
print(dir(torchao.ops))


if __name__ == "__main__":
main()
Loading