Use Docker for Bazel Build #1232
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,43 @@ | ||
| ARG BASE_IMG=ubuntu:18.04 | ||
| FROM ${BASE_IMG} as dev-base | ||
|
|
||
| ARG ARCH="x86_64" | ||
| ARG TARGETARCH="amd64" | ||
| ARG BAZEL_VERSION=4.2.1 | ||
|
|
||
| # Install basic packages | ||
| RUN apt-get update && \ | ||
| apt-get install -y \ | ||
| python3.8 \ | ||
| python3.8-dev \ | ||
| cmake \ | ||
| ccache \ | ||
| ninja-build \ | ||
| git \ | ||
| python3-pip \ | ||
| wget \ | ||
| clang-10 \ | ||
| automake \ | ||
| libtool \ | ||
| curl \ | ||
| make \ | ||
| unzip | ||
|
|
||
| RUN update-alternatives --install /usr/bin/python python /usr/bin/python3.8 10 | ||
| RUN update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.8 10 | ||
|
|
||
| RUN update-alternatives --install /usr/bin/clang clang /usr/bin/clang-10 10 | ||
| RUN update-alternatives --install /usr/bin/clang++ clang++ /usr/bin/clang++-10 10 | ||
|
Comment on lines
+29
to
+30
There was a problem hiding this comment. @asaadaldien with this we can avoid having to configure both bazel and cmake builds to pick the specific version. |
||
|
|
||
| # Install bazel | ||
| RUN wget -q https://github.com/bazelbuild/bazel/releases/download/${BAZEL_VERSION}/bazel-${BAZEL_VERSION}-linux-${ARCH} -O /usr/bin/bazel \ | ||
| && chmod a+x /usr/bin/bazel | ||
|
|
||
| COPY externals/llvm-project/mlir/python/requirements.txt /opt/app/mlir-requirements.txt | ||
| COPY requirements.txt /opt/app/torch-mlir-requirements.txt | ||
| WORKDIR /opt/app | ||
| RUN python -m pip install --upgrade pip | ||
| RUN python -m pip install -r mlir-requirements.txt | ||
| RUN python -m pip install -r torch-mlir-requirements.txt | ||
|
|
||
| WORKDIR /opt/src/torch-mlir | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| #!/usr/bin/env bash | ||
|
|
||
| cd "$(pwd)/utils/bazel" && bazel build @torch-mlir//... |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,27 @@ | ||
| #!/usr/bin/env bash | ||
|
|
||
| torch_binary="${TORCH_BINARY:-ON}" | ||
|
|
||
|
|
||
| # Configure cmake to build torch-mlir in-tree | ||
| cmake -GNinja -Bbuild \ | ||
| -DCMAKE_BUILD_TYPE=Release \ | ||
| -DCMAKE_C_COMPILER=clang \ | ||
| -DCMAKE_CXX_COMPILER=clang++ \ | ||
| -DCMAKE_C_COMPILER_LAUNCHER=ccache \ | ||
| -DCMAKE_CXX_COMPILER_LAUNCHER=ccache \ | ||
| -DCMAKE_LINKER=lld \ | ||
| -DLLVM_ENABLE_ASSERTIONS=ON \ | ||
| -DLLVM_ENABLE_PROJECTS=mlir \ | ||
| -DLLVM_EXTERNAL_PROJECTS="torch-mlir;torch-mlir-dialects" \ | ||
| -DLLVM_EXTERNAL_TORCH_MLIR_SOURCE_DIR="$(pwd)" \ | ||
| -DLLVM_EXTERNAL_TORCH_MLIR_DIALECTS_SOURCE_DIR="$(pwd)/externals/llvm-external-projects/torch-mlir-dialects" \ | ||
| -DLLVM_TARGETS_TO_BUILD=host \ | ||
| -DMLIR_ENABLE_BINDINGS_PYTHON=ON \ | ||
| -DTORCH_MLIR_ENABLE_LTC=ON \ | ||
| -DTORCH_MLIR_USE_INSTALLED_PYTORCH="${torch_binary}" \ | ||
| -DPython3_EXECUTABLE="$(which python)" \ | ||
| externals/llvm-project/llvm | ||
|
|
||
| # Build torch-mlir | ||
| cmake --build build |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,39 @@ | ||
| #!/usr/bin/env bash | ||
|
|
||
| torch_binary="${TORCH_BINARY:-ON}" | ||
|
|
||
|
|
||
| # Configure cmake to build torch-mlir out-of-tree | ||
| cmake -GNinja -Bllvm-build \ | ||
| -DCMAKE_BUILD_TYPE=Release \ | ||
| -DCMAKE_C_COMPILER=clang \ | ||
| -DCMAKE_CXX_COMPILER=clang++ \ | ||
| -DCMAKE_C_COMPILER_LAUNCHER=ccache \ | ||
| -DCMAKE_CXX_COMPILER_LAUNCHER=ccache \ | ||
| -DCMAKE_LINKER=lld \ | ||
| -DLLVM_ENABLE_ASSERTIONS=ON \ | ||
| -DLLVM_ENABLE_PROJECTS=mlir \ | ||
| -DLLVM_TARGETS_TO_BUILD=host \ | ||
| -DMLIR_ENABLE_BINDINGS_PYTHON=ON \ | ||
| -DPython3_EXECUTABLE="$(which python)" \ | ||
| externals/llvm-project/llvm | ||
|
|
||
| # Build llvm | ||
| cmake --build llvm-build | ||
|
|
||
| # TODO: Reenable LTC once OOT build is successful (https://github.com/llvm/torch-mlir/issues/1154) | ||
| cmake -GNinja -Bbuild \ | ||
| -DCMAKE_C_COMPILER=clang \ | ||
| -DCMAKE_CXX_COMPILER=clang++ \ | ||
| -DCMAKE_C_COMPILER_LAUNCHER=ccache \ | ||
| -DCMAKE_CXX_COMPILER_LAUNCHER=ccache \ | ||
| -DCMAKE_LINKER=lld \ | ||
| -DLLVM_DIR="$(pwd)/llvm-build/lib/cmake/llvm/" \ | ||
| -DMLIR_DIR="$(pwd)/llvm-build/lib/cmake/mlir/" \ | ||
| -DMLIR_ENABLE_BINDINGS_PYTHON=OFF \ | ||
| -DTORCH_MLIR_USE_INSTALLED_PYTORCH="${torch_binary}" \ | ||
| -DPython3_EXECUTABLE="$(which python)" \ | ||
| . | ||
|
|
||
| # Build torch-mlir | ||
| cmake --build build |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,10 @@ | ||
| #!/usr/bin/env bash | ||
|
|
||
| docker build -f build_tools/docker/Dockerfile \ | ||
| -t torch-mlir-cmake:dev \ | ||
| . | ||
|
|
||
| docker run -it \ | ||
| -v "$(pwd)":/opt/src/torch-mlir \ | ||
| -e CCACHE_DIR=/opt/src/torch-mlir/.ccache \ | ||
| torch-mlir-cmake:dev |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,15 @@ | ||
| #!/usr/bin/env bash | ||
|
|
||
| export PYTHONPATH="$(pwd)/build/tools/torch-mlir/python_packages/torch_mlir" | ||
|
|
||
| # refbackend e2e tests | ||
| python -m e2e_testing.torchscript.main --config=refbackend -v -s | ||
|
|
||
| # eagermode backend e2e tests | ||
| python -m e2e_testing.torchscript.main --config=eager_mode -v -s | ||
|
|
||
| # tosa backend e2e tests | ||
| python -m e2e_testing.torchscript.main --config=tosa -v -s | ||
|
|
||
| # ltc backend e2e tests | ||
| python -m e2e_testing.torchscript.main --config=lazy_tensor_core -v -s |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,13 @@ | ||
| #!/usr/bin/env bash | ||
|
|
||
| # Full suite | ||
| # cmake --build build --target check-torch-mlir-all | ||
|
|
||
| # Run torch-mlir unit tests. | ||
| cmake --build build --target check-torch-mlir | ||
|
|
||
| # Run torch-mlir-python unit tests. | ||
| cmake --build build --target check-torch-mlir-python | ||
|
|
||
| # Run torch-mlir-dialects unit tests. | ||
| cmake --build build --target check-torch-mlir-dialects |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@asaadaldien PTAL. Previously we'd decided to not send this upstream because clang on GHA VMs was already new, but with Ubuntu 18 container the default clang is old and fails with the AVX errors.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actually, this breaks CMake workflow because it is looking for
clangand notclang-10:) Looking into a way to symbolic link so as to not update every-DCMAKE_C_COMPILER=clangwith-DCMAKE_C_COMPILER=clang-10.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks like this is what I need: https://man7.org/linux/man-pages/man1/update-alternatives.1.html (and we're already doing this for python).