Make build environment manylinux2014 compatible #57

nitins17 · 2021-12-02T22:35:00Z

Modifies the build environment to be manylinux2014 compatible

Changes include:

Added devtoolset-9 to the Dockerfile
Changed to the manylinux2014 toolchain in {cpu,gpu}.bazelrc
Added manylinux2014 compatible glibc and libstdc++ libraries in devtoolset-9 (/dt9)
Use the new libstdcxx ABI

github-actions · 2021-12-02T23:29:27Z

I pushed these containers:

gcr.io/tensorflow-sigs/build:57-python3.9
gcr.io/tensorflow-sigs/build:57-python3.8
gcr.io/tensorflow-sigs/build:57-python3.7

Re-apply the build and push to gcr.io for staging label to
rebuild and push again. This comment will only be posted once.

github-actions · 2022-01-07T01:15:38Z

I pushed these containers:

gcr.io/tensorflow-sigs/build:57-python3.10
gcr.io/tensorflow-sigs/build:57-python3.9
gcr.io/tensorflow-sigs/build:57-python3.8
gcr.io/tensorflow-sigs/build:57-python3.7

Re-apply the build and push to gcr.io for staging label to rebuild and push again. This comment will only be posted once.

guizili0 · 2022-02-15T10:56:13Z

Hi did this PR has relationships with -cxxopt="-D_GLIBCXX_USE_CXX11_ABI=1" and do you plan enable it in TensorFlow release binary? Thanks!

bhack · 2022-02-15T20:16:31Z

We could also check https://discuss.python.org/t/how-to-set-glibcxx-use-cxx11-abi-for-manylinux2014-and-manylinux2010-wheels/10551

nitins17 · 2022-02-16T00:39:33Z

Hi did this PR has relationships with -cxxopt="-D_GLIBCXX_USE_CXX11_ABI=1" and do you plan enable it in TensorFlow release binary? Thanks!

Hi,

This PR is for updating the TensorFlow's Linux wheel to be manylinux2014 compliant. As far as I can tell, D_GLIBCXX_USE_CXX11_ABI would still be equal to 0 because we set --with-default-libstdcxx-abi="gcc4-compatible"; see with-default-libstdcxx-abi in https://gcc.gnu.org/onlinedocs/libstdc++/manual/configure.html

bhack · 2022-02-16T14:34:16Z

@nitins17 Do you know what was the reason for including --with-default-libstdcxx-abi="gcc4-compatible" flag?

nitins17 · 2022-03-01T21:24:10Z

@nitins17 Do you know what was the reason for including --with-default-libstdcxx-abi="gcc4-compatible" flag?

I'm not sure. @mihaimaruseac wdyt?

bhack · 2022-03-07T19:49:11Z

@nitins17 Do you know what was the reason for including --with-default-libstdcxx-abi="gcc4-compatible" flag?

I'm not sure. @mihaimaruseac wdyt?

What was the conclusion about this point in the last SIG build meeting?

tf_sig_build_dockerfiles/setup.python.sh

angerson · 2022-03-15T23:13:51Z

tf_sig_build_dockerfiles/builder.devtoolset/build_devtoolset.sh

+# Note that the installation path for libstdc++ here is /${TARGET}/usr/lib64/
+mv "/${TARGET}/usr/lib64/libstdc++.so.${LIBSTDCXX_VERSION}" \
+   "/${TARGET}/usr/lib64/libstdc++.so.${LIBSTDCXX_VERSION}.backup"
+echo -e "OUTPUT_FORMAT(elf64-x86-64)\nINPUT ( libstdc++.so.6.0.18 -lstdc++_nonshared44 )" \


@perfinion Here's the part we didn't really understand where nonshared44 works but nonshared48 doesn't

I would obviously prefer we use 48, since that looks like it should be the correct version. but the .so is used first and only uses the nonshared for symbols that dont exist in the .so.6.0.18. so if somehow there are symbols in both of them they just wont get used so i dont think there will be any issues.

the critical part in this toolchain is libstdc++.so.6.0.18, that is fine so im good with rolling this out. lets try and figure it out later and maybe update again tho

Set up symlink for devtoolset-8 Combine Docker GCR presubmits and also push main to gcr Commit missed files Log in to GCR Fix conditional, hopefully Clarify Add Python 3.10 support (#58) Adds Python 3.10 support to the containers. Python 3.10 changes some library behavior and, for now, needs an alternative installation method to work. Upgrade gcrpio for fast build and cleanup setup Add utilities for running release tests (#56) This adds the dependencies and notably bazelrc config options to run TensorFlow's Nightly and Release tests, which I've been working on replicating on internal CI. I still have documentation and migration work to do, but the major portion of the support work is here. add gdb to the system packages change to gcc 8.3.1 from centos7 for devtoolset8 fix libstdc++ symlink in devtoolset-8 environment Undo ignoring other xml files Update README Deduplicate repeated messages Squash long runfiles paths Lock nvidia driver to 460 libtensorflow work Fix libtensorflow script and start prelim check Update Test Requirements to have same versions as tf_sig_build_dockerfiles/devel.requirements.txt (#65) * Add additional gitignore files * Update requirements with same versions Keep versions consistent with tf_sig_build_dockerfiles/devel.requirements.txt Cleanup Fix Build issue from `python_include` (#67) * Remove Python 3.10 pip special handling * Link usr/include to usr/local/include * Update location of python include * Update setup.python.sh Assorted changes -- see details - Remove installation of nvidia-profiler, which depends on libcuda1, which ultimately installs an nvidia driver package, which we don't want because we're running in docker, in which the drivers are mounted. I hope nvidia-profiler isn't necessary for anything important; otherwise we'll need to synchronize driver versions between the containers and VM images. - Add less, colordiff and a newer version of clang-format - Add code_check_changed_files, which is intended to replace the "incremental" parts of ci_sanity. Still a work in progress because we need to decide on valuable configurations (clang-format and pylint cannot be run the same way as we have them configured internally and currently have a lot of findings) - Add code_check_full, which is intended to replace the "across entire code base" parts of ci_sanity. I rewrote many of the clunkier tests. Still a work in progress because we must verify that the changed tests will still fail. - Fix bad "bazel test " expansion for libtensorflow - Fix bad chmod for libtensoflow repacker Change libtensorflow config values to fix target selection Fix a typo in venv installation (Thanks to reedwm) Remove extra lines (Thanks again to reedwm) Clarify ctrl-s warning Correctly remove extra test filters Make it possible to run isolated pip tests More work on code checks Fix a typo Clean up code check full Remove clang-format Cleanup changed_files and move one to full Add a missing test Clean up and fix code_check_full Update docs and create experimental RBE configs Update docs and create experimental RBE configs Update dependencies to 2.9.0.dev Update Go API installation guide for TensorFlow 2.8.0 (#74) Clarify usage of nightly commit Fix mistaken 'test' command Update docs and create experimental RBE configs Update docs and create experimental RBE configs Update dependencies to 2.9.0.dev Update Go API installation guide for TensorFlow 2.8.0 (#74) Clarify usage of nightly commit Fix mistaken 'test' command change to devtoolset-9 and gcc 9.3.1 for manylinux2014 change cachebuster value for ml2014 remote cache change to new libstdcxx abi for devtoolset-9 change cachebuster value to use the new libstdcxx abi link against nonshared44 in devtoolset-9 update the cachebuster value change CACHEBUSTER value for gpu builds remove redudant commands during build environment setup change cachbuster variable name for gpu builds store manylinux2014 cache in a different location amend comment for accuracy

perfinion

Looks good to me. lower version is fine with manylinux. I'd prefer the correct version obviously but lets work on that later as an unrelated change

perfinion · 2022-03-16T17:29:03Z

tf_sig_build_dockerfiles/builder.devtoolset/build_devtoolset.sh

+# Note that the installation path for libstdc++ here is /${TARGET}/usr/lib64/
+mv "/${TARGET}/usr/lib64/libstdc++.so.${LIBSTDCXX_VERSION}" \
+   "/${TARGET}/usr/lib64/libstdc++.so.${LIBSTDCXX_VERSION}.backup"
+echo -e "OUTPUT_FORMAT(elf64-x86-64)\nINPUT ( libstdc++.so.6.0.18 -lstdc++_nonshared44 )" \


I would obviously prefer we use 48, since that looks like it should be the correct version. but the .so is used first and only uses the nonshared for symbols that dont exist in the .so.6.0.18. so if somehow there are symbols in both of them they just wont get used so i dont think there will be any issues.

the critical part in this toolchain is libstdc++.so.6.0.18, that is fine so im good with rolling this out. lets try and figure it out later and maybe update again tho

nitins17 · 2022-03-16T22:10:03Z

Hi did this PR has relationships with -cxxopt="-D_GLIBCXX_USE_CXX11_ABI=1" and do you plan enable it in TensorFlow release binary? Thanks!

Hi,

This PR is for updating the TensorFlow's Linux wheel to be manylinux2014 compliant. As far as I can tell, D_GLIBCXX_USE_CXX11_ABI would still be equal to 0 because we set --with-default-libstdcxx-abi="gcc4-compatible"; see with-default-libstdcxx-abi in https://gcc.gnu.org/onlinedocs/libstdc++/manual/configure.html

On this, after the discussion in the March 2022 edition of the TF Sig Build meeting, the manylinux2014 upgrade will also include the change to new libstdcxx ABI. For more information on this and the manylinux2014 rollout, please take a look at this community thread.

Thanks!

guizili0 · 2022-03-17T00:54:16Z

Hi did this PR has relationships with -cxxopt="-D_GLIBCXX_USE_CXX11_ABI=1" and do you plan enable it in TensorFlow release binary? Thanks!

Hi,
This PR is for updating the TensorFlow's Linux wheel to be manylinux2014 compliant. As far as I can tell, D_GLIBCXX_USE_CXX11_ABI would still be equal to 0 because we set --with-default-libstdcxx-abi="gcc4-compatible"; see with-default-libstdcxx-abi in https://gcc.gnu.org/onlinedocs/libstdc++/manual/configure.html

On this, after the discussion in the March 2022 edition of the TF Sig Build meeting, the manylinux2014 upgrade will also include the change to new libstdcxx ABI. For more information on this and the manylinux2014 rollout, please take a look at this community thread.

Thanks!

That's great, thanks for this!!

angerson · 2022-03-21T19:19:37Z

Note that the images, when not cached, take quite a long time to build both dt7 and dt9 -- eventually we'll remove dt7, tentatively "after the TF 2.9 release."

nitins17 requested review from angerson and perfinion as code owners December 2, 2021 22:35

nitins17 added the build and push to gcr.io for staging Create a staging release on gcr.io label Dec 2, 2021

github-actions bot removed build and push to gcr.io for staging Create a staging release on gcr.io labels Dec 2, 2021

nitins17 added build and push to gcr.io for staging Create a staging release on gcr.io and removed build and push to gcr.io for staging Create a staging release on gcr.io labels Dec 3, 2021

nitins17 mentioned this pull request Dec 3, 2021

Make build environment manylinux2014 compatible #52

Closed

nitins17 added the build and push to gcr.io for staging Create a staging release on gcr.io label Jan 7, 2022

nitins17 removed the build and push to gcr.io for staging Create a staging release on gcr.io label Jan 7, 2022

nitins17 added build and push to gcr.io for staging Create a staging release on gcr.io and removed build and push to gcr.io for staging Create a staging release on gcr.io labels Jan 29, 2022

bhack mentioned this pull request Feb 10, 2022

Does TensorFlow 2.x still need the bazel build option of --cxxopt="-D_GLIBCXX_USE_CXX11_ABI=0" ? tensorflow/tensorflow#34946

Closed

nitins17 mentioned this pull request Feb 11, 2022

Test Bazel remote cache issue #73

Closed

nitins17 added build and push to gcr.io for staging Create a staging release on gcr.io and removed build and push to gcr.io for staging Create a staging release on gcr.io labels Feb 14, 2022

nitins17 force-pushed the nitin/manylinux2014 branch from 8044782 to df6cd72 Compare February 16, 2022 00:18

nitins17 force-pushed the nitin/manylinux2014 branch 2 times, most recently from 219a8dc to 6773a82 Compare March 2, 2022 11:56

nitins17 force-pushed the nitin/manylinux2014 branch from 87dca1d to ff4320f Compare March 15, 2022 21:43

angerson approved these changes Mar 15, 2022

View reviewed changes

nitins17 force-pushed the nitin/manylinux2014 branch from ff4320f to b1ae52e Compare March 15, 2022 23:21

perfinion approved these changes Mar 16, 2022

View reviewed changes

Merge branch 'master' into nitin/manylinux2014

1923ea0

angerson merged commit 807ce87 into master Mar 21, 2022

mihaimaruseac deleted the nitin/manylinux2014 branch March 21, 2022 21:38

mohantym mentioned this pull request Jun 3, 2022

-D_GLIBCXX_USE_CXX11_ABI=1 increases a lot RAM usage tensorflow/tensorflow#35160

Open

sachinprasadhs mentioned this pull request Sep 21, 2022

issue with building manylinux wheel tensorflow/tensorflow#53772

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make build environment manylinux2014 compatible #57

Make build environment manylinux2014 compatible #57

nitins17 commented Dec 2, 2021 •

edited

Loading

github-actions bot commented Dec 2, 2021

github-actions bot commented Jan 7, 2022

guizili0 commented Feb 15, 2022

bhack commented Feb 15, 2022

nitins17 commented Feb 16, 2022

bhack commented Feb 16, 2022 •

edited

Loading

nitins17 commented Mar 1, 2022

bhack commented Mar 7, 2022

angerson Mar 15, 2022

perfinion Mar 16, 2022

perfinion left a comment

perfinion Mar 16, 2022

nitins17 commented Mar 16, 2022

guizili0 commented Mar 17, 2022

angerson commented Mar 21, 2022

Make build environment manylinux2014 compatible #57

Make build environment manylinux2014 compatible #57

Conversation

nitins17 commented Dec 2, 2021 • edited Loading

github-actions bot commented Dec 2, 2021

github-actions bot commented Jan 7, 2022

guizili0 commented Feb 15, 2022

bhack commented Feb 15, 2022

nitins17 commented Feb 16, 2022

bhack commented Feb 16, 2022 • edited Loading

nitins17 commented Mar 1, 2022

bhack commented Mar 7, 2022

angerson Mar 15, 2022

Choose a reason for hiding this comment

perfinion Mar 16, 2022

Choose a reason for hiding this comment

perfinion left a comment

Choose a reason for hiding this comment

perfinion Mar 16, 2022

Choose a reason for hiding this comment

nitins17 commented Mar 16, 2022

guizili0 commented Mar 17, 2022

angerson commented Mar 21, 2022

nitins17 commented Dec 2, 2021 •

edited

Loading

bhack commented Feb 16, 2022 •

edited

Loading