[BUG] setuptools does not protect against race conditions on parallel builds #3119

pradyunsg · 2022-02-18T16:16:38Z

setuptools version

setuptools == 60.9.3

Python version

3.9, although this is Python version agnostic

OS

MacOS, although this is OS-agnostic

Additional environment information

No response

Description

I have a Makefile that tries to build multiple wheels for a Python package, in parallel. This fails, with weird errors such as the following (from a pip wheel . --log out.log output):

2022-02-18T15:49:53,923   running clean
2022-02-18T15:49:53,923   'build/lib' does not exist -- can't clean it

2022-02-18T15:49:53,620   Copying test_package.egg-info to build/bdist.macosx-10.15-x86_64/wheel/test_package-1.0.0-py3.9.egg-info
2022-02-18T15:49:53,621   error: [Errno 2] No such file or directory

2022-02-18T16:12:39,443   creating build/bdist.macosx-10.15-x86_64/wheel/test_package-1.0.0.dist-info/WHEEL
2022-02-18T16:12:39,444   error: [Errno 2] No such file or directory: 'build/bdist.macosx-10.15-x86_64/wheel/test_package-1.0.0.dist-info/WHEEL'

2022-02-18T16:12:39,409   Copying test_package.egg-info to build/bdist.macosx-10.15-x86_64/wheel/test_package-1.0.0-py3.9.egg-info
2022-02-18T16:12:39,412   running install_scripts
2022-02-18T16:12:39,440   error: [Errno 17] File exists: 'build/bdist.macosx-10.15-x86_64/wheel/test_package-1.0.0.dist-info'

And various other errors related to files in the build directory.

This is because setuptools does not have any isolation/protection against parallel builds.

Expected behavior

setuptools works properly with parallel builds that use the same working directory. Either through some sort of separation within the build directory, or through filesystem based locks for the build directory.

A more targetted behaviour change would be to include the running Python version into the build directory paths, which would enable building extensions in parallel for a Python package; which is my main use case. I do think a more general solution would be nicer, eg for the example I have below. :)

How to Reproduce

Take a basic setuptools package.
Try to perform parallel builds on it (eg: building wheels for different Python versions, for C extensions).
Notice that all these builds trample over each other, non-deterministically fail or have incorrect contents.

I'll take an example of a pure-Python wheel below, but the behavior for platform-specific wheels is similar.

Output

This is more of a reproducer -- examples of setuptools' failure messages are shown above and change depending on what had a race condition in each run. :)

/private/tmp/setuptools-parallel  setuptools-parallel
❯ pip --version
pip 22.0.3 from /private/tmp/setuptools-parallel/.venv/lib/python3.9/site-packages/pip (python 3.9)
/private/tmp/setuptools-parallel  setuptools-parallel
❯ pip list
Package    Version
---------- -------
pip        22.0.3
setuptools 60.9.3
wheel      0.37.1
/private/tmp/setuptools-parallel  setuptools-parallel
❯ tree .
.
├── run.sh
└── setup.py

0 directories, 2 files
/private/tmp/setuptools-parallel  setuptools-parallel
❯ cat setup.py
from setuptools import setup

setup(
    name="test-package",
    version="1.0.0",
)
/private/tmp/setuptools-parallel  setuptools-parallel
❯ cat run.sh
pip wheel . --wheel-dir one/ --log one/out.txt &
pip wheel . --wheel-dir two/ --log two/out.txt &
pip wheel . --wheel-dir three/ --log three/out.txt &
/private/tmp/setuptools-parallel  setuptools-parallel
❯ . ./run.sh
[expect overlapping pip output below]
Processing /private/tmp/setuptools-parallel
Processing /private/tmp/setuptools-parallel
Processing /private/tmp/setuptools-parallel
  Preparing metadata (setup.py) ... done
  Preparing metadata (setup.py) ... done
  Preparing metadata (setup.py) ... done
Building wheels for collected packages: test-package
Building wheels for collected packages: test-package
Building wheels for collected packages: test-package
  error: subprocess-exited-with-error
  
  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> See above for output.
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  Building wheel for test-package (setup.py) ... error
  ERROR: Failed building wheel for test-package
  Running setup.py clean for test-package
  error: subprocess-exited-with-error
  
  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> See above for output.
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  error: subprocess-exited-with-error
  
  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> See above for output.
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  Building wheel for test-package (setup.py) ... error
  Building wheel for test-package (setup.py) ... error  ERROR: Failed building wheel for test-package

  ERROR: Failed building wheel for test-package
  Running setup.py clean for test-package
  Running setup.py clean for test-package
Failed to build test-package
Failed to build test-package
Failed to build test-package
ERROR: Failed to build one or more wheels
ERROR: Failed to build one or more wheels
ERROR: Failed to build one or more wheels

Some example errors are mentioned in the description. If you run this yourself, you can see the failure output and wheels in one/, two/ and three/ directories.

The text was updated successfully, but these errors were encountered:

pradyunsg · 2022-02-18T16:22:44Z

FWIW, I could add a C extension build to the reproducer to cause version-specific wheels to be built, use different parallel builds with multiple Python versions to more closely match my workflow -- but that feels unnecessary given that the error behaviour is going to be largely the same.

jaraco · 2022-02-19T00:19:19Z

Does the issue exist with pep517-isolated builds?

jaraco · 2022-02-19T00:41:01Z

I was able to replicate the behavior, and setting PIP_USE_PEP517=1 dosen't work around the issue. Is that because build isolation isolates the Python environment but not the source? I thought it also copied the source (though I understand why it might not). Aha! I see that pip 21 dropped support for isolated builds. If I downgrade to pip 20.0.2 and set PIP_USE_PEP517=1, the repro no longer occurs.

It'll take some work to get setuptools to support parallel builds, and I won't be surprised if it requires changes to distutils as well. It may prove challenging before setuptools has fully adopted distutils (and no longer relies on stdlib).

pradyunsg · 2022-02-19T06:55:13Z

Aha! I see that pip 21 dropped support for isolated builds.

Yea, 21.3 switched to in-tree builds unconditionally, instead of copying to a randomised build directory.

Yay295 · 2024-04-24T23:27:41Z

I don't know if this is the same issue since it looks like you're running setuptools in parallel rather than using setuptools to build extensions in parallel, but I've at least ran into a similar issue with that.

setup(
    ext_modules=[
        Extension("A", ["a.c", "common.c"]),
        Extension("B", ["b.c", "common.c"])
    ]
)

When this runs in parallel, each module will compile its own copy of common.c into the same location, and if they step on each other at the wrong time the builds can fail.

abravalheri · 2024-04-25T11:02:48Z

Hi @Yay295 I believe that is a separated thing.
My impression is that what is being discussed here is parallel build for different Python versions.

I think your use case might require something like the following:

setup(
    # ...
    libraries=[
        # Compile a binary archive once and re-use in extensions
        ("common", {"sources": ["common.c"]}),
    ],
    ext_modules=[
        Extension("A", sources=["a.c"], libraries=["common"]),
        Extension("B", sources=["b.c"], libraries=["common"]),
    ],
)

Yay295 · 2024-04-25T14:32:52Z

Thanks, that seems to work. Though it doesn't look like the libraries argument for setup() is documented.

https://setuptools.pypa.io/en/latest/references/keywords.html

abravalheri · 2024-04-25T14:49:40Z

Yes, that is probably something inherited from distutils (and also a less common/used feature) :(

PR for doc improvements are always welcome!

## Summary I don't love this, but it turns out that setuptools is not robust to parallel builds: pypa/setuptools#3119. As a result, if you run uv from multiple processes, and they each attempt to build the same source distribution, you can hit failures. This PR applies an advisory lock to the source distribution directory. We apply it unconditionally, even if we ultimately find something in the cache and _don't_ do a build, which helps ensure that we only build the distribution once (and wait for that build to complete) rather than kicking off builds from each thread. Closes #3512. ## Test Plan Ran: ```sh #!/bin/bash make_venv(){ target/debug/uv venv $1 source $1/bin/activate target/debug/uv pip install opentracing --no-deps --verbose } for i in {1..8} do make_venv ./$1/$i & done ```

pradyunsg added bug Needs Triage Issues that need to be evaluated for severity and status. labels Feb 18, 2022

jaraco added enhancement help wanted and removed bug Needs Triage Issues that need to be evaluated for severity and status. labels Feb 19, 2022

abravalheri mentioned this issue May 11, 2023

test_cygwin job failing after adding ruff #3921

Open

This was referenced May 10, 2024

Wheel Build failures when installing via multiple simultaneous UV instances astral-sh/uv#3512

Closed

Apply advisory locks when building source distributions astral-sh/uv#3525

Merged

leofang mentioned this issue Dec 6, 2024

Building cuda.bindings in parallel might fail non-deterministically NVIDIA/cuda-python#271

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] setuptools does not protect against race conditions on parallel builds #3119

[BUG] setuptools does not protect against race conditions on parallel builds #3119

pradyunsg commented Feb 18, 2022 •

edited

Loading

pradyunsg commented Feb 18, 2022

jaraco commented Feb 19, 2022

jaraco commented Feb 19, 2022

pradyunsg commented Feb 19, 2022 •

edited

Loading

Yay295 commented Apr 24, 2024

abravalheri commented Apr 25, 2024

Yay295 commented Apr 25, 2024

abravalheri commented Apr 25, 2024

[BUG] setuptools does not protect against race conditions on parallel builds #3119

[BUG] setuptools does not protect against race conditions on parallel builds #3119

Comments

pradyunsg commented Feb 18, 2022 • edited Loading

setuptools version

Python version

OS

Additional environment information

Description

Expected behavior

How to Reproduce

Output

pradyunsg commented Feb 18, 2022

jaraco commented Feb 19, 2022

jaraco commented Feb 19, 2022

pradyunsg commented Feb 19, 2022 • edited Loading

Yay295 commented Apr 24, 2024

abravalheri commented Apr 25, 2024

Yay295 commented Apr 25, 2024

abravalheri commented Apr 25, 2024

pradyunsg commented Feb 18, 2022 •

edited

Loading

pradyunsg commented Feb 19, 2022 •

edited

Loading