Issues with `PYTHONPATH` resolution in recent python/rules_python versions #1221

phlax · 2023-05-12T10:27:22Z

🐞 bug report

Affected Rule

The issue is caused by the rule:

afaict ~any python rules but ive tested or hit the issue with py_binary, py_library

Is this a regression?

Yes, the previous version in which this bug was not present was: ....

We had trouble upgrading to 0.19.x versions iirc due to this issue

We have been able to upgrade to 0.20.0 but when upgrading the python version to 3.11 the issue reappears

Description

A clear and concise description of the problem...

Expected paths are not being added to sys.path

🔬 Minimal Reproduction

py_library(
    name = "mylib",
    srcs = ["mylib.py"],
)

py_binary(
    name = "mybin",
    srcs = ["mybin.py"],
    deps = [":mylib"],
)

def myfun():
    print("BOOM!")

from mylib import myfun


def main():
    myfun()


if __name__ == "__main__":
    main()

🔥 Exception or Error

$ bazel run //tools/base:mybin
Starting local Bazel server and connecting to it...
INFO: Invocation ID: 19b6db42-4a15-4858-be40-7cf804c798de
INFO: Analyzed target //tools/base:mybin (126 packages loaded, 3458 targets configured).
INFO: Found 1 target...
Target //tools/base:mybin up-to-date:
  bazel-bin/tools/base/mybin
INFO: Elapsed time: 4.511s, Critical Path: 0.28s
INFO: 4 processes: 4 internal.
INFO: Build completed successfully, 4 total actions
INFO: Running command line: bazel-bin/tools/base/mybin
Traceback (most recent call last):
  File "/home/worker/.cache/bazel/_bazel_worker/f704bab1b165ed1368cb88f9f49e7532/execroot/envoy/bazel-out/k8-fastbuild/bin/tools/base/mybin.runfiles/envoy/tools/base/mybin.py", line 2, in <module>
    from mylib import myfun
ModuleNotFoundError: No module named 'mylib'

🌍 Your Environment

Operating System:

  
Linux

Output of bazel version:

  
$ bazel version
Bazelisk version: v1.11.0
Build label: 6.1.0
Build target: bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Mon Mar 6 17:09:47 2023 (1678122587)
Build timestamp: 1678122587
Build timestamp as int: 1678122587

Rules_python version:

  
0.20.0

Anything else relevant?

Python version: 3.11

The text was updated successfully, but these errors were encountered:

phlax · 2023-05-12T10:38:38Z

cc @htuch

phlax · 2023-05-12T12:02:34Z

adding a shim - ie:

import os
import sys

sys.path += [os.path.dirname(__file__)]

works, and is what i have done to workaround, either in our py targets, or for py files in dep rules

kevinaud · 2023-06-09T12:00:15Z

@phlax Can you elaborate on how you hooked that shim in? I think I'm currently running into this same issue. I'm on rules_python version 0.21.0 and python 3.11.1.

Things are working on my dev machine but during my automated build/tests in github actions I'm getting this (I have a python tool for running these tests which is why the logger prefix is present):

INFO: Repository third_party_docopt instantiated at:
2023-06-09 11:37:53,037 - tools.presubmit.cli - INFO - /home/runner/work/app/app/WORKSPACE:100:13: in <toplevel>
2023-06-09 11:37:53,037 - tools.presubmit.cli - INFO - /home/runner/.cache/bazel/_bazel_runner/9bb8a7cb6[522](https://github.com/company/app/actions/runs/5221416166/jobs/9425648690#step:12:523)6a357f750dd2451a3938/external/third_party/requirements.bzl:47:20: in install_deps
2023-06-09 11:37:53,037 - tools.presubmit.cli - INFO - Repository rule whl_library defined at:
2023-06-09 11:37:53,037 - tools.presubmit.cli - INFO - /home/runner/.cache/bazel/_bazel_runner/9bb8a7cb65226a357f750dd2451a3938/external/rules_python/python/pip_install/pip_repository.bzl:712:30: in <toplevel>
2023-06-09 11:37:53,046 - tools.presubmit.cli - INFO - ERROR: An error occurred during the fetch of repository 'third_party_docopt':
2023-06-09 11:37:53,046 - tools.presubmit.cli - INFO - Traceback (most recent call last):
2023-06-09 11:37:53,046 - tools.presubmit.cli - INFO - File "/home/runner/.cache/bazel/_bazel_runner/9bb8a7cb65226a357f750dd2451a3938/external/rules_python/python/pip_install/pip_repository.bzl", line 688, column 13, in _whl_library_impl
2023-06-09 11:37:53,046 - tools.presubmit.cli - INFO - fail("whl_library %s failed: %s (%s)" % (rctx.attr.name, result.stdout, result.stderr))
2023-06-09 11:37:53,046 - tools.presubmit.cli - INFO - Error in fail: whl_library third_party_docopt failed: Collecting docopt==0.6.2
2023-06-09 11:37:53,046 - tools.presubmit.cli - INFO - Downloading docopt-0.6.2.tar.gz (25 kB)
2023-06-09 11:37:53,046 - tools.presubmit.cli - INFO - Preparing metadata (setup.py): started
2023-06-09 11:37:53,046 - tools.presubmit.cli - INFO - Preparing metadata (setup.py): finished with status 'error'
2023-06-09 11:37:53,046 - tools.presubmit.cli - INFO - (  error: subprocess-exited-with-error
2023-06-09 11:37:53,046 - tools.presubmit.cli - INFO - 
2023-06-09 11:37:53,046 - tools.presubmit.cli - INFO - × python setup.py egg_info did not run successfully.
2023-06-09 11:37:53,046 - tools.presubmit.cli - INFO - │ exit code: 1
2023-06-09 11:37:53,046 - tools.presubmit.cli - INFO - ╰─> [6 lines of output]
2023-06-09 11:37:53,046 - tools.presubmit.cli - INFO - Traceback (most recent call last):
2023-06-09 11:37:53,046 - tools.presubmit.cli - INFO - File "<string>", line 2, in <module>
2023-06-09 11:37:53,046 - tools.presubmit.cli - INFO - File "<pip-setuptools-caller>", line 34, in <module>
2023-06-09 11:37:53,046 - tools.presubmit.cli - INFO - File "/tmp/pip-wheel-yuurxaqt/docopt_0109a8f2a8f74fe78074f6032e6f0359/setup.py", line 3, in <module>
2023-06-09 11:37:53,046 - tools.presubmit.cli - INFO - from docopt import __version__
2023-06-09 11:37:53,046 - tools.presubmit.cli - INFO - ModuleNotFoundError: No module named 'docopt'
2023-06-09 11:37:53,046 - tools.presubmit.cli - INFO - [end of output]
2023-06-09 11:37:53,046 - tools.presubmit.cli - INFO - 
2023-06-09 11:37:53,046 - tools.presubmit.cli - INFO - note: This error originates from a subprocess, and is likely not a problem with pip.
2023-06-09 11:37:53,046 - tools.presubmit.cli - INFO - error: metadata-generation-failed
2023-06-09 11:37:53,046 - tools.presubmit.cli - INFO - 
2023-06-09 11:37:53,046 - tools.presubmit.cli - INFO - × Encountered error while generating package metadata.
2023-06-09 11:37:53,046 - tools.presubmit.cli - INFO - ╰─> See above for output.
2023-06-09 11:37:53,046 - tools.presubmit.cli - INFO - 
2023-06-09 11:37:53,046 - tools.presubmit.cli - INFO - note: This is an issue with the package mentioned above, not pip.
2023-06-09 11:37:53,046 - tools.presubmit.cli - INFO - hint: See above for details.
2023-06-09 11:37:53,046 - tools.presubmit.cli - INFO - Traceback (most recent call last):
2023-06-09 11:37:53,046 - tools.presubmit.cli - INFO - File "<frozen runpy>", line 198, in _run_module_as_main
2023-06-09 11:37:53,046 - tools.presubmit.cli - INFO - File "<frozen runpy>", line 88, in _run_code
2023-06-09 11:37:53,046 - tools.presubmit.cli - INFO - File "/home/runner/.cache/bazel/_bazel_runner/9bb8a7cb65226a357f750dd2451a3938/external/rules_python/python/pip_install/tools/wheel_installer/wheel_installer.py", line 450, in <module>
2023-06-09 11:37:53,046 - tools.presubmit.cli - INFO - main()
2023-06-09 11:37:53,046 - tools.presubmit.cli - INFO - File "/home/runner/.cache/bazel/_bazel_runner/9bb8a7cb65226a357f750dd2451a3938/external/rules_python/python/pip_install/tools/wheel_installer/wheel_installer.py", line 427, in main
2023-06-09 11:37:53,046 - tools.presubmit.cli - INFO - subprocess.run(pip_args, check=True, env=env)
2023-06-09 11:37:53,046 - tools.presubmit.cli - INFO - File "/home/runner/.cache/bazel/_bazel_runner/9bb8a7cb65226a357f750dd2451a3938/external/python3_11_x86_64-unknown-linux-gnu/lib/python3.11/subprocess.py", line 571, in run
2023-06-09 11:37:53,046 - tools.presubmit.cli - INFO - raise CalledProcessError(retcode, process.args,
2023-06-09 11:37:53,047 - tools.presubmit.cli - INFO - subprocess.CalledProcessError: Command '['/home/runner/.cache/bazel/_bazel_runner/9bb8a7cb65226a357f750dd2451a3938/external/python3_11_x86_64-unknown-linux-gnu/bin/python3', '-m', 'pip', '--isolated', 'wheel', '--no-deps', '-r', '/tmp/tmpck70_4v8']' returned non-zero exit status 1.
2023-06-09 11:37:53,047 - tools.presubmit.cli - INFO - )
2023-06-09 11:37:53,047 - tools.presubmit.cli - INFO - ERROR: /home/runner/work/app/app/WORKSPACE:100:13: fetching whl_library rule //external:third_party_docopt: Traceback (most recent call last):

Not these lines in particular:

2023-06-09 11:37:53,046 - tools.presubmit.cli - INFO - File "/tmp/pip-wheel-yuurxaqt/docopt_0109a8f2a8f74fe78074f6032e6f0359/setup.py", line 3, in <module>
2023-06-09 11:37:53,046 - tools.presubmit.cli - INFO - from docopt import __version__
2023-06-09 11:37:53,046 - tools.presubmit.cli - INFO - ModuleNotFoundError: No module named 'docopt'

Can't figure out how to resolve this for the life of me.

phlax · 2023-06-09T12:05:14Z

Can you elaborate on how you hooked that shim in?

very hackily - see this PR for details https://github.com/envoyproxy/envoy/pull/25359/files

seems to have survived a rules_python upgrade, i didnt check to see if the hacks could be removed yet

alexeagle · 2023-10-02T18:17:25Z

I expect this is resolved by https://github.com/aspect-build/rules_py because it doesn't mess with the PYTHONPATH, it just lays out a python-idiomatic virtualenv with a site-packages folder.

phlax · 2023-10-02T18:20:34Z

interesting we just landed a workaround to inject namespacing init pys to our repo folders to avoid pypi namespace clashing - seems like we could avoid that workaround - ill look further when i get chance

github-actions · 2024-03-30T22:50:17Z

This issue has been automatically marked as stale because it has not had any activity for 180 days. It will be closed if no further activity occurs in 30 days.
Collaborators can add an assignee to keep this open indefinitely. Thanks for your contributions to rules_python!

phlax · 2024-03-31T09:39:41Z

this is still an issue

github-actions · 2024-09-27T22:50:39Z

This issue has been automatically marked as stale because it has not had any activity for 180 days. It will be closed if no further activity occurs in 30 days.
Collaborators can add an assignee to keep this open indefinitely. Thanks for your contributions to rules_python!

alexeagle · 2024-10-04T18:17:34Z

Note, I deleted the stale bot :)

aignas · 2024-12-26T01:36:37Z

Well, this is actually working as intended. the py_library is missing an imports attribute in the reproduction, so if I modify the example and run it, it works:

$ bazel run //tools/base:mybin
INFO: Analyzed target //tools/base:mybin (1 packages loaded, 4 targets configured).
INFO: Found 1 target...
Target //tools/base:mybin up-to-date:
  bazel-bin/tools/base/mybin
INFO: Elapsed time: 1.004s, Critical Path: 0.05s
INFO: 2 processes: 3 action cache hit, 2 internal.
INFO: Build completed successfully, 2 total actions
INFO: Running command line: bazel-bin/tools/base/mybin
BOOM!

$ git diff
diff --git a/tools/base/BUILD.bazel b/tools/base/BUILD.bazel
index 5c40070..63324c4 100644
--- a/tools/base/BUILD.bazel
+++ b/tools/base/BUILD.bazel
@@ -4,6 +4,7 @@ load("@rules_python//python:py_library.bzl", "py_library")
 py_library(
     name = "mylib",
     srcs = ["mylib.py"],
+    imports = ["."],
 )

 py_binary(

So I'll close as won't do. The previous behaviour might have been due to 3.10 allowing the interpreter to escape sandbox and with 3.11 there was an extra safe_path feature added that the entrypoint template in the bazel repository (this is the version when the issue was raised) was using.

The correct way to fix this would be to either wait until #2156 is resolved or add imports attributes to your py_library instances.

phlax mentioned this issue May 15, 2023

deps: Bump python -> 3.11.1 envoyproxy/envoy#25359

Merged

github-actions bot added the Can Close? Will close in 30 days if there is no new activity label Mar 30, 2024

aignas removed the Can Close? Will close in 30 days if there is no new activity label Mar 31, 2024

github-actions bot added the Can Close? Will close in 30 days if there is no new activity label Sep 27, 2024

alexeagle removed the Can Close? Will close in 30 days if there is no new activity label Oct 4, 2024

aignas closed this as not planned Won't fix, can't repro, duplicate, stale Dec 26, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Issues with `PYTHONPATH` resolution in recent python/rules_python versions #1221

Issues with `PYTHONPATH` resolution in recent python/rules_python versions #1221

phlax commented May 12, 2023 •

edited

Loading

phlax commented May 12, 2023

phlax commented May 12, 2023

kevinaud commented Jun 9, 2023

phlax commented Jun 9, 2023

alexeagle commented Oct 2, 2023

phlax commented Oct 2, 2023

github-actions bot commented Mar 30, 2024

phlax commented Mar 31, 2024

github-actions bot commented Sep 27, 2024

alexeagle commented Oct 4, 2024

aignas commented Dec 26, 2024

Issues with PYTHONPATH resolution in recent python/rules_python versions #1221

Issues with PYTHONPATH resolution in recent python/rules_python versions #1221

Comments

phlax commented May 12, 2023 • edited Loading

🐞 bug report

Affected Rule

Is this a regression?

Description

🔬 Minimal Reproduction

🔥 Exception or Error

🌍 Your Environment

phlax commented May 12, 2023

phlax commented May 12, 2023

kevinaud commented Jun 9, 2023

phlax commented Jun 9, 2023

alexeagle commented Oct 2, 2023

phlax commented Oct 2, 2023

github-actions bot commented Mar 30, 2024

phlax commented Mar 31, 2024

github-actions bot commented Sep 27, 2024

alexeagle commented Oct 4, 2024

aignas commented Dec 26, 2024

Issues with `PYTHONPATH` resolution in recent python/rules_python versions #1221

Issues with `PYTHONPATH` resolution in recent python/rules_python versions #1221

phlax commented May 12, 2023 •

edited

Loading