[bazel, rules_python] Unable to get correct Python path from whl?

[EricCousineau-TRI]

Describe the bug
Relates bazel-contrib/rules_python#2071

To Reproduce
See repro in linked issue, i.e,.
https://github.com/EricCousineau-TRI/repro/tree/e084a7434286cf426a71350f79755c7cbab6d6eb/bazel/rules_python_rerun

Workaround: EricCousineau-TRI/repro@96e1336

Expected behavior
Things are declared such that rules_python generates rules that have correct PYTHONPATH

Screenshots
N/A

Backtrace
N/A

Desktop (please complete the following information):

• OS: Ubuntu 22.04

Rerun version
0.17.0

Additional context
using bazel build system + rules_python

[Wumpf]

I unfortunately don't know anything about Bazel build setups, but I suppose there's some issue about the package being called rerun-sdk, but the module is rerun?

[abey79]

I know nothing about bezel, but I'm kind of surprised by that __init__.py file there. I dont have it when I manually install rerun in a venv:

$ ls external/pip_deps_rerun_sdk/site-packages/rerun_sdk
__init__.py  rerun  rerun_cli

Also, for records, this what I have in my site-packages:

…/lib/python3.11/site-packages 
❯ ls | grep rerun
rerun_bindings
rerun_sdk
rerun_sdk-0.17.0.dist-info
rerun_sdk.pth

You may want to check if you have that .pth in particular.

[EricCousineau-TRI]

but I suppose there's some issue about the package being called rerun-sdk, but the module is rerun

yeah, I think that it's because the module is under rerun_sdk/{rerun,rerun_cli} as abey79 mentioned.

You may want to check if you have that .pth in particular.

yeah, it seems like rules_python may not correctly recognize .pth files:
bazel-contrib/rules_python#2071 (comment)

closing this issue, as it seems like a bazel rules_python issue.
thanks!

[abey79]

Reopening this issue, if only for visibility. Although this technically seems to be a Bazel problem, it's is triggered by some packaging trick we are doing using that pth file.

[jleibs]

I'm kind of surprised our approach there has held up as long as it has. For context, the number one thing our .pth hackery exists to achieve is to save us from collateral damage being caused by the rerun package in pypi-managed environments.

Especially in the early days of Rerun it was very common for people to pip install rerun instead of pip install rerun-sdk. This wouldn't have been quite so bad in-and-of-itself, but we also felt strongly that import rerun was superior to import rerun_sdk.

In theory we CAN just install rerun as a top-level package, but if a user ever installs rerun package accidentally, the files stomp on each other and to get back in a clean state you have to uninstall rerun and then force-reinstall rerun-sdk. And since pypi provides no way of declaring a mutual exclusion with another package, there was no way to prevent this.

Scoping rerun as a sub-package of rerun_sdk and injecting it into the top-level package path via a .pth file keeps us from having any conflicting files, and avoiding the need for that reinstall.

We need some way to provide a target for bazel users that users an alternative layout that avoids the need for .pth files, under the assunption they won't be adding rerun to their projects.

[david-dorf]

A simple workaround is to add aspect_rules_py to the MODULE.bazel and the respective py_binary to the BUILD.bazel, as shown in the screenshots attached.

[abey79]

Thanks for that. Some background information can be found here: https://github.com/aspect-build/rules_py

[fabiannagel]

Hi everyone. Stumbled upon this issue when trying to get ROS2 to accept dependencies from a uv venv. Managed to get it to run like this:

export PYTHONPATH=$PYTHONPATH:/root/.venv/lib/python3.12/site-packages/

# first venv, then ROS
source /root/.venv/bin/activate
source /opt/ros/jazzy/setup.bash

colcon build ...
ros2 run ...

This works for all my dependencies - except rerun. After some digging, I came across the .pth trick which apparently does not work well with manually configured PYTHONPATHs. Given the targeted composability of rerun witH ROS2, I'd appreciate a workaround. Happy to add some documentation also.

edit: Tried importing rerun_sdk instead. Works for my own code, but breaks internally (which still looks for rerun).

[jleibs]

@fabiannagel You could work-around this by doing what the .pth file does manually in your python-path by setting:

export PYTHONPATH=$PYTHONPATH:/root/.venv/lib/python3.12/site-packages/:/root/.venv/lib/python3.12/site-packages/rerun_sdk

[fabiannagel]

@jleibs Thanks! I managed to get ROS to respect venvs such that I don't need any more PYTHONPATH hacking :)