feat(venv): Static site-packages trees

[arrdem]

This PR introduces a new suite of virtualenv machinery building on the existing py_binary virtualenv tree builder to create such trees "statically" at build time rather than "dynamically" at runtime. The original reason this is difficult is that the $VENV/bin/python file is usually a link to a static interpreter, and the interpreter specified in pyvenv.cfg is also usually an absolute path to a real interpreter outside of the venv.

The introduction of #546 or something like it makes it possible to create a virtualenv tree which does not refer to an interpreter with a fixed path. Instead an appropriate interpreter can be dynamically discovered by $PATH inspection. Support for other configuration ala /etc/pexrc to facilitate interpreter discovery is possible but beyond the scope of an initial implementation.

By using an interpreter shim, the pyenv.cfg file can use a relative path to ./bin/python as its interpreter reference, and the ./bin/python3 and ./bin/python3.X links can be created statically pointing to the shim. As long as other bin/ entrypoint scripts reference only #!/usr/bin/env python3 as is fairly conventional, the constructed virtual environment will contain no absolute references to a specific interpreter and will be fully relocatable.

A relocatable virtualenv tree containing dependencies installed into the site-packages tree may be built using the new experimental py_venv rule.

py_venv(
    name = "venv",
    deps = [
        ... # PyInfo targets to be integrated into the site-packages tree
    ],
)

Files are selected for inclusion into virtualenv based on import paths.

• Any import path which contains a *.dist-info child directory will have its contents included.
• Any included path which ends in site-packages and for which there exist files in a sibling bin or other conventional directory will have those assets included into the virtualenv.

These heuristics are chosen to both handle how rules_python currently emulates package installation, and to avoid dependencies on external workspace specific behavior so that internal or vendored libraries can be "installed" into the venv.

A py_venv is not intended to be a .runfiles-style packaging of all source code, it is intended to be a packaging of dependency code which is compliant with being treated as if pip installed. This may not be generally true of firstparty application code, depending on usage of runfiles.Runfiles or repository-relative logics. Because of this py_venv targets provide files not relocated into the venv as runfiles dependencies.

Applications wishing to leverage a static virtualenv should leverage the py_venv_binary rule.

py_venv_binary(
    name = "demo",
    srcs = ["demo.py"],
    deps = [ ... ],
    main = "demo.py",
)

A py_venv_binary defines a more conventional .runfiles-based Python application, but one which will use the .runfiles embedded virtualenv as the source of thirdparty packages. Firstparty packages will receive the normal .runfiles workspace structure preserving treatment.

A py_venv_test is also provided which behaves the same as a py_venv_binary.

TODO

• Get the implementation fully aligned with this writeup
• Convert the existing example into a test
• Create an example of using py_venv_binary with rules_oci together to create a static application

For follow-up work

• Fix for [Bug]: py_venv doesn't include dependency executables in venv bin/ #423 Not currently possible, but we do have support when rules_python catches up
• Get the zip machinery working nicely, either with a Rust impl or via rules_pkg

Changes are visible to end-users: yes

• Searched for relevant documentation and updated as needed: yes
• Breaking change (forces users to change their own code or config): no
• Suggested release notes appear below: yes

Test plan

• New test cases added

[github-actions]

e2e/use_release folder: LCOV of commit 22c91c7 during CI #1739

Summary coverage rate:
  lines......: 100.0% (2 of 2 lines)
  functions..: 100.0% (1 of 1 function)
  branches...: no data found

Files changed coverage rate: n/a

[aspect-workflows]

Test

All tests were cache hits

32 tests (100.0%) were fully cached saving 59s.

[thesayyn]

can this transition live in a single place?

[thesayyn]

previously this rule was used for IDE support via bazel run :venv to create the .venv symlink in the source tree. we are probably breaking those people here.

[thesayyn]

Overall looking good, a few things, rust can be better but i think its good enough for now.

[thesayyn]

Suggested change

	pth_lines.use_param_file("%s/", use_always = True)
	pth_lines.use_param_file("%s", use_always = True)

[thesayyn]

lets double check if this is breaking

[thesayyn]

.unwrap() here sounds dangerous. do we want to check and print an error message?

[thesayyn]

LGTM, nice job!