[Bug]: Python 3.11

[siddharthab]

What happened?

When using a standalone Python 3.11 toolchain, the environment setup in the entrypoint template is not right.

If we set up the Python interpreter to be version 3.11 as in the reproduction section below, we get an error from the Python interpreter at launch.

% bazel test //examples/pytest:pytest_test
DEBUG: Rule 'python_toolchain_aarch64-apple-darwin' indicated that a canonical reproducible form can be obtained by modifying arguments url = ["https://github.com/indygreg/python-build-standalone/releases/download/20230116/cpython-3.11.1+20230116-aarch64-apple-darwin-install_only.tar.gz"] and dropping ["urls"]
DEBUG: Repository python_toolchain_aarch64-apple-darwin instantiated at:
  /Users/sidb/Workspaces/rules_py/WORKSPACE:19:27: in <toplevel>
  /private/var/tmp/_bazel_sidb/a21ec54da3d04a4699759ed6246d5320/external/rules_python/python/repositories.bzl:533:26: in python_register_toolchains
Repository rule python_repository defined at:
  /private/var/tmp/_bazel_sidb/a21ec54da3d04a4699759ed6246d5320/external/rules_python/python/repositories.bzl:366:36: in <toplevel>
INFO: Analyzed target //examples/pytest:pytest_test (0 packages loaded, 0 targets configured).
INFO: Found 1 test target...
FAIL: //examples/pytest:pytest_test (see /private/var/tmp/_bazel_sidb/a21ec54da3d04a4699759ed6246d5320/execroot/aspect_rules_py/bazel-out/darwin_arm64-fastbuild/testlogs/examples/pytest/pytest_test/test.log)
INFO: From Testing //examples/pytest:pytest_test:
==================== Test output for //examples/pytest:pytest_test:
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Python path configuration:
  PYTHONHOME = (not set)
  PYTHONPATH = (not set)
  program name = '/private/var/tmp/_bazel_sidb/a21ec54da3d04a4699759ed6246d5320/sandbox/darwin-sandbox/28/execroot/aspect_rules_py/bazel-out/darwin_arm64-fastbuild/bin/examples/pytest/pytest_test.runfiles/pytest_test.venv/bin/python'
  isolated = 1
  environment = 0
  user site = 0
  safe_path = 1
  import site = 1
  is in build tree = 0
  stdlib dir = '/install/lib/python3.11'
  sys._base_executable = '/private/var/tmp/_bazel_sidb/a21ec54da3d04a4699759ed6246d5320/execroot/aspect_rules_py/external/python_toolchain_aarch64-apple-darwin/bin/python3.11'
  sys.base_prefix = '/install'
  sys.base_exec_prefix = '/install'
  sys.platlibdir = 'lib'
  sys.executable = '/private/var/tmp/_bazel_sidb/a21ec54da3d04a4699759ed6246d5320/sandbox/darwin-sandbox/28/execroot/aspect_rules_py/bazel-out/darwin_arm64-fastbuild/bin/examples/pytest/pytest_test.runfiles/pytest_test.venv/bin/python'
  sys.prefix = '/install'
  sys.exec_prefix = '/install'
  sys.path = [
    '/install/lib/python311.zip',
    '/install/lib/python3.11',
    '/install/lib/python3.11/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00000001e27f5b40 (most recent call first):
  <no Python frame>
================================================================================

Version

Development (host) and target OS/architectures:

Output of bazel --version: bazel 5.4.0 (but same error with latest version of Bazel)

Version of the Aspect rules, or other relevant rules from your
WORKSPACE or MODULE.bazel file: HEAD of this repo -- commit 913676de2f89b8272cdd14bc53f719534e58c31f

Language(s) and/or frameworks involved: Python

How to reproduce

To use a Python 3.11 toolchain with an example, we can modify this repo itself and test one of the examples here.

diff --git a/WORKSPACE b/WORKSPACE
index fc4261e..a5028a3 100644
--- a/WORKSPACE
+++ b/WORKSPACE
@@ -18,7 +18,7 @@ load("@rules_python//python:repositories.bzl", "python_register_toolchains")
 
 python_register_toolchains(
     name = "python_toolchain",
-    python_version = "3.9",
+    python_version = "3.11",
 )
 
 ############################################
diff --git a/examples/pytest/BUILD.bazel b/examples/pytest/BUILD.bazel
index 9398436..3decfc4 100644
--- a/examples/pytest/BUILD.bazel
+++ b/examples/pytest/BUILD.bazel
@@ -1,5 +1,4 @@
-load("@rules_python//python:defs.bzl", "py_test")
-load("@//py:defs.bzl", "py_pytest_main")
+load("@//py:defs.bzl", "py_pytest_main", "py_test")
 
 py_pytest_main(name = "__test__")
 
diff --git a/internal_deps.bzl b/internal_deps.bzl
index f0b12e8..ca2b071 100644
--- a/internal_deps.bzl
+++ b/internal_deps.bzl
@@ -12,6 +12,13 @@ def http_archive(name, **kwargs):
 
 def rules_py_internal_deps():
     "Fetch deps needed for local development"
+    http_archive(
+        name = "rules_python",
+        sha256 = "94750828b18044533e98a129003b6a68001204038dc4749f40b195b24c38f49f",
+        strip_prefix = "rules_python-0.21.0",
+        url = "https://github.com/bazelbuild/rules_python/archive/refs/tags/0.21.0.tar.gz",
+    )
+
     http_archive(
         name = "build_bazel_integration_testing",
         urls = [

Any other information?

This issue may be relevant here -- https://discuss.python.org/t/python3-11-seems-to-break-venv-created-from-non-system-python-with-zipped-stdlib/20874.

[Ubehebe]

I ran into the same problem when trying to switch from rules_python.

[siddharthab]

Any ETA on this? Python 3.11 has been out for quite some time. And there is no evidence that these rules support it. Taking such a long time to support a Python release reduces confidence that these rules are being maintained.

[RyanDraves]

I'm still experiencing this bug using rules_python = 0.29.0 and both aspect_rules_py = 0.5.0 and HEAD. This exact error occurs when trying to use py_venv:

load("@aspect_rules_py//py:defs.bzl", "py_venv")

py_venv(
    name = "venv",
)

Then a simple:

• bazel run //:venv
• .venv/bin/python -> Error occurs

The error goes away when downgrading the interpreter constraint to 3.10