Make the interpreter paths initialization more consistent

[FFY00]

As Steve pointed out in #98790 (comment), it would be really nice to make paths initialization more consistent across platforms.

I am creating this issue to start documenting all the quirks and inconsistencies.

• Embedded Python 3.11 on windows warns about <exec_prefix> #98790

[davidhewitt]

One quirk I have noticed seems to be that launching an embedded interpreter with Py_Initialize behaves differently with respect to virtualenvs depending on OS.

(Specifically here I'm talking about Rust applications embedding Python via PyO3 - which basically just calls Py_InitializeEx under the hood & doesn't try to do anything smart.)

On Linux, I've never had any problems - launching an embedded interpreter with a virtualenv loaded will access the packages from that environment.

On macOS and Windows, users have reported that the system Python appears to always be preferred, and while there's been some discussion about possible causes / solutions I wonder if you think it's relevant to this issue?
PyO3/pyo3#1741
PyO3/pyo3#1896

[FFY00]

Yep, I think that's definitely relevant to this issue. Just beware that I have no estimate of when this work might be completed, so I'd advise to still try to work things out downstream if these issues are affecting your users.

[davidhewitt]

Appreciate that, thanks. I think even if this work landed in 3.12, given we support all supported Python versions in PyO3 we probably will want to find some solution downstream before Python 3.11 hits EOL ;)

[zooba]

There's a nice long list of quirks marked in getpath.py already. I don't know whether they can all be resolved, or if they should.

Many differences are due to / being inherently secure on POSIX and inherently insecure on Windows (so e.g., we don't want to search up through all directories because the top level is a free-for-all). Most of the rest are just because getpath.c and getpathp.c had different implementations.

For building alternate Python entry points¹ that recognise venv, you will really want to look at getpath.py for the logic (search venv). Chances are it's not working because you're calling any one of the functions that override normal search, since an embedded Python shouldn't be using a venv.²

Footnotes

1. I call it this because venv is a feature of the pythonX.Y command, not the language. So once you replace the command, if you want it to behave the same, you're actually building an entrypoint. A hosted/embedded Python runtime should have its own set of library directories, rather than trying to rely on the discovery mechanisms used by pythonX.Y. ↩

2. See the previous footnote. ↩