Test that venvs created in tests are properly isolated

[pfmoore]

See the discussion in #11320 for context. It appears that virtual environments created via the stdlib venv module in tests are not properly isolated. This PR adds a test case to reproduce the issue.

[sbidoul]

I can make this test fail locally on Ubuntu 20 with Python 3.7, 3.9, 3.10 (from deadsnakes). But it succeeds with the system Python 3.8. That is weird indeed.

[pfmoore]

Interesting. I'm extremely glad someone can reproduce it. I was starting to think I was going crazy!

[sbidoul]

Correction: after removing the .nox directory, it fails with all Python versions.

[sbidoul]

And the venv built by the test has a pyvenv.cfg with a home = <the bin dir of the nox session virtualenv>.

So we have a virtualenv whose python is from a "parent" virtualenv, and the paths of that parent virtualenv leaks in the child virtualenv.

That is something I have noticed before in other circumstances and never tried to understand why.
So I generally avoid to create virtualenvs with a python that is itself in a virtualenv.

I can reproduce this in a ubuntu 20 container:

# apt update
# apt install python3 python3-pip python3-venv --no-install-recommends
# pip install --user "virtualenv<20"
# cd /tmp
# python3 -m virtualenv testvenv
# testvenv/bin/python -c "import venv; venv.EnvBuilder().create('/tmp/testvenv-child')"
# ./testvenv-child/bin/python -m site
sys.path = [
    '/tmp',
    '/tmp/testvenv/lib/python38.zip',
    '/tmp/testvenv/lib/python3.8',
    '/tmp/testvenv/lib/python3.8/lib-dynload',
    '/usr/lib/python3.8',
    '/tmp/testvenv/lib/python3.8/site-packages',
]
USER_BASE: '/root/.local' (exists)
USER_SITE: '/root/.local/lib/python3.8/site-packages' (exists)
ENABLE_USER_SITE: False

In the result of python -m site we see the paths of the the parent virtualenv.

That does not explain why nox does it, but not always...

[pfmoore]

Interesting. I thought "home" (which AIUI corresponds to base_prefix) was meant to inherit the parent's base_prefix in the case of a venv created from a venv. So that sounds to me like a bug (probably in something Debian patched, as it doesn't seem to happen in a python.org build. But it's hard for me to be sure, the Windows logic is significantly different. But I was tesring with pyenv builds of Python on Ubuntu, which I believe are built from source (and hence are not patched).

If this were reproducible in a version of Python created from python.org sources, I'd be tempted to call it a bug, although it might get rejected as the docs for sys.base_prefix are a little vague on the matter.

However, I'm 100% sure that the behaviour of "leaking" packages from the base env into the child env is a bug.

[pfmoore]

I raised python/cpython#95469 to get the sys.base_prefix docs clarified.

[sbidoul]

@pfmoore the problem does not occur when the parent virtualenv has been created with venvor virtualenv>=20.

So we are back to #11288.

A possible reason for the problem not occurring locally for you is that you have virtualenv>=20 in the environment where nox is installed.

[pfmoore]

Wow, that's weird, given that I thought the parent virtualenv was the one created by nox. Maybe I missed a layer somewhere... But ultimately, I agree, if this is basically #11288, we've already got that in hand. I feel that maybe it's worth retaining this PR, if only to remind us to check the basic assumption that venvs are isolated once #11288 is fixed. But at that point, I'll probably change the test for --python to assert that the venv starts empty, so we'll have it covered there. So let's leave this PR open for now, but no need to do anything more with it.

Thanks for the diagnostic work here - I'd reached a point where I was completely stumped, and I doubt I'd have got to the bottom of it without your help.

[SpecLad]

In the result of python -m site we see the paths of the the parent virtualenv.

I believe this is because venv relies on the site module to update sys.prefix to point to the current (child) virtual environment, but virtualenv<20 adds a custom site module to the parent virtual environment that doesn't have this functionality. So when ./testvenv-child/bin/python imports site, it gets /tmp/testvenv/lib/python3.x/site.py, which fails to update sys.prefix, and so the site packages from /tmp/testvenv are added to sys.path instead of the ones from /tmp/testvenv-child.

Interesting. I thought "home" (which AIUI corresponds to base_prefix) was meant to inherit the parent's base_prefix in the case of a venv created from a venv.

FWIW, the parent's base_prefix isn't set correctly either:

$ ./testvenv/bin/python -S -c 'import sys; print(sys.base_prefix)'
/tmp/testvenv