Skip to content

bpo-41011: venv -- add more variables to pyvenv.cfg (GH-30382) #30382

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 17 commits into from
Jan 7, 2022
Merged

bpo-41011: venv -- add more variables to pyvenv.cfg (GH-30382) #30382

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
9857b28
added env_exec_cmd and env_create_command; added them to the test
akulakov Jan 4, 2022
34f7943
add news entry
akulakov Jan 4, 2022
0c7643c
update var names
akulakov Jan 4, 2022
e4bb85e
unskip a few tests
akulakov Jan 4, 2022
f93cae0
use sys.executable not sys.argv
akulakov Jan 4, 2022
fabe6eb
update news
akulakov Jan 4, 2022
74bbfeb
update executable to be sys.executable and command to include args ba…
akulakov Jan 5, 2022
abef4f4
unskip tests
akulakov Jan 5, 2022
1184ffb
fix test_defaults
akulakov Jan 5, 2022
19d825e
fix test_defaults again
akulakov Jan 5, 2022
ac22946
update news
akulakov Jan 5, 2022
c891cac
update to generate --symlinks and --copies based on the OS; add promp…
akulakov Jan 6, 2022
5afeb8a
add prompt to test
akulakov Jan 6, 2022
727c515
quotes for orig_prompt
akulakov Jan 6, 2022
0d44a7d
fix basic test for NT
akulakov Jan 6, 2022
9f06b55
fix test config file for NT + copies flag
akulakov Jan 6, 2022
0e519ec
mock setup pip for speed
akulakov Jan 7, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 37 additions & 1 deletion Lib/test/test_venv.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@
from test.support.os_helper import (can_symlink, EnvironmentVarGuard, rmtree)
import unittest
import venv
from unittest.mock import patch
from unittest.mock import patch, Mock

try:
import ctypes
Expand Down Expand Up @@ -114,13 +114,49 @@ def test_defaults(self):
executable = sys._base_executable
path = os.path.dirname(executable)
self.assertIn('home = %s' % path, data)
self.assertIn('executable = %s' %
os.path.realpath(sys.executable), data)
copies = '' if os.name=='nt' else ' --copies'
cmd = f'command = {sys.executable} -m venv{copies} --without-pip {self.env_dir}'
self.assertIn(cmd, data)
fn = self.get_env_file(self.bindir, self.exe)
if not os.path.exists(fn): # diagnostics for Windows buildbot failures
bd = self.get_env_file(self.bindir)
print('Contents of %r:' % bd)
print(' %r' % os.listdir(bd))
self.assertTrue(os.path.exists(fn), 'File %r should exist.' % fn)

def test_config_file_command_key(self):
attrs = [
(None, None),
('symlinks', '--copies'),
('with_pip', '--without-pip'),
('system_site_packages', '--system-site-packages'),
('clear', '--clear'),
('upgrade', '--upgrade'),
('upgrade_deps', '--upgrade-deps'),
('prompt', '--prompt'),
]
for attr, opt in attrs:
rmtree(self.env_dir)
if not attr:
b = venv.EnvBuilder()
else:
b = venv.EnvBuilder(
**{attr: False if attr in ('with_pip', 'symlinks') else True})
b.upgrade_dependencies = Mock() # avoid pip command to upgrade deps
b._setup_pip = Mock() # avoid pip setup
self.run_with_capture(b.create, self.env_dir)
data = self.get_text_file_contents('pyvenv.cfg')
if not attr:
for opt in ('--system-site-packages', '--clear', '--upgrade',
'--upgrade-deps', '--prompt'):
self.assertNotRegex(data, rf'command = .* {opt}')
elif os.name=='nt' and attr=='symlinks':
pass
else:
self.assertRegex(data, rf'command = .* {opt}')

def test_prompt(self):
env_name = os.path.split(self.env_dir)[1]

Expand Down
24 changes: 24 additions & 0 deletions Lib/venv/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,7 @@ def __init__(self, system_site_packages=False, clear=False,
self.symlinks = symlinks
self.upgrade = upgrade
self.with_pip = with_pip
self.orig_prompt = prompt
if prompt == '.': # see bpo-38901
prompt = os.path.basename(os.getcwd())
self.prompt = prompt
Expand Down Expand Up @@ -178,6 +179,29 @@ def create_configuration(self, context):
f.write('version = %d.%d.%d\n' % sys.version_info[:3])
if self.prompt is not None:
f.write(f'prompt = {self.prompt!r}\n')
f.write('executable = %s\n' % os.path.realpath(sys.executable))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(I appreciate that this PR is long since merged, but just following-up on a surprising pyvenv.cfg file that I saw recently, and git-blamed it back to here)

I'm not at all sure why realpath is important here - the sys.executable should be good enough when being called directly. When called through the EnvBuilder, I don't see a reason for this either. 🤔

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right, it is a difference when you're not on Windows. But thinking about it, you could view the config as recording the actual interpreter used instead of the path that could change on you in the future (e.g., some symlink later gets changed to point to another interpreter).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this perspective (I hadn't considered it)! I'm not sure that using a config file to log information about how a venv was created is a good idea, but can appreciate the idea. Since this is a machine readable file, I would be concerned that it eventually gets used for non-log/audit purposes.

I've been digging into symlink issues with venv a lot recently, and think calling realpath is quite problematic in other context (i.e. when determining home). More details of that in #106045.

No follow-up needed here - just flagging it, and depending on the outcome of the linked issue, perhaps in the future I may request that this information become non realpath, or under a "history of how this venv was created" section of the config (perhaps in the form of a comment).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See also https://discuss.python.org/t/virtual-environments-vs-nix-python-upgrades/12588

args = []
nt = os.name == 'nt'
if nt and self.symlinks:
args.append('--symlinks')
if not nt and not self.symlinks:
args.append('--copies')
if not self.with_pip:
args.append('--without-pip')
if self.system_site_packages:
args.append('--system-site-packages')
if self.clear:
args.append('--clear')
if self.upgrade:
args.append('--upgrade')
if self.upgrade_deps:
args.append('--upgrade-deps')
if self.orig_prompt is not None:
args.append(f'--prompt="{self.orig_prompt}"')

args.append(context.env_dir)
args = ' '.join(args)
f.write(f'command = {sys.executable} -m venv {args}\n')

if os.name != 'nt':
def symlink_or_copy(self, src, dst, relative_symlinks_ok=False):
Expand Down
3 changes: 3 additions & 0 deletions Misc/NEWS.d/next/Library/2022-01-03-21-03-50.bpo-41011.uULmGi.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
Added two new variables to *pyvenv.cfg* which is generated by :mod:`venv`
module: *executable* for the executable and *command* for the command line used
to create the environment.