Skip to content

Add --seed-packages. #2015

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Conversation

AllSeeingEyeTolledEweSew

Hello,

I added a --seed-packages option to the builtin seeders. This tells them to seed additional packages, with the intent being to provide a way to bootstrap any packages required to make pip work.

My use case is that my machine can only access the internet through a socks proxy (in fact I use tails, which routes all internet traffic through tor). This causes problems for most virtualenvs, because pip requires PySocks to download anything, so I have to bootstrap every virtualenv with a .whl file downloaded from a web browser.

This is fine for my own virtualenvs, but tools like tox and pre-commit create virtualenvs with little opportunity to customize bootstrapping. See pre-commit/pre-commit#1697 for some discussion.

I considered writing a custom seeder plugin, but I expect that virtualenv-creating tools would override my plugin with --seeder, based on considerations of how to seed, rather than what to seed. So I thought it was better to modify the builtin seeders.

For my case, I can do

export VIRTUALENV_SEED_PACKAGE=PySocks
export VIRTUALENV_EXTRA_SEARCH_DIR=/path/to/wheel

then, pre-commit and tox work normally everywhere.

I'm not sure if this is immediately useful for other cases, but I see a lot of try-import blocks in pip's source code, so I expect it's useful to someone somewhere.

This PR is still WIP while I add tests, but I thought I should open it now for discussion purposes.

@codecov
Copy link

codecov bot commented Nov 19, 2020

Codecov Report

Merging #2015 (cf254ab) into main (87877f8) will decrease coverage by 0.28%.
The diff coverage is 72.22%.

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #2015      +/-   ##
==========================================
- Coverage   94.36%   94.08%   -0.29%     
==========================================
  Files          86       86              
  Lines        4280     4295      +15     
==========================================
+ Hits         4039     4041       +2     
- Misses        241      254      +13     
Flag Coverage Δ
tests 94.08% <72.22%> (-0.29%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
src/virtualenv/seed/embed/base_embed.py 90.90% <71.42%> (-7.21%) ⬇️
src/virtualenv/seed/wheels/embed/__init__.py 91.66% <75.00%> (-8.34%) ⬇️
...ualenv/create/via_global_ref/builtin/pypy/pypy2.py 92.20% <0.00%> (-6.50%) ⬇️
src/virtualenv/util/path/_pathlib/via_os_path.py 89.62% <0.00%> (-1.89%) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 87877f8...cf254ab. Read the comment docs.

@@ -98,6 +109,15 @@ def add_parser_arguments(cls, parser, interpreter, app_data):
help="disable the periodic (once every 14 days) update of the embedded wheels",
default=not PERIODIC_UPDATE_ON_BY_DEFAULT,
)
parser.add_argument(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMHO this should be a separate plugin. This needs a lot more thought, on e.g. how this interacts with periodic update feature, transitive dependencies and where do we get these custom wheels you're injecting here. Perhaps you're after a pip functionality here, not virtualenv one. In general would be nice to open an issue to discuss such proposal before you lead on with a solution.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMHO this should be a separate plugin. This needs a lot more thought, on e.g. how this interacts with periodic update feature, transitive dependencies and where do we get these custom wheels you're injecting here. Perhaps you're after a pip functionality here, not virtualenv one. In general would be nice to open an issue to discuss such proposal before you lead on with a solution.

Should the discussion be here? I find it's often useful to propose a problem and solution together.

The wheels are supplied externally, hence my VIRTUALENV_EXTRA_SEARCH_DIR.

It shouldn't interact with periodic update; AFAICT the intent of that is only for the embedded wheels (maybe I should ensure that code is explicitly restricted to those).

Based on #1923 it seems like virtualenv won't support traversing dependencies for now. Until it does, you need to specify your whole dependency set in --seed-package, but this seems fine for now.

Updating pip would solve my problem as well; I thought that --seed-package was a smaller ask than pip vendoring more code. I thought they would want to be as lightweight as possible. If you think I'm more likely to succeed with pip, I can try there instead.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Based on #1923 it seems like virtualenv won't support traversing dependencies for now.

This is not true, just waiting for implementation. I'd consider this a blocking functionality request for this PR.

Until it does, you need to specify your whole dependency set in --seed-package, but this seems fine for now.

I disagree.

Updating pip would solve my problem as well; I thought that --seed-package was a smaller ask than pip vendoring more code.

Wishful thinking. 😊

If you want to go ahead with the current works for your half-baked solution I recommend using a plugin, but for acceptance in the core, we'd need at the very least traversing and handling dependencies too.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you want to go ahead with the current works for your half-baked solution I recommend using a plugin, but for acceptance in the core, we'd need at the very least traversing and handling dependencies too.

Fair enough! thanks for your time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants