Shared modules across venvs

[jaggzh]

What's the problem this feature will solve?

Duplication/disk-space use of redundant modules across venvs, installation time, etc.

Describe the solution you'd like

This might be more appropriate for python venv, conda, etc. projects... BUT... what if we have a location where the versions of our modules are installed, and pip (and python/venvs) use this location, and the required version, OR use symlinks to those versions.
This would:

1. Allow the re-use of already-installed modules
2. Leading to preventing the need to duplicate them several, or even dozens of times across projects/venvs
3. Increase system speed

With links/shortcuts, one wouldn't even need to do reference count tracking.

Alternative Solutions

Hmm. No.

Additional context

I'm not sure how it would best be implemented, while [potentially] honoring different virtual environment types, except if the user sets up some optional paths to use. But I think the cleanliness of it requires the ability for the links to be handled by the pip system. Not sure.

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[uranusjr]

I think this has been raised before, but pip is in the wrong abstraction layer to do this since it lives inside a virtual environment and doesn’t have knowledge to the global system—that’s what virtual environments are for. I would raised this on discuss.python.org instead to propose this to a higher level tool that can manage multiple virtual environments instead.

[notatallshaw]

By the way, conda does this already with hardlinks for conda packages when it can. It mentions it somewhere in the dev guide: https://docs.conda.io/projects/conda/en/22.9.x/dev-guide/deep-dive-install.html#generating-the-transaction-and-the-corresponding-actions

[jaggzh]

I think this has been raised before, but pip is in the wrong abstraction layer to do this since it lives inside a virtual environment and doesn’t have knowledge to the global system—that’s what virtual environments are for. I would raised this on discuss.python.org instead to propose this to a higher level tool that can manage multiple virtual environments instead.

I believe that's the whole point -- pip merely needs to manage the virtual environments, but have the added capability of a centralized list of module storage locations to use in those virtual envs. If it doesn't have this available, it operates the same as it always has. Attempting to re-implement the version testing "outside" pip would require something else to manage the version tests and resolution, and I don't see a way to elegantly do this in another layer of abstraction. (Granted, if someone is using conda, then conda's system "is that system", but if they're using pip ...)

[pfmoore]

pip isn't (and won't become) a virtual environment manager. You'd need a higher level tool to handle this. Tools like PDM, hatch and poetry exist at this level, and they work with pip, so I'm not sure your concern that this can't be done at a higher layer is valid.

[pradyunsg]

Consolidating into #11092