Specify substitute sources for given packages, dump list of packages installed during "pip install"

[klmitch]

We have a need to do repeatable builds of a virtual environment, with substitutions of certain packages that may be discovered during the install. There are two essential enhancements I'm looking for here:

1. Have some means to inform pip—either command line or some sort of configuration file—that, when a dependency is discovered for package foo, we would prefer that pip install github.com/foolabs/foo.git instead of a version found on PyPi or from a VCS repository link. Think of this as an "override" feature.
2. Have a command line argument that causes pip to emit a file in requirements.txt format that, when passed to pip with "--no-deps -r file.txt", will cause pip to install exactly the same packages and versions that it did last time. For packages installed from PyPi, of course, this could be done with a "pip freeze"; but when some packages are installed from git repositories (or overrides, as suggested above), "pip freeze" won't generate enough information. (In terms of VCS repositories, I would prefer that the originally specified branch or tag be used, rather than a specific commit ID; that is, if the repository link is "git+https://github.com/foolabs/foo.git@devel#egg=foo", this should be the link included in the emitted file, rather than "git+https://github.com/foolabs/foo.git@a736e…#egg=foo".)

(I saw an open issue against "pip freeze" that, if resolved, could address my enhancement 2; but I'd still prefer that the file be emitted by "pip install" itself, if at all possible.)

Without these enhancements, it is necessary for us to do the requisite work to generate the requirements file manually; this has, needless to say, proven to be somewhat error-prone. I am contemplating building a tool that tries to use pip.req.parse_requirements() directly, but I'm not yet certain that that approach will net me the information I desire here…

[klmitch]

I've thought about this some more, and I've realized that the first element is the most important. I've also figured out the UI interface I'd like to see: I propose adding a "--prefer" option to command line and requirements file processing that would add the option's argument to a "preferences" dictionary (mapping package name to the InstallRequirement instance). Then, when pip encounters a requirement matching the name, it would substitute the value from the preferences dictionary for the InstallRequirement it would normally construct at that point.

The operative points here are that the preferences declared with --prefer would not be installed if they were not referenced somewhere else in the requirements chain; that would enable me to create a "preferences.txt" containing preferences for a large number of virtual environments without worrying about installing unnecessary requirements into the venv.

For the second point above, I figure giving an InstallRequirement instance a str() method if it doesn't already have one would probably be sufficient; then, you just dump the full list of installed requirements at the end of installation if the "--dump" CLI option is given.

[piyush0101]

👍
+1

[dstufft]

So right now this should be already possible if I understand what you're asking for here. At least the first item.

If you do pip install thing something-that-depends-on-thing then pip will prefer the definition you used in "thing" over the definition in the dependency information for something-that-depends-on-thing. This holds true for requirements files as well, so you can do:

thing>=2.0
something-that-depends-on-thing  # This depends on thing <2

And in this case the effective dependency spec will be thing>=2.0.

This also holds true for things from repositories:

git+https://github.com/foo/thing.git#egg=thing
something-that-depends-on-thing

Then "thing" will be installed from the git repository instead of from PyPI.

[klmitch]

OK, cool. I have two basic comments here, though. My first comment is that, as I describe it above, --prefer wouldn't actually install "thing"; the goal here is that I might have one file with a bunch of --prefer selections, where only some may be relevant to one particular "something-that-depends-on-thing", while a partially disjoint set may apply to "something-that-depends-on-thing-2".

The second comment is a behavior I saw from pip; I haven't tested this with the method you lay out above, and I'll be looking into trying that, but since you didn't mention it as an example, I wanted to bring it up. The situation was that "something-that-depends-on-thing" had "thing" specified as a git URL. We had generated a virtual env by installing our version of "thing" from a different git repository, then invoked pip again to install the requirements.txt for "something-that-depends-on-thing"—and watched in consternation as pip actually uninstalled our version of "thing" and installed the version of "thing" from the git URL specified in the requirements.txt file for "something-that-depends-on-thing". So, do you know if this would work as expected if we had kept everything all together in one requirements file?

[dstufft]

It might! This behavior isn't well tested :(

[dstufft]

Closing this, the ask appears to already be possible and there has been no movement in 3 years.