pip install -U breaks dependencies

[illbebach]

Environment

• pip version: 20.0.2
• Python version: 3.8.2
• OS: MacOS
• python installed by brew to avoid conflict with OS's python.

$ uname -a

Darwin illbebachs-iMac 19.4.0 Darwin Kernel Version 19.4.0: Wed Mar  4 22:28:40 PST 2020; root:xnu-6153.101.6~15/RELEASE_X86_64 x86_64

Related Issue

#7744 Handle conflicts from installed packages when updating other packages

Description

pip install -U breaks dependencies on installed packages, even when --upgrade-strategy=only-if-needed is specified on the command line. In the script below, I create a clean environment using venv and install pylint. pip install -U breaks dependencies of installed packages.

pip recognizes and reports the error regarding the dependency, but performs the upgrade anyway.

Expected behavior

pip should report the error and not perform the upgrade.

How to Reproduce

I ran the following script to demonstrate the error

#!/bin/bash

which python3.8
python3.8 --version

# setup a clean environment for testing
rm -rf clean_slate
python3.8 -m venv clean_slate
. clean_slate/bin/activate
which pip
which python

# upgrade basic packages to latest version
pip install --upgrade pip setuptools
pip --version
pip list

pip install pylint

pip list
pip check
pip list --outdated

# now perform the upgrade, where pip should fail
pip install --upgrade --upgrade-strategy=only-if-needed wrapt
pip check

Output

$ which python3.8
/usr/local/bin/python3.8

$ python3.8 --version
Python 3.8.2

$ rm -rf clean_slate

$ python3.8 -m venv clean_slate

$ . clean_slate/bin/activate

$ which pip
/Users/illbebach/clean_slate/bin/pip

$ which python
/Users/illbebach/clean_slate/bin/python

$ pip install --upgrade pip setuptools
... verbose output omitted ...
Successfully installed pip-20.0.2 setuptools-46.1.3

$ pip --version
pip 20.0.2 from /Users/illbebach/clean_slate/lib/python3.8/site-packages/pip (python 3.8)

$ pip list
Package    Version
---------- -------
pip        20.0.2 
setuptools 46.1.3 

$ pip install pylint
... verbose output omitted ...
Successfully installed astroid-2.3.3 isort-4.3.21 lazy-object-proxy-1.4.3 mccabe-0.6.1 pylint-2.4.4 six-1.14.0 wrapt-1.11.2

$ pip list
Package           Version
----------------- -------
astroid           2.3.3  
isort             4.3.21 
lazy-object-proxy 1.4.3  
mccabe            0.6.1  
pip               20.0.2 
pylint            2.4.4  
setuptools        46.1.3 
six               1.14.0 
wrapt             1.11.2 

$ pip check
No broken requirements found.

$ pip list --outdated
Package Version Latest Type 
------- ------- ------ -----
wrapt   1.11.2  1.12.1 sdist

$ pip install --upgrade --upgrade-strategy=only-if-needed wrapt
Collecting wrapt
  Using cached wrapt-1.12.1.tar.gz (27 kB)
ERROR: astroid 2.3.3 has requirement wrapt==1.11.*, but you'll have wrapt 1.12.1 which is incompatible.
Installing collected packages: wrapt
  Attempting uninstall: wrapt
    Found existing installation: wrapt 1.11.2
    Uninstalling wrapt-1.11.2:
      Successfully uninstalled wrapt-1.11.2
    Running setup.py install for wrapt: started
    Running setup.py install for wrapt: finished with status 'done'
Successfully installed wrapt-1.12.1

$ pip check
astroid 2.3.3 has requirement wrapt==1.11.*, but you have wrapt 1.12.1.

[uranusjr]

This was (incidentally) discussed quite extensively in discussions on implementing the new dependency resolver (notes from last week, search for installed packages for the relevant parts; follow-up discussion is around here). The opinions are conflicting, unfortunately, and has a lot of technical and UX implications.

[WSLUser]

Recommend using poetry as a work-around. https://github.com/python-poetry/poetry

[uranusjr]

It’s not a workaround, but a correct solution 🙂 More specifically, the reason pip doese not do the same is because it lacks a way to know the user’s intent, while a more workflow-oriented tool would be able to better derive the best action to take when conflicts happen.

[sbidoul]

@uranusjr @pfmoore @pradyunsg have you considered PEP 376 REQUESTED in that discussion? It is a way to mark top level (i.e. user supplied) dependencies. Although that would require a standardization round to update PEP 376, It could conceivably be extended to contain the version specifier that the user requested. So there would be a way to initialize constraints on what is allowed as version changes.

[uranusjr]

We did, but REQUESTED does not really solve the problem, since it does not tell pip whether the user allows the package to change version, and how much if so. It does help in certain upgrade situations, but does not align with all of them (although not orthogontal either). So the current approach is to keep looking for a more general solution without committing to restrictions too early, so we don’t find ourselves cornered when (if) the solution comes up.

[pradyunsg]

@uranusjr I think @sbidoul is suggesting augmenting/improving the REQUESTED file, to include that information -- about what the user requested -- to serve as a distributed manifest.

[pradyunsg]

Honestly, this is a known issue with pip's current resolver (see #988) and grant funded work is being undertaken right now to fix the broader issue.

I'm personally inclined to close this issue and classify it as "will be tackled as part of funded work on #988" + duplicate of #7744.

[uranusjr]

Thanks, I didn’t read the message clearly. Extending REQUESTED sounds like a good idea, we’ll need to talk about it a some point. There are some unresolved UX issues from the top of my head: Would pip install -U a only upgrade a within the previously specified range? If not, how do I do it? If so, how do user “re-request” a package with a different range?

---

I'm personally inclined to close this issue and classify it as "will be tackled as part of funded work on #988" + duplicate of #7744.

Does this mean we need to have a conclusion at the end of the resolver rollout—and does “stick to the current behaviour for now” count? Because I can easily imagine this being dragged out long after the resolver is implemented due to its many implications. I also don’t consider this a hard part of the resolver. I don’t mind discussing this as a part of the resolver work (or merging this into #7744), but I would hate to see #988 being kept open because we haven’t answered all the resolver-related issues.

[pfmoore]

I would hate to see #988 being kept open because we haven’t answered all the resolver-related issues.

Strong +1 on this. I think the goal of #988 should be simply to implement the new resolver. That would trigger a review of resolver-related issues to see which, if any, are still relevant with the new resolver, but we should not assume that the new resolver can't be "finished" before all existing resolver issues are closed.

[pradyunsg]

I didn't really express myself super clearly; mostly because I was trying to not go too far off-topic on this issue. A lot of this feels a lot like a misunderstanding, or at least a communication gap, caused by that.

I really wanted to not go off-topic here, but that's a lost cause now. So here goes... :)

---

We need to choose what we do instead of printing messages like:

ERROR: astroid 2.3.3 has requirement wrapt==1.11.*, but you'll have wrapt 1.12.1 which is incompatible.

I do think solving the user confusion that such messages (and their location in the logs) cause, is the right thing to do here. This issue is being filed not because pip doesn't know it didn't do the right thing -- it's because it doesn't communicate with the user about why. We've had users complain about this before: start digging from #6200 for getting all the history on this.

The fact that there's no way for pip to know what the correct thing to do during an upgrade is (which is what @uranusjr is understandably talking about) is orthogonal to this specific case -- where pip is doing the wrong thing and being OK with it.

Why is the message so poorly located? Because a college student hacked it together knowing fully well that it's not perfect but it was better than not communicating anything about the breakages (see #5000).

Why does the message not contain more detail? Because... the current resolver is horrible. Also, we don't actually have useful information (i.e. what the user requested initially, what extras need to be installed etc) to tell the user/pip itself what the correct thing to do would be. This is what @uranusjr is talking about.

Why does pip not straight up abort in these situations? It should, but backwards compatibility is a painful thing. Without a proper rollout with communication etc, it wouldn't make sense to change something so fundamental (and common?). We have that with the new resolver, and IMO we should do that.

All of this is exactly what #7744 is for -- choosing a behavior for dealing with already-installed packages for the new resolver, and clearly communicating about that behavior to the user. Solving #7744 (which I think is a very important thing to do as part of rolling out this new resolver) will solve one of the underlying issues that resulted in this issue being filed -- the not-so-great handling of conflicts with already-installed packages. pip has no way of knowing what the correct thing to do is... which it should communicate.

Also, in case I haven't been clear about this somehow: No, I don't consider the lack of "what the user wants", as something we should solve as part of #988 -- that's a metadata problem, not a problem we can solve with a new dependency resolver. Fixing the metadata problem is a separate task, that we should tackle as a separate task.

There's tracking issues for sub-tasks and Discourse discussions on various bits and pieces related to this. Those are discoverable. I don't think it's super discoverable if we discuss that stuff here.

---

I would hate to see #988 being kept open because we haven’t answered all the resolver-related issues.

I'm the last person on the planet who wants that to stay open. :)

I don't think we'll magically fix every dependency resolution related problem in the Python ecosystem that involves pip, with this new resolver. Once we have this new resolver rolled out (i.e. the default), #988 will be closed.

That said, it has not and does not make sense to me to keep 10s of issues on pip's issue tracker about "the resolver doesn't resolve dependencies correctly because <reasons>" (that's known -- #988 exists) so I've been closing them all and cross-linking them to #988. To aid with...

a review of resolver-related issues to see which, if any, are still relevant with the new resolver, but we should not assume that the new resolver can't be "finished" before all existing resolver issues are closed.

And the point is, #988 should be the one place we'd have to look for doing this review instead of searching for issues in the issue tracker, or elsewhere.

Does this mean we need to have a conclusion at the end of the resolver rollout—and does “stick to the current behaviour for now” count?

This specific issue -- I think making a choice in #7744 and implementing+rolling it out will be enough.

More broadly -- feature parity w/ current resolver and fixing a bunch of tricky-if-we-do-not-do-them-now issues (like #7744) would be what I'd like us to look into, before calling it the end.

Note that I said "look into" and not "resolve" -- I think it's important to consider these issues, and check if/what we can do to improve the situation without waiting on additional standardization. If the answer is nothing, that's perfectly fine and doing nothing is also OK.

---

Given that this issue has been cross-linked to #988, I don't see much value in keeping this open, and we've gone very off-topic here now.

I certainly consider discussing how we're gonna solve the metadata issue to be off-topic for this issue. OTOH, I didn't wanna be too pushy earlier, and just click the close button without waiting on responses to the clarification I posted about @sbidoul's suggestion.

Finally, I hate closing issues immediately after a wall-of-text comment (it feels equivalent to flips table and walks away). I'm not gonna do that here either, but I'll note that I think this discussion has gone off the rails and nearly everything we've talked about here has significantly better locations to talk about them instead (I'm thinking #7744, dev-syncup meetings, a new issue for discussing @sbidoul's suggestion, etc).

[pfmoore]

Thanks @pradyunsg for the context. I'm not sure it's completely off-topic here.

I note that @illbebach hasn't followed up on this discussion at all. Possibly because it veered into much deeper issues than they were expecting... But if I understand the original request correctly, it's basically saying that pip should treat the message

ERROR: astroid 2.3.3 has requirement wrapt==1.11.*, but you'll have wrapt 1.12.1 which is incompatible.

as an error, which is after all what it's describing itself as, and abort rather than continuing (and in effect treating the message as a warning).

I'm not sure if that's exactly what we'll end up doing, but I think there are two important points that we should address in any resolution here:

1. If the message says ERROR, we should abort. (Change the text otherwise)
2. This issue argues for aborting, which is maybe not what everyone wants (skirting the edge of the deep discussions here again) but is nevertheless a useful data point from an end user's point of view.

[pradyunsg]

I think both of those concerns are in-scope for #7744.

[illbebach]

Forgive the lateness of my reply. During the discussion, it quickly became apparent I do not have a full context of the history of this issue. That said, several times above, others stated "pip does not know what the user wants". To me, "what the user wants" can be stated in this simplistic statement:

Update everything to the newest version possible, without breaking anything.

If I were king, I would specify the proper behavior be much as the description of poetry, as mentioned by @WSLUser, behaves. Without other inputs, "pip install -U" should upgrade all installed packages, then give a note stating what happened. "pip install -U" is my typical use-case where I periodically want to update my system. A contrived example output would be

$ pip install -U
package update complete
NOTE: the following packages could not be updated to the latest available version, due to version conflicts with other installed packages:
package1 - updated to 1.2.3, even though 1.3.4 is available
package2 - updated to 3.4, even though 3.4.8 is available
...

Perhaps if —verbose is specified, for each package1,2,... above, print the newly installed version, the latest available version, and a possibly long list of packages which caused the conflict.

Even if the user specifies "pip install —update A B", it is possible that A or B cannot be updated to the latest available, due to conflicts with installed packages. In that case, it seems to me, the proper thing to do is still "Update A and B to the newest version possible, without breaking anything".

I recently became aware of the pipdeptree package, which I find helpful in seeing dependencies among packages.

Thank you all for adding so much context to the discussion. I was not aware of the longstanding issues or the broader context.

[illbebach]

One closing thought. Since only-if-needed is what we're discussing, my opinion is that pip should never break any dependencies. Only when the user says DWIM (i.e. eager) should pip have the right to break dependencies.

[pradyunsg]

Adding a cross reference to #9094