pip install -U might reinstall same version of package

[remram44]

My project consists of several Python packages that depend on each other: pkgA, and pkgB that depends on pkgA.

I install pkgA with pip install -U -e ./pkgA

Then I install pkgB with pip install -U -e ./pkgB. Doing that, pip removes pkgA, downloads one from pypi (the exact same version that was installed) and installs it. Then I lose half an hour trying to figure out things in a broken dev environment.

I'm going back to running setup.py develop because setuptools work, but it sounds like something you might want to know.

[pfmoore]

Why are you using -U? It seems to me that this would work fine if you just didn't ask for an upgrade. (I'm not saying that what pip does is right, though).

[remram44]

I want to upgrade pkgA and pkgB, in case they got new dependencies, or a version change, or new entry points/package data; I also want to upgrade the external dependencies.

[xavfernandez]

pip install -U -e ./pkgB -e ./pkbA should work.

[remram44]

It's what I have right now (allsetups.sh), we'll see. It's the fourth attempt already.

[qwcode]

closing. I think @xavfernandez 's advice is correct.

[remram44]

The fact that it re-installs the same version of a package over the editable one is of no concern to anyone?

[qwcode]

oh, I see, reopened. will try to recreate locally and see where this breaks down

[pfmoore]

Personally, my feeling is that there's a conceptual difference between "normal" requirements and requirements based on a VCS URL or editables. A lot of things have problematic edge cases when the 2 come into contact. That's not much help in this particular case, and a "tactical" fix is better for now, but longer term I think pip needs to come up with a better classification of the different types of requirements and how they work together...

[remram44]

pip has code to determine the version of an editable package (using things like git tags, see get_src_requirement()), so it makes sense to use that information. But I agree that it might be clunky in some situations.

[qwcode]

I tried to recreate this with local clones of:

• deform : https://github.com/Pylons/deform
• peppercorn: https://github.com/Pylons/peppercorn

deform depends on peppercorn.

things worked as we would want, not as you describe

git clone [email protected]:Pylons/peppercorn.git
git clone [email protected]:Pylons/deform.git
pip install -e peppercorn/
pip install -U -e deform/

the final install reports:

Requirement already up-to-date: peppercorn>=0.3 in ./peppercorn (from deform==2.0a3.dev0)

and pip show peppercorn reports the local editable install

Name: peppercorn
Version: 0.5
Location: /tmp/peppercorn

[remram44]

I can still reproduce it here:

C:\Users\Remi\Desktop\testpip>pip --version
pip 6.0.8 from C:\Python2.7\lib\site-packages (python 2.7)

C:\Users\Remi\Desktop\testpip>git describe
0.6.1-1-g6dd6844

C:\Users\Remi\Desktop\testpip>pip install -U -e ./pkga
Obtaining file:///C:/Users/Remi/Desktop/testpip/pkga
Installing collected packages: pkga
  Running setup.py develop for pkga
    Creating c:\python2.7\lib\site-packages\pkga.egg-link (link to .)
    Adding pkga 0.6.1 to easy-install.pth file
    Installed c:\users\remi\desktop\testpip\pkga
Successfully installed pkga

C:\Users\Remi\Desktop\testpip>pip install -U -e ./pkgb
Obtaining file:///C:/Users/Remi/Desktop/testpip/pkgb
Could not find any downloads that satisfy the requirement pkga>=0.6 in c:\users\remi\desktop\testpip\pkga (from pkgb==0.6.1)
Collecting pkga>=0.6 (from pkgb==0.6.1)
  No distributions at all found for pkga>=0.6 in c:\users\remi\desktop\testpip\pkga (from pkgb==0.6.1)

C:\Users\Remi\Desktop\testpip>

pkga/setup.py:

from setuptools import setup

setup(name='pkga',
      version='0.6.1',
      py_modules=['pkga'],
      description="Descripted",
      author="Remi Rampin",
      author_email='[email protected]',
      classifiers=['Development Status :: 3 - Alpha'])

pkgb/setup.py:

from setuptools import setup

setup(name='pkgb',
      version='0.6.1',
      py_modules=['pkgb'],
      install_requires=['pkga>=0.6'],
      description="Descripted",
      author="Remi Rampin",
      author_email='[email protected]',
      classifiers=['Development Status :: 3 - Alpha'])

[remram44]

On Debian:

(venv27)vagrant@debian-78-amd64:/vagrant$ pip --version
pip 6.0.8 from /home/vagrant/venv27/local/lib/python2.7/site-packages (python 2.7)
(venv27)vagrant@debian-78-amd64:/vagrant$ pip install -U -e ./pkga
Obtaining file:///vagrant/pkga
Installing collected packages: pkga
  Running setup.py develop for pkga
    Creating /home/vagrant/venv27/lib/python2.7/site-packages/pkga.egg-link (link to .)
    Adding pkga 0.6.1 to easy-install.pth file
    Installed /vagrant/pkga
Successfully installed pkga
(venv27)vagrant@debian-78-amd64:/vagrant$ pip install -U -e ./pkgb
Obtaining file:///vagrant/pkgb
Could not find any downloads that satisfy the requirement pkga>=0.6 in ./pkga (from pkgb==0.6.1)
Collecting pkga>=0.6 (from pkgb==0.6.1)
  No distributions at all found for pkga>=0.6 in ./pkga (from pkgb==0.6.1)

See here: https://github.com/remram44/_pip-install-issue
Travis: https://travis-ci.org/remram44/_pip-install-issue/builds/53297517

[qwcode]

I can still reproduce it here:

what you reproduced is different than the original description.
the original description has a local install being overwritten by the pypi version.
in your fake pkga/pkgb scenario, it falters because it can't find pkga on pypi.

I think your 2nd reproduction is a real bug, but not as concerning as your first claim.

can you reproduce the first claim in a way I can reproduce?

[remram44]

This is the exact same bug, pip looks up a package on pypi although it's already available locally. I have no doubt that if pkga had been there it would have been installed.

If you replace 'pkga' with the name of any package that's on pypi (with the appropriate version) you'll get the first error I showed.

[qwcode]

If you replace 'pkga' with the name of any package that's on pypi
(with the appropriate version) you'll get the first error I showed.

I replaced:

install_requires=['pkga>=0.6']

with

install_requires=['peppercorn>=0.5']

and did not recreate what you describe (where peppercorn was already installed from local path)

I got

Requirement already up-to-date: peppercorn>=0.5 in ./peppercorn (from pkgb==0.6.1)

[remram44]

Yes, I see that as well with peppercorn: https://travis-ci.org/remram44/_pip-install-issue/builds/53298176

I don't know what to tell you, I see what I see, so does Travis. Can't debug further right now but something is definitely wrong.

[qwcode]

I see what I see, so does Travis

where's the travis link that shows this? I see 2 travis links in this issue. one is the case with peppercorn working correctly, the other is the case of pkga not existing on pypi.

[msabramo]

I might be misunderstanding this because this is fairly long and difficult for me to follow, but I feel like this might be the expected behavior.

pip's default place to look for packages is PyPI; I don't think that because you installed a package from a local file on your system that this will change that and not make it look at PyPI.

So you're doing:

pip install -U -e ./pkga

and getting pkga==0.6.1. Nothing controversial there.

Now you do:

pip install -U -e ./pkgb

even though you specified -U and only ./pkgb, to pip this doesn't mean only to upgrade pkgb. If you pass -U/--upgrade, it tries to upgrade everything (for better or worse). So it also wants to try to upgrade pkga. So it looks on PyPI for it because that's pip's default place to look. Pip is not what I would call conservative -- just because pkgb only needs pkga>=0.6, doesn't mean that pip will be satisfied with the pkga==0.6.1 already installed, especially since you specified -U and explicitly told it to try to upgrade all the things. So I think looking on PyPI is not so surprising.

It seems a little odd/wrong to me that if it doesn't find pkga on PyPI that it would output an error and abort the whole thing. After all, it's trying to upgrade, but you already have something for pkga that meets the requirement so it seems it should fail to upgrade pkga but then continue to upgrade pkgb.

As for the fact that it installs it from PyPI if it finds it -- I guess this could be right or wrong depending on how you look at it. Intuitively, I can see why this seems wrong to install something that you already have. On the other hand, and this is speculation, I think that probably --upgrade does this so that you can easily reinstall a local package without bumping the version number -- e.g.:

python setup.py sdist
pip install --upgrade foo.tar.gz
<make some changes but don't bump version number>
python setup.py sdist
pip install --upgrade foo.tar.gz

If --upgrade didn't reinstall despite the version number being the same, that would be kind of a pain, because you'd have to bump the version whenever you make a change.

But I'll let someone who knows things better say whether this is right or not.

[remram44]

especially since you specified -U and explicitly told it to try to upgrade all the things.

Yes, but how do you justify replacing a package with the same version number?

After all, it's trying to upgrade, but you already have something for pkga that meets the requirement so it seems it should fail to upgrade pkga but then continue to upgrade pkgb.

We agree. Failing because something you didn't ask for and that isn't required didn't work is definitely a peculiar approach.

[qwcode]

Yes, but how do you justify replacing a package with the same version number?

but so far, you haven't shown that pip does that. still waiting for a public example that I can recreate. everything so far, shows that it doesn't do that.

seems it should fail to upgrade pkga but then continue to upgrade pkgb
we agree

and I agree to. my plan at this point is to open a bug for this lesser issue and close this issue, unless you can provide an example for the original description.

[remram44]

Do that, I probably won't lose any more time arguing with the pip team again. (do check out #1829 someday though)

[qwcode]

arguing? just trying to get a reproducible example...

[qwcode]

ok, closing as "can't reproduce" for now.
created #2502 for the other issue mentioned.