attempt an optimistic localhost-only search for "==" requirements #2114

bukzor · 2014-11-04T02:20:52Z

This is mostly for discussion.

In our very large project (>300 requirements) we're finding that the round-trips to the pypi server make the pip portion of our build frustratingly slow, even though we've ensured that all wheels are built and available on local disk.

Here I attempt to leverage the fact that there's no need to talk to the pypi server if we have a == requirement and a match is found locally.

There's two parts of this implementation that are (probably) objectionable, and I'd like advise on how to implement them more cleanly:

_find_requirement is called twice in the case of a package with pinned version, not found locally. This results in a series of messages "Do distributions at all found for foolib! Downloading foolib. Unpacking foolib. Successfully installed foolib."
The fake html page (line 751). This doesn't seem to cause any problems, but does seem like a pretty terrible hack.

On the other hand, if you don't mind these problems as much as I do, don't let me stop you from pulling :)

bukzor · 2014-11-04T20:42:19Z

All tests pass, after rebasing onto #2116 .

achembarpu · 2014-11-25T02:27:54Z

+1 - along with the new default caching mechanism, this is very useful for projects!

piotr-dobrogost · 2014-11-25T17:59:00Z

I think passing --no-index with --find-links should prevent reaching pypi server, doesn't it?

bukzor · 2014-11-25T18:00:33Z

@piotr Yes, and then it fails if anything is not yet cached. The objective
is to speed up normal operation.

On Tue, Nov 25, 2014 at 9:59 AM, Piotr Dobrogost [email protected]
wrote:

I think passing --no-index with --find-links should prevent reaching pypi
server, doesn't it?

—
Reply to this email directly or view it on GitHub
#2114 (comment).

Buck Golemon
Cell: (408) 202-3901

piotr-dobrogost · 2014-11-25T18:05:09Z

@bukzor
But you write

(...) even though we've ensured that all wheels are built and available on local disk

msabramo · 2014-12-16T05:22:11Z

tests/data/packages/symlinks/setup.cfg

@@ -1,3 +1,2 @@
 [egg_info]
 tag_build = dev
-tag_svn_revision = true


Is this related to this change or was it committed by accident?

msabramo · 2014-12-16T05:26:44Z

This seems useful as it will speed things up when you pin with == but not fail if you don't like it would with --no-index.

msabramo · 2014-12-21T17:15:04Z

Can you fix the merge conflicts? I'd love to play with this.

…equals Conflicts: pip/index.py

bukzor · 2014-12-22T16:59:04Z

@msabramo: merged. The conflict was trivial, an unrelated edit on the next line.

msabramo · 2014-12-22T18:14:10Z

Cool, thanks! There's a couple of test failures on Python 3, but I still will probably pull this down and play with it.

msabramo · 2014-12-24T01:06:20Z

How do I test this?

I did pip install --find-links=/opt/pip/wheels zope.sqlalchemy==0.7.3 and I have a wheel for zope.sqlalchemy==0.7.3 in /opt/pip/wheels so I was expecting it to use that not download, but it did do a download.

A repeat of that command didn't fetch it but that was seeming because of the caching and I saw the same behavior on the develop branch.

bukzor · 2014-12-24T02:21:48Z

You have to use --wheel-dir iirc

--phone is hard.
On Dec 23, 2014 5:06 PM, "Marc Abramowitz" [email protected] wrote:

How do I test this?

I did pip install --find-links=/opt/pip/wheels zope.sqlalchemy==0.7.3 and
I have a wheel for zope.sqlalchemy==0.7.3 in /opt/pip/wheels so I was
expecting it to use that not download, but it did do a download.

A repeat of that command didn't fetch it but that was seeming because of
the caching and I saw the same behavior on the develop branch.

—
Reply to this email directly or view it on GitHub
#2114 (comment).

msabramo · 2015-03-07T20:03:23Z

Shoot I need to look at this again. Quite similar to #2493 - right? Sorry, browsing on phone so hard to check in detail.

piotr-dobrogost · 2015-03-14T21:03:45Z

@msabramo

In 2493 you write

If an exact version is specified for install, and that version is
already installed, then there is no point going to pypi as no install
is needed.

whereas here there's this statement:

Here I attempt to leverage the fact that there's no need to talk to the pypi server if we have a == requirement and a match is found locally.

Looking briefly on this PR it seems that found locally above refers also to packages available locally for install and not only installed ones. If that's the case then features in both PRs are different. Btw, skipping pypi if there's locally available package for install would be nice to have. #2090 asks about similar feature but there there's no restriction to requirements with == operator.

piotr-dobrogost · 2015-03-14T21:53:47Z

Shouldn't this feature work also for <= requirements?

dstufft · 2015-04-07T04:15:47Z

Going to close this, we've decided in another PR that trying to do this is not something we generally want to do. It's an optimization and one that's likely to break in unexpected ways, especially with the new PEP 440 meaning of ==.

bukzor · 2015-04-07T16:00:44Z

link?

On Mon, Apr 6, 2015 at 9:15 PM, Donald Stufft [email protected]
wrote:

Going to close this, we've decided in another PR that trying to do this is
not something we generally want to do. It's an optimization and one that's
likely to break in unexpected ways, especially with the new PEP 440 meaning
of ==.

—
Reply to this email directly or view it on GitHub
#2114 (comment).

Buck Golemon
Cell: (408) 202-3901

pfmoore · 2015-04-07T16:25:37Z

#2631 and #2493

bukzor force-pushed the optimistic-double-equals branch from 3d8fc1f to ec5fd8c Compare November 4, 2014 18:57

Buck Golemon added 2 commits November 4, 2014 11:00

attempt an optimistic localhost-only search for "==" requirements

0ae9562

_get_pages grew a third argument

2cdf43a

bukzor force-pushed the optimistic-double-equals branch from ec5fd8c to 2cdf43a Compare November 4, 2014 19:01

msabramo reviewed Dec 16, 2014
View reviewed changes

Merge branch 'develop' of github.com:pypa/pip into optimistic-double-…

cf263e4

…equals Conflicts: pip/index.py

dstufft closed this Apr 7, 2015

asottile mentioned this pull request Jan 6, 2018

Investigate pip-faster & venv-update pypa/pipenv#1264

Closed

asottile mentioned this pull request Nov 12, 2018

Incompatible with pip 18.1 Yelp/venv-update#214

Closed

lock bot added the auto-locked Outdated issues that have been locked by automation label Jun 4, 2019

lock bot locked as resolved and limited conversation to collaborators Jun 4, 2019

attempt an optimistic localhost-only search for "==" requirements #2114

attempt an optimistic localhost-only search for "==" requirements #2114

Uh oh!

Conversation

bukzor commented Nov 4, 2014

Uh oh!

bukzor commented Nov 4, 2014

Uh oh!

achembarpu commented Nov 25, 2014

Uh oh!

piotr-dobrogost commented Nov 25, 2014

Uh oh!

bukzor commented Nov 25, 2014

Uh oh!

piotr-dobrogost commented Nov 25, 2014

Uh oh!

msabramo Dec 16, 2014

Choose a reason for hiding this comment

Uh oh!

msabramo commented Dec 16, 2014

Uh oh!

msabramo commented Dec 21, 2014

Uh oh!

bukzor commented Dec 22, 2014

Uh oh!

msabramo commented Dec 22, 2014

Uh oh!

msabramo commented Dec 24, 2014

Uh oh!

bukzor commented Dec 24, 2014

Uh oh!

msabramo commented Mar 7, 2015

Uh oh!

piotr-dobrogost commented Mar 14, 2015

Uh oh!

piotr-dobrogost commented Mar 14, 2015

Uh oh!

dstufft commented Apr 7, 2015

Uh oh!

bukzor commented Apr 7, 2015

Uh oh!

pfmoore commented Apr 7, 2015

Uh oh!

Uh oh!