pip wheel should not build if the wheel already exists

[pfmoore]

Currently

    pip wheel mock
    pip wheel mock

downloads and builds mock twice. It would be good if it checked whether the wheel was already present, and if so did not rebuild it. This would be extremely useful for automatically maintaining local wheel caches. (A flag to force a build would also be useful, to allow building from a development source that has changed, but the version has not changed).

[pfmoore]

So it is - I missed that, sorry.

[dholth]

Works great as its own issue though. Michele's original wheel feature behaved in the non-rebuilding way.

If you think about "pip wheel" as not "build a wheel" but "obtain a wheel", or introduce the confusing "pip can build a wheel from an existing wheel by doing nothing" strategy then there is less rebuilding. This strategy is also going to be important when some packages are published wheel-only and have no sdist.

Does it even rebuild if you add the wheel directory to --find-links and pass --use-wheel: pip wheel --use-wheel --find-links=wheelhouse ?

Clearly we'll be adding a --rebuild flag at some point.

We might also want to start using the build numbers feature: wheel-1.4.0-1.tag-tag-tag.whl wheel-1.4.0-2.tag-tag-tag.whl ...

[trifonov-bc]

@qwcode This feature will be really helpful. When to expect this feature?

[rassie]

I would like to rephrase this issue a little bit: pip 1.4.1 actually skips re-building a wheel if it's already built.

$ pip wheel --find-links wheels --use-wheel --wheel-dir=wheels -v mock
Downloading/unpacking mock
  Local files found: $cwd/wheels/mock-1.0.1-py26-none-any.whl
  Ignoring link $local_pypi_mirror/packages/mock/download/3124/mock-1.0b1.tar.gz#md5=bf7e9acc7215eeb805bef1d77864e5f2 (from $local_pypi_mirror/simple/mock/), version 1.0b1 is a pre-release (use --pre to allow).
  Ignoring link $local_pypi_mirror/packages/mock/download/3125/mock-1.0b1.zip#md5=93495a8376e8fc9a3f6cb36524ebe15b (from $local_pypi_mirror/simple/mock/), version 1.0b1 is a pre-release (use --pre to allow).
  Ignoring link $local_pypi_mirror/packages/mock/download/3132/mock-0.7.0rc1.tar.gz#md5=cb4f34459e45607b75a304bde6353ac3 (from $local_pypi_mirror/simple/mock/), version 0.7.0rc1 is a pre-release (use --pre to allow).
  Ignoring link $local_pypi_mirror/packages/mock/download/3133/mock-0.7.0rc1.zip#md5=84b71a6ce1e328e27fc4bdc37afd8029 (from $local_pypi_mirror/simple/mock/), version 0.7.0rc1 is a pre-release (use --pre to allow).
  Ignoring link $local_pypi_mirror/packages/mock/download/3134/mock-0.7.0b4.tar.gz#md5=12af216898f08f85963139efe5d9582b (from $local_pypi_mirror/simple/mock/), version 0.7.0b4 is a pre-release (use --pre to allow).
  Ignoring link $local_pypi_mirror/packages/mock/download/3135/mock-0.7.0b4.zip#md5=0d85979ef556456aaf5a0766a2152831 (from $local_pypi_mirror/simple/mock/), version 0.7.0b4 is a pre-release (use --pre to allow).
  Ignoring link $local_pypi_mirror/packages/mock/download/3136/mock-0.7.0b3.tar.gz#md5=2799e896f528f5cecec8cc55a5d5da21 (from $local_pypi_mirror/simple/mock/), version 0.7.0b3 is a pre-release (use --pre to allow).
  Ignoring link $local_pypi_mirror/packages/mock/download/3137/mock-0.7.0b3.zip#md5=af4a1053a9ad7e935ff86d1de9d1042c (from $local_pypi_mirror/simple/mock/), version 0.7.0b3 is a pre-release (use --pre to allow).
  Ignoring link $local_pypi_mirror/packages/mock/download/3138/mock-0.7.0b2.tar.gz#md5=a411b39ef90120a4992a5d5dfdd0a52d (from $local_pypi_mirror/simple/mock/), version 0.7.0b2 is a pre-release (use --pre to allow).
  Ignoring link $local_pypi_mirror/packages/mock/download/3139/mock-0.7.0b2.zip#md5=e83f8af34d80e0491c19789aa9925267 (from $local_pypi_mirror/simple/mock/), version 0.7.0b2 is a pre-release (use --pre to allow).
  Ignoring link $local_pypi_mirror/packages/mock/download/3140/mock-0.7.0b1.tar.gz#md5=a21ba6804db7e30c7b5dc7025f72abac (from $local_pypi_mirror/simple/mock/), version 0.7.0b1 is a pre-release (use --pre to allow).
  Ignoring link $local_pypi_mirror/packages/mock/download/3141/mock-0.7.0b1.zip#md5=ba5124c632e0394a6e9ec25d681fd374 (from $local_pypi_mirror/simple/mock/), version 0.7.0b1 is a pre-release (use --pre to allow).
  Using version 1.0.1 (newest of versions: 1.0.1, 1.0.1, 1.0.1, 1.0.0, 1.0.0, 0.8.0, 0.8.0, 0.7.2, 0.7.2, 0.7.1, 0.7.1, 0.7.0, 0.7.0, 0.6.0, 0.6.0, 0.5.0)
Building wheels for collected packages: mock
  *Skipping building wheel*: file:///$cwd/wheels/mock-1.0.1-py26-none-any.whl
Cleaning up...
  Removing temporary dir $home/.virtualenvs/wheel/build...

As you see, building is skipped, but the roundtrip to PyPI is still there even though it has found a valid wheel already. --no-index is not an option here, since some dependencies might still be missing, but there should be an option to disable PyPI querying for existing wheels.

[dstufft]

That's to be expected. It doesn't know if the wheel it has is the highest version that matches the spec. It needs to discover versions. Otherwise if you built a Wheel today, in a year you'll still be building that same old version.

[rassie]

Fair enough, I agree in this case. However, the same happens with exact version specification pip wheel -f wheels/ --use-wheel --wheel-dir=wheels -v mock==1.0.1 -- I hope we can assume that a released version doesn't change? (except with dependency_links pointing to Github's master snapshots, but I would keep this special case out of pip)

[dstufft]

Unfortunately a released version can change. People are free to delete a version from PyPI and then upload a new one with the same version with different code. In pip install we use the hash from the page to detect if this happens. I'm not familiar with how pip wheel works to be able to say if it does the same hash check but it probably should.

[rassie]

If I understand you correctly, in a perfect system a versioned requirement would also include the exact checksum of the sdist, which also would be recorded in the built wheel or a checksum of the wheel itself, in case of sdist-less wheels. In that case a check can be performed offline, not needing a roundtip to PyPI. Unversioned requirements would be checked against PyPI and rebuilt if either a new release is available or the checksum has changed. Would this be a desirable (and implementable) scenario?

[dholth]

I'm sure pip is not checking any hashes when it is "Skipping building wheel" above. Instead it will be looking for all possible links (for both sdists and wheels) for the package. It will pick the best match out of all the collected links. At this phase pip is considering individual archives and there can be multiple archives per package version.

Pip would need to short-circuit the "collecting links" phase to consider only your local wheel and not the active indices.

IIRC pip does not consider it but wheels also specify a build number in which case a wheel on the index could have a greater build number than your local wheel and would be preferred.

[fdemmer]

while something like this works fine for updating only missing wheels:

pip wheel -w /tmp/wheelhouse/ --use-wheel -f file:///tmp/wheelhouse/ -r requirements.txt

i found it odd, when first using wheel, that it did not behave like "install" does with the "-d" argument, which downloads only missing source packages using this simple command (and does not install anything):

pip install -d /tmp/packages/ -r requirements.txt

i use this to create and maintain a source package cache. imho, maintaining a wheel cache should work very similar.

maybe it makes sense to change wheel to act in a similar way as "install" with only the "-w" argument and use a "--rebuild" switch to force rebuilding (or just fetching) existing wheels (even with pinned version and best using a build number instead of overwriting files).

[qwcode]

I looked at this briefly a while back. 2 points to consider

1. we have to go through the process of unpacking the sdist to determine if the project's setup.cfg has universal=1
2. ideally, we'd use the same code bdist_wheel uses, to determine the filename we'll be looking for (pip is not in control of the built filename). the filename code is currently an instance method of wheel.bdist_wheel.bdist_wheel which is a distutils command class, so it's not in the most friendly location.

[qwcode]

maybe it makes more sense for bdist_wheel to own this logic, and have pip just use the right flags when calling it.

[meshy]

i found it odd, when first using wheel, that it did not behave like "install" does with the "-d" argument

This is what I'm aiming for too. Am I right in saying that there's currently no combination of flags that will give me that behaviour?

[SpotlightKid]

Is this still being worked on? Almost a year with no progress...

I find the default behavior to rebuild existing wheels very surprising and AFAICS there is currently no way to use the local wheelhouse as a cache and not rebuild existing wheels but still fetch missing packages from PyPI with pip alone.

[techtonik]

+1 for caching by default and explicit --rebuild to force rebuilding when developing.

[ssbarnea]

Is there a way to tell wheel to compile and cache builds if they do not already exists in the current cache?

I am trying to build a cache that contains binaries but it seems that for at least some packages, wheel wants to recompile them every time.

pip3 wheel -w ~/.cache/wheels --use-wheel -f https://repo/wheels/ -r requirements.txt
# rsync to upload the content of ~/.cache/wheels to the web server https://repo/wheels/
# now, if I run again the first command I would expect to cache the binaries from the webserver, but it seems that it does recompile them

How can I make wheel to recompile only what's needed?

[mgeisler]

I've had good success with using both --find-link and --wheel-dir. This takes just 5 seconds compared to ~10 minutes without the cache:

$ pip wheel --find-links wheels --wheel-dir wheels .
Processing /home/user/foo
Collecting pandas (from foo==0.1)
  File was already downloaded /home/user/foo/wheels/pandas-0.17.1-cp35-cp35m-linux_x86_64.whl
Collecting matplotlib (from foo==0.1)
  File was already downloaded /home/user/foo/wheels/matplotlib-1.5.1-cp35-cp35m-linux_x86_64.whl
Collecting click (from foo==0.1)
  File was already downloaded /home/user/foo/wheels/click-6.3-py2.py3-none-any.whl
Collecting zabbix-api (from foo==0.1)
  File was already downloaded /home/user/foo/wheels/zabbix_api-0.4-py3-none-any.whl
Collecting numpy>=1.7.0 (from pandas->foo==0.1)
  File was already downloaded /home/user/foo/wheels/numpy-1.10.4-cp35-cp35m-linux_x86_64.whl
Collecting pytz>=2011k (from pandas->foo==0.1)
  File was already downloaded /home/user/foo/wheels/pytz-2015.7-py2.py3-none-any.whl
Collecting python-dateutil>=2 (from pandas->foo==0.1)
  File was already downloaded /home/user/foo/wheels/python_dateutil-2.5.0-py2.py3-none-any.whl
Collecting cycler (from matplotlib->foo==0.1)
  File was already downloaded /home/user/foo/wheels/cycler-0.10.0-py2.py3-none-any.whl
Collecting pyparsing!=2.0.0,!=2.0.4,>=1.5.6 (from matplotlib->foo==0.1)
  File was already downloaded /home/user/foo/wheels/pyparsing-2.1.0-py2.py3-none-any.whl
Collecting six>=1.5 (from python-dateutil>=2->pandas->foo==0.1)
  File was already downloaded /home/user/foo/wheels/six-1.10.0-py2.py3-none-any.whl
Skipping pandas, due to already being wheel.
Skipping matplotlib, due to already being wheel.
Skipping click, due to already being wheel.
Skipping zabbix-api, due to already being wheel.
Skipping numpy, due to already being wheel.
Skipping pytz, due to already being wheel.
Skipping python-dateutil, due to already being wheel.
Skipping cycler, due to already being wheel.
Skipping pyparsing, due to already being wheel.
Skipping six, due to already being wheel.
Building wheels for collected packages: foo
  Running setup.py bdist_wheel for foo
  Stored in directory: /home/user/foo/wheels
Successfully built foo

[Tinche]

We, for example, wipe our wheelhouse before every build. We do this because, later in the build process, the wheelhouse is sent to a Docker container and we don't want old dependencies accumulating in there. So the --find-links/--wheel-dir approach doesn't really work in our case.

Now, if pip wheel actually populated the cache (the global pip cache, not the wheelhouse) with a wheel when it builds one (which it doesn't, as far as I know), and if it would fetch the wheel from the cache instead of building it (which it does, I think), that'd speed up our build process substantially.

Pip will cache a wheel it builds on pip install, why not on pip wheel too?

[xavfernandez]

Pip will cache a wheel it builds on pip install, why not on pip wheel too?

I see no particular reason, this could be implemented.

[chadrik]

I've had good success with using both --find-link and --wheel-dir.

@mgeisler Thanks for the suggestion, that worked perfectly. You saved me a lot of headache.

[dstufft]

This appears to have been fixed:

$ pip wheel mock
Collecting mock
  Using cached mock-2.0.0-py2.py3-none-any.whl
  Saved ./mock-2.0.0-py2.py3-none-any.whl
Collecting six>=1.9 (from mock)
  Using cached six-1.10.0-py2.py3-none-any.whl
  Saved ./six-1.10.0-py2.py3-none-any.whl
Collecting pbr>=0.11 (from mock)
  Using cached pbr-2.0.0-py2.py3-none-any.whl
  Saved ./pbr-2.0.0-py2.py3-none-any.whl
Skipping mock, due to already being wheel.
Skipping six, due to already being wheel.
Skipping pbr, due to already being wheel.
$ pip wheel mock
Collecting mock
  File was already downloaded /Users/dstufft/.virtualenvs/tmp-d841adaf4c274c1/t/mock-2.0.0-py2.py3-none-any.whl
Collecting pbr>=0.11 (from mock)
  File was already downloaded /Users/dstufft/.virtualenvs/tmp-d841adaf4c274c1/t/pbr-2.0.0-py2.py3-none-any.whl
Collecting six>=1.9 (from mock)
  File was already downloaded /Users/dstufft/.virtualenvs/tmp-d841adaf4c274c1/t/six-1.10.0-py2.py3-none-any.whl
Skipping mock, due to already being wheel.
Skipping pbr, due to already being wheel.
Skipping six, due to already being wheel

[sublee]

Is "File was already downloaded" of pip wheel enough fast? "Requirement already satisfied" of pip install is much faster. I tried pip wheel -w wheelhouse -f wheelhouse as mentioned in this issue.

In my case I have almost 50 requirements. pip wheel's skipping behavior is too slow for me.