pip wheel should not build if the wheel already exists

[pfmoore]

Currently

    pip wheel mock
    pip wheel mock

downloads and builds mock twice. It would be good if it checked whether the wheel was already present, and if so did not rebuild it. This would be extremely useful for automatically maintaining local wheel caches. (A flag to force a build would also be useful, to allow building from a development source that has changed, but the version has not changed).

[pfmoore]

So it is - I missed that, sorry.

[dholth]

Works great as its own issue though. Michele's original wheel feature behaved in the non-rebuilding way.

If you think about "pip wheel" as not "build a wheel" but "obtain a wheel", or introduce the confusing "pip can build a wheel from an existing wheel by doing nothing" strategy then there is less rebuilding. This strategy is also going to be important when some packages are published wheel-only and have no sdist.

Does it even rebuild if you add the wheel directory to --find-links and pass --use-wheel: pip wheel --use-wheel --find-links=wheelhouse ?

Clearly we'll be adding a --rebuild flag at some point.

We might also want to start using the build numbers feature: wheel-1.4.0-1.tag-tag-tag.whl wheel-1.4.0-2.tag-tag-tag.whl ...

[trifonov-bc]

@qwcode This feature will be really helpful. When to expect this feature?

[rassie]

I would like to rephrase this issue a little bit: pip 1.4.1 actually skips re-building a wheel if it's already built.

$ pip wheel --find-links wheels --use-wheel --wheel-dir=wheels -v mock
Downloading/unpacking mock
  Local files found: $cwd/wheels/mock-1.0.1-py26-none-any.whl
  Ignoring link $local_pypi_mirror/packages/mock/download/3124/mock-1.0b1.tar.gz#md5=bf7e9acc7215eeb805bef1d77864e5f2 (from $local_pypi_mirror/simple/mock/), version 1.0b1 is a pre-release (use --pre to allow).
  Ignoring link $local_pypi_mirror/packages/mock/download/3125/mock-1.0b1.zip#md5=93495a8376e8fc9a3f6cb36524ebe15b (from $local_pypi_mirror/simple/mock/), version 1.0b1 is a pre-release (use --pre to allow).
  Ignoring link $local_pypi_mirror/packages/mock/download/3132/mock-0.7.0rc1.tar.gz#md5=cb4f34459e45607b75a304bde6353ac3 (from $local_pypi_mirror/simple/mock/), version 0.7.0rc1 is a pre-release (use --pre to allow).
  Ignoring link $local_pypi_mirror/packages/mock/download/3133/mock-0.7.0rc1.zip#md5=84b71a6ce1e328e27fc4bdc37afd8029 (from $local_pypi_mirror/simple/mock/), version 0.7.0rc1 is a pre-release (use --pre to allow).
  Ignoring link $local_pypi_mirror/packages/mock/download/3134/mock-0.7.0b4.tar.gz#md5=12af216898f08f85963139efe5d9582b (from $local_pypi_mirror/simple/mock/), version 0.7.0b4 is a pre-release (use --pre to allow).
  Ignoring link $local_pypi_mirror/packages/mock/download/3135/mock-0.7.0b4.zip#md5=0d85979ef556456aaf5a0766a2152831 (from $local_pypi_mirror/simple/mock/), version 0.7.0b4 is a pre-release (use --pre to allow).
  Ignoring link $local_pypi_mirror/packages/mock/download/3136/mock-0.7.0b3.tar.gz#md5=2799e896f528f5cecec8cc55a5d5da21 (from $local_pypi_mirror/simple/mock/), version 0.7.0b3 is a pre-release (use --pre to allow).
  Ignoring link $local_pypi_mirror/packages/mock/download/3137/mock-0.7.0b3.zip#md5=af4a1053a9ad7e935ff86d1de9d1042c (from $local_pypi_mirror/simple/mock/), version 0.7.0b3 is a pre-release (use --pre to allow).
  Ignoring link $local_pypi_mirror/packages/mock/download/3138/mock-0.7.0b2.tar.gz#md5=a411b39ef90120a4992a5d5dfdd0a52d (from $local_pypi_mirror/simple/mock/), version 0.7.0b2 is a pre-release (use --pre to allow).
  Ignoring link $local_pypi_mirror/packages/mock/download/3139/mock-0.7.0b2.zip#md5=e83f8af34d80e0491c19789aa9925267 (from $local_pypi_mirror/simple/mock/), version 0.7.0b2 is a pre-release (use --pre to allow).
  Ignoring link $local_pypi_mirror/packages/mock/download/3140/mock-0.7.0b1.tar.gz#md5=a21ba6804db7e30c7b5dc7025f72abac (from $local_pypi_mirror/simple/mock/), version 0.7.0b1 is a pre-release (use --pre to allow).
  Ignoring link $local_pypi_mirror/packages/mock/download/3141/mock-0.7.0b1.zip#md5=ba5124c632e0394a6e9ec25d681fd374 (from $local_pypi_mirror/simple/mock/), version 0.7.0b1 is a pre-release (use --pre to allow).
  Using version 1.0.1 (newest of versions: 1.0.1, 1.0.1, 1.0.1, 1.0.0, 1.0.0, 0.8.0, 0.8.0, 0.7.2, 0.7.2, 0.7.1, 0.7.1, 0.7.0, 0.7.0, 0.6.0, 0.6.0, 0.5.0)
Building wheels for collected packages: mock
  *Skipping building wheel*: file:///$cwd/wheels/mock-1.0.1-py26-none-any.whl
Cleaning up...
  Removing temporary dir $home/.virtualenvs/wheel/build...

As you see, building is skipped, but the roundtrip to PyPI is still there even though it has found a valid wheel already. --no-index is not an option here, since some dependencies might still be missing, but there should be an option to disable PyPI querying for existing wheels.

[dstufft]

That's to be expected. It doesn't know if the wheel it has is the highest version that matches the spec. It needs to discover versions. Otherwise if you built a Wheel today, in a year you'll still be building that same old version.

[rassie]

Fair enough, I agree in this case. However, the same happens with exact version specification pip wheel -f wheels/ --use-wheel --wheel-dir=wheels -v mock==1.0.1 -- I hope we can assume that a released version doesn't change? (except with dependency_links pointing to Github's master snapshots, but I would keep this special case out of pip)

[dstufft]

Unfortunately a released version can change. People are free to delete a version from PyPI and then upload a new one with the same version with different code. In pip install we use the hash from the page to detect if this happens. I'm not familiar with how pip wheel works to be able to say if it does the same hash check but it probably should.

[rassie]

If I understand you correctly, in a perfect system a versioned requirement would also include the exact checksum of the sdist, which also would be recorded in the built wheel or a checksum of the wheel itself, in case of sdist-less wheels. In that case a check can be performed offline, not needing a roundtip to PyPI. Unversioned requirements would be checked against PyPI and rebuilt if either a new release is available or the checksum has changed. Would this be a desirable (and implementable) scenario?

[dholth]

I'm sure pip is not checking any hashes when it is "Skipping building wheel" above. Instead it will be looking for all possible links (for both sdists and wheels) for the package. It will pick the best match out of all the collected links. At this phase pip is considering individual archives and there can be multiple archives per package version.

Pip would need to short-circuit the "collecting links" phase to consider only your local wheel and not the active indices.

IIRC pip does not consider it but wheels also specify a build number in which case a wheel on the index could have a greater build number than your local wheel and would be preferred.

[fdemmer]

while something like this works fine for updating only missing wheels:

pip wheel -w /tmp/wheelhouse/ --use-wheel -f file:///tmp/wheelhouse/ -r requirements.txt

i found it odd, when first using wheel, that it did not behave like "install" does with the "-d" argument, which downloads only missing source packages using this simple command (and does not install anything):

pip install -d /tmp/packages/ -r requirements.txt

i use this to create and maintain a source package cache. imho, maintaining a wheel cache should work very similar.

maybe it makes sense to change wheel to act in a similar way as "install" with only the "-w" argument and use a "--rebuild" switch to force rebuilding (or just fetching) existing wheels (even with pinned version and best using a build number instead of overwriting files).

[qwcode]

I looked at this briefly a while back. 2 points to consider

1. we have to go through the process of unpacking the sdist to determine if the project's setup.cfg has universal=1
2. ideally, we'd use the same code bdist_wheel uses, to determine the filename we'll be looking for (pip is not in control of the built filename). the filename code is currently an instance method of wheel.bdist_wheel.bdist_wheel which is a distutils command class, so it's not in the most friendly location.