Drop support for self upgrade/installation

[jaraco]

I'd like to consider dropping support for setuptools installing or upgrading itself, including the removal of the bootstrap script, and instead, rely entirely on pip for installation of setuptools. Doing so would (a) allow for the simple declaration of dependencies over vendoring them, (b) eliminate the need for a bootstrap process besides simple 'pip install', (c) eliminate the special handling for self installation, and (d) limit setuptools installation to the single-version installs that pip allows.

I expect some environments like bootstrap or other systems might be affected, so I'm cautious about this possibility. This ticket serves to host the conversation and considerations.

[bhaugen]

I'd suggest you also put some official upgrade instructions in the readme and docs. I don't see anything there, and I see the usual conflicting advice on the internet. Same for pypi.

I tried this, which I found in some comments here:
sudo pip install setuptools –upgrade –no-use-wheel
which seemed to work fine.

[jaraco]

The no-use-wheel recommendation is concerning, because that option indicates to setuptools that it needs to build itself, which is contrary to the goal of this ticket. I think one assumption is that setuptools would always be installed from wheels, and the build step would only be invoked on systems that already had a setuptools installation.

But you're right, the upgrade story needs to be well understood and documented.

[bhaugen]

I hope it's clear that I was not recommending –no-use-wheel, just copying and pasting.

[jayvdb]

I have recently needed to install setuptools manually because pip couldn't do it.

https://github.com/PyCQA/pyflakes/pull/76/files#diff-11c909939117928998b102a1fff7d363R33

[pombredanne]

I think it makes a lot of sense for a cleaner and consistent experience and clean separation of duties. It will make the overall package landscape more coherent: setuptools builds (and optionally installs from sources) eventually programmatically while pip bootstraps and installs.

The only area where I can fathom a possible issue for some rare tools is the lack of a stable function API in pip when using it as a library. But even then, spawning a pip subprocess is not a big issue (instead of calling its functions) from Python and this is something I routinely do.

[sigmavirus24]

@jayvdb let's be entirely forthright here, pyflakes needed to install setuptools manually on AppVeyor and pypy. That also should have been a separate bug report to fix that. It's related to this, yes, but not something that negates this goal.

[jaraco]

In the feature/581-depend-not-bundle branch, I've drafted an implementation that passes tests and installs nicely as a wheel. If you would like to try it out, for a limited time, the wheel of g767dcea0 can be installed as so:

pip install https://dl.dropboxusercontent.com/s/nn2nh33fvqkz2u5/setuptools-33.0.0.post20170101-py2.py3-none-any.whl

[AraHaan]

hmm I think having it upgrade itself (as an optional configuration option) should be added and default to not set to do it. So that way those who prefers this self upgrades can do it but before it upgrades itself it checks for upgrades to the other dependencies instead of shipping them with setuptools. But wouldn't this break pip for a little while as some parts of pip use setuptools?

So basically (as an optional config) have to self updates go in this order:

1. Any updated dependencies that setuptools requires? install them.
2. Any updates for setuptools? if so install them.

This way for those who (like me) does not always remember to pip install modules to know when there is a new version it would be easier to handle. Besides I have been working on things like this for my code so I dont have to remember to update them constantly to robot pip.

[jaraco]

@AraHaan I don't think there's any way to achieve what you describe. Part of the problem is that easy_install of any package over a pip-installed version of that package is liable to break. easy_install doesn't know how to uninstall pip-installed packages, so I wouldn't even expect a self-upgrade to work.

Now if you're talking about as strictly easy_install-managed environment, perhaps your suggestion would be plausible, but still very tricky. The way easy_install works is it downloads the sdist of the target package (i.e. setuptools), expands it to a temporary folder, builds a bdist_egg of that package, installs that egg, then determines which unsatisfied dependencies exist and installs them. The problem lies in the 'build' step, which because a 'setuptools' directory is present in that build directory, means that it's already using the newer version of setuptools to build itself. One would have to specifically special-case setuptools to inspect that package metadata and determine its dependencies to install those in advance. While possible, this approach would have additional pitfalls. What if this new version of setuptools relies on a dependency that's not compatible with the dependency that the current version of setuptools requires? Then this pre-install of dependencies might succeed but the install of the new version of setuptools might not succeed, leaving the environment in a broken state. Additionally, the way that setuptools isolates itself from the installation of the target package is to use in-process tricks (sandbox module) to hide existing imports of setuptools - that approach would also need to be expanded to include hiding of existing imports of dependencies.

In other words, it may be possible, but it's an awful lot of work and investment in a tool chain that's long been superseded by pip. If others wish to invest this effort, I won't resist it, but personally, my focus will be on pushing to the future rather than dragging forward the past.