Monkey patches in setuptools

[j0057]

Distutils is getting duck punched left and right by setuptools, which makes it much harder to contribute to or even comprehend the code.

It isn't made easier by the fact that one piece of the puzzle lives in the python stdlib (distutils), another lives on bitbucket (setuptools) and yet another lives on github (virtualenv/pip).

I think getting rid of the monkey patch business is very important, because it slows down development of the packaging ecosystem.

One way I can think of is to assimilate/fork distutils into setuptools, clean up the code and to officially deprecate distutils. End result: a puzzle with three pieces instead of four.

Possibly the setuptools/distutils dependency could be reversed to maintain backwards compatibility.

But all this does presume that it's even possible to get rid of distutils, I'm not sure if that's feasible.

[ncoghlan]

The core problem is unfortunately the combination of the relatively weak test suite for pkg_resources/setuptools with the arcane structure of the project internals.

The approach we're currently taking is progressively moving functionality out of setuptools into a new support package ("packaging") which is then shared by both pip and setuptools. Eventually we'll hopefully get to a point where both distutils & setuptools become optional dependencies that projects only use if they really want to, but it's going to be a long journey.

[pradyunsg]

2018 update: We have PEP 517/518 which would allow for the use of non-setuptools/distutils build systems.

See also: pypa/setuptools#417 and #127

---

Given that we've since figured out how to move things forward on one front (build systems) and the other front (improving setuptools) is in discussion elsewhere, I think this can be closed.

[Uzume]

I will be happy to see distutils disappear from the stdlib (being integrated/dissolved into setuptools).

Too bad we cannot get PEP 518, PEP 453 and friends extended/changed to allow for something more along the lines of "ensurepm" (package manager) so the Python stdlib would know how to install the appropriate package manager (e.g., pip) based on pyproject.toml and then the package manager could install the build system (e.g., setuptools + distutils + pkgresource + wheel, etc.) and the build system could install project dependencies and build and install/upload all the requested formats (sdists, wheels, etc.).

I basically see two key requirements: an ensurepm in stdlib for bootstrapping the package manager of choice and a specification for pyproject.toml (or similar so one knows what choices are made) that ensurepm, a package manager, a build system, etc. could use to configure requirements along the way.

Without a pyproject.toml, e.g., in situations where you just want to install an application, package managers could have metadata saying they were package managers and ensurepm could query pypi and provide a list of options and/or defaults, etc.

After this the use of requirements.txt, setup.cfg, setup.py, etc. would all be possible based on choosen/configured tools.

I do not really care what it is called--maybe ensurepackaging is better or bootstrappackager but hopefully you can understand my vision.

[ncoghlan]

@Uzume What you're describing sounds a lot like what we actually did with ensurepip: https://www.python.org/dev/peps/pep-0453/

For the original issue reported here, I'm going to close it as being superseded by #127 (which goes in the direction of having setuptools absorb distutils entirely, such that the way to get access to the distutils API is to run pip install setuptools).

[Uzume]

@ncoghlan In some ways yes, however, ensurepip does not allow one a choice in package managers. I believe generalizing ensurepip and extending pyproject.toml would be the right direction and similar to generalizing the choice in build tools that PEP 518 seems to be finally making possible. I got here from #127 but decided to respond here.

[ncoghlan]

No, we have zero intention of generalizing the default choice of installer, as that's part of the reference interpreter's user experience. Other interpreter distributions can, and do, offer their own choice of default dependency manager, and end users are always free to use pip to bootstrap something else, and then uninstall pip.

Generalising the choice of build tools is driven by the design goal of wanting to make the use of setuptools and distutils optional for new Python users, without breaking the world for developers and users of existing setuptools and distutils based projects.

[Uzume]

So what you are telling me is that there is no drive for a design goal of wanting to make the use of pip optional for new users without breaking the world for developers and users of existing pip installations.

It is possible to bootstrap another build system using setuptools, that does not mean that is optimal.

[ncoghlan]

pip is already optional, there's no need to make it more optional.

It just means folks are on their own bootstrapping their package manager if they choose not to use the one we provide - typically this is done using platform native packaging tools (like the Windows installers provided by ActiveState, Enthought and Anaconda for their respective Python distributions).

[Uzume]

Frankly I see pip as optional as setuptools (so why are build systems getting special treatment over packager managers in PEP 518?). On the other hand, as I expressed here originally, I will be glad to see distutils retired from stdlib.

[ncoghlan]

@Uzume I think there are some fundamental misconceptions in the way you're viewing the packaging problem space. I'm done commenting on it here, but you may want to read http://www.curiousefficiency.org/posts/2016/09/python-packaging-ecosystem.html and the preamble sections in the overly ambitious https://www.python.org/dev/peps/pep-0426/#development-distribution-and-deployment-of-python-software

[njsmith]

@Uzume Say you want to install both Django and NumPy into the same python environment. The django developers picked one build system to use for django, and wrote that down in django's pyproject.toml. The numpy developers picked a different build system and wrote that down in numpy's pyproject.toml. Now when you go to install them, your package manager can read django's pyproject.toml and numpy's pyproject.toml and figure out how to build both of them for you automatically.

But you wouldn't want Django and NumPy to be telling you what package manager to use. What if they disagreed? Then you couldn't even install both of them.

[Uzume]

@ncoghlan It is funny you mention that as I had just finished reading your post on the python packaging ecosystem before originally coming here. I shall go over the suggested reading again though. Thanks.

@njssmith Why not!? If I can have multiple build systems, I can also have multiple package management systems. I agree dependency databases would have to be agreed upon but we already have pypi repository metadata (which is a single format unless we get many more repos) and installed metadata with each distribution based on PEP 376 dist-info (and/or the earlier setuptools eggs formats). I agree having the project specify a package management system might have been a bit overzealous but I do think generalizing ensurepip to just ensurepm for some package manager (and not specifically pip) is probably a good idea.

[njsmith]

Yes, the whole point of having PEP 376 (and other similar PEPs) is to have a written standard for how package management works inside python environments, so that we're not stuck using just pip.

If ensurepm could install arbitrary package managers, that would mean it was... itself a package manager. The only reason ensurepip is simpler than pip is because it can't install anything except pip. Or put another way, you can already install any package manager you like, by running python -m ensurepip && python -m pip install $MY_FAVORITE_PACKAGE_MANAGER.

Also, as it happens, right now there aren't really any direct competitors to pip. (Probably the closest thing is conda, but it's actually totally different architecturally: it uses different package formats, different metadata, different repositories, and works at a different level of abstraction.) If one shows up and it turns out that changing ensurepip somehow would make it work better, then we can certainly have that discussion then, but until that happens the current situation seems reasonable enough.

[Uzume]

Yes, hopefully easy_install will die (or become something entirely different and better) and distutils will get integrated and/or dissolve into setuptools. Things like distlib are interesting for potentially creating other build tools and package managers. I wonder about the possibility of making a tool (or an add-on/change to an existing tool) to expunge egg formats converting all installed distributions to PEP 376 dist-info.