[bootstrap] Fix building setuptools #3211
Closed
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #3200
Fixes #2428
This started as a fix for #3200 but falls a bit short of solving the installation for a Python3 environment without setuptools already installed.
What works/got fixed:
Problems to be resolved in follow-up:
TLDR: Don't use easy_install anymore
The
distribute
package used installs an incredibly ancient version of setuptools which doesn't work out of the box anymore (see above and the need for patching setuptools and the setup_distribute script). There was/is even an ez_setup.py script which seems to superseed this: https://bootstrap.pypa.io/ez_setup.pyHowever even that is deprecated: pypa/setuptools#581
I started to use the https://bootstrap.pypa.io/get-pip.py to install pip (if missing) and then use
pip
to install setuptools just asdistribute
would have. But I hit another, currently hidden problem: The EBbootstrap
script has a functionprep
which sets up PYTHONPATH for some temporary location wherestage{0,1,2}
are going to be installed to. It then proceeds to import easy_install and start installing dependencies and packages of EB (e.g. vsc-*) into that environment. The problem: This environment is not part of the current environment (sys.path
). Hence easy_install fails to find the dependencies it just installed and tries to redownload them, which then fails because it doesn't find thewheels
package which is also part of the stage0 environment (and would be part of asystem()
invoked environment) This is (at least partially) due to the mismatch of the environments, i.e. python core libs check for support of features and then runsystem()
but those features are not part of that sub-process to to mismatch of PYTHONPATH vs sys.path.IMO the easiest solution would be just use pip run via
system
and install pip if it is missing. Potentially (very probably required) also upgrade pip and setuptools into the stage1 directory.Summary of proposed EB bootstrap procedure:
--prefix
set to the temporary folder using pipsystem("eb ...")
to install the real EasyBuild via stage1 EBpip
instead to do that.In addition: Setup CI for the boostrap script