Skip to content

[bootstrap] Fix building setuptools #3211

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed

Conversation

Flamefire
Copy link
Contributor

@Flamefire Flamefire commented Feb 24, 2020

Fixes #3200
Fixes #2428

This started as a fix for #3200 but falls a bit short of solving the installation for a Python3 environment without setuptools already installed.

What works/got fixed:

Problems to be resolved in follow-up:
TLDR: Don't use easy_install anymore

The distribute package used installs an incredibly ancient version of setuptools which doesn't work out of the box anymore (see above and the need for patching setuptools and the setup_distribute script). There was/is even an ez_setup.py script which seems to superseed this: https://bootstrap.pypa.io/ez_setup.py
However even that is deprecated: pypa/setuptools#581

I started to use the https://bootstrap.pypa.io/get-pip.py to install pip (if missing) and then use pip to install setuptools just as distribute would have. But I hit another, currently hidden problem: The EB bootstrap script has a function prep which sets up PYTHONPATH for some temporary location where stage{0,1,2} are going to be installed to. It then proceeds to import easy_install and start installing dependencies and packages of EB (e.g. vsc-*) into that environment. The problem: This environment is not part of the current environment (sys.path). Hence easy_install fails to find the dependencies it just installed and tries to redownload them, which then fails because it doesn't find the wheels package which is also part of the stage0 environment (and would be part of a system() invoked environment) This is (at least partially) due to the mismatch of the environments, i.e. python core libs check for support of features and then run system() but those features are not part of that sub-process to to mismatch of PYTHONPATH vs sys.path.

IMO the easiest solution would be just use pip run via system and install pip if it is missing. Potentially (very probably required) also upgrade pip and setuptools into the stage1 directory.

Summary of proposed EB bootstrap procedure:

  • Prepare stage1 environment in new temporary folder
    • Check for pip and if non-existant install into stage1 via get-pip script
    • Upgrade pip and setuptools with --prefix set to the temporary folder using pip
    • Install EB and its dependencies into that stage1 folder using pip
  • In stage2
    • Use system("eb ...") to install the real EasyBuild via stage1 EB
    • double-check if the workarounds for pre 4.0 EB setting PYTHONPATH and force-installing vsc-install and setuptools are still required for this setup. I'd expect EB to handle that. If that is required use pip instead to do that.

In addition: Setup CI for the boostrap script

  • Use a dedicated .github file for those tests instead of duplicates in travis and github testing in non-clean environments
  • For all python versions test in default environment AND in stripped environment where all packages are removed

@Flamefire Flamefire force-pushed the fix_building_setuptools branch from 8f0df01 to 802f600 Compare September 14, 2020 07:30
@boegel boegel modified the milestones: next release (4.3.1), 4.x Oct 14, 2020
@Flamefire
Copy link
Contributor Author

As the bootstrap script gets deprecated closing this

@Flamefire Flamefire closed this Jun 23, 2021
@Flamefire Flamefire deleted the fix_building_setuptools branch August 8, 2024 08:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

bootstrap_eb.py, version 20200203.01, error installing distribute_setup.py Bootstrap fails on CentOS 6
3 participants