Skip to content

Rewrite bootstrap script using pip #3223

Open
@Flamefire

Description

@Flamefire

There are currently 15 open issues that the boostrap script is broken on some environments: https://github.com/easybuilders/easybuild-framework/issues?utf8=%E2%9C%93&q=is%3Aissue+is%3Aopen+bootstrap+in%3Atitle

One of the main reasons is the patched distribute and setuptools package used. Furthermore it makes the whole install process very hacky and has several issues I described in a partial fix PR: #3211

Summary:

  • PYTHONPATH and sys.path not kept in sync leading to different behavior based on the setuptools used (spawned by loading it inside the current env [sys.path] but it may spawn a new process which uses PYTHONPATH)
  • Resulting instillation of unwanted versions is possible (partially due to the above but also because EB < 4 uses vsc-utils with a minimum version but requires a maximum version [manually installed on boostrap])
  • Using setup.py install is deprecated as is using the ancient disutils, easy_install or ez_setup.py, see e.g. Drop support for self upgrade/installation pypa/setuptools#581
  • The only valid way of installing something is pip install

Proposal: Use pip and virtualenv for installation

I created a Proof Of Concept in a branch: https://github.com/Flamefire/easybuild-framework/blob/test_pip_for_bootstrap/easybuild/scripts/bootstrap_eb.py

Details:

  • If pip exists, use it to install a specific, up-to-date pip into a temporary folder, else use get-pip.py (the official way to install pip) to do the same
  • Use this pip to add setuptools and wheels packages required to build source packages (stage 0)
  • install easybuild and its prerequisites into a new temporary folder (stage 1)
  • reset PYTHONPATH to stage0 (maybe not even required if the EC is correctly handled)
  • import easybuild from stage1 and build EasyBuild from generated EasyConfig (stage 2)

Potential problem: System installed packages. If a package exists, pip will uninstall it before installing the new one. There is --ignore-installed which avoids this, but this will also ignore existing dependencies leading to the following situation:

  • pip install vsc-install<0.11.4 --ignore-installed -> No dependencies, all good
  • pip install vsc-base<2.9.0 --ignore-installed -> Ignores installed vsc-install and pulls in the latest available one because vsc-base requests vsc-install>=0.9.0
  • Finally fails due to incompatibilites

Potential solution: Use a virtualenv. I never really used it but it seems to solve exactly this problem: Create a new environment which is free of any python packages and hence we don't need --ignore-installed. virtualenv itself has no dependencies so we can install that with --ignore-installed first and run it.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions