Description
There are currently 15 open issues that the boostrap script is broken on some environments: https://github.com/easybuilders/easybuild-framework/issues?utf8=%E2%9C%93&q=is%3Aissue+is%3Aopen+bootstrap+in%3Atitle
One of the main reasons is the patched distribute and setuptools package used. Furthermore it makes the whole install process very hacky and has several issues I described in a partial fix PR: #3211
Summary:
- PYTHONPATH and sys.path not kept in sync leading to different behavior based on the setuptools used (spawned by loading it inside the current env [sys.path] but it may spawn a new process which uses PYTHONPATH)
- Resulting instillation of unwanted versions is possible (partially due to the above but also because EB < 4 uses vsc-utils with a minimum version but requires a maximum version [manually installed on boostrap])
- Using
setup.py install
is deprecated as is using the ancient disutils, easy_install or ez_setup.py, see e.g. Drop support for self upgrade/installation pypa/setuptools#581 - The only valid way of installing something is
pip install
Proposal: Use pip and virtualenv for installation
I created a Proof Of Concept in a branch: https://github.com/Flamefire/easybuild-framework/blob/test_pip_for_bootstrap/easybuild/scripts/bootstrap_eb.py
Details:
- If
pip
exists, use it to install a specific, up-to-date pip into a temporary folder, else useget-pip.py
(the official way to install pip) to do the same - Use this pip to add setuptools and wheels packages required to build source packages (stage 0)
- install easybuild and its prerequisites into a new temporary folder (stage 1)
- reset PYTHONPATH to stage0 (maybe not even required if the EC is correctly handled)
- import easybuild from stage1 and build EasyBuild from generated EasyConfig (stage 2)
Potential problem: System installed packages. If a package exists, pip
will uninstall it before installing the new one. There is --ignore-installed
which avoids this, but this will also ignore existing dependencies leading to the following situation:
pip install vsc-install<0.11.4 --ignore-installed
-> No dependencies, all goodpip install vsc-base<2.9.0 --ignore-installed
-> Ignores installed vsc-install and pulls in the latest available one because vsc-base requestsvsc-install>=0.9.0
- Finally fails due to incompatibilites
Potential solution: Use a virtualenv. I never really used it but it seems to solve exactly this problem: Create a new environment which is free of any python packages and hence we don't need --ignore-installed
. virtualenv
itself has no dependencies so we can install that with --ignore-installed
first and run it.