Use pipenv to create venv for development

[mkoeppe]

We modify src/setup.py to support editable installs via pipenv (pipenv install -e ., which invokes pip install --editable). Since distribution continues using build/pkgs/sagelib/src/setup.py, the changes to src/setup.py made in this ticket do not affect the existing build process of the Sage library.

With this ticket, the following code works:

./bootstrap
./configure --enable-editable
make build-local
cd src
mkdir .venv
pipenv install --dev --ignore-pipfile
pipenv run pip install -e . --upgrade --exists-action=i --no-build-isolation
pipenv run sage-runtests --all

Instead of the first three commands, one can of course also use tox -e local-sudo-standard build-local.
It generates a virtual env in src/.venv that contains all necessary Python packages, and an editable install of sage.
Moreover, the installation of some packages might fail (depending on your system configuration). In this case one can manually build it using sage and then install the sage-built wheel in the pipenv. For example,

make fpylll
cd src && pipenv run pip install <sage>/local/var/lib/sage/venv-python3.8/var/lib/sage/wheels/fpylll-0.5.6-cp38-cp38-linux_x86_64.whl

The output of the added GH workflow is available here: https://github.com/sagemath/sagetrac-mirror/actions?query=workflow%3A%22Build+%26+Test+using+pipenv%22+branch%3Apublic%2Fbuild%2Finplace
Most tests are working, but a few are failing. Only a few tests are failing:

sage -t --random-seed=0 sage/misc/sagedoc.py  # 4 doctests failed
sage -t --random-seed=0 sage/docs/conf.py  # 1 doctest failed
sage -t --random-seed=0 sage/matrix/matrix_integer_dense.pyx  # 1 doctest failed
sage -t --random-seed=0 sage/cpython/dict_del_by_value.pyx  # 2 doctests failed
sage -t --random-seed=0 sage/tests/cmdline.py  # 8 doctests failed

TODO (as followup tickets):

• Expand on the added documentation on how to set up the virtual environment, using either plain python -m venv, virtualenv (20.x), or pipenv. This is Extend documentation on how to set up the virtual environment #31342.
• Consider not commit pipfile.lock but instead generate it as build artifact on ci (so that build errors can be easily reproduced by downloading the lock file, and re-creating the exact venv locally): Proposal: pipenv patterns and antipatterns for python library project pypa/pipenv#1911 (comment). However, normally you want to put the lock file under version control for the reasons described in https://trac.sagemath.org/timeline?from=2021-02-03T13%3A15%3A22Z. The official statement is "always include in version control", see git - Should Pipfile.lock be committed to version control? pypa/pipenv#598.
• Replace manual definition of build variables (PKG_CONFIG_PATH, LIBRARY_PATH etc) by sage-env.
• Fix failing doctests.

Previous tickets on in-place builds, editable installs:

• build the sage library in place #12659
• Make src/setup.py respect --build-base and --inplace, independent of SAGE_CYTHONIZED #21535

Depends on #30779
Depends on #30672
Depends on #30673
Depends on #30709
Depends on #30706
Depends on #30901
Depends on #31080
Depends on #29850
Depends on #29314
Depends on #30937
Depends on #30935
Depends on #30731
Depends on #30859
Depends on #30923
Depends on #30910
Depends on #30944
Depends on #30770
Depends on #31216
Depends on #31029
Depends on #30912
Depends on #31357
Depends on #31362
Depends on #31365
Depends on #31377

CC: @tobiasdiez @kiwifb @jhpalmieri @isuruf

Component: build

Keywords: sd111

Author: Tobias Diez

Branch/Commit: public/build/inplace @ c736e0c

Issue created by migration from https://trac.sagemath.org/ticket/30371

[tobiasdiez]

comment:1

I've worked a bit on it. The current commit is a semi-working prototype, in the sense that all cython files are successfully compiled to C files if invoked via python src/setup.py build_ext --inplace. The second compilation of the c files does not yet work.

I had problems that none of the source cython files were correctly found on my system (wsl) and that a few dependencies in the current build files were not met ('linbox', 'Singular', 'm4ri', 'zlib', 'cblas'). For this reason, I started from scratch and only included what was needed so far. Any pointers to how to install these dependencies (I think they are needed for the second compilation).

---

New commits:

bc6e272

Semi-working prototype for inplace compilation

[tobiasdiez]

Commit: bc6e272

[tobiasdiez]

Branch: public/build/inplace

[mkoeppe]

comment:2

I don't really have time to look into this before the 9.2 release. But if you are looking for where include directories are determined, take a look at sage_setup.command.sage_build_cython

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

58452ce

Make in-place compilation work

[sagetrac-git]

Changed commit from bc6e272 to 58452ce

[tobiasdiez]

comment:4

The command ./sage src/setup.py build_ext --inplace runs now fine for me. I had to change quite a few things to make it work, and found it easier to completely start over using cythonize instead of using the existing build_cython code. The resulting code is a minimal working version. Please let me know which features of the existing build_cython file you would like to keep (to be honest, I don't really understand what some of the code there is doing, and why its needed).

The code definitely needs cleanup, but otherwise is ready for a first round of review.

[sagetrac-git]

Changed commit from 58452ce to 1a99228

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

1a99228

Update gitignore

[mkoeppe]

comment:7

--- a/.gitignore
+++ b/.gitignore
@@ -99,7 +99,18 @@ gitlab-build-docker.log
 /src/build
 /src/Makefile
 /src/bin/sage-env-config
-/build/bin/sage-build-env-config
+/build

this is not good. Our /build/ contains important files of the sage distribution

[mkoeppe]

comment:8

And the changes removing old python2 code really need to go on a separate ticket

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

22b27a0

Remove global gitignore of build folder

[sagetrac-git]

Changed commit from 1a99228 to 22b27a0

[tobiasdiez]

comment:10

Ok, I was under the assumption that the build folder is for build artifacts. If this would be the case, then I suppose it would be better to whitelist those files in the build folder that needs to be checked-in. I've now replaced the global build ignore, by a more fine-grained statement which only ignores the "temporary" files.

Concerning the changes about old Python2 code, I've created #30397.