sage.numerical.backends: Replace use of TestSuite by pytest

[mkoeppe]

sage.numerical.backends provides an opportunity to study use of pytest as a replacement for TestSuite (proposed in #30738) in a small self-contained codebase.

GenericBackend defines an API and a small number of _test_ methods to check that the API is implemented correctly by the backend implementations. We do this at the example of _test_ncols_nonnegative, leaving the other test methods for follow-up tickets.

Important design constraint:

• The testing framework must be extensible by user-defined classes (or classes from separately installed Python packages - such as the existing sage_numerical_backends_coin).

This is accomplished by introducing a new GenericBackendTests class which defines the checks that should hold for all backend implementations. Then for each backend we introduce a class that derives from GenericBackendTests and returns in the backend test fixture the backend under test.

The pytests are now also executed in sage -t after the doctests, e.g. bin/sage-runtests --verbose . prints at the end

===================================================================================================================== test session starts ======================================================================================================================
platform linux -- Python 3.8.6, pytest-6.2.1, py-1.10.0, pluggy-0.13.1 -- /mnt/d/Programming/sage/src/.venv/bin/python3
cachedir: .pytest_cache
rootdir: /mnt/d/Programming/sage/src
collected 10 items                                                                                                                                                                                                                                             

sage/numerical/backends/cvxopt_backend_test.py::TestCVXOPTBackend::test_old_testsuite PASSED                                                                                                                                                             [ 10%]
sage/numerical/backends/cvxopt_backend_test.py::TestCVXOPTBackend::test_ncols_nonnegative PASSED                                                                                                                                                         [ 20%]
sage/numerical/backends/glpk_backend_test.py::TestGLPKBackend::test_ncols_nonnegative PASSED                                                                                                                                                             [ 30%]
sage/numerical/backends/glpk_backend_test.py::TestGLPKBackend::test_old_testsuite PASSED                                                                                                                                                                 [ 40%]
sage/numerical/backends/glpk_exact_backend_test.py::TestGLPKExactBackend::test_ncols_nonnegative PASSED                                                                                                                                                  [ 50%]
sage/numerical/backends/glpk_exact_backend_test.py::TestGLPKExactBackend::test_old_testsuite PASSED                                                                                                                                                      [ 60%]
sage/numerical/backends/interactivelp_backend_test.py::TestInteractiveLPBackend::test_ncols_nonnegative PASSED                                                                                                                                           [ 70%]
sage/numerical/backends/interactivelp_backend_test.py::TestInteractiveLPBackend::test_old_testsuite PASSED                                                                                                                                               [ 80%]
sage/numerical/backends/ppl_backend_test.py::TestPPLBackend::test_ncols_nonnegative PASSED                                                                                                                                                               [ 90%]
sage/numerical/backends/ppl_backend_test.py::TestPPLBackend::test_old_testsuite PASSED                                                                                                                                                                   [100%]

Depends on #30901
Depends on #31003
Depends on #30551

CC: @tobiasdiez @nthiery @tscrim @fchapoton @yuan-zhou

Component: doctest framework

Author: Tobias Diez, Matthias Koeppe

Branch: 0592226

Reviewer: Matthias Koeppe, Tobias Diez

Issue created by migration from https://trac.sagemath.org/ticket/31103

[tobiasdiez]

comment:1

What should happen with the generic tests methods like _test_new, _test_not_implemented_methods, _test_category, _test_pickling that sadly make the "small self-contained" part hard?

[mkoeppe]

comment:2

Great point. This brings me to the question whether it would be possible to either

• invoke the existing TestSuite as one pytest method or
• dynamically generate adapter methods that invoke the existing _test methods.

[tobiasdiez]

Dependencies: #30901, #31003

[tobiasdiez]

Branch: public/refactoring/pytest-numerical-backends

[tobiasdiez]

Description changed:

--- 
+++ 
@@ -7,3 +7,26 @@
 Important design constraint:
 - The testing framework must be extensible by user-defined classes (or classes from separately installed Python packages - such as the existing `sage_numerical_backends_coin`).
 
+
+This is accomplished by introducing a new `GenericBackendTests` class which defines the checks that should hold for all backend implementations. Then for each backend we introduce a class that derives from `GenericBackendTests` and returns in the `backend` test fixture the backend under test.
+
+The pytests are now also executed in `sage -t` after the doctests, e.g. `bin/sage-runtests --verbose .` prints at the end
+
+```
+===================================================================================================================== test session starts ======================================================================================================================
+platform linux -- Python 3.8.6, pytest-6.2.1, py-1.10.0, pluggy-0.13.1 -- /mnt/d/Programming/sage/src/.venv/bin/python3
+cachedir: .pytest_cache
+rootdir: /mnt/d/Programming/sage/src
+collected 10 items                                                                                                                                                                                                                                             
+
+sage/numerical/backends/cvxopt_backend_test.py::TestCVXOPTBackend::test_old_testsuite PASSED                                                                                                                                                             [ 10%]
+sage/numerical/backends/cvxopt_backend_test.py::TestCVXOPTBackend::test_ncols_nonnegative PASSED                                                                                                                                                         [ 20%]
+sage/numerical/backends/glpk_backend_test.py::TestGLPKBackend::test_ncols_nonnegative PASSED                                                                                                                                                             [ 30%]
+sage/numerical/backends/glpk_backend_test.py::TestGLPKBackend::test_old_testsuite PASSED                                                                                                                                                                 [ 40%]
+sage/numerical/backends/glpk_exact_backend_test.py::TestGLPKExactBackend::test_ncols_nonnegative PASSED                                                                                                                                                  [ 50%]
+sage/numerical/backends/glpk_exact_backend_test.py::TestGLPKExactBackend::test_old_testsuite PASSED                                                                                                                                                      [ 60%]
+sage/numerical/backends/interactivelp_backend_test.py::TestInteractiveLPBackend::test_ncols_nonnegative PASSED                                                                                                                                           [ 70%]
+sage/numerical/backends/interactivelp_backend_test.py::TestInteractiveLPBackend::test_old_testsuite PASSED                                                                                                                                               [ 80%]
+sage/numerical/backends/ppl_backend_test.py::TestPPLBackend::test_ncols_nonnegative PASSED                                                                                                                                                               [ 90%]
+sage/numerical/backends/ppl_backend_test.py::TestPPLBackend::test_old_testsuite PASSED                                                                                                                                                                   [100%]
+```

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

6dd6e5c	Fix compilation
eceefb3	Remove string wrap
d345bff	Fix test
c47c4bf	Correct indent
3fcaf5f	Merge branch 'develop' of git://github.com/sagemath/sage into public/build/multiarchsimple
090e6f1	Simplify code
fa4556a	Remove _get_sage_local
4cdf94d	Merge branch 'public/build/multiarchsimple' of git://trac.sagemath.org/sage into public/refactoring/pytest-numerical-backends
5fd49b7	Migrate some tests to pytest
719e442	Run pytest as part of sage -t

[sagetrac-git]

Commit: 719e442

[tobiasdiez]

Description changed:

--- 
+++ 
@@ -1,8 +1,6 @@
 `sage.numerical.backends` provides an opportunity to study use of `pytest` as a replacement for `TestSuite` (proposed in #30738) in a small self-contained codebase.
 
-`GenericBackend` defines an API and a small number of `_test_` methods to check that the API is implemented correctly by the backend implementations. 
-
-Apart from converting from `TestSuite` to pytest, this would be a good opportunity to expand the tests to replace what is currently tested using copy-pasted doctests.
+`GenericBackend` defines an API and a small number of `_test_` methods to check that the API is implemented correctly by the backend implementations. We do this at the example of `_test_ncols_nonnegative`, leaving the other test methods for follow-up tickets.
 
 Important design constraint:
 - The testing framework must be extensible by user-defined classes (or classes from separately installed Python packages - such as the existing `sage_numerical_backends_coin`).

[tobiasdiez]

comment:6

Replying to @mkoeppe:

• invoke the existing TestSuite as one pytest method or

That's a great idea, making it possible to step-by-step migrate more test methods without the need to do everything at once! I've used it now.

• dynamically generate adapter methods that invoke the existing _test methods.

That might also work, but is more work since the _test methods are instance methods, and for pytest the adapter method would need to be provided with the instance (which in the current design is passed to the TestSuite). It's probably easier to simply rewrite the test method in pytest.

[mkoeppe]

comment:7

Thanks a lot!

[mkoeppe]

comment:8

Some quick remarks (I haven't had a chance to test it yet):

• I'd suggest to rename test_old_testsuite to something more specific like test_sage_unittest_testsuite.
• In the addition to src/bin/sage-runtests - this needs to work without error if pytest is not installed (currently it is only an optional package - see Make pytest a standard package #31110)

[mkoeppe]

Author: Tobias Diez