[FR] Add support for running 62.3.4 directly with pytest and a installed version of setuptools.

[kloczek]

setuptools version

62.3.4

Python version

3.8.13

OS

Linux/x86_64

Additional environment information

list of module sinstalled in build env

Package                       Version
----------------------------- -----------------
alabaster                     0.7.12
appdirs                       1.4.4
asttokens                     2.0.5
attrs                         21.4.0
autocommand                   2.2.1
Babel                         2.10.1
backcall                      0.2.0
Brlapi                        0.8.3
build                         0.8.0
charset-normalizer            2.0.12
codespell                     2.1.0
coverage                      6.4.1
cycler                        0.11.0
decorator                     5.1.1
distlib                       0.3.4
distro                        1.7.0
docutils                      0.17.1
entrypoints                   0.4
execnet                       1.9.0
executing                     0.8.3
extras                        1.0.0
fastjsonschema                2.15.3
filelock                      3.7.1
fixtures                      4.0.0
flake8                        4.0.1
fonttools                     4.33.3
gpg                           1.17.1-unknown
html5lib                      1.1
idna                          3.3
imagesize                     1.3.0
importlib-metadata            4.11.3
importlib-resources           5.7.1
ini2toml                      0.10
iniconfig                     1.1.1
jaraco.envs                   2.4.0
jaraco.packaging              9.0.0
jaraco.path                   3.4.0
jedi                          0.18.1
Jinja2                        3.1.1
jsonschema                    4.5.1
jupyter-client                7.3.4
jupyter-core                  4.10.0
kiwisolver                    1.4.2
libcomps                      0.1.18
louis                         3.21.0
lxml                          4.9.0
MarkupSafe                    2.1.1
matplotlib                    3.5.1
matplotlib-inline             0.1.3
mccabe                        0.7.0
mock                          4.0.3
mypy                          0.960
mypy-extensions               0.4.3
nbformat                      5.4.0
nest-asyncio                  1.5.5
numpy                         1.22.4
olefile                       0.46
packaging                     21.3
parso                         0.8.3
path                          16.4.0
pbr                           5.8.1
pep517                        0.12.0
pexpect                       4.8.0
pickleshare                   0.7.5
Pillow                        9.1.1
pip                           22.0.4
pip-run                       8.8.0
platformdirs                  2.5.2
pluggy                        1.0.0
ply                           3.11
prompt-toolkit                3.0.29
ptyprocess                    0.7.0
pure-eval                     0.2.2
py                            1.11.0
pycodestyle                   2.8.0
pycparser                     2.21
pyflakes                      2.4.0
Pygments                      2.12.0
PyGObject                     3.42.1
pyparsing                     3.0.9
pyrsistent                    0.18.1
pytest                        7.1.2
pytest-checkdocs              2.7.1
pytest-cov                    3.0.0
pytest-fixture-config         1.7.0
pytest-flake8                 1.1.1
pytest-forked                 1.4.0
pytest-shutil                 1.7.0
pytest-virtualenv             1.7.0
pytest-xdist                  2.5.0
python-dateutil               2.8.2
pytz                          2022.1
pyzmq                         23.1.0
requests                      2.28.0
rpm                           4.17.0
rst                           0.1
rst.linker                    2.3.0
scour                         0.38.2
six                           1.16.0
snowballstemmer               2.2.0
soupsieve                     2.3.2.post1
Sphinx                        5.0.1
sphinx-favicon                0.2
sphinx_inline_tabs            2022.1.2b11
sphinx-reredirects            0.1.1
sphinxcontrib-applehelp       1.0.2.dev20220410
sphinxcontrib-devhelp         1.0.2.dev20220410
sphinxcontrib-htmlhelp        2.0.0
sphinxcontrib-jsmath          1.0.1.dev20220410
sphinxcontrib-qthelp          1.0.3.dev20220410
sphinxcontrib-serializinghtml 1.1.5
stack-data                    0.2.0
termcolor                     1.1.0
testtools                     2.5.0
toml                          0.10.2
tomli                         2.0.1
tomli_w                       1.0.0
tornado                       6.1
tox                           3.24.5
traitlets                     5.2.2
typing_extensions             4.2.0
urllib3                       1.26.9
virtualenv                    20.13.4
wcwidth                       0.2.5
webencodings                  0.5.1
wheel                         0.37.1
zipp                          3.8.0

Description

I'm trying to package your module as an rpm package. So I'm using the typical PEP517 based build, install and test cycle used on building packages from non-root account.

• python3 -sBm build -w --no-isolation
• because I'm calling build with --no-isolation I'm using during all processes only locally installed modules
• install .whl file in </install/prefix>
• run pytest with PYTHONPATH pointing to sitearch and sitelib inside </install/prefix>

Recently I found that some progress has been made on "testing as installed" scenario and it is possible now execute pytest with small patch

--- a/pytest.ini~       2022-06-11 11:09:46.000000000 +0000
+++ b/pytest.ini        2022-06-12 01:15:20.396239151 +0000
@@ -1,4 +1,5 @@
 [pytest]
+testpaths = setuptools/tests
 norecursedirs=dist build .tox .eggs
 addopts=
        --doctest-modules

With that patch pytest is ble to collect units and start testing and it fails in few units.

Expected behavior

pytest should not fail.

How to Reproduce

• apply above patch
• run pytest

Output

Will add pytest output in next comment.

[kloczek]

Here is pytest output:

+ PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-setuptools-62.3.4-2.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-setuptools-62.3.4-2.fc35.x86_64/usr/lib/python3.8/site-packages
+ /usr/bin/pytest -ra -p no:randomly
==================================================================== test session starts =====================================================================
platform linux -- Python 3.8.13, pytest-7.1.2, pluggy-1.0.0
rootdir: /home/tkloczko/rpmbuild/BUILD/setuptools-62.3.4, configfile: pytest.ini, testpaths: setuptools/tests
plugins: forked-1.4.0, shutil-1.7.0, cov-3.0.0, virtualenv-1.7.0, xdist-2.5.0, flake8-1.1.1, checkdocs-2.7.1
collected 875 items / 2 skipped

setuptools/tests/test_archive_util.py X                                                                                                                [  0%]
setuptools/tests/test_bdist_deprecations.py F                                                                                                          [  0%]
setuptools/tests/test_bdist_egg.py .x                                                                                                                  [  0%]
setuptools/tests/test_build_clib.py .                                                                                                                  [  0%]
setuptools/tests/test_build_ext.py ....                                                                                                                [  1%]
setuptools/tests/test_build_meta.py .................................................................................................................. [ 14%]
................................................................                                                                                       [ 21%]
setuptools/tests/test_build_py.py ....x                                                                                                                [ 21%]
setuptools/tests/test_config_discovery.py ................................................................................                             [ 31%]
setuptools/tests/test_dep_util.py .                                                                                                                    [ 31%]
setuptools/tests/test_depends.py .                                                                                                                     [ 31%]
setuptools/tests/test_develop.py s....F                                                                                                                [ 32%]
setuptools/tests/test_dist.py F......x...x.............................................                                                                [ 38%]
setuptools/tests/test_dist_info.py ...................................                                                                                 [ 42%]
setuptools/tests/test_distutils_adoption.py .............                                                                                              [ 44%]
setuptools/tests/test_easy_install.py .F...FFF......FF.F...FFFFFFFFFFFFFFFFFFF..........F                                                              [ 49%]
setuptools/tests/test_editable_install.py .x                                                                                                           [ 50%]
setuptools/tests/test_egg_info.py .........................x...............................................                                            [ 58%]
setuptools/tests/test_extern.py ...                                                                                                                    [ 58%]
setuptools/tests/test_find_packages.py ...................                                                                                             [ 60%]
setuptools/tests/test_find_py_modules.py .........                                                                                                     [ 61%]
setuptools/tests/test_glob.py ....                                                                                                                     [ 62%]
setuptools/tests/test_install_scripts.py .s.s                                                                                                          [ 62%]
setuptools/tests/test_integration.py sssss                                                                                                             [ 63%]
setuptools/tests/test_logging.py ..                                                                                                                    [ 63%]
setuptools/tests/test_manifest.py ...................................................................                                                  [ 71%]
setuptools/tests/test_msvc14.py ssss                                                                                                                   [ 71%]
setuptools/tests/test_namespaces.py ....                                                                                                               [ 72%]
setuptools/tests/test_packageindex.py ....................                                                                                             [ 74%]
setuptools/tests/test_register.py .                                                                                                                    [ 74%]
setuptools/tests/test_sandbox.py ..........                                                                                                            [ 75%]
setuptools/tests/test_sdist.py .................                                                                                                       [ 77%]
setuptools/tests/test_setopt.py ..                                                                                                                     [ 77%]
setuptools/tests/test_setuptools.py .......................                                                                                            [ 80%]
setuptools/tests/test_test.py .                                                                                                                        [ 80%]
setuptools/tests/test_unicode_utils.py .                                                                                                               [ 80%]
setuptools/tests/test_upload.py .                                                                                                                      [ 80%]
setuptools/tests/test_virtualenv.py ..x...X..                                                                                                          [ 81%]
setuptools/tests/test_wheel.py ........................                                                                                                [ 84%]
setuptools/tests/test_windows_wrappers.py sss                                                                                                          [ 85%]
setuptools/tests/config/test_apply_pyprojecttoml.py .....................................                                                              [ 89%]
setuptools/tests/config/test_expand.py ..................                                                                                              [ 91%]
setuptools/tests/config/test_pyprojecttoml.py .......................                                                                                  [ 93%]
setuptools/tests/config/test_setupcfg.py .............................................                                                                 [ 99%]
setuptools/tests/integration/test_pip_install_sdist.py ssssssss                                                                                        [100%]

[..]

-------------------------------------------------------------------- Captured stdout call --------------------------------------------------------------------
running develop
Checking .pth file support in /tmp/pytest-of-tkloczko/pytest-40/test_editable_user_and_build_i0/.home/.local/lib/python3.8/site-packages
/usr/bin/python3 -E -c pass
TEST PASSED: /tmp/pytest-of-tkloczko/pytest-40/test_editable_user_and_build_i0/.home/.local/lib/python3.8/site-packages appears to support .pth files
-------------------------------------------------------------------- Captured stderr call --------------------------------------------------------------------
WARNING: The user site-packages directory is disabled.
================================================================== short test summary info ===================================================================
SKIPPED [2] setuptools/tests/test_msvc.py:17: could not import 'distutils.msvc9compiler': No module named 'winreg'
SKIPPED [1] setuptools/tests/test_develop.py:66: TODO: needs a fixture to cause 'develop' to be invoked without mutating environment.
SKIPPED [1] setuptools/tests/test_install_scripts.py:50: Windows only
SKIPPED [1] setuptools/tests/test_install_scripts.py:78: Windows only
SKIPPED [5] setuptools/tests/test_integration.py:31: Integration tests cannot run when pbr is installed
SKIPPED [1] setuptools/tests/test_msvc14.py:16: These tests are only for win32
SKIPPED [1] setuptools/tests/test_msvc14.py:34: These tests are only for win32
SKIPPED [1] setuptools/tests/test_msvc14.py:52: These tests are only for win32
SKIPPED [1] setuptools/tests/test_msvc14.py:68: These tests are only for win32
SKIPPED [1] setuptools/tests/test_windows_wrappers.py:80: Windows only
SKIPPED [1] setuptools/tests/test_windows_wrappers.py:121: Windows only
SKIPPED [1] setuptools/tests/test_windows_wrappers.py:180: Windows only
SKIPPED [8] conftest.py:51: skipping integration tests
XFAIL setuptools/tests/test_bdist_egg.py::Test::test_exclude_source_files
  Byte code disabled
XFAIL setuptools/tests/test_build_py.py::test_excluded_subpackages
  reason: #3260
XFAIL setuptools/tests/test_dist.py::test_read_metadata[Metadata Version 1.2: Project-Url-attrs5]
  Issue #1578: project_urls not read
XFAIL setuptools/tests/test_dist.py::test_read_metadata[Metadata Version 2.1: Provides Extra-attrs9]
  provides_extras not read
XFAIL setuptools/tests/test_editable_install.py::test_editable_with_pyproject[None]
  Editable install is currently only supported with `setup.py`
XFAIL setuptools/tests/test_egg_info.py::TestEggInfo::test_requires[extras_require_with_marker_in_setup_cfg]
XFAIL setuptools/tests/test_virtualenv.py::test_pip_upgrade_from_source[pip<20]
  pypa/pip#6599
XPASS setuptools/tests/test_archive_util.py::test_unicode_files #710 and #712
XPASS setuptools/tests/test_virtualenv.py::test_pip_upgrade_from_source[https://github.com/pypa/pip/archive/main.zip] #2975
FAILED setuptools/tests/test_bdist_deprecations.py::test_bdist_rpm_warning - TypeError: issubclass() arg 1 must be a class
FAILED setuptools/tests/test_develop.py::TestNamespaces::test_editable_prefix - subprocess.CalledProcessError: Command '['/usr/bin/python3', '-m', 'pip', '...
FAILED setuptools/tests/test_dist.py::test_dist_fetch_build_egg - DeprecationWarning: Creating a LegacyVersion has been deprecated and will be removed in t...
FAILED setuptools/tests/test_easy_install.py::TestEasyInstallTest::test_no_find_links - DeprecationWarning: Creating a LegacyVersion has been deprecated an...
FAILED setuptools/tests/test_easy_install.py::TestEasyInstallTest::test_unicode_filename_in_sdist - DeprecationWarning: Creating a LegacyVersion has been d...
FAILED setuptools/tests/test_easy_install.py::TestEasyInstallTest::test_unicode_content_in_sdist - DeprecationWarning: Creating a LegacyVersion has been de...
FAILED setuptools/tests/test_easy_install.py::TestEasyInstallTest::test_script_install - DeprecationWarning: Creating a LegacyVersion has been deprecated a...
FAILED setuptools/tests/test_easy_install.py::TestUserInstallTest::test_user_install_not_implied_user_site_enabled - DeprecationWarning: Creating a LegacyV...
FAILED setuptools/tests/test_easy_install.py::TestUserInstallTest::test_user_install_not_implied_user_site_disabled - DeprecationWarning: Creating a Legacy...
FAILED setuptools/tests/test_easy_install.py::TestUserInstallTest::test_local_index - DeprecationWarning: Creating a LegacyVersion has been deprecated and ...
FAILED setuptools/tests/test_easy_install.py::TestSetupRequires::test_setup_requires_overrides_version_conflict[use_setup_cfg0] - DeprecationWarning: Creat...
FAILED setuptools/tests/test_easy_install.py::TestSetupRequires::test_setup_requires_overrides_version_conflict[use_setup_cfg1] - DeprecationWarning: Creat...
FAILED setuptools/tests/test_easy_install.py::TestSetupRequires::test_setup_requires_overrides_version_conflict[use_setup_cfg2] - DeprecationWarning: Creat...
FAILED setuptools/tests/test_easy_install.py::TestSetupRequires::test_setup_requires_overrides_version_conflict[use_setup_cfg3] - DeprecationWarning: Creat...
FAILED setuptools/tests/test_easy_install.py::TestSetupRequires::test_setup_requires_override_nspkg[use_setup_cfg0] - DeprecationWarning: Creating a Legacy...
FAILED setuptools/tests/test_easy_install.py::TestSetupRequires::test_setup_requires_override_nspkg[use_setup_cfg1] - DeprecationWarning: Creating a Legacy...
FAILED setuptools/tests/test_easy_install.py::TestSetupRequires::test_setup_requires_override_nspkg[use_setup_cfg2] - DeprecationWarning: Creating a Legacy...
FAILED setuptools/tests/test_easy_install.py::TestSetupRequires::test_setup_requires_override_nspkg[use_setup_cfg3] - DeprecationWarning: Creating a Legacy...
FAILED setuptools/tests/test_easy_install.py::TestSetupRequires::test_setup_requires_with_attr_version[use_setup_cfg0] - DeprecationWarning: Creating a Leg...
FAILED setuptools/tests/test_easy_install.py::TestSetupRequires::test_setup_requires_with_attr_version[use_setup_cfg1] - DeprecationWarning: Creating a Leg...
FAILED setuptools/tests/test_easy_install.py::TestSetupRequires::test_setup_requires_with_attr_version[use_setup_cfg2] - DeprecationWarning: Creating a Leg...
FAILED setuptools/tests/test_easy_install.py::TestSetupRequires::test_setup_requires_with_attr_version[use_setup_cfg3] - DeprecationWarning: Creating a Leg...
FAILED setuptools/tests/test_easy_install.py::TestSetupRequires::test_setup_requires_honors_pip_env - DeprecationWarning: Creating a LegacyVersion has been...
FAILED setuptools/tests/test_easy_install.py::TestSetupRequires::test_setup_requires_with_pep508_url - DeprecationWarning: Creating a LegacyVersion has bee...
FAILED setuptools/tests/test_easy_install.py::TestSetupRequires::test_setup_requires_with_allow_hosts - DeprecationWarning: Creating a LegacyVersion has be...
FAILED setuptools/tests/test_easy_install.py::TestSetupRequires::test_setup_requires_with_python_requires - DeprecationWarning: Creating a LegacyVersion ha...
FAILED setuptools/tests/test_easy_install.py::TestSetupRequires::test_setup_requires_with_find_links_in_setup_cfg[False] - DeprecationWarning: Creating a L...
FAILED setuptools/tests/test_easy_install.py::TestSetupRequires::test_setup_requires_with_find_links_in_setup_cfg[True] - DeprecationWarning: Creating a Le...
FAILED setuptools/tests/test_easy_install.py::TestSetupRequires::test_setup_requires_with_transitive_extra_dependency - DeprecationWarning: Creating a Lega...
FAILED setuptools/tests/test_easy_install.py::test_editable_user_and_build_isolation - DeprecationWarning: Creating a LegacyVersion has been deprecated and...
======================================= 30 failed, 813 passed, 25 skipped, 7 xfailed, 2 xpassed in 1216.08s (0:20:16) ========================================

And here is full pytest output
setuptools-pytes,txt

[abravalheri]

Hi @kloczek, thank you very much for the information.

I don't consider this issue to be a bug for the reason that no testing methodology is supported by setuptools other than the one implemented for the CI (this methodology is encoded in the tox.ini and .github/workflows/... files). Instead this issue would be a kind of "feature request", i.e. a request to add support for running the tests using a different methodology.

Under this perspective, this issue seems to me like it is a duplication of already the existing issue #2318.

---

Unfortunately the suggested patch is not applicable in the main repository due to the following reason:

• setuptools own implementation contains doctests that are expected to run alongside the test files. pkg_resources tests are also in a different folder.
  Therefore the correct value for testpaths is the root directory, which happens to be the default value for this configuration anyway (unless I am wrong...). So I don't think there is anything to be changed here.

---

Please note that although this project does not support other testing methodologies directly, it is open to any PRs that would make other testing methodologies easier, as long as:

1. Any proposed change is "backwards compatible" with the tests executed by setuptools CI (e.g. the proposed change don't exclude tests just for the sake of making the test suite run in a different setup).
2. Any folder re-organization needs to be discussed before implemented (e.g. there seems to be a strong preference in setuptools for keeping a flat-layout instead of adopting a src-layout, if a PR wants to change that, it should seek for a common agreement first).

The change proposed in the text of this issue, seems to violate condition 1.

---

Please consider the following tips when creating an issue in setuptools:

• Please select the appropriate form. If something is not a bug please refrain from using the bug form or referring to the observed behaviour as a bug. It might lead to misleading conclusions.
• When reporting a behaviour that you believe is inappropriate, please try to isolate the behaviour as much as possible:
  • It is important to perform the experiment in an environment that does not contain any other package other than the strictly necessary ones.
  • If you detect a problem that affects only a subset of files, please create a minimal reproduction that exercises only this minimal subset of files.
  • Please have a look on the methodology described in https://stackoverflow.com/help/minimal-reproducible-example.

[kloczek]

BTW looks like to testpaths can be added pkg_resources/tests and with that path things are not so well as with setuptools/tests

[kloczek]

Recently I found that some progress has been made on "testing as installed" scenario and it is possible now execute pytest with small patch

--- a/pytest.ini~       2022-06-11 11:09:46.000000000 +0000
+++ b/pytest.ini        2022-06-12 01:15:20.396239151 +0000
@@ -1,4 +1,5 @@
 [pytest]
+testpaths = setuptools/tests
 norecursedirs=dist build .tox .eggs
 addopts=
        --doctest-modules

Should I submit that patch as PR? 🤔

[abravalheri]

Please don't, this patch has problems described in #3362 (comment).

[kloczek]

Hmm I'm uning that patch in my rpm build procedure and despite some number of units failing at least it is possible to use pytest. Currently without that patch it is not possible (pytest fails on collectiing units)
CI is unisg tox.

[abravalheri]

You know that by adding this patch you are skipping a bunch of tests, right? Please refer to the other comment I linked above.

Setuptools test suite is designed to run the way it is encoded in our tox/ci files. If you try to run it differently, it will probably break, unfortunately.

I listed in a previous comment some conditions that must be observed if you want to submit a PR.

[kloczek]

You kiow .. I completly do not undrerstand that situation.
Do you have ANY intention to provide possiblity to test setuptools using pytest? 🤔

[abravalheri]

You kiow .. I completly do not undrerstand that situation.

This is explained in #3362 (comment)

Do you have ANY intention to provide possiblity to test setuptools using pytest? 🤔

Setuptools already uses pytest. The circumstances of the call are what differ.

We don't have any plans to officially support any test methodology other than the one encoded via the tox.ini and CI-related files.

We would consider a positive change if we manage to test setuptools as if it was loaded from a test virtual environment, instead of the source tree directly (this is pointed out in the original issue #2318). However this is not trivial. After spending a lot of time trying to achieve this, I can say that right now there is no quick and easy modification that we can do to make it possible. It will take time and maybe require a lot of discussions.

But as you noticed, we have already started to make some progress, some parts of the test suite already work with a proper installation (but not all of it).

[kloczek]

Issue ids that thios methot is radically different with typical methodology used on packaging python modules.