pytest discovers tests only from subfolders

[klauerm]

Description

I have the following project structure

my-project/
    test_1.py
    test_2.py
    test_3.py
    subproject/
       2501/
         test_1.py
       2502/
         test_1.py

I want to run the all 3 tests in my-project and the test for the subproject release 2502
So I try

pytest --ignore my-project/subproject my-project my-project/subproject/2502

However, only my-project/subproject/2502/test_1.py is run. According to the documentation, all tests except my-project/subproject/2501/test_1.py shall be run.

output of pip list

pytest 8.3.3

Package                       Version
----------------------------- -----------
alabaster                     1.0.0
alerta                        8.5.3
aliyun-python-sdk-core        2.16.0
aliyun-python-sdk-core-v3     2.13.33
aliyun-python-sdk-kms         2.16.5
aliyun-python-sdk-ram         3.2.0
apache-libcloud               3.8.0
asttokens                     2.4.1
attrs                         24.2.0
awscli                        1.35.22
azure-common                  1.1.28
azure-core                    1.32.0
azure-identity                1.19.0
azure-mgmt-core               1.5.0
azure-mgmt-storage            21.2.1
azure-nspkg                   3.0.2
azure-storage                 0.36.0
babel                         2.16.0
backcall                      0.2.0
backports.tarfile             1.2.0
beautifulsoup4                4.13.4
bleach                        6.2.0
boto3                         1.35.56
botocore                      1.35.56
build                         1.2.2.post1
bump2version                  1.0.1
bumpversion                   0.6.0
cachetools                    5.5.0
cassandra-driver              3.29.2
certifi                       2024.8.30
cffi                          1.17.1
charset-normalizer            3.4.0
click                         8.1.7
colorama                      0.4.6
confluent-kafka               2.6.1
coverage                      7.6.4
crcmod                        1.7
cryptography                  43.0.3
decorator                     5.1.1
defusedxml                    0.7.1
Deprecated                    1.2.14
distlib                       0.3.9
docopt                        0.6.2
docutils                      0.16
durationpy                    0.9
exceptiongroup                1.2.2
executing                     2.1.0
fastjsonschema                2.21.1
fido2                         2.0.0
filelock                      3.16.1
flake8                        7.1.1
flaky                         3.8.1
freezegun                     1.5.1
geomet                        0.2.1.post1
google-api-core               2.22.0
google-api-python-client      2.151.0
google-auth                   2.36.0
google-auth-httplib2          0.2.0
google-cloud-compute          1.20.1
google-cloud-core             2.4.1
google-cloud-storage          2.18.2
google-crc32c                 1.6.0
google-resumable-media        2.7.2
googleapis-common-protos      1.65.0
grpcio                        1.67.1
grpcio-status                 1.67.1
httplib2                      0.22.0
idna                          3.10
imagesize                     1.4.1
importlib_metadata            8.5.0
influxdb                      5.3.2
iniconfig                     2.0.0
ipython                       8.12.3
isodate                       0.7.2
jaraco.classes                3.4.0
jaraco.context                6.0.1
jaraco.functools              4.1.0
jedi                          0.19.2
jeepney                       0.8.0
Jinja2                        3.1.4
jmespath                      0.10.0
jsonschema                    4.23.0
jsonschema-specifications     2024.10.1
jupyter_client                8.6.3
jupyter_core                  5.8.1
jupyterlab_pygments           0.3.0
jwcrypto                      1.5.6
kafka-python                  2.0.6
kazoo                         2.10.0
keyring                       25.5.0
kubernetes                    31.0.0
lxml                          5.3.0
Mako                          1.3.6
markdown-it-py                3.0.0
MarkupSafe                    3.0.2
matplotlib-inline             0.1.7
mccabe                        0.7.0
mdurl                         0.1.2
mistune                       3.1.3
mockito                       1.5.3
mohawk                        1.1.0
more-itertools                10.5.0
msal                          1.31.0
msal-extensions               1.2.0
msgpack                       1.1.0
nbclient                      0.10.2
nbconvert                     7.16.6
nbformat                      5.10.4
netaddr                       1.3.0
nh3                           0.2.18
nodeenv                       1.9.1
oauthlib                      3.2.2
oss2                          2.19.1
packaging                     24.1
pandocfilters                 1.5.1
parso                         0.8.4
pexpect                       4.9.0
pickleshare                   0.7.5
pip                           25.1.1
pip-tools                     7.4.1
pipenv                        2024.3.1
pipreqs                       0.5.0
pkginfo                       1.10.0
platformdirs                  4.3.6
pluggy                        1.5.0
portalocker                   2.10.1
powerline-status              2.7
prompt_toolkit                3.0.48
proto-plus                    1.25.0
protobuf                      5.28.3
ptyprocess                    0.7.0
pure_eval                     0.2.3
pyasn1                        0.6.1
pyasn1_modules                0.4.1
pycodestyle                   2.12.1
pycparser                     2.22
pycryptodome                  3.21.0
pyflakes                      3.2.0
PyGithub                      2.5.0
Pygments                      2.18.0
PyJWT                         2.9.0
PyNaCl                        1.5.0
pyOpenSSL                     24.2.1
pyparsing                     3.2.0
pyproject_hooks               1.2.0
pyright                       1.1.390
pyscard                       2.2.2
pytest                        8.3.3
pytest-cov                    6.1.1
pytest-localserver            0.9.0.post0
python-dateutil               2.9.0.post0
pytz                          2024.2
PyYAML                        6.0.2
pyzmq                         27.0.0
readme_renderer               44.0
referencing                   0.35.1
requests                      2.32.3
requests-hawk                 1.2.1
requests-oauthlib             2.0.0
requests-toolbelt             1.0.0
responses                     0.25.3
retrying                      1.3.4
rfc3986                       2.0.0
rich                          13.9.4
rpds-py                       0.21.0
rsa                           4.7.2
ruamel.yaml                   0.18.6
ruamel.yaml.clib              0.2.12
s3transfer                    0.10.3
sapbd                         1.861.0
SecretStorage                 3.3.3
setuptools                    75.3.0
six                           1.16.0
snowballstemmer               2.2.0
soupsieve                     2.7
Sphinx                        8.1.3
sphinxcontrib-applehelp       2.0.0
sphinxcontrib-devhelp         2.0.0
sphinxcontrib-htmlhelp        2.1.0
sphinxcontrib-jsmath          1.0.1
sphinxcontrib-qthelp          2.0.0
sphinxcontrib-serializinghtml 2.0.0
stack-data                    0.6.3
tabulate                      0.9.0
tinycss2                      1.4.0
tomli                         2.0.2
tornado                       6.5.1
traitlets                     5.14.3
twine                         5.1.1
typing_extensions             4.12.2
uritemplate                   4.1.1
urllib3                       2.2.3
virtualenv                    20.27.1
wcwidth                       0.2.13
webencodings                  0.5.1
websocket-client              1.8.0
Werkzeug                      3.1.3
wheel                         0.45.0
wrapt                         1.16.0
yarg                          0.1.9
yubikey-manager               5.7.0
zipp                          3.21.0

[kai-wei-mo]

i think this is the same issue as #12605. i've tried recreating your project structure locally and running pytest my-project my-project/subproject/2502 collects only my-project/subproject/2502/test_5.py (using pytest 8.4.1).

downgrading to pytest <=7.4.4 would give you the behavior you expect, if downgrading is an option for you.

[klauerm]

Ah, this explains why this feature was originally introduced by my colleagues several years ago. The feature stopped working with upgrading pytest. For us, downgrading pytest is not an option, but I could convince my fellow developers that we should run all unit tests in our pipeline and not exclude ones from an older release.

Do you plan to fix this error anyway?

[RonnyPfannschmidt]

there is a plan to fix this but it needs some extensive debugging

[karlicoss]

I think this might have been fixed together with #13426 -- but you might need to use consider_namespace_packages = true flag

[RonnyPfannschmidt]

this is a case where that flag ought not to be needed

[klauerm]

I agree with @RonnyPfannschmidt . It worked in pytest <=7.4.4. If now a flag is needed, every users of this feature need to introduce that flag into their pytest config written with that old release.