addopts behavior with coverage flag

[miketheman]

(uncertain if this is specific to py.test config, or pytest-cov, apologies if I'm putting it in the wrong place)

• Include a detailed description of the bug or suggestion

It appears that the addopts section of config file appends positional arguments to option flags following, making specifying a path of tests to include dependent on the order of the option flags.

Note: I use the --collect-only flag as an example, in my project I've got others like --duration=10, etc - I tried to use a simple reproduction approach.

• pip list of the virtual environment you are using

coverage (4.3.1)
pip (9.0.1)
py (1.4.32)
pytest (3.0.5)
pytest-cov (2.4.0)
setuptools (32.3.0)
wheel (0.30.0a0)

• pytest and operating system versions

$ python --version
Python 2.7.13
$ py.test --version
This is pytest version 3.0.5, imported from ..../lib/python2.7/site-packages/pytest.pyc
setuptools registered plugins:
  pytest-cov-2.4.0 at ..../lib/python2.7/site-packages/pytest_cov/plugin.py

• Minimal example if possible

Given a setup.cfg section like:

[tool:pytest]
addopts = --collect-only --cov

And a directory tree structure like:

.
├── app
│   ├── one
│   │   └── test_sample1.py
│   └── two
│       └── test_sample2.py
└── setup.cfg

3 directories, 3 files

And test_sample?.py contains this example:

def func(x):
    return x + 1

def test_answer():
    assert func(3) == 5

When running py.test app/one/
Then both test cases are collected, instead of only one.

Output:

============================= test session starts ==============================
platform darwin -- Python 2.7.13, pytest-3.0.5, py-1.4.32, pluggy-0.4.0
rootdir: /Users/miketheman/tmp/pytest-repro, inifile: setup.cfg
plugins: cov-2.4.0
collected 2 items
<Module 'app/one/test_sample1.py'>
  <Function 'test_answer'>
<Module 'app/two/test_sample2.py'>
  <Function 'test_answer'>


---------- coverage: platform darwin, python 2.7.13-final-0 ----------
Name                      Stmts   Miss  Cover
---------------------------------------------
app/one/test_sample1.py       4      2    50%

========================= no tests ran in 0.02 seconds =========================

It appears that the path specified on the commandline is being passed to --cov

Workaround

Changing the setup.cfg to reverse the options like so:

[tool:pytest]
addopts = --cov --collect-only

Produces the correct result:

============================= test session starts ==============================
platform darwin -- Python 2.7.13, pytest-3.0.5, py-1.4.32, pluggy-0.4.0
rootdir: /Users/miketheman/tmp/pytest-repro, inifile: setup.cfg
plugins: cov-2.4.0
collected 1 items
<Module 'app/one/test_sample1.py'>
  <Function 'test_answer'>


---------- coverage: platform darwin, python 2.7.13-final-0 ----------
Name                      Stmts   Miss  Cover
---------------------------------------------
app/one/test_sample1.py       4      2    50%

========================= no tests ran in 0.01 seconds =========================

[nicoddemus]

Hi @miketheman thanks for writing.

You assessment is correct, addopts just appends the arguments to the command line. So you get in the end addopts + command line, so your first example is running:

py.test --collect-only --cov app/one/

I agree it is confusing. Perhaps it would be possible to interpret the options in two separate passes?