pytest.mark.only as debug marker to limit testsuite on existence

[captain-kark]

Although it's not much work to put this into a conftest.py file, I've put this into nearly every pytest project I've touched, and wouldn't mind publishing it if it weren't so small.

Do you think it would belong in the standard marks? It would complement pytest.mark.skip, but would be the inverse. It's useful for when you've isolated a small batch of tests in a single file for quick debugging or troubleshooting, to be removed before committing. It's faster than updating a config file, or updating a test run through command line switches.

See https://mochajs.org/#exclusive-tests for prior art, and where I first encountered this pattern. Note at the bottom of the section, it says:

Be mindful not to commit usages of .only() to version control, unless you really mean it! To do so one can run mocha with the option --forbid-only in the continuous integration test command (or in a git precommit hook).

The implementation I use is simple enough as a top-level conftest.py hook, but honestly I'm not sure how well it would behave as a generally available mark. Things like pytest.mark.parameterize could make this tricky, as an example.

def pytest_collection_modifyitems(session, config, items):
    items = [item for item in items if 'only' in [m.name for m in item.own_markers]]


def test_one():
    assert True

@pytest.mark.only
def test_two():
    assert False

def test_three():
    assert True

[RonnyPfannschmidt]

at first glance i would say this is unacceptable, this is simply a relatively intense uncontextualized trim down of tests

at projects i work on there tend to be specific markers for the intent of the test subset like smoke, slow, fast, health_indicator - breaking it down to a conceptual only is no longer helpful from my pov

so without the surrounding context making it look sensible, it does not to me

[captain-kark]

I've updated my issue to include more context on why this mark would be useful, as well as a link to prior art.

[RonnyPfannschmidt]

now that makes a lot more sense with the actual surrounding context

so the idea here is to have a markers, that if exists, it breaks down the test set

the actual selection would break down to something like the following

def pytest_collection_modifyitems(config, items):
    only_marked = []
    deselected = []
    for item in items:
        if any(item.iter_markers("only")):
            only_marked.append(item)
        else:
            deselected.append(item)
    if only_marked:
        items[:] = only_marked
        config.hook.pytest_deselected(items=deselected)

[RonnyPfannschmidt]

maini question is wheether to provide this as a small plugin or directly in pytest

[nicoddemus]

I like the use case, but the potential for users committing this by accident and suddenly tests quietly getting discard feels too dangerous to me to be included in the core.

This is a nice case for a plugin, and @RonnyPfannschmidt even included a functional code which seems spot on.

I suggest creating and publishing this as a small plugin (pytest-exclusive-tests, pytest-only?), which using https://github.com/pytest-dev/cookiecutter-pytest-plugin should be really simple. I'm all for adding a note about the plugin to the official docs as a suggestion once the plugin is published too. 👍

[RonnyPfannschmidt]

@nicoddemus i also feel that with an addition to the debug comand pre-commmit plugin as well as a additional ci option to ignore/error on it would be a great help

[blueyed]

        config.hook.pytest_deselected(items=deselected)

btw: #5104

IMHO it would be nice to have this in the core (assuming that people would use it - I'd rather use -k etc probably still, or things like commenting parts of test files, renaming test functions, or putting / copying a set of selected tests into a new file etc still - but mostly because there is no only)

There was also an idea about having something like --only-fixture(s) foo, which could be done also for marks then - i.e. it might be more flexible that way.

[asottile]

I'm -1 on including this in core

I don't think we need first class support for this as we already have pytest -m only without the special code

[blueyed]

we already have pytest -m only without the special code

True.
It's not a registered mark though.

[asottile]

weird, I thought -m made it auto-register, maybe it should?

[RonnyPfannschmidt]

no, it shouldn't

[captain-kark]

The mocha.js reference I linked to includes tips on automating the task of ensuring that the .only mark doesn't get committed. I could bundle that with the plugin, but what (in your opinion) makes the most sense from a user interface perspective?

Explicit opt-in is always safest, but having local developers include a special entry in their pytest.ini defeats the purpose of quick debugging test sessions, and is equivalent to existing solutions.

I am leaning towards checking for a "well known" env var (CI=true comes to mind) that the tool refuses to act on when present?

I'm trying to think of a seamless way to integrate something like this into the larger ecosystem, so your input is appreciated @RonnyPfannschmidt @nicoddemus @asottile @blueyed thanks.

I'll see how much time I have this weekend to whip up a pytest plugin from cookiecutter and publish it under my pypi account, once that's finished I'll link to it here and close the issue.

[blueyed]

I would not bother checking if it's left in code accidentally - after all you might want to use it to debug something on CI..!
And I still think that having something that small/useful in the core would be better.
After all there is -m already indeed - and even if "only" is not a registered marker, it's easier to register it yourself, instead of installing (or building ;)) a plugin.

[captain-kark]

Weird, I thought I searched for this before filing this issue, but I guess I didn't look hard enough...?

https://github.com/theY4Kman/pytest-only

[nicoddemus]

Thanks @captain-kark for the follow up. 👍

[ilyakamens]

I just found this after being surprised that @pytest.mark.only doesn't exist. (Others on my team were as well.)

The repo mentioned above, pytest-only, hasn't been changed in two years. Would you consider a PR to add this into core?

The motivation is the same as the OP's, and the syntax came to me from it.only() in JavaScript. I'm familiar with -k, but in conjunction with pytest-watch, -k is much less efficient than being able to move (or remove or add) @pytest.mark.only in a given test file, as -k requires CTRL-C'ing the watcher.

Either way, thanks for making a great testing framework.

[RonnyPfannschmidt]

If we include it in core, it should require a opt in and/or ensure its failing on ci

[The-Compiler]

I'm still -1 at adding somewhat magic behavior like this to the core. If the plugin above still works, no big need to change anything about it. And in either case, if you want the "if an only marker exists, ignore all other tests" behavior without a CLI switch, this would be some 5-10 lines in a conftest.py probably.