pytest.skip in pytest_itemcollected hook raises INTERNALERROR

[bechaos]

Hello,

I have tried to find anything in documentation about hooks and skip, but i was not able to. Calling pytest.skip in other hooks e.g. pytest_runtest_call, pytest_generate_tests or pytest_runtest_setup skips item fine.
However when called in pytest_itemcollected i get:

INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "...\pytest-3.0.6-py2.7.egg\_pytest\main.py", line 98, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "...\pytest-3.0.6-py2.7.egg\_pytest\main.py", line 132, in _main
INTERNALERROR>     config.hook.pytest_collection(session=session)
INTERNALERROR>   File "...\pytest-3.0.6-py2.7.egg\_pytest\vendored_packages\pluggy.py", line 745, in __call__
INTERNALERROR>     return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
INTERNALERROR>   File "...\pytest-3.0.6-py2.7.egg\_pytest\vendored_packages\pluggy.py", line 339, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "...\pytest-3.0.6-py2.7.egg\_pytest\vendored_packages\pluggy.py", line 334, in <lambda>
INTERNALERROR>     _MultiCall(methods, kwargs, hook.spec_opts).execute()
INTERNALERROR>   File "...\pytest-3.0.6-py2.7.egg\_pytest\vendored_packages\pluggy.py", line 614, in execute
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "...\pytest-3.0.6-py2.7.egg\_pytest\main.py", line 141, in pytest_collection
INTERNALERROR>     return session.perform_collect()
INTERNALERROR>   File "...\pytest-3.0.6-py2.7.egg\_pytest\main.py", line 600, in perform_collect
INTERNALERROR>     items = self._perform_collect(args, genitems)
INTERNALERROR>   File "...\pytest-3.0.6-py2.7.egg\_pytest\main.py", line 636, in _perform_collect
INTERNALERROR>     self.items.extend(self.genitems(node))
INTERNALERROR>   File "...\pytest-3.0.6-py2.7.egg\_pytest\main.py", line 772, in genitems
INTERNALERROR>     for x in self.genitems(subnode):
INTERNALERROR>   File "...\pytest-3.0.6-py2.7.egg\_pytest\main.py", line 765, in genitems
INTERNALERROR>     node.ihook.pytest_itemcollected(item=node)
INTERNALERROR>   File "...\pytest-3.0.6-py2.7.egg\_pytest\vendored_packages\pluggy.py", line 745, in __call__
INTERNALERROR>     return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
INTERNALERROR>   File "...\pytest-3.0.6-py2.7.egg\_pytest\vendored_packages\pluggy.py", line 339, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "...\pytest-3.0.6-py2.7.egg\_pytest\vendored_packages\pluggy.py", line 334, in <lambda>
INTERNALERROR>     _MultiCall(methods, kwargs, hook.spec_opts).execute()
INTERNALERROR>   File "...\pytest-3.0.6-py2.7.egg\_pytest\vendored_packages\pluggy.py", line 614, in execute
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "D:\workspace\sdt\sdt-test\us_test\conftest.py", line 208, in pytest_itemcollected
INTERNALERROR>     pytest.skip()
INTERNALERROR>   File "...\pytest-3.0.6-py2.7.egg\_pytest\runner.py", line 529, in skip
INTERNALERROR>     raise Skipped(msg=msg)
INTERNALERROR> Skipped: <Skipped instance>

Thank you for your time.

[twmr]

Have you tried testing your code with the latest pytest version? Please provide a minimal example.

[bechaos]

With attached example.zip:

============================= test session starts =============================
platform win32 -- Python 2.7.10, pytest-3.4.0, py-1.5.2, pluggy-0.6.0
rootdir: D:\workspace\sdt\sdt-test\pt_example, inifile:
plugins: faulthandler-1.4.1, cov-2.4.0
collecting 0 items                                                             INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "...\pytest-3.4.0-py2.7\_pytest\main.py", line 100, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "...\pytest-3.4.0-py2.7\_pytest\main.py", line 137, in _main
INTERNALERROR>     config.hook.pytest_collection(session=session)
INTERNALERROR>   File "build\bdist.win-amd64\egg\pluggy\__init__.py", line 617, in __call__
INTERNALERROR>     return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
INTERNALERROR>   File "build\bdist.win-amd64\egg\pluggy\__init__.py", line 222, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "build\bdist.win-amd64\egg\pluggy\__init__.py", line 216, in <lambda>
INTERNALERROR>     firstresult=hook.spec_opts.get('firstresult'),
INTERNALERROR>   File "build\bdist.win-amd64\egg\pluggy\callers.py", line 201, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "build\bdist.win-amd64\egg\pluggy\callers.py", line 77, in get_result
INTERNALERROR>     _reraise(*ex)  # noqa
INTERNALERROR>   File "build\bdist.win-amd64\egg\pluggy\callers.py", line 180, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "...\pytest-3.4.0-py2.7\_pytest\main.py", line 147, in pytest_collection
INTERNALERROR>     return session.perform_collect()
INTERNALERROR>   File "...\pytest-3.4.0-py2.7\_pytest\main.py", line 328, in perform_collect
INTERNALERROR>     items = self._perform_collect(args, genitems)
INTERNALERROR>   File "...\pytest-3.4.0-py2.7\_pytest\main.py", line 365, in _perform_collect
INTERNALERROR>     self.items.extend(self.genitems(node))
INTERNALERROR>   File "...\pytest-3.4.0-py2.7\_pytest\main.py", line 509, in genitems
INTERNALERROR>     for x in self.genitems(subnode):
INTERNALERROR>   File "...\pytest-3.4.0-py2.7\_pytest\main.py", line 502, in genitems
INTERNALERROR>     node.ihook.pytest_itemcollected(item=node)
INTERNALERROR>   File "build\bdist.win-amd64\egg\pluggy\__init__.py", line 617, in __call__
INTERNALERROR>     return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
INTERNALERROR>   File "build\bdist.win-amd64\egg\pluggy\__init__.py", line 222, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "build\bdist.win-amd64\egg\pluggy\__init__.py", line 216, in <lambda>
INTERNALERROR>     firstresult=hook.spec_opts.get('firstresult'),
INTERNALERROR>   File "build\bdist.win-amd64\egg\pluggy\callers.py", line 201, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "build\bdist.win-amd64\egg\pluggy\callers.py", line 77, in get_result
INTERNALERROR>     _reraise(*ex)  # noqa
INTERNALERROR>   File "build\bdist.win-amd64\egg\pluggy\callers.py", line 180, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "D:\workspace\sdt\sdt-test\pt_example\conftest.py", line 4, in pytest_itemcollected
INTERNALERROR>     pytest.skip()
INTERNALERROR>   File "...\pytest-3.4.0-py2.7\_pytest\outcomes.py", line 79, in skip
INTERNALERROR>     raise Skipped(msg=msg, allow_module_level=allow_module_level)
INTERNALERROR> Skipped: <Skipped instance>

======================== no tests ran in 0.09 seconds =========================

If you comment pytest_itemcollected in conftest.py:

============================= test session starts =============================
platform win32 -- Python 2.7.10, pytest-3.4.0, py-1.5.2, pluggy-0.6.0
rootdir: D:\workspace\sdt\sdt-test\pt_example, inifile:
plugins: faulthandler-1.4.1, cov-2.4.0
collected 1 item

test_A.py s

========================== 1 skipped in 0.14 seconds ==========================

[twmr]

Do you really need to "skip" tests, or would it be possible to "deselect" them ?

[bechaos]

I have scenario where test need to be skipped, and i cant just mark them, because skip decision is a bit more complex than in attached example.
If you were suggesting alternative or workaround, adding skip mark item.add_marker(pytest.mark.skip()) in pytest_itemcollected works, but than item teardown hooks are (unnecessary) called and I'd like to avoid that.
Question still remains is internalerror a bug?

[twmr]

Would it be possible if you deselect the tests in pytest_collection_modifyitems and implement the complicated skip logic there?

[bechaos]

pytest_collection_modifyitems is called after pytest_itemcollected. I wanted to skip test as soon as i can and best case would be if I could skip test even before i.e in pytest_generate_tests. However i need information if the test is parametrized and I dont know how to get it from metafunc (do you know is it possible and if so how?). Fact is, i'd prefer to skip test in pytest_generate_tests so the answer to prior question would be much appreciated.

Regarding pytest_collection_modifyitems, it is called just once on all items, so i couldnt call pytest.skip but would have to do exactly the same as i do in pytest_itemcollected, so i see no difference.

As I wrote, i have workaround for my case, so question is just do you see internalerror as a bug or not?

[bechaos]

Regarding my question about metafunc, i guess found it myself, metafunc.function.parametrize, but if you have anything to add, please do.

[RonnyPfannschmidt]

for this issue - using skip in that hook is a programmer error, i consider it perfectly valid for pytest to fall over there - a skip happens in a test execution - the but where the code is at is test selection

[The-Compiler]

Should we close this then?

[RonnyPfannschmidt]

i believe so, but i'd like to hear back from @bechaos first

[bechaos]

I dont understand why is using pytest.skip in pytest_itemcollected a programmer error and in e.g. pytest_collection_modifyitems not. But if you see it as such, there should be some information about that in documentation. (not to write bug report again)

[nicoddemus]

for this issue - using skip in that hook is a programmer error, i consider it perfectly valid for pytest to fall over there - a skip happens in a test execution - the but where the code is at is test selection

Makes sense, but why I wonder it works with pytest_collection_modifyitems?

[RonnyPfannschmidt]

hmm, if it works, then we are likely looking at a bug in pytest there, it shouldnt

[nicoddemus]

It might be sufficient to just mention in the docs that pytest.skip is meant to be used during test execution (test runs, fixtures, pytest_runtest_* hooks only). Usage in other hooks is not explicitly supported (even if it works in some cases).

[ceridwen]

Is pytest.mark.skip supposed to be allowed or disallowed in the params argument of pytest.fixture?

[nicoddemus]

@ceridwen currently it works:

import pytest

@pytest.fixture(params=[pytest.mark.skip('foo')(1), 2, 3])
def fix(request):
    pass

def test(fix):
    pass

def test2():
    pass

.tmp\test_fixture_marks.py::test[1] SKIPPED          [ 25%]
.tmp\test_fixture_marks.py::test[2] PASSED           [ 50%]
.tmp\test_fixture_marks.py::test[3] PASSED           [ 75%]
.tmp\test_fixture_marks.py::test2 PASSED             [100%]

=========== 3 passed, 1 skipped in 0.03 seconds ===========

[ceridwen]

I just wondering if it's going to continue to work in the future. Actually, I should say, it usually works. I haven't been able to put together a minimal example for it yet, but I've been using pytest.mark.skipif("not pytest.config.getoption('option')") in a parameters list for a session-scoped fixture and discovered that under certain circumstances the tests will refuse to run because pytest.config.getoption doesn't exist, presumably because of some internal ordering issue.

[nicoddemus]

pytest.config.getoption doesn't exist, presumably because of some internal ordering issue.

Actually string-expression are evaluated using the globals available where the test function executes, here's the related code:

pytest/_pytest/skipping.py

Lines 110 to 114 in 3909225

	def _getglobals(self):
	d = {'os': os, 'sys': sys, 'config': self.item.config}
	if hasattr(self.item, 'obj'):
	d.update(self.item.obj.__globals__)
	return d

This might explain why it works in some cases (pytest is imported in the module of the test functions) and not in others.

That's one of the reasons why using a boolean is preferred than string expressions, see string conditions in the docs.

[ceridwen]

That explains it. Naturally, in my use-case it has to be in conftest.py because it's a session fixture that's used by every test but has to be configured by command-line arguments.

[nicoddemus]

@ceridwen in that case try using pytest.mark.skipif("not config.getoption('option')") instead, because config is added by pytest itself as part of the namespace to be evaluated.