pytest.skip in pytest_itemcollected hook raises INTERNALERROR

[bechaos]

Hello,

I have tried to find anything in documentation about hooks and skip, but i was not able to. Calling pytest.skip in other hooks e.g. pytest_runtest_call, pytest_generate_tests or pytest_runtest_setup skips item fine.
However when called in pytest_itemcollected i get:

INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "...\pytest-3.0.6-py2.7.egg\_pytest\main.py", line 98, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "...\pytest-3.0.6-py2.7.egg\_pytest\main.py", line 132, in _main
INTERNALERROR>     config.hook.pytest_collection(session=session)
INTERNALERROR>   File "...\pytest-3.0.6-py2.7.egg\_pytest\vendored_packages\pluggy.py", line 745, in __call__
INTERNALERROR>     return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
INTERNALERROR>   File "...\pytest-3.0.6-py2.7.egg\_pytest\vendored_packages\pluggy.py", line 339, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "...\pytest-3.0.6-py2.7.egg\_pytest\vendored_packages\pluggy.py", line 334, in <lambda>
INTERNALERROR>     _MultiCall(methods, kwargs, hook.spec_opts).execute()
INTERNALERROR>   File "...\pytest-3.0.6-py2.7.egg\_pytest\vendored_packages\pluggy.py", line 614, in execute
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "...\pytest-3.0.6-py2.7.egg\_pytest\main.py", line 141, in pytest_collection
INTERNALERROR>     return session.perform_collect()
INTERNALERROR>   File "...\pytest-3.0.6-py2.7.egg\_pytest\main.py", line 600, in perform_collect
INTERNALERROR>     items = self._perform_collect(args, genitems)
INTERNALERROR>   File "...\pytest-3.0.6-py2.7.egg\_pytest\main.py", line 636, in _perform_collect
INTERNALERROR>     self.items.extend(self.genitems(node))
INTERNALERROR>   File "...\pytest-3.0.6-py2.7.egg\_pytest\main.py", line 772, in genitems
INTERNALERROR>     for x in self.genitems(subnode):
INTERNALERROR>   File "...\pytest-3.0.6-py2.7.egg\_pytest\main.py", line 765, in genitems
INTERNALERROR>     node.ihook.pytest_itemcollected(item=node)
INTERNALERROR>   File "...\pytest-3.0.6-py2.7.egg\_pytest\vendored_packages\pluggy.py", line 745, in __call__
INTERNALERROR>     return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
INTERNALERROR>   File "...\pytest-3.0.6-py2.7.egg\_pytest\vendored_packages\pluggy.py", line 339, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "...\pytest-3.0.6-py2.7.egg\_pytest\vendored_packages\pluggy.py", line 334, in <lambda>
INTERNALERROR>     _MultiCall(methods, kwargs, hook.spec_opts).execute()
INTERNALERROR>   File "...\pytest-3.0.6-py2.7.egg\_pytest\vendored_packages\pluggy.py", line 614, in execute
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "D:\workspace\sdt\sdt-test\us_test\conftest.py", line 208, in pytest_itemcollected
INTERNALERROR>     pytest.skip()
INTERNALERROR>   File "...\pytest-3.0.6-py2.7.egg\_pytest\runner.py", line 529, in skip
INTERNALERROR>     raise Skipped(msg=msg)
INTERNALERROR> Skipped: <Skipped instance>

Thank you for your time.

[twmr]

Have you tried testing your code with the latest pytest version? Please provide a minimal example.

[bechaos]

With attached example.zip:

============================= test session starts =============================
platform win32 -- Python 2.7.10, pytest-3.4.0, py-1.5.2, pluggy-0.6.0
rootdir: D:\workspace\sdt\sdt-test\pt_example, inifile:
plugins: faulthandler-1.4.1, cov-2.4.0
collecting 0 items                                                             INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "...\pytest-3.4.0-py2.7\_pytest\main.py", line 100, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "...\pytest-3.4.0-py2.7\_pytest\main.py", line 137, in _main
INTERNALERROR>     config.hook.pytest_collection(session=session)
INTERNALERROR>   File "build\bdist.win-amd64\egg\pluggy\__init__.py", line 617, in __call__
INTERNALERROR>     return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
INTERNALERROR>   File "build\bdist.win-amd64\egg\pluggy\__init__.py", line 222, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "build\bdist.win-amd64\egg\pluggy\__init__.py", line 216, in <lambda>
INTERNALERROR>     firstresult=hook.spec_opts.get('firstresult'),
INTERNALERROR>   File "build\bdist.win-amd64\egg\pluggy\callers.py", line 201, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "build\bdist.win-amd64\egg\pluggy\callers.py", line 77, in get_result
INTERNALERROR>     _reraise(*ex)  # noqa
INTERNALERROR>   File "build\bdist.win-amd64\egg\pluggy\callers.py", line 180, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "...\pytest-3.4.0-py2.7\_pytest\main.py", line 147, in pytest_collection
INTERNALERROR>     return session.perform_collect()
INTERNALERROR>   File "...\pytest-3.4.0-py2.7\_pytest\main.py", line 328, in perform_collect
INTERNALERROR>     items = self._perform_collect(args, genitems)
INTERNALERROR>   File "...\pytest-3.4.0-py2.7\_pytest\main.py", line 365, in _perform_collect
INTERNALERROR>     self.items.extend(self.genitems(node))
INTERNALERROR>   File "...\pytest-3.4.0-py2.7\_pytest\main.py", line 509, in genitems
INTERNALERROR>     for x in self.genitems(subnode):
INTERNALERROR>   File "...\pytest-3.4.0-py2.7\_pytest\main.py", line 502, in genitems
INTERNALERROR>     node.ihook.pytest_itemcollected(item=node)
INTERNALERROR>   File "build\bdist.win-amd64\egg\pluggy\__init__.py", line 617, in __call__
INTERNALERROR>     return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
INTERNALERROR>   File "build\bdist.win-amd64\egg\pluggy\__init__.py", line 222, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "build\bdist.win-amd64\egg\pluggy\__init__.py", line 216, in <lambda>
INTERNALERROR>     firstresult=hook.spec_opts.get('firstresult'),
INTERNALERROR>   File "build\bdist.win-amd64\egg\pluggy\callers.py", line 201, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "build\bdist.win-amd64\egg\pluggy\callers.py", line 77, in get_result
INTERNALERROR>     _reraise(*ex)  # noqa
INTERNALERROR>   File "build\bdist.win-amd64\egg\pluggy\callers.py", line 180, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "D:\workspace\sdt\sdt-test\pt_example\conftest.py", line 4, in pytest_itemcollected
INTERNALERROR>     pytest.skip()
INTERNALERROR>   File "...\pytest-3.4.0-py2.7\_pytest\outcomes.py", line 79, in skip
INTERNALERROR>     raise Skipped(msg=msg, allow_module_level=allow_module_level)
INTERNALERROR> Skipped: <Skipped instance>

======================== no tests ran in 0.09 seconds =========================

If you comment pytest_itemcollected in conftest.py:

============================= test session starts =============================
platform win32 -- Python 2.7.10, pytest-3.4.0, py-1.5.2, pluggy-0.6.0
rootdir: D:\workspace\sdt\sdt-test\pt_example, inifile:
plugins: faulthandler-1.4.1, cov-2.4.0
collected 1 item

test_A.py s

========================== 1 skipped in 0.14 seconds ==========================

[twmr]

Do you really need to "skip" tests, or would it be possible to "deselect" them ?

[bechaos]

I have scenario where test need to be skipped, and i cant just mark them, because skip decision is a bit more complex than in attached example.
If you were suggesting alternative or workaround, adding skip mark item.add_marker(pytest.mark.skip()) in pytest_itemcollected works, but than item teardown hooks are (unnecessary) called and I'd like to avoid that.
Question still remains is internalerror a bug?

[twmr]

Would it be possible if you deselect the tests in pytest_collection_modifyitems and implement the complicated skip logic there?

[bechaos]

pytest_collection_modifyitems is called after pytest_itemcollected. I wanted to skip test as soon as i can and best case would be if I could skip test even before i.e in pytest_generate_tests. However i need information if the test is parametrized and I dont know how to get it from metafunc (do you know is it possible and if so how?). Fact is, i'd prefer to skip test in pytest_generate_tests so the answer to prior question would be much appreciated.

Regarding pytest_collection_modifyitems, it is called just once on all items, so i couldnt call pytest.skip but would have to do exactly the same as i do in pytest_itemcollected, so i see no difference.

As I wrote, i have workaround for my case, so question is just do you see internalerror as a bug or not?

[bechaos]

Regarding my question about metafunc, i guess found it myself, metafunc.function.parametrize, but if you have anything to add, please do.

[RonnyPfannschmidt]

for this issue - using skip in that hook is a programmer error, i consider it perfectly valid for pytest to fall over there - a skip happens in a test execution - the but where the code is at is test selection

[The-Compiler]

Should we close this then?