report.nodeid is not proper in pytest_runtest_logreport() hook for setup/teardown

[raoyitao]

I ran pytest 3.6.0 with python 3.4 on redhat 6.

I have a module pytest_tasks.py and a test class named 'TestClassOne', in which there's a testcase named 'test_success'. And I wrote a plugin with pytest_runtest_logreport() hook.

If I define a setup_method(self, method) in TestClassOne, then in pytest_runtest_logreport(self, report) I can get:
report.when == 'setup'
report.nodeid == 'pytest_tasks.py::TestClassOne::()::test_success'
I think this information is correct.

If I define a classmethod setup_class(cls), or a setup_module(module) in module scope, then I would expect to have report.nodeid equal to 'pytest_tasks.py::TestClassOne' and 'pytest_tasks.py' respectively, because the setup() works in suite scope or module scope, but the value is always 'pytest_tasks.py::TestClassOne::()::test_success'. So when I get a setup report I don't know which level it is, how can I get the correct information ?

[RonnyPfannschmidt]

please show the actual code - and use the github code formatting so we can actually read it

[RonnyPfannschmidt]

oh, also all setup of higher scopes currently happens at the setup stage of one item, which is the reported item

[raoyitao]

My test is really simple:

=============pytest_task.py===============

import pytest

def setup_module(module):

    #raise Exception('wrong')
    pass

def teardown_module(module):

    #raise Exception('wrong')
    pass

class TestClassOne(object):

    @classmethod
    def setup_class(cls):
        #raise Exception('wrong')
        pass

    @classmethod
    def teardown_class(cls):
        #raise Exception('wrong')
        pass

    def setup_method(self, method):
        #raise Exception('wrong')
        pass

    def teardown_method(self, method):
        #raise Exception('wrong')
        pass

    def test_success(self):
        assert 1 == 1

    def test_failure(self):
        assert True is False

================py_test.py=============

import pytest

class PyTestPlugin(object):

    def pytest_runtest_logreport(self, report):
        # Here I want to log something

_pytest_plugin = PyTestPlugin()
return_code = pytest.main(['pytest_task.py'], plugins=[_pytest_plugin])

===========================
The setup and teardown function may fail (i.e. exception raised). In function pytest_runtest_logreport I want to log the test result in tree Structure, for example:

Module Pytest_task - Fail
____setup - pass
____TestClassOne - Fail
________setup - pass
________test_success - pass
____________setup - pass
____________teardown - Pass
________test_failure - Fail
____________setup - Fail
________teardown - pass
____teardown - fail

also, I can only print the setup/teardown if they failed. But first of all, I need to know if the result of setup/teardown in each scope. Now, in pytest_runtest_logreport() hook, while report.when is setup or teardown, it only means something went wrong, but I don't know where it is.

[RonnyPfannschmidt]

currently all setup of outer scopes happens at the time of the first setup, so indeed there is no direct way to tell where it went wrong

with fixtures, or the upcoming #4091 you can tell
based on fixture setup hooks,

[nicoddemus]

@raoyitao can you try with the branch from #4091 and see if it fixes the issue for you? You can install it with:

pip install git+https://github.com/nicoddemus/pytest@setup-methods-as-fixtures-3094 

[Zac-HD]

Closing as inactive.