Teardown for parametrized module scope fixtures is not executed in the teardown phase

[samjl]

I am writing a plugin to track saved results during the different test phases (setup, call, teardown) and use the pytest_runtest_setup, pytest_pyfunc_call, pytest_runtest_teardown functions to track the current test phase.

However, I have encountered a problem tracking the test phase when parametrizing module scoped fixtures.

The intermediate module teardown (all but the last module fixture parameter teardown) is executed after both pytest_runtest_teardown and pytest_report_teststatus (for the teardown phase) have executed but it would be expected to execute when the pytest_runtest_teardown yields. This is shown in the annotated output below.

=========================================================================================================== test session starts ============================================================================================================
platform linux2 -- Python 2.7.12, pytest-3.3.1, py-1.5.2, pluggy-0.6.0
rootdir: /home/slea1/workspace/pytest_teardown, inifile:
collected 6 items                                                                                                                                                                                                                          

test_pytest_teardown_order.py 
SETUP phase starting for test test_no_fixture[x]
module_scoped_fix-setup
SETUP phase complete for test test_no_fixture[x]

REPORT stage for phase setup, outcome: passed

CALL phase starting for test test_no_fixture[x]
test_no_fixture
CALL complete for test test_no_fixture[x]

REPORT stage for phase call, outcome: passed
.
TEARDOWN phase starting for test test_no_fixture[x]
TEARDOWN phase complete for test test_no_fixture[x]
TEARDOWN next item is test_func_fixture[x-a]

REPORT stage for phase teardown, outcome: passed

SETUP phase starting for test test_func_fixture[x-a]
function_params_fix-setup
SETUP phase complete for test test_func_fixture[x-a]

REPORT stage for phase setup, outcome: passed

CALL phase starting for test test_func_fixture[x-a]
test_func_fixture
CALL complete for test test_func_fixture[x-a]

REPORT stage for phase call, outcome: passed
.
TEARDOWN phase starting for test test_func_fixture[x-a]
function_params_fix-teardown
TEARDOWN phase complete for test test_func_fixture[x-a]
TEARDOWN next item is test_func_fixture[x-b]

REPORT stage for phase teardown, outcome: passed

SETUP phase starting for test test_func_fixture[x-b]
function_params_fix-setup
SETUP phase complete for test test_func_fixture[x-b]

REPORT stage for phase setup, outcome: passed

CALL phase starting for test test_func_fixture[x-b]
test_func_fixture
CALL complete for test test_func_fixture[x-b]

REPORT stage for phase call, outcome: passed
.
TEARDOWN phase starting for test test_func_fixture[x-b]
function_params_fix-teardown                                   Should be executed here
TEARDOWN phase complete for test test_func_fixture[x-b]         ^
TEARDOWN next item is test_no_fixture[y]                        |
                                                                |
REPORT stage for phase teardown, outcome: passed                |   Also means this report is incorrect - passes if module teardown fails
                                                                |
SETUP phase starting for test test_no_fixture[y]                |
module_scoped_fix-teardown                                     Module teardown executed in setup phase of following test
module_scoped_fix-setup
SETUP phase complete for test test_no_fixture[y]

REPORT stage for phase setup, outcome: passed

CALL phase starting for test test_no_fixture[y]
test_no_fixture
CALL complete for test test_no_fixture[y]

REPORT stage for phase call, outcome: passed
.
TEARDOWN phase starting for test test_no_fixture[y]
TEARDOWN phase complete for test test_no_fixture[y]
TEARDOWN next item is test_func_fixture[y-a]

REPORT stage for phase teardown, outcome: passed

SETUP phase starting for test test_func_fixture[y-a]
function_params_fix-setup
SETUP phase complete for test test_func_fixture[y-a]

REPORT stage for phase setup, outcome: passed

CALL phase starting for test test_func_fixture[y-a]
test_func_fixture
CALL complete for test test_func_fixture[y-a]

REPORT stage for phase call, outcome: passed
.
TEARDOWN phase starting for test test_func_fixture[y-a]
function_params_fix-teardown
TEARDOWN phase complete for test test_func_fixture[y-a]
TEARDOWN next item is test_func_fixture[y-b]

REPORT stage for phase teardown, outcome: passed

SETUP phase starting for test test_func_fixture[y-b]
function_params_fix-setup
SETUP phase complete for test test_func_fixture[y-b]

REPORT stage for phase setup, outcome: passed

CALL phase starting for test test_func_fixture[y-b]
test_func_fixture
CALL complete for test test_func_fixture[y-b]

REPORT stage for phase call, outcome: passed
.                                                                                                                                                                                                 [100%]
TEARDOWN phase starting for test test_func_fixture[y-b]
function_params_fix-teardown
module_scoped_fix-teardown                                  Final parameter module teardown is correctly executed here
TEARDOWN phase complete for test test_func_fixture[y-b]

REPORT stage for phase teardown, outcome: passed


========================================================================================================= 6 passed in 0.01 seconds =========================================================================================================

The test module and conftest files required to reproduce the issue are also included.

Note that:

• the teardown is executed as expected for module scope fixtures that are not parametrized (as is the case for the final parameter of the example provided)
• the test includes a parametrized function scoped fixture and this behaves as expected (teardown executed during pytest_runtest_teardown yield).

test script:

from pytest import fixture


@fixture(scope='module', params=["x", "y"])  # same effect with autouse=True
def module_scoped_fix(request):
    def module_teardown():
        print("module_scoped_fix-teardown")
    request.addfinalizer(module_teardown)

    def module_setup():
        print("module_scoped_fix-setup")
    module_setup()


@fixture(scope='function', params=["a", "b"])
# @fixture(scope='function')
def function_params_fix(request):
    def function_teardown():
        print("function_params_fix-teardown")
    request.addfinalizer(function_teardown)

    def function_setup():
        print("function_params_fix-setup")
    function_setup()


def test_no_fixture(module_scoped_fix):
    print("test_no_fixture")


def test_func_fixture(module_scoped_fix, function_params_fix):
    print("test_func_fixture")

conftest:

import pytest


@pytest.hookimpl(hookwrapper=True)
def pytest_runtest_setup(item):
    print("\nSETUP phase starting for test {}".format(item.name))
    # Here I set the current phase to setup
    yield
    print("SETUP phase complete for test {}".format(item.name))
    # Here I set the current phase to call. Do it here because
    # pytest_pyfunc_call doesn't always execute (if setup fails)


@pytest.hookimpl(hookwrapper=True)
def pytest_pyfunc_call(pyfuncitem):
    print("\nCALL phase starting for test {}".format(pyfuncitem.name))
    yield
    print("CALL complete for test {}".format(pyfuncitem.name))


@pytest.hookimpl(hookwrapper=True)
def pytest_runtest_teardown(item, nextitem):
    print("\nTEARDOWN phase starting for test {}".format(item.name))
    # Here I set the current phase to teardown
    yield
    print("TEARDOWN phase complete for test {}".format(item.name))
    if nextitem:
        print("TEARDOWN next item is {}".format(nextitem.name))


def pytest_report_teststatus(report):
    print("\nREPORT stage for phase {0.when}, outcome: {0.outcome}".format(
        report))

pytest_teardown.zip

OS: kubuntu 16.04

pip list:
Package          Version
---------------- -------
attrs            17.3.0 
decorator        4.1.2  
funcsigs         1.0.2  
future           0.16.0 
pip              9.0.1  
pluggy           0.6.0  
py               1.5.2  
pytest           3.3.1  
setuptools       38.2.4 
six              1.11.0 
tox              2.9.1  
virtualenv       15.1.0 
wheel            0.30.0

[pytestbot]

GitMate.io thinks the contributors most likely able to help are @nicoddemus, and @RonnyPfannschmidt.

[nicoddemus]

Hi @samjl, really sorry about the delay.

SETUP phase starting for test test_no_fixture[y]     
module_scoped_fix-teardown   # <--- destroys module_scoped_fix[x]
module_scoped_fix-setup # <--- creates module_scoped_fix[y]
SETUP phase complete for test test_no_fixture[y]

(comments added by me)

Your reasoning makes sense, but that's how pytest works currently: at the point that a fixture is requested is the point where pytest decides if the previous fixture instance must be teardown and a new one created. This is done at this point:

pytest/_pytest/fixtures.py

Lines 771 to 794 in 794fb19

	def execute(self, request):
	# get required arguments and register our own finish()
	# with their finalization
	for argname in self.argnames:
	fixturedef = request._get_active_fixturedef(argname)
	if argname != "request":
	fixturedef.addfinalizer(functools.partial(self.finish, request=request))
	my_cache_key = request.param_index
	cached_result = getattr(self, "cached_result", None)
	if cached_result is not None:
	result, cache_key, err = cached_result
	if my_cache_key == cache_key:
	if err is not None:
	py.builtin._reraise(*err)
	else:
	return result
	# we have a previous but differently parametrized fixture instance
	# so we need to tear it down before creating a new one
	self.finish(request)
	assert not hasattr(self, "cached_result")
	hook = self._fixturemanager.session.gethookproxy(request.node.fspath)
	return hook.pytest_fixture_setup(fixturedef=self, request=request)

In line 783 it compares the current "cache key" of the fixture, and if the new requested cache key differs, it tears down the current instance (790) and then will afterwards create and store a new fixture instance. The side effect of this design is that during a fixture setup call (y) we may actually call the fixture teardown hook for the previous fixture instance (x).

This is the current design and unfortunately I don't think it is easy to change; it would probably require a rewrite of the fixture code to be more proactive (look at the current items and decide which fixtures to create and destroy) rather than reactive (try to create fixtures and decide to teardown previous ones based on the ones being created).

I hope the explanation makes sense.

Unless I'm mistaken I think we will have to close this as is because it will require a major rewrite as I explained above. 😕

@RonnyPfannschmidt any comments?

[samjl]

That makes sense and I agree it sounds like a lot of effort for not much gain...
Also, I have found a workaround for my case using the pytest_fixture_setup and pytest_fixture_post_finalizer hooks to help me track the currently executing fixture or test function. The plugin I am writing is almost complete and I will submit it soon and add an explanation of the workaround here.

I will try to find some time today to add the documentation and some example tests so you can get an idea of how it works. What I have so far is located here samjl/pytest-phases.

I have no issues with closing this issue.
Thanks very much for your reply.

[nicoddemus]

Cool @samjl thanks for understanding. I look forward to take a look at your plugin.

Unrelated question: do you use the request argument of the pytest_fixture_post_finalizer for something?

[samjl]

@nicoddemus Yes, I use the param attribute of request argument.
If a fixture is parametrized I use request.param to retrieve the parameter that was used for the completed teardown fixture.

[nicoddemus]

Cool, thanks for the answer.