Wrong session scoped fixtures parametrization

[janbudinsky]

Consider following testing code:

import pytest


@pytest.fixture(scope="session")
def session_wrapper(param1, param2):
    return param1, param2


class TestClass:
    @pytest.fixture(scope="session",
                    params=["p1 inside A", "p1 inside B"])
    def param1(self, request):
        return request.param

    @pytest.fixture(scope="session")
    def param2(self):
        return "p2 inside"

    def test_inside(self, session_wrapper, param1, param2):
        assert session_wrapper == (param1, param2)


@pytest.fixture(scope="session",
                params=["p1 outside A", "p1 outside B"])
def param1(request):
    return request.param


@pytest.fixture(scope="session")
def param2():
    return "p2 outside"


def test_outside(session_wrapper, param1, param2):
    assert session_wrapper == (param1, param2)

Session-scoped fixture session_wrapper is parametrized by param1 and param2 fixtures, which the fixture returns as tuple. Tests check if session_wrapper really returns current param1 and param2 instances. Which is not the case - first session_wrapper instance in new context returns wrong params (from previous context):

$ py.test -vv test_f.py
============================= test session starts =============================
platform win32 -- Python 3.4.3, pytest-3.2.3, py-1.4.34, pluggy-0.4.0 -- c:\program files (x86)\python34\python.exe
cachedir: .cache
rootdir: D:\workspace\businessFactory\calistos, inifile: setup.cfg
plugins: cov-2.5.1
collecting ... collected 4 items

test_f.py::TestClass::test_inside[p1 inside A] PASSED
test_f.py::test_outside[p1 outside A] FAILED
test_f.py::TestClass::test_inside[p1 inside B] PASSED
test_f.py::test_outside[p1 outside B] PASSED

================================== FAILURES ===================================
_________________________ test_outside[p1 outside A] __________________________
test_f.py:37: in test_outside
    assert session_wrapper == (param1, param2)
E   AssertionError: assert ('p1 inside', 'p2 inside') == ('p1 outside A', 'p2 outside')
E     At index 0 diff: 'p1 inside' != 'p1 outside A'
E     Full diff:
E     - ('p1 inside', 'p2 inside')
E     ?      ^^           ^^
E     + ('p1 outside A', 'p2 outside')
E     ?      ^^^    ++       ^^^
===================== 1 failed, 3 passed in 0.13 seconds ======================
Coverage.py warning: No data was collected. (no-data-collected)

If TestClass is at the end of the script (first outside test and context, then inside), equivalent result is achieved:

test_f.py::test_outside[p1 outside A] PASSED
test_f.py::TestClass::test_inside[p1 inside A] FAILED
test_f.py::test_outside[p1 outside B] PASSED
test_f.py::TestClass::test_inside[p1 inside B] PASSED

When session_wrapper has class or function scope, everything works as expected.

[nicoddemus]

Thanks for the report. This is really strange.

I managed to reduce the example further:

import pytest


@pytest.fixture(scope="session")
def session_wrapper(param1):
    return param1


class TestClass:
    @pytest.fixture(scope="session")
    def session_wrapper(self, session_wrapper):
        return session_wrapper

    @pytest.fixture(scope="session",
                    params=["inside A", "inside B"])
    def param1(self, request):
        return request.param

    def test_inside(self, session_wrapper, param1):
        assert session_wrapper == param1


@pytest.fixture(scope="session",
                params=["outside A", "outside B"])
def param1(request):
    return request.param


def test_outside(session_wrapper, param1):
    assert session_wrapper == param1

And here is the full execution and failure:

============================= test session starts =============================
platform win32 -- Python 3.6.0, pytest-3.2.4.dev10+gae4e596, py-1.4.34, pluggy-0.4.0 -- c:\pytest\.env36\scripts\python.exe
cachedir: .tmp\.cache
rootdir: C:\pytest\.tmp, inifile: pytest.ini
plugins: hypothesis-3.25.0
collected 4 items

.tmp\test_param.py::TestClass::test_inside[inside A] PASSED
.tmp\test_param.py::test_outside[outside A] FAILED
.tmp\test_param.py::TestClass::test_inside[inside B] PASSED
.tmp\test_param.py::test_outside[outside B] PASSED

================================== FAILURES ===================================
___________________________ test_outside[outside A] ___________________________

session_wrapper = 'inside A', param1 = 'outside A'

    def test_outside(session_wrapper, param1):
>       assert session_wrapper == param1
E       AssertionError: assert 'inside A' == 'outside A'
E         - inside A
E         + outside A

.tmp\test_param.py:30: AssertionError
===================== 1 failed, 3 passed in 0.05 seconds ======================

[janbudinsky]

The same issue happens across files too.
tests/conftest.py:

import pytest


@pytest.fixture(scope="session")
def session_wrapper(param):
    return param

tests/test_file1:

import pytest


@pytest.fixture(scope="session",
                params=["f1 A", "f1 B"])
def param(request):
    return request.param


def test_1(session_wrapper, param):
    assert session_wrapper == param

tests/test_file2:

import pytest


@pytest.fixture(scope="session",
                params=["f2 A", "f2 B"])
def param(request):
    return request.param


def test_2(session_wrapper, param):
    assert session_wrapper == param

Execution and failure:

$ py.test -vv tests
============================= test session starts =============================
platform win32 -- Python 3.4.3, pytest-3.2.3, py-1.4.34, pluggy-0.4.0 -- c:\program files (x86)\python34\python.exe
cachedir: ..\.cache
rootdir: D:\workspace\businessFactory\calistos, inifile: setup.cfg
plugins: cov-2.5.1
collecting ... collected 4 items

tests\test_file1.py::test_1[f1 A] PASSED
tests\test_file2.py::test_2[f2 A] FAILED
tests\test_file1.py::test_1[f1 B] PASSED
tests\test_file2.py::test_2[f2 B] PASSED

================================== FAILURES ===================================
________________________________ test_2[f2 A] _________________________________
tests\test_file2.py:13: in test_2
    assert session_wrapper == param
E   AssertionError: assert 'f1 A' == 'f2 A'
E     - f1 A
E     ?  ^
E     + f2 A
E     ?  ^
===================== 1 failed, 3 passed in 0.09 seconds ======================

[ghost]

The problem occurs for session and module scoped fixtures. It seems like session_wrapper is somehow cached during the 1st context switch - from module-level test_outside to class-level test_inside (and vice versa) and its return value from previous execution is preserved.

I put import pdb; pdb.set_trace() at the top of session_wrapper:

@pytest.fixture(scope="session")
def session_wrapper(param1):
    import pdb; pdb.set_trace()
    return param1

and it drops into the debugger every time BUT NOT this one after scope switching (from class test <-> module test):

============================= test session starts =============================
platform linux2 -- Python 2.7.12, pytest-3.2.4.dev38+g5631a86.d20171024, py-1.4.34,
pluggy-0.4.0 -- /home/kris/.virtualenvs/pytest-dev/bin/python
cachedir: .cache
rootdir: /home/kris/projects/pytest, inifile: tox.ini
plugins: hypothesis-3.33.0
collected 6 items                                                              

t.py::test_outside[outside A] 
>>>>>>>>>>>>>>>>>>> PDB set_trace (IO-capturing turned off) >>>>>>>>>>>>>>>>>>>
[55] > /home/kris/projects/pytest/t.py(7)session_wrapper()
-> return param1
(Pdb++) continue  # OK - dropped into pdb
PASSED
t.py::TestClass::test_inside[inside A] FAILED  # DIDN'T drop into pdb
t.py::test_outside[outside B] 
>>>>>>>>>>>>>>>>>>> PDB set_trace (IO-capturing turned off) >>>>>>>>>>>>>>>>>>>
[55] > /home/kris/projects/pytest/t.py(7)session_wrapper()
-> return param1
(Pdb++) continue  # OK - dropped into pdb
PASSED
t.py::TestClass::test_inside[inside B] 
>>>>>>>>>>>>>>>>>>> PDB set_trace (IO-capturing turned off) >>>>>>>>>>>>>>>>>>>
[55] > /home/kris/projects/pytest/t.py(7)session_wrapper()
-> return param1
(Pdb++) continue  # OK - dropped into pdb
PASSED
t.py::test_outside[outside C] 
>>>>>>>>>>>>>>>>>>> PDB set_trace (IO-capturing turned off) >>>>>>>>>>>>>>>>>>>
[55] > /home/kris/projects/pytest/t.py(7)session_wrapper()
-> return param1
(Pdb++) continue  # OK - dropped into pdb
PASSED
t.py::TestClass::test_inside[inside C] 
>>>>>>>>>>>>>>>>>>> PDB set_trace (IO-capturing turned off) >>>>>>>>>>>>>>>>>>>
[55] > /home/kris/projects/pytest/t.py(7)session_wrapper()
-> return param1
(Pdb++) continue  # OK - dropped into pdb
PASSED

================================== FAILURES ===================================
_______________________ TestClass.test_inside[inside A] _______________________

self = <t.TestClass instance at 0x7f1bbb0fedd0>, session_wrapper = 'outside A'
param1 = 'inside A'

    def test_inside(self, session_wrapper, param1):
>       assert session_wrapper == param1
E       AssertionError: assert 'outside A' == 'inside A'
E         - outside A
E         + inside A

t.py:37: AssertionError
===================== 1 failed, 5 passed in 4.42 seconds ======================

[golvok]

Discovered this today... I think it's a variation on this ticket -- no parameterization needed? It seems like just instantiating the session-scope fixture once (not surprising?), but when it does, it uses the overrides available at the time of first use.
Which is test-order-dependent behaviour, so not good. I should also note that the right fixtures get run, but the value seen is not what you might expect from looking at just the containing scopes.

It could be surprising that sesion_wrapper isn't restarted with the new param, or even more surprising that there are two params running (say, they manage the same resource, but clash). My work-around is to say "when defining a fixture, it's scope parameter should not be greater than it's declared-in scope". E.g., it is encouraged that the only file to declare session-scope fixtures is the root conftest.py, other conftests max-out at "package", and other files at "module". This has the effect of making this a collect-time error.

IMO, the fix for this is either:

• Detect the incompatible override + error. (possibly a big breaking change?)
• Restart session_wrapper. (sounds complicated? could be surprising?)

# conftest.py
@pytest.fixture(scope='session')
def session_wrapper(param):
   print("\nroot: session_wrapper: start")
   yield param
   print("\nroot: session_wrapper: done")

@pytest.fixture(scope='session')
def param():
   print("\nroot: param: start") # always printed if test_file_2 is run
   yield 'root'
   print("\nroot: param: done") # always printed if test_file_2 is run

# test_1.py
@pytest.fixture(scope='session')
def param():
   print("\nfile 1: param: start") # always printed if test_file_1 is run
   yield 'file 1'
   print("\nfile 1: param: done") # always printed if test_file_1 is run

def test_file_1(session_wrapper):
   print(session_wrapper) # prints either 'file 1' or 'root', depending on which test ran first

# test_2.py
def test_file_2(session_wrapper):
   print(session_wrapper) # prints either 'file 1' or 'root', depending on which test ran first

Run all tests: both get the test_1.py value. Note that both of the param fixtures are active at the same time, but only one sesion_wrapper.

(python_venv) % pytest test_?.py -sv
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.4.0, pluggy-1.2.0
plugins: split-0.8.1, timeout-2.1.0, cov-4.1.0
collected 2 items

test_1.py::test_file_1
file 1: param: start

root: session_wrapper: start
file 1
PASSED
test_2.py::test_file_2
root: param: start
file 1
PASSED
root: session_wrapper: done

root: param: done

file 1: param: done

============================== 2 passed in 0.01s ===============================

Run just test_1.py: seems normal (test gets local override, 'file 1').

(python_venv) % pytest test_1.py -sv
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.4.0, pluggy-1.2.0
plugins: split-0.8.1, timeout-2.1.0, cov-4.1.0
collected 1 item

test_1.py:test_file_1
file 1: param: start

root: session_wrapper: start
file 1
PASSED
root: session_wrapper: done

file 1: param: done

============================== 1 passed in 0.01s ===============================

Run just test_2.py: The test gets the 'root' fixture's value. Seems normal.

(python_venv) % pytest test_2.py -sv
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.4.0, pluggy-1.2.0
plugins: split-0.8.1, timeout-2.1.0, cov-4.1.0
collected 1 item

test_2.py:test_file_2
root: param: start

root: session_wrapper: start
root
PASSED
root: session_wrapper: done

root: param: done

============================== 1 passed in 0.01s ==============================