pytester configuration leaks in main pytest configuration

[youtux]

I discover this while debugging a failure in pytest-bdd (see pytest-dev/pytest-bdd#276).
It seems that the ini configuration created by testdir.makeini function leaks in the main pytest configuration.
Here it is a test that reproduce the issue:

$ pytest test_bug.py::test_1 test_bug.py::test_2

# test_bug.py
import pytest

pytest_plugins = "pytester"


def test_1(testdir):
    """Creating some configuration in the testdir. We set the `minversion` option, but it can be any option."""
    testdir.makeini(
        """
        [pytest]
        minversion=3.0.0
    """
    )

    testdir.makepyfile(
        """
        import pytest

        def test_expected_minversion():
            assert pytest.config.getini('minversion') == '3.0.0'
    """
    )

    result = testdir.runpytest()
    result.assert_outcomes(passed=1)


def test_2():
    """The configuration made in test_1 should not leak here."""
    assert pytest.config.getini('minversion') != '3.0.0'

This is the configuration I am using:

Package        Version
-------------- -------
atomicwrites   1.2.1
attrs          18.2.0
more-itertools 4.3.0
pip            18.1
pluggy         0.8.0
py             1.7.0
pytest         4.0.1
setuptools     40.6.2
six            1.11.0
wheel          0.32.3

Python 3.7.1 on Mac OS 10.14.1

• Include a detailed description of the bug or suggestion
• pip list of the virtual environment you are using
• pytest and operating system versions
• Minimal example if possible

[RonnyPfannschmidt]

its not so much leaking, but rather a rather massive/painful bug about pytests internal state

as pytest does in fact patch the object pytest.config, but never undoes the patch

[RonnyPfannschmidt]

pytest/src/_pytest/main.py

Lines 170 to 172 in ecc5c84

	def pytest_configure(config):
	__import__("pytest").config = config # compatibility

[RonnyPfannschmidt]

also ref #3050

[youtux]

It would be quite unfortunate if pytest.config is removed, since pytest-bdd requires it at test import time (cannot just switch to request.config).

[RonnyPfannschmidt]

@youtux why would it need it that way?

[youtux]

Because pytest-bdd generates test items from .feature files at import time, but in order to parse the .feature files it needs to know where they are stored. This value is currently taken from the ini configuration of pytest.
Before pytest-dev/pytest-bdd#255, pytest-bdd used to get this value by calling a fixture, but since fixture cannot be called directly anymore, we changed it to use the pytest.config.getini().

[RonnyPfannschmidt]

all that happens past pytest-configure time, just configure your plugin at that time instead of getting the globals later

[youtux]

I don't really get what you are suggesting, @RonnyPfannschmidt

[asottile]

does this become "wontfix" given the resolution of #3050 (deprecating pytest.config global)?

[RonnyPfannschmidt]

@asottile we can debate on whether we want pytester to remember/reset the value

[sliwinski-milosz]

all that happens past pytest-configure time, just configure your plugin at that time instead of getting the globals later

@RonnyPfannschmidt could you please tell us what you meant? Please give us some hint or link to the documentation.
Does the bug only affects pytest.config? Does it mean that if we will switch to solution mentioned by you we will not be affected by it anymore?

Because of the problem with pytest.config one pytest-bdd test is failing and that blocks our releases. I would like to fix it asap.

[RonnyPfannschmidt]

@sliwinski-milosz to fix pytest-bdd, the pytester plugin would have t fix the reference - note that pytest.config will go away with the next major release

[sliwinski-milosz]

I think this bug is even worse. It is not only about undoing the patch of the config. If I understand correctly hooks are affected too.

Lets say for now that we configure our plugin using pytest_configure hook.

Please take a look at following steps:

1. We run our test suite that has one normal test and one test that uses testdir plugin.
2. We start from test that uses testdir.
3. When we run testdir.runpytest it runs pytest_configure hook and it configures our plugin with some custom config.
4. Now we switch to normal test. But it will not run pytest_configure again (and other hooks that are meant to be started once) so still our internal state is broken even if we undone config patch.

Important part is that the testdir test is testing config.

One solution for above is to run our testdir test in subprocess by using --runpytest=subprocess option which guarantees test separation when we try to test config values. But that means those tests will run slower.

[RonnyPfannschmidt]

@sliwinski-milosz for correctness you need pytest_configure and pytest_unconfigure paired

[sliwinski-milosz]

Ah ok. In our case unconfigure means bring previous values back.
To do that I need to be able to access the "previous" config. Can I get it from somewhere?
Would be great if it would be passed as an argument to pytest_unconfigure but as far as I can see it provides current config.

[RonnyPfannschmidt]

@sliwinski-milosz with the current layering of pytest, you have to track it yourself, its quite a pain,

[sliwinski-milosz]

Thanks! All clear now!

I have opened pull request for pytest-bdd in which I use config stack. It seems that it works fine :)

[RonnyPfannschmidt]

@sliwinski-milosz great job

[RonnyPfannschmidt]

Closing as we removed the config global