Skip to content

Add new INI config key 'required_plugins' that defines a list of plugins that must be present for a run of pytest #7330

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 12 commits into from
Jun 12, 2020
Merged
1 change: 1 addition & 0 deletions changelog/7305.feature.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
New ``required_plugins`` configuration option allows the user to specify a list of plugins required for pytest to run. An error is raised if any required plugins are not found when running pytest.
11 changes: 11 additions & 0 deletions doc/en/reference.rst
Original file line number Diff line number Diff line change
Expand Up @@ -1561,6 +1561,17 @@ passed multiple times. The expected format is ``name=value``. For example::
See :ref:`change naming conventions` for more detailed examples.


.. confval:: required_plugins

A space separated list of plugins that must be present for pytest to run.
If any one of the plugins is not found, emit an error.

.. code-block:: ini

[pytest]
required_plugins = pytest-html pytest-xdist


.. confval:: testpaths


Expand Down
40 changes: 34 additions & 6 deletions src/_pytest/config/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -952,6 +952,12 @@ def _initini(self, args: Sequence[str]) -> None:
self._parser.extra_info["inifile"] = self.inifile
self._parser.addini("addopts", "extra command line options", "args")
self._parser.addini("minversion", "minimally required pytest version")
self._parser.addini(
"required_plugins",
"plugins that must be present for pytest to run",
type="args",
default=[],
)
self._override_ini = ns.override_ini or ()

def _consider_importhook(self, args: Sequence[str]) -> None:
Expand Down Expand Up @@ -1035,7 +1041,8 @@ def _preparse(self, args: List[str], addopts: bool = True) -> None:
self.known_args_namespace = ns = self._parser.parse_known_args(
args, namespace=copy.copy(self.option)
)
self._validatekeys()
self._validate_plugins()
self._validate_keys()
if self.known_args_namespace.confcutdir is None and self.inifile:
confcutdir = py.path.local(self.inifile).dirname
self.known_args_namespace.confcutdir = confcutdir
Expand Down Expand Up @@ -1078,12 +1085,33 @@ def _checkversion(self):
)
)

def _validatekeys(self):
def _validate_keys(self) -> None:
for key in sorted(self._get_unknown_ini_keys()):
message = "Unknown config ini key: {}\n".format(key)
if self.known_args_namespace.strict_config:
fail(message, pytrace=False)
sys.stderr.write("WARNING: {}".format(message))
self._warn_or_fail_if_strict("Unknown config ini key: {}\n".format(key))

def _validate_plugins(self) -> None:
required_plugins = sorted(self.getini("required_plugins"))
if not required_plugins:
return
Comment on lines +1094 to +1095
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd omit these two lines as they're not really necessary.

Suggested change
if not required_plugins:
return

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was intentional. The point is to not fetch plugin info if it’s not necessary. The function is useless if there are no required plugins, so we exit

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this optimization is worth it, the code below doesn't do anything expensive. I think it is better to avoid special cases when they are not necessary. But it's OK if you want to keep it.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The call to get dist info is expensive ( as per @nicoddemus’ earlier comment )

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I was under that impression, but I was wrong, sorry for not doing the proper research before.

Regardless I think the change is harmless and particularly I would prefer to bail out early. 😁

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 on bailing early. It’s unnecessary otherwise. We’re not writing high frequency trading software but at the same time we should do basic optimizations

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The main thing is that the condition makes it look like a Falsey required_plugins has a special meaning. If it is removed, the reader doesn't need to figure out that it doesn't.

But just in general, if I see e.g. sum implemented like this:

def sum(xs):
    if not xs:
        return 0
    s = 0
    for x in xs:
        s += x
    return s

that seems redundant...

Copy link
Member Author

@gnikonorov gnikonorov Jun 12, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mean, technically speaking a falsey required_plugins means to the function and the reader of the function "nothing important to do here now, just carry on"

In this case we also don't have to assemble plugin_dist_names plugin_dist_names = [dist.project_name for _, dist in plugin_info] which we still would if we don't bail early

Also, doing the None check informs future readers that this item is not guaranteed to be populated which isn’t obvious otherwise


plugin_info = self.pluginmanager.list_plugin_distinfo()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any particular reason to only include setuptools plugins here?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This isn’t just setup tools it is all plugins, but with the proper distribution info. Correct me if I’m wrong @nicoddemus

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Resolving as above. Please reopen if I’m wrong

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

list_plugin_distinfo only lists setuptools plugins, at least this is what I understand from the pluggy code. That might be what we want here, not sure myself, but unresolving so can figure it out.

Copy link
Member

@nicoddemus nicoddemus Jun 12, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point @bluetech.

Should we consider somehow plugins specified via -p and PYTEST_PLUGINS? How would that look like?

Copy link
Member Author

@gnikonorov gnikonorov Jun 12, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we want to add support for -p and PYTEST_PLUGINS in a follow up PR or does that feel like we’re submitting a half complete feature @nicoddemus @bluetech ( asking since I'm not familiar with the usage of -p and PYTEST_PLUGINS

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven't checked what each means, but just from perusing the PlugingManager code, list_name_plugin() seems good?

I guess it depends on why the code uses dist.project_name instead of the name the plugin was registered with? (Sorry if was discussed before)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was discussed here: #7330 (comment)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From reading the help, I think we should defer -p if we add it at all. It sounds like a user is intentionally fiddling around with the run

-p name               early-load given plugin module name or entry point
                            (multi-allowed).

The same goes for the environment variable PYTEST_PLUGINS. I'm not opposed to adding them in, but to me it really sounds like they're a different use case than this and would warrant their own PR

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree, not sure this makes sense when used with -p or PYTEST_PLUGINS.

plugin_dist_names = [dist.project_name for _, dist in plugin_info]

missing_plugins = []
for plugin in required_plugins:
if plugin not in plugin_dist_names:
missing_plugins.append(plugin)

if missing_plugins:
fail(
"Missing required plugins: {}".format(", ".join(missing_plugins)),
pytrace=False,
)

def _warn_or_fail_if_strict(self, message: str) -> None:
if self.known_args_namespace.strict_config:
fail(message, pytrace=False)
sys.stderr.write("WARNING: {}".format(message))

def _get_unknown_ini_keys(self) -> List[str]:
parser_inicfg = self._parser._inidict
Expand Down
2 changes: 1 addition & 1 deletion src/_pytest/main.py
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@ def pytest_addoption(parser: Parser) -> None:
group._addoption(
"--strict-config",
action="store_true",
help="invalid ini keys for the `pytest` section of the configuration file raise errors.",
help="any warnings encountered while parsing the `pytest` section of the configuration file raise errors.",
)
group._addoption(
"--strict-markers",
Expand Down
59 changes: 59 additions & 0 deletions testing/test_config.py
Original file line number Diff line number Diff line change
Expand Up @@ -212,6 +212,60 @@ def test_invalid_ini_keys(
with pytest.raises(pytest.fail.Exception, match=exception_text):
testdir.runpytest("--strict-config")

@pytest.mark.parametrize(
"ini_file_text, exception_text",
[
(
"""
[pytest]
required_plugins = fakePlugin1 fakePlugin2
""",
"Missing required plugins: fakePlugin1, fakePlugin2",
),
(
"""
[pytest]
required_plugins = a pytest-xdist z
""",
"Missing required plugins: a, z",
),
(
"""
[pytest]
required_plugins = a q j b c z
""",
"Missing required plugins: a, b, c, j, q, z",
),
(
"""
[some_other_header]
required_plugins = wont be triggered
[pytest]
minversion = 5.0.0
""",
"",
),
(
"""
[pytest]
minversion = 5.0.0
""",
"",
),
],
)
def test_missing_required_plugins(self, testdir, ini_file_text, exception_text):
pytest.importorskip("xdist")

testdir.tmpdir.join("pytest.ini").write(textwrap.dedent(ini_file_text))
testdir.monkeypatch.delenv("PYTEST_DISABLE_PLUGIN_AUTOLOAD")

if exception_text:
with pytest.raises(pytest.fail.Exception, match=exception_text):
testdir.parseconfig()
else:
testdir.parseconfig()


class TestConfigCmdlineParsing:
def test_parsing_again_fails(self, testdir):
Expand Down Expand Up @@ -610,6 +664,7 @@ class PseudoPlugin:

class Dist:
files = ()
metadata = {"name": "foo"}
entry_points = (EntryPoint(),)

def my_dists():
Expand Down Expand Up @@ -640,6 +695,7 @@ def load(self):
class Distribution:
version = "1.0"
files = ("foo.txt",)
metadata = {"name": "foo"}
entry_points = (DummyEntryPoint(),)

def distributions():
Expand All @@ -664,6 +720,7 @@ def load(self):
class Distribution:
version = "1.0"
files = None
metadata = {"name": "foo"}
entry_points = (DummyEntryPoint(),)

def distributions():
Expand All @@ -689,6 +746,7 @@ def load(self):
class Distribution:
version = "1.0"
files = ("foo.txt",)
metadata = {"name": "foo"}
entry_points = (DummyEntryPoint(),)

def distributions():
Expand Down Expand Up @@ -720,6 +778,7 @@ def load(self):
return sys.modules[self.name]

class Distribution:
metadata = {"name": "foo"}
entry_points = (DummyEntryPoint(),)
files = ()

Expand Down