pytest-cov is not tested with master branch of pytest

[ssbarnea]

Summary

pytest-cov needs a a job (tox env) that tests it using pytest from master branch, so we find bugs before a new release is made.

Details

At this moment there is an environment that tests compatibility with pytest-xdist from master, which is very good, but this needs to be extended.

As CI/CD resources are limited it could make sense to create a single devel job that is testing impact of using master branches of all dependencies we want to test (pytest and pytest-xdist for now).

As the project is not very active, we just need to assure that we enable daily builds of master, so we know when something breaks CI. Travis make this very easy, two clicks.

Please note that I raised this bug because I discovered that master branch of pytest-cov is incompatible with pytest master, see failure at pytest-dev/pytest#7721 -- as it crashes with

  File "/home/runner/work/pytest/pytest/.tox/py37-plugins/lib/python3.7/site-packages/_pytest/config/__init__.py", line 998, in pytest_cmdline_parse
    self.parse(args)
  File "/home/runner/work/pytest/pytest/.tox/py37-plugins/lib/python3.7/site-packages/_pytest/config/__init__.py", line 1272, in parse
    self._preparse(args, addopts=addopts)
  File "/home/runner/work/pytest/pytest/.tox/py37-plugins/lib/python3.7/site-packages/_pytest/config/__init__.py", line 1178, in _preparse
    early_config=self, args=args, parser=self._parser
  File "/home/runner/work/pytest/pytest/.tox/py37-plugins/lib/python3.7/site-packages/pluggy/hooks.py", line 286, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "/home/runner/work/pytest/pytest/.tox/py37-plugins/lib/python3.7/site-packages/pluggy/manager.py", line 93, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/home/runner/work/pytest/pytest/.tox/py37-plugins/lib/python3.7/site-packages/pluggy/manager.py", line 87, in <lambda>
    firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
  File "/home/runner/work/pytest/pytest/.tox/py37-plugins/lib/python3.7/site-packages/pluggy/callers.py", line 208, in _multicall
    return outcome.get_result()
  File "/home/runner/work/pytest/pytest/.tox/py37-plugins/lib/python3.7/site-packages/pluggy/callers.py", line 80, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/home/runner/work/pytest/pytest/.tox/py37-plugins/lib/python3.7/site-packages/pluggy/callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "/home/runner/work/pytest/pytest/.tox/py37-plugins/lib/python3.7/site-packages/pytest_cov/plugin.py", line 125, in pytest_load_initial_conftests
    if early_config.known_args_namespace.cov_source:
AttributeError: 'Namespace' object has no attribute 'cov_source'
ERROR: InvocationError for command /home/runner/work/pytest/pytest/.tox/py37-plugins/bin/coverage run -m pytest (exited with code 1)

This happens with any project, as soon you install pytest from master, if you have pytest-cov also installed, you cannot use it anymore.

[ionelmc]

We could have a cronjob for such an environment but what would happen when a failure occurs? I'm not involved in pytest's core development not am I an expert on the pytest internals. Is that failure a pytest-cov or a pytest bug? I don't know.

What I would like to do is add envs for beta/rc releases of pytest - if there's such a release let me know and I'll add it in the test matrix, no problem.

[nicoddemus]

Is that failure a pytest-cov or a pytest bug? I don't know.

It is a pytest bug, which sadly no test caught on time before merging.

In pytest-dev/pytest#7721 we are introducing some simple integration testing to avoid this kind of breakage to happen in the future: while it is not perfect, it should prevent most basic regressions from happening in the first place.