Add coverage reporting

[obestwalter]

I am aware that there are different opinions about coverage, but to me it is a helpful data point, so I would like to add coverage reporting to our CI builds.

To me https://codecov.io looks good

• free for open source
• plays nicely with pytest/tox (https://github.com/codecov/example-python#testing-with-tox)
• browser extension to visualize coverage directly on Github
• PR coverage

[The-Compiler]

FWIW I use codecov for qutebrowser too and I'm quite happy with it.

[obestwalter]

Problem: A lot of the tests are run in a subprocess. There is a special wrapper class implementing that as part of the pytest plugin. Coverage needs to be made aware of that as described here.

I am still in the process of learning about it and trying to find out how others are dealing with this. At the moment I am tempted to circumvent the proposed and just bake it right into the wrapper by checking if this is a coverage run and then replacing the python invocation with a coverage run invocation. Don't know if good idea or stupid yet though.

[The-Compiler]

Why not use pytest-cov which takes care of subprocess coverage automagically?

[obestwalter]

Does it? Oh great. I wasn't aware of that. Thanks!

[obestwalter]

This plugin produces coverage reports. It supports centralised testing and distributed testing in both load and each modes. It also supports coverage of subprocesses.

Wow - first paragraph in the README :) I started reading through the coverage docs instead and from that angle it looks much more involved :)

[gaborbernat]

yeah I use pytest-cov for all my projects, works great 👍

[obestwalter]

Branch coverage would be nice but has a problem atm: pytest-dev/pytest-cov#168

keeping this open until resolved.

[obestwalter]

Found a workaround for that (pass config file on command line), so we can close this.