Using pytest-cov for coverage results in top-level package name of "." when using coverage.py doesn't

[beamerblvd]

First, versions:

$ python --version
Python 2.7.11
$ pip freeze|grep coverage
coverage==4.4.1
$ pip freeze|grep pytest-cov
pytest-cov==2.5.1

Happens on Ubuntu 12.04, Ubuntu 16.04, and Mac OS X El Capitan 10.11.6. Can't test anything else, but seems safe to say it would be the same.

I have a project laid out like this:

proj/ex_secure/__init__.py
proj/ex_secure/base.py
proj/ex_secure/metrics.py
proj/ex_secure/keys.py
proj/ex_secure/utils.py
proj/tests/test_base.py
proj/tests/test_metrics.py
proj/tests/test_keys.py
proj/.gitignore
proj/.pep8
proj/README.rst
proj/setup.cfg
proj/setup.py

All the example commands below I run from within the proj directory.

First, I run coverage directly:

$ coverage run --branch --source=ex_secure -m pytest -s --junitxml=pytests.xml
... output that doesn't matter ...
$ coverage xml
$ coverage report
Name                              Stmts   Miss Branch BrPart  Cover
-------------------------------------------------------------------
ex_secure/__init__.py                 1      0      0      0   100%
ex_secure/base.py                   274     55     74     12    74%
ex_secure/keys.py                    31      6      6      0    78%
ex_secure/metrics.py                 10      2      4      0    57%
ex_secure/util.py                   238    238     98      0     0%

And the top package of the XML file looks like so:

<package branch-rate="0.3058" complexity="0" line-rate="0.4769" name="ex_secure">

Next, I run pytest with the pytest-cov plugin:

$ pytest -s --junitxml=pytests.xml --cov-report xml --cov-report term --cov-branch --cov=ex_secure
... output that doesn't matter ...
---------- coverage: platform darwin, python 2.7.11-final-0 ----------
Name                              Stmts   Miss Branch BrPart  Cover
-------------------------------------------------------------------
ex_secure/__init__.py                 1      0      0      0   100%
ex_secure/base.py                   274     55     74     12    74%
ex_secure/keys.py                    31      6      6      0    78%
ex_secure/metrics.py                 10      2      4      0    57%
ex_secure/util.py                   238    238     98      0     0%

And the top package of the XML file looks like so:

<package branch-rate="0.3058" complexity="0" line-rate="0.4769" name=".">

Notice the name of the package is now . instead of ex_secure. However, if I then run coverage xml to re-generate the report:

$ coverage xml

The XML file looks right again:

<package branch-rate="0.3058" complexity="0" line-rate="0.4769" name="ex_secure">

So I tried running pytest a different way:

$ pytest -s --junitxml=pytests.xml --cov-report xml --cov-report term --cov-branch --cov=ex_secure.base --cov=ex_secure.keys --cov=ex_secure.metrics --cov=ex_secure.util
... output that doesn't matter ...
---------- coverage: platform darwin, python 2.7.11-final-0 ----------
Name                              Stmts   Miss Branch BrPart  Cover
-------------------------------------------------------------------
ex_secure/base.py                   274     55     74     12    74%
ex_secure/keys.py                    31      6      6      0    78%
ex_secure/metrics.py                 10      2      4      0    57%

Now the XML also looks right:

<package branch-rate="0.4722" complexity="0" line-rate="0.6766" name="ex_secure">

But this has several downsides:

• I have to enumerate everything, and I miss coverage info for new files added if they are not enumerated.
• Can't cover __init__.py
• utils.py is not in the report anymore because none of its lines are covered, resulting in incorrectly high coverage results
• Branch rate and line rate are now different (because of missing __init__.py and utils.py

[ionelmc]

Did you install the project (setup.py install) and do you happen to have any .coveragerc around?

Or if you have this stuff in a repo I can look there directly ...

[beamerblvd]

Unfortunately, this code is NDA/proprietary and I am not allowed to share it with you.

Before I double-checked that it was setup.py installed, I tried this:

setup.cfg

[aliases]
test=pytest

[tool:pytest]
addopts = -s --junitxml=pytests.xml --cov-report xml --cov-report term --cov-branch --cov=ex_secure

$ python setup.py test
... output that doesn't matter ...
---------- coverage: platform darwin, python 2.7.11-final-0 ----------
Name                              Stmts   Miss Branch BrPart  Cover
-------------------------------------------------------------------
ex_secure/__init__.py                 1      0      0      0   100%
ex_secure/base.py                   274     55     74     12    74%
ex_secure/keys.py                    31      6      6      0    78%
ex_secure/metrics.py                 10      2      4      0    57%
ex_secure/util.py                   238    238     98      0     0%

<package branch-rate="0.3058" complexity="0" line-rate="0.4769" name=".">

And the behavior is still the same as originally reported above with multiple --covs in setup.cfg and coverage run without the coverage options in setup.cfg (coverage run --branch --source=ex_secure setup.py test).

I ran setup.py install and then tested all the ways again, and, as I expected, the results were still no different than originally reported above. 😕

I generated two .coverage files—one from pytest with just --cov=ex_secure (so coverage.xml says package name is .) and one from coverage (so coverage.xml says package name is ex_secure)—and they are identical:

$ shasum -a 256 coverage.coverage 
2be5502e366603510de1ce40ab5e3f6284e6b988db73ec0b39682406af72c450  coverage.coverage
$ shasum -a 256 pytest.coverage 
2be5502e366603510de1ce40ab5e3f6284e6b988db73ec0b39682406af72c450  pytest.coverage

It would seem the flaw is in the step that converts .coverage to an XML report (notice the terminal report is always fine), which is supported by the fact that running coverage xml after pytest ... --cov... corrects the coverage.xml file from . to ex_secure. I can share the .coverage files if truly needed, but I will have to redact/replace actual paths/file names in them, since I cannot divulge the actual details of this project.

[beamerblvd]

Additional note, since I just realized that I misunderstood your question: I do not have any .coveragerc files.

[ionelmc]

Before I double-checked that it was setup.py installed, I tried ...

Well that's the thing - since you have ad-hoc project layout you shouldn't install it at all. Eg, your code comes from site-packages and not your cwd, but your coverage configuration assumes cwd. Since you already installed your code you probably have the right tooling for a src-layout.

If the xml has '.' as the root package then perhaps it covers multiple packages (ex_secure and something else? what exactly?). Perhaps you have multiple --cov=something (like addopts in pytest.ini/setup.cfg/tox.ini)?

[beamerblvd]

No, none of those are the cases. Under the . root package are these "classes" (the way Coverage handles Python files):

<package branch-rate="0.3058" complexity="0" line-rate="0.4769" name=".">
    <classes>
        <class branch-rate="1" complexity="0" filename="ex_secure/__init__.py" line-rate="1" name="__init__.py">...</class>
        <class branch-rate="0.5135" complexity="0" filename="ex_secure/base.py" line-rate="0.7993" name="base.py">...</class>
        <class branch-rate="0.6667" complexity="0" filename="ex_secure/keys.py" line-rate="0.8065" name="keys.py">...</class>
        <class branch-rate="0" complexity="0" filename="ex_secure/metrics.py" line-rate="0.8" name="metrics.py">...</class>
        <class branch-rate="0" complexity="0" filename="ex_secure/util.py" line-rate="0" name="util.py">...</class>
    </classes>
</package>

And that's it. No other classes/files. No other projects.

And, again, to reiterate, the .coverage files are identical, and, if, after generating the name="." XML with pytest, I then run coverage xml JUST to regenerate the report from the existing .coverage, the coverage.xml is updated to name="ex_secure", just as if I had generated it with coverage run... followed by coverage xml. There is something different in how pytest-cov generates the XML report vs how coverage xml generates the report.

[beamerblvd]

I did some debugging and I might have an answer:

This is the XML report generation using pytest-cov:

# In coverage/control.py xml_report()
ipdb> self.source_match
<TreeMatcher [u'/Users/nwilliams/proj/ex_secure']>
# In coverage/xmlreport.py report()
ipdb> morfs, outfile
(None, <open file 'coverage.xml', mode 'w' at 0x10cdd3e40>)
ipdb> self.packages
{}
# In coverage/xmlreport.py xml_file()
ipdb> filename
u'/Users/nwilliams/proj/ex_secure/__init__.py'
ipdb> self.source_paths
set([u'/Users/nwilliams/proj/ex_secure'])
ipdb> dirname
u'.'
ipdb> package_name
u'.'
# Back in coverage/xmlreport.py report()
ipdb> self.packages
{'.': [{}, 0, 0, 0, 0]}

This is the XML report generation using coverage xml immediately after running pytest with pytest-cov:

# In coverage/control.py xml_report()
ipdb> self.source_match
None
# In coverage/xmlreport.py report()
ipdb> morfs, outfile
([], <open file 'coverage.xml', mode 'w' at 0x10537e5d0>)
ipdb> self.packages
{}
# In coverage/xmlreport.py xml_file()
ipdb> filename
u'/Users/nwilliams/proj/ex_secure/__init__.py'
ipdb> self.source_paths
set([])
ipdb> dirname
u'ex_secure'
ipdb> package_name
u'ex_secure'
# Back in coverage/xmlreport.py report()
ipdb> self.packages
{u'ex_secure': [{}, 0, 0, 0, 0]}

Upon further examination of the code, coverage xml does not allow the --sources argument and has no way to pass sources into xml_report. That's why it works. On the other hand, pytest-cov is creating a single Coverage object and setting its sources, which get passed on to the xml_report, and so it strips that from all the file names before figuring out their package names.

So one could argue Coverage should be a little smarter here, but also pytest-cov is not calling xml_report the same way that coverage xml would, and that seems like a legit bug in pytest-cov to me.

[beamerblvd]

bump Any thoughts on resolving this? I can submit a PR, but my approach would simply be to rip out sources in the code that calls xml_report, and that might not be the approach you want to take.

[ionelmc]

Uh ... I wonder if https://github.com/pytest-dev/pytest-cov/archive/pr142.zip would solve your issue. Could you give it a try? Assuming the problem is the broken combining that pr142 fixes.

[beamerblvd]

Just tried it. It runs without error, but I have the same problem:

<package branch-rate="0" complexity="0" line-rate="1" name=".">

[beamerblvd]

Note that, in addition to the above, #142 actually breaks my workaround. Using multiple --covs no longer results in a top-level element like this:

<package branch-rate="0" complexity="0" line-rate="0.7273" name="ex_secure">

Now it results in multiple top-level elements like this:

<package branch-rate="0" complexity="0" line-rate="0.7273" name="ex_secure.base">
...
<package branch-rate="0" complexity="0" line-rate="0.7273" name="ex_secure.keys">
...
<package branch-rate="0" complexity="0" line-rate="0.7273" name="ex_secure.metrics">

Ignore this comment. It always did that. It's just the HTML view in Jenkins that turned that into a top-level project of ex_secure.

[beamerblvd]

I'm going to submit a PR this week to disconnect sources from the XML report generation, unless you have a better solution on the way that we can use very soon. We're about to have to drop pytest-cov altogether and switch to using coverage directly/manually, as this is skewing our coverage results undesirably, but I would rather not do that if we can improve a project many people use.

[tonnydourado]

Just to add, I'm having the same problem. We have a slightly weird package layout, with a main package and an extra module in the same level, and pytest-cov does the same name='.' thing, which breaks our Sonar integration.

[ericescalante]

Having the same issue as well, in the meantime just going with:

sed -i -e 's/name="."/name="package_name"/' package/coverage.xml

as part of the build script :)

[Jnesselr]

I'm running into this same issue and having difficulties with the workarounds above. I'm trying to understand why "writing out to an xml coverage file" is handled by two different classes in two different ways above. Maybe if I understand that, I can help out a bit/submit a pull request, etc.

[ionelmc]

Can't hurt to try @beamerblvd 's suggestion (not passing sources to the xml generation parts) I guess ...