coverage is wrong when running with xdist's --boxed

[diefans]

I need to run my tests with xdist's --boxed feature.
there is a big difference in coverage compared to running unboxed:

• unboxed: 93.48 %
• boxed: 26.93 %

35 % of all modules are completely uncovered.
I also noticed, that despite that I run the tests with py.test tests/ --boxed -d -n 8 --random the uncovered modules are always the same.

When I run one single test module in boxed mode coverage complains that the module to cover was not imported:

py.test tests/unit/security/test_features.py --boxed -d -n 8 --cov bm.security
===================================================================================================== test session starts =====================================================================================================
platform linux2 -- Python 2.7.5 -- pytest-2.5.0
Tests are shuffled using seed number 358684703390.
plugins: random, bdd, cov, capturelog, ipdb, cache, pep8, greendots, xdist
gw0 [49] / gw1 [49] / gw2 [49] / gw3 [49] / gw4 [49] / gw5 [49] / gw6 [49] / gw7 [49]
scheduling tests via LoadScheduling
................................s..Coverage.py warning: Module bm.security was never imported.
Coverage.py warning: No data was collected.
.......Coverage.py warning: Module bm.security was never imported.
Coverage.py warning: No data was collected.
.Coverage.py warning: Module bm.security was never imported.
Coverage.py warning: No data was collected.
.Coverage.py warning: Module bm.security was never imported.
Coverage.py warning: No data was collected.
.Coverage.py warning: Module bm.security was never imported.
Coverage.py warning: No data was collected.
.Coverage.py warning: Module bm.security was never imported.
Coverage.py warning: No data was collected.
.Coverage.py warning: Module bm.security was never imported.
Coverage.py warning: No data was collected.
.Coverage.py warning: Module bm.security was never imported.
Coverage.py warning: No data was collected.
.
--------------------------------------------------------------------------------------- coverage: platform linux2, python 2.7.5-final-0 ---------------------------------------------------------------------------------------
Name    Stmts   Miss     Cover   Missing
----------------------------------------
============================================================================================ 48 passed, 1 skipped in 3.60 seconds =============================================================================================

[schlamar]

Can you please test against latest master, there are various improvements/fixes not yet released.

[diefans]

Sorry, I get the same results...

[mattupstate]

I was experiencing a similar issue where running:

$ py.test --cov mymodule --cov-report=term-missing

was reporting correct coverage. Then when running through setuptools like:

$ python setup.py test

The coverage was incorrect. However, installing the latest from master branch has fixed my issue.

[schlamar]

@diefans Confirmed: https://travis-ci.org/schlamar/pytest-cov/builds/26548730

@mattupstate That seems unrelated, but thanks for confirming that master solves a few issues.

[schlamar]

This is an upstream issue: https://bitbucket.org/hpk42/py/issue/45/forkedfunc-skips-exit-handlers-breaks

[schlamar]

@diefans I have a hacky but working solution by patching py.process.ForkedFunc. Not sure if I want to roll that out officially. But it should work if you add this to your conftest.py: fb100db

[schlamar]

I have prepared a much nicer solution upstream:

• py pull request
• fix in cov-core schlamar/cov-core@52086bc

[sontek]

@schlamar Since this is upstream bug, does this mean you can release without the monkeypatching? Or are you going to wait for upstream anyways?

[schlamar]

@sontek I'm going to wait for the decision on my pull request for py. If it is merged I can release because pytest-cov doesn't need any further changes.

[schlamar]

@sontek 1.7.0 just released

@diefans this will be fixed as soon as a new py version is released, my pull request was merged

[schlamar]

@diefans should be fixed, you just need to upgrade py to 1.4.21 and cov-core to 1.13.0.

[mocksoul]

Broken again with py 1.4.22
https://bitbucket.org/hpk42/py/diff/py/_process/forkedfunc.py?diff2=37f557844498&at=default

[schlamar]

Yes, I know. We are working something out. For now, you can pin py to 1.4.21.

[mocksoul]

pytest 2.6.0 depends on py 1.4.22, so you require to downgrade them both =)

My 2 years old patch is still working:

1. In session start record current PID
2. Before test start compare current pid with session pid and if they are different -- make multiprocessing start
3. Before teardown check is we used multiprocessing and stop it if needed

But I if you want things to make right you probably should ask xdist to provide box_start/box_stop hooks which you will be able to use.

[schlamar]

@mocksoul Do you have some code? =)

[aconrad]

Do we have a fix for this?

[schlamar]

@aconrad #15 should fix it, I might find some time at the weekend to have a look at it.

[schlamar]

fixed (again) in 1633bd3, will be released soon

[aconrad]

Sweet! @schlamar can you ping this ticket when a new release is available? Thanks!

[mocksoul]

As being said, it is fixed in pytest-cov-1.8.0, https://pypi.python.org/pypi/pytest-cov/1.8.0