Travis: report coverage with all builds #2800

blueyed · 2017-09-26T22:49:15Z

https://codecov.io/gh/pytest-dev/pytest/tree/coverage

TODO:

tox: coveralls: also report to codecov #3877 should be merged first, then this one rebased.
bring back AppVeyor

blueyed · 2017-09-27T15:12:53Z

It seems to be problematic with py27/py26 (https://travis-ci.org/pytest-dev/pytest/builds/280216101).
I've triggered a full build now again.

Is this something we want to have in general?

nicoddemus

I'm in favor of improving our coverage testing. 👍

Your approach also seems to be possible to implement on AppVeyor.

Ultimately we should strive to obtain 100% coverage, even if by explicitly marking parts of the code which are not currently tested (like java specific parts). Even if we "cheat", we gain the benefit of demanding that new code from now on has 100% coverage.

nicoddemus · 2017-09-27T15:49:39Z

.travis.yml

@@ -5,11 +5,15 @@ python:
 # command to install dependencies
 install:
  - pip install --upgrade --pre tox
+  - |
+    pip install tox


tox is already installed in the line above, so this is not really needed

nicoddemus · 2017-09-27T15:58:40Z

.travis.yml

@@ -5,11 +5,15 @@ python:
 # command to install dependencies
 install:
  - pip install --upgrade --pre tox
+  - |
+    pip install tox
+    if [[ "${TOXENV#py}" != "$TOXENV" ]]; then


Pardon my lack of bash-fu, what does this expression inside if does?

TOXENV starts with "py".

Thanks, and what does "${TOXENV#py}" do? (I tried googling it but it wasn't entirely obvious what to search for).

It substitutes "py" at the beginning.
Yes, shell programming can be ugly. See "Parameter Expansion" in bash's man page.

nicoddemus · 2017-09-27T15:59:53Z

tox.ini

 commands =
-    pytest -ra test_pdb.py test_terminal.py test_unittest.py
+    {env:TOX_COVERAGE_RUN:} pytest -ra test_pdb.py test_terminal.py test_unittest.py


Looking at the Travis build, py27/py36-trial seem to be missing this configuration

nicoddemus · 2017-09-27T16:01:17Z

.coveragerc

@@ -1,4 +1,5 @@
 [run]
+source = _pytest
 omit = 
    # standlonetemplate is read dynamically and tested by test_genscript


Oh this does not exist anymore and can be removed. 😁

~~standlonetemplate?~~ Fixed (not pushed).

Apart from that I think we should also include testing (the tests themselves). Anything else?

"standlonetemplate" was used by the removed genscript feature.

Apart from that I think we should also include testing (the tests themselves). Anything else?

Not sure, then xfail tests which are never meant to run because they reproduce issues not yet fixed will drop the coverage. In my head "coverage" does not include the tests. What's your opinion?

I think unexpected missing coverage is a good sign of something being wrong.
Those xfail tests should be marked with "pragma: no cover" then.

nicoddemus · 2017-09-27T16:04:55Z

.travis.yml

+  - |
+    pip install tox
+    if [[ "${TOXENV#py}" != "$TOXENV" ]]; then
+      export TOX_COVERAGE_RUN="coverage run -m"


At first glance I thought TOX_EXTRA_DEPS was a feature of tox itself. Perhaps we should add a PYTEST_ prefix to avoid confusion? While at it, perhaps there is a way to add extra dependencies using some tox feature?

Just remembered that we used _PYTEST_SETUP_SKIP_PLUGGY_DEP in setup.py for this type of "setup/buid only flag used internally".

_PYTEST_TOX_COVERAGE_RUN?

Makes sense. Will rename it to _PYTEST_TOX_COVERAGE_RUN and _PYTEST_TOX_EXTRA_DEPS then?

Sounds good, thanks

blueyed · 2017-09-27T17:58:02Z

Failures on py2, pypy and py33 are all:

=========================== short test summary info ============================
FAIL testing/code/test_excinfo.py::TestTraceback_f_g_h::()::test_traceback_cut_excludepath
FAIL testing/python/collect.py::TestModule::()::test_show_traceback_import_error[0]
FAIL testing/python/collect.py::TestModule::()::test_show_traceback_import_error[1]
FAIL testing/python/collect.py::TestTracebackCutting::()::test_traceback_argsetup

e.g. https://travis-ci.org/pytest-dev/pytest/jobs/280479131#L599

Any idea? (likely related to the assertions there not being true when run on coverage likely?!)

blueyed · 2017-09-27T18:05:15Z

@nicoddemus
Can you enable the codecov bot user on https://codecov.io/gh/pytest-dev/pytest?

nicoddemus · 2017-09-27T18:22:03Z

Can you enable the codecov bot user on https://codecov.io/gh/pytest-dev/pytest?

Done! If we will use codecov we should disable coveralls (after this is merged)

blueyed · 2017-09-27T19:51:00Z

The problem with test_traceback_cut_excludepath is that the traceback entry looks different when run with -m, also when using python -m pytest (and we use coverage run -m).

<TracebackEntry _pytest/python_api.py:580> vs <TracebackEntry /home/user/Vcs/pytest/.tox/py27/lib/python2.7/site-packages/_pytest/python_api.py:580>.
Is that something that should get fixed for excinfo.traceback?
Looks like it uncovered an actual issue?!

blueyed · 2017-09-27T20:05:23Z

This works around it:

diff --git i/testing/code/test_excinfo.py w/testing/code/test_excinfo.py
index f8f8a036..cc512a2d 100644
--- i/testing/code/test_excinfo.py
+++ w/testing/code/test_excinfo.py
@@ -162,9 +162,12 @@ def test_traceback_cut(self):
         assert len(newtraceback) == 1
 
     def test_traceback_cut_excludepath(self, testdir):
+        ptf = py.path.local(pytest.__file__)
+        if not ptf.exists():
+            pytest.skip('pytest.__file__ does not point to an actual file.')
+        basedir = ptf.dirpath()
         p = testdir.makepyfile("def f(): raise ValueError")
         excinfo = pytest.raises(ValueError, "p.pyimport().f()")
-        basedir = py.path.local(pytest.__file__).dirpath()
         newtraceback = excinfo.traceback.cut(excludepath=basedir)
         for x in newtraceback:
             if hasattr(x, 'path'):

Looks like an issue with Python 2 where pytest.__file__ is not correct in this case.

nicoddemus · 2017-09-28T02:36:35Z

I gotta say I like codecov's report better. 😄

nicoddemus

There's some environments failing, could you take a look at those?

nicoddemus · 2017-10-05T02:39:26Z

.travis.yml

+      pip install codecov coveralls
+      coverage xml
+      coverage report -m
+      coveralls


Should we drop coveralls if we are migrating to codecov?

I think it is good to keep it for comparison, at least for a while.

nicoddemus · 2017-10-05T02:42:40Z

tox.ini

 commands =
-    pytest -n3 -ra --runpytest=subprocess {posargs:testing}
+    {env:TOX_COVERAGE_RUN:} pytest -n3 -ra --runpytest=subprocess {posargs:testing}


I'm seeing this warning from tox:

WARNING:test command found but not installed in testenv cmd: /usr/bin/env env: /home/travis/build/pytest-dev/pytest/.tox/py27

Do we have to whitelist env?

Couldn't we just use pytest-cov instead?

pytest-cov does incomplete reports due to late plugin startup, its a longstanding pytest issue

nicoddemus · 2017-10-05T02:43:52Z

tox.ini

-    coverage run --source=_pytest -m pytest testing
-    coverage report -m
-    coveralls
+    {env:TOX_COVERAGE_RUN:} {envpython} create_executable.py


We can drop coverage from this env I believe

RonnyPfannschmidt · 2017-10-05T06:18:15Z

tox.ini

@@ -19,29 +19,32 @@ envlist =
    docs

 [testenv]
-commands = pytest --lsof -ra {posargs:testing}
+commands = {env:TOX_COVERAGE_RUN:} pytest --lsof -ra {posargs:testing}


can we just introduce a coverage factor for this?

@RonnyPfannschmidt
There is no way to have a command by default and then another for a factor, is there? (where the command comes not from the default testenv).

Will do it via setenv:

[testenv] commands = {env:_TOX_COVERAGE_RUN:} pytest --lsof -ra {posargs:testing} coverage: coverage report -m --skip-covered setenv = coverage: _TOX_COVERAGE_RUN=env COVERAGE_FILE={toxinidir}/.coverage coverage run -a -m whitelist_externals = coverage: env

RonnyPfannschmidt · 2017-10-05T06:19:16Z

.travis.yml

@@ -5,11 +5,14 @@ python:
 # command to install dependencies
 install:
  - pip install --upgrade --pre tox
+  - |
+    if [[ "${TOXENV#py}" != "$TOXENV" ]]; then
+      export TOX_COVERAGE_RUN="env COVERAGE_FILE=$PWD/.coverage coverage run -a -m"


at frist glance this "breaks" at least for the xdist envs

RonnyPfannschmidt · 2017-10-05T06:20:10Z

tox.ini

 passenv = USER USERNAME
 deps =
    hypothesis>=3.5.2
    nose
    mock
    requests
+    {env:TOX_EXTRA_DEPS:}


blueyed · 2017-10-12T00:32:08Z

@nicoddemus
See #2800 (comment).

RonnyPfannschmidt · 2017-10-12T07:01:23Z

instead of falling out for pytest in some cases

lets use _pytest instead - its not being used as module

nicoddemus · 2017-10-16T23:03:10Z

lets use _pytest instead - its not being used as module

You mean in the production code or the test?

RonnyPfannschmidt · 2017-10-17T05:27:21Z

in testing/code/test_excinfo.py

nicoddemus · 2018-08-28T15:13:16Z

.travis.yml

+      coverage report -m
+      codecov --required -X gcov pycov search -f coverage.xml --flags ${TOXENV//-/ }
+
+      # Coveralls does not support merged reports.


I believe we can drop coveralls then. Or would you rather do it in another PR @blueyed?

I've thought to keep it for comparison.

Oh OK. But I think we should remove it after a while to avoid confusion. 👍

nicoddemus · 2018-08-28T15:15:40Z

appveyor.yml

@@ -1,53 +0,0 @@
-environment:


I assume you removed AppVeyor so the builds are faster, please remember to bring it back before merging.

nicoddemus · 2018-08-28T15:18:26Z

tox.ini

+    {env:_PYTEST_TOX_COVERAGE_RUN:} pytest --lsof -ra {env:_PYTEST_TEST_OPTS:} {posargs:testing}
+    coverage: coverage report -m --skip-covered
+setenv =
+    coverage: _PYTEST_TOX_COVERAGE_RUN=coverage run -a -m


Why do we handle _PYTEST_TOX_COVERAGE_RUN and _PYTEST_TOX_EXTRA_DEP both here and in the travis file?

This is meant to allow for tox -e py36-coverage.
I've thought about/tried using only -coverage, but it is not trivial/feasible due to the many different testenvs.

I see, OK, thanks

blueyed · 2018-08-28T20:28:07Z

Squashed (old HEAD: 78d2c4d).
I think this is good from my side for now.

blueyed · 2018-08-28T20:30:59Z

Meh.. AppVeyor will fail - still using TOXENV=coveralls..

I guess the same mechanism should be used with AppVeyor then? Basically the before_script and after_success from Travis would need to be ported then.
@nicoddemus can you do it? It should also add an extra flag with codecov then, "win" or "windows".

nicoddemus · 2018-08-28T20:36:40Z

@blueyed sure I can give it a try. For now I suggest to comment out the TOXENV=coveralls in AppVeyor so we can get this merged.

codecov · 2018-08-28T21:15:11Z

Codecov Report

Merging #2800 into master will increase coverage by 4.05%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #2800      +/-   ##
==========================================
+ Coverage   92.18%   96.24%   +4.05%     
==========================================
  Files          52      108      +56     
  Lines       10009    23353   +13344     
==========================================
+ Hits         9227    22475   +13248     
- Misses        782      878      +96

Flag	Coverage Δ
#doctesting	`31.93% <ø> (?)`
#nobyte	`93.5% <ø> (?)`
#numpy	`31.48% <ø> (?)`
#pexpect	`53.58% <ø> (?)`
#pluggymaster	`95.09% <ø> (?)`
#py27	`94.61% <ø> (?)`
#py34	`93.9% <ø> (?)`
#py35	`93.9% <ø> (?)`
#py36	`94.82% <ø> (?)`
#py37	`94.07% <ø> (?)`
#trial	`34.15% <ø> (?)`
#xdist	`94.64% <ø> (?)`

Impacted Files	Coverage Δ
testing/test_warnings.py	`98.43% <0%> (ø)`
testing/test_pastebin.py	`100% <0%> (ø)`
testing/python/fixture.py	`99.23% <0%> (ø)`
testing/test_terminal.py	`99.38% <0%> (ø)`
testing/python/setup_plan.py	`100% <0%> (ø)`
testing/test_pdb.py	`53.48% <0%> (ø)`
testing/test_helpconfig.py	`100% <0%> (ø)`
testing/logging/test_fixture.py	`100% <0%> (ø)`
testing/test_modimport.py	`90% <0%> (ø)`
testing/test_assertion.py	`98.21% <0%> (ø)`
... and 69 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d76fb83...417516c. Read the comment docs.

blueyed · 2018-08-28T21:15:51Z

Rebased it on master (where the other ci improvements are already), and re-added coveralls for AppVeyor. Since it also runs codecov there it should be good for Windows for now.

nicoddemus · 2018-08-28T21:17:29Z

Since it also runs codecov there

You mean coveralls right?

blueyed · 2018-08-28T21:18:14Z

Both, since 4d19b94.

blueyed · 2018-08-28T23:50:12Z

Fixed linting, squashed the keeping of coveralls for AppVeyor:

diff --git a/.travis.yml b/.travis.yml
index 9279334e..2c9deb3e 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -60,7 +60,7 @@ jobs:
       env: TOXENV=py27
     - env: TOXENV=py34
     - env: TOXENV=py36
-    - env: TOXENV=linting
+    - env: TOXENV=linting PYTEST_NO_COVERAGE=1

     - stage: deploy
       python: '3.6'

- Skips pypy for coverage, reports only py37 to coveralls - tox: allow for TOXENV=py37-coverage - tracks coverage in subprocesses, using coverage-enable-subprocess, and parallel=1 - removes usedevelop with doctesting to match `--source` being used with coverage - keep coveralls for now, used with AppVeyor

doctesting: remove changedir With coverage 5 we could use COVERAGE_RCFILE to make it find the .coveragerc, or we could add `--rcfile` to _PYTEST_TOX_COVERAGE_RUN, but I've thought that this should not be the job that has to test if `--pyargs` actually works.

blueyed · 2018-08-29T21:40:11Z

doctesting is working now (https://travis-ci.org/pytest-dev/pytest/jobs/422263747#L677), but I've changed the tox testenv quite a bit.

nicoddemus · 2018-08-30T22:41:29Z

Thanks a ton @blueyed!

blueyed · 2018-08-31T00:26:36Z

Yay! :)

blueyed changed the title ~~Travis: report coverage with all builds, include codecov~~ [WIP/RFC] Travis: report coverage with all builds, include codecov Sep 27, 2017

blueyed requested review from nicoddemus and RonnyPfannschmidt September 27, 2017 15:13

nicoddemus reviewed Sep 27, 2017

View reviewed changes

pytest-dev deleted a comment from coveralls Sep 27, 2017

blueyed mentioned this pull request Sep 29, 2017

Travis: add coverage reporting via pytest-cov on codecov.io pytest-dev/pytest-django#520

Merged

blueyed force-pushed the coverage branch from 2e2b96d to 1ed81ed Compare October 5, 2017 00:52

blueyed changed the title ~~[WIP/RFC] Travis: report coverage with all builds, include codecov~~ [RFC] Travis: report coverage with all builds, include codecov Oct 5, 2017

blueyed requested a review from a team October 5, 2017 00:53

nicoddemus previously requested changes Oct 5, 2017

View reviewed changes

RonnyPfannschmidt reviewed Oct 5, 2017

View reviewed changes

tox.ini Outdated

passenv = USER USERNAME

deps =

hypothesis>=3.5.2

nose

mock

requests

{env:TOX_EXTRA_DEPS:}

Copy link

Member

RonnyPfannschmidt Oct 5, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

dito

nicoddemus reviewed Aug 28, 2018

View reviewed changes

blueyed force-pushed the coverage branch from 78d2c4d to 84e4909 Compare August 28, 2018 20:27

blueyed changed the title ~~[RFC] Travis: report coverage with all builds, include codecov~~ Travis: report coverage with all builds Aug 28, 2018

blueyed changed the base branch from features to master August 28, 2018 21:11

pytest-dev deleted a comment from codecov bot Aug 28, 2018

blueyed force-pushed the coverage branch from 84e4909 to e1dbbba Compare August 28, 2018 21:14

nicoddemus approved these changes Aug 28, 2018

View reviewed changes

blueyed force-pushed the coverage branch from e1dbbba to 6a74957 Compare August 28, 2018 23:49

blueyed force-pushed the coverage branch from 6a74957 to 950630c Compare August 29, 2018 14:49

This comment has been minimized.

Sign in to view

blueyed force-pushed the coverage branch from 1f12e92 to df7758f Compare August 29, 2018 19:18

blueyed force-pushed the coverage branch 2 times, most recently from 9d883e1 to 794fd05 Compare August 29, 2018 20:40

blueyed force-pushed the coverage branch from 794fd05 to 417516c Compare August 29, 2018 20:53

nicoddemus merged commit f0e852b into pytest-dev:master Aug 30, 2018

blueyed deleted the coverage branch August 31, 2018 00:25

Uh oh!

Travis: report coverage with all builds #2800

Travis: report coverage with all builds #2800

Uh oh!

Conversation

blueyed commented Sep 26, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

blueyed commented Sep 27, 2017

Uh oh!

nicoddemus left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

blueyed Sep 27, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

blueyed commented Sep 27, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

blueyed commented Sep 27, 2017

Uh oh!

nicoddemus commented Sep 27, 2017

Uh oh!

blueyed commented Sep 27, 2017

Uh oh!

blueyed commented Sep 27, 2017

Uh oh!

nicoddemus commented Sep 28, 2017

Uh oh!

nicoddemus left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

blueyed commented Sep 26, 2017 •

edited

Loading

blueyed Sep 27, 2017 •

edited

Loading

blueyed commented Sep 27, 2017 •

edited

Loading

codecov bot commented Aug 28, 2018 •

edited

Loading