coverage regression with Python 3.7 ?

[0LL13]

I posted a question on Stackoverflow because the time for my tests running with coverage and Python 3.7 were about 8 times longer than with Python 3.5 and 3.6. Hoefling suggested I run the tests without coverage and times for testing were about the same for the different versions. Hoefling suggested that it might be a coverage regression and that I post an issue.

[nedbat]

@0LL13 Can you verify for me the versions of coverage you have running in each Python version? "coverage --version" should do it. In particular, that will tell me whether you are using the C tracer (fast) or the Python tracer (slow).

[0LL13]

(vEnv)$ coverage --version
Coverage.py, version 4.0.3.

[nedbat]

@0LL13 Can you upgrade to the latest version? 4.0.3 is over 2.5 years old. The latest version is 4.5.1. Once you've done that, if there is still a difference in performance, show the coverage version for each Python environment. Thanks.

[0LL13]

(vEnv)$ pip install -U coverage
(vEnv)$ coverage --version
Coverage.py, version 4.5.1 with C extension

#tox.ini
...
[testenv]
basepython =
    py35 = python3.5
    py36 = python3.6
    py37 = python3.7
deps = 
    pytest
    pytest-cov
commands =
    py.test --cov=pdf2textbox --duration=0 tests

(vEnv)$ tox
GLOB sdist-make: /home/frodo/Python-Projekte/.virtualenvs/pdf2textbox/setup.py
py35 recreate: /home/frodo/Python-Projekte/.virtualenvs/pdf2textbox/.tox/py35
py35 installdeps: check-manifest, pytest, requests, pdfminer2, coverage, pytest-cov
py35 inst: /home/frodo/Python-Projekte/.virtualenvs/pdf2textbox/.tox/dist/pdf2textbox-0.3.1.zip
py35 installed: atomicwrites==1.1.5,attrs==18.1.0,certifi==2018.4.16,chardet==3.0.4,check-manifest==0.37,coverage==4.5.1,idna==2.7,more-itertools==4.2.0,pdf2textbox==0.3.1,pdfminer2==20151206,pluggy==0.6.0,py==1.5.4,pytest==3.6.3,pytest-cov==2.5.1,requests==2.19.1,six==1.11.0,urllib3==1.23
py35 runtests: PYTHONHASHSEED='2082557123'
py35 runtests: commands[0] | check-manifest
lists of files in version control and sdist match
py35 runtests: commands[1] | python --version
Python 3.5.5
py35 runtests: commands[2] | py.test --cov=pdf2textbox --duration=0 tests
================================= test session starts =================================
platform linux -- Python 3.5.5, pytest-3.6.3, py-1.5.4, pluggy-0.6.0
rootdir: /home/frodo/Python-Projekte/.virtualenvs/pdf2textbox, inifile:
plugins: cov-2.5.1
collected 15 items                                                                    

tests/test_cli_slice_option.py ...                                              [ 20%]
tests/test_convert_pdf.py ......                                                [ 60%]
tests/test_get_pdf.py ....                                                      [ 86%]
tests/test_types.py ..                                                          [100%]

----------- coverage: platform linux, python 3.5.5-final-0 -----------
Name                                   Stmts   Miss  Cover
pdf2textbox/pdf2textbox.py     506    201    60%


=============================== slowest test durations ================================
54.32s call     tests/test_cli_slice_option.py::test_pdf_to_text_slice
1.26s call     tests/test_types.py::test_textbox_structure
1.25s call     tests/test_types.py::test_pdf_to_text_all
============================ 15 passed in 59.82 seconds ==============================

py36 recreate: /home/frodo/Python-Projekte/.virtualenvs/pdf2textbox/.tox/py36
py36 installdeps: check-manifest, pytest, requests, pdfminer2, coverage, pytest-cov
py36 inst: /home/frodo/Python-Projekte/.virtualenvs/pdf2textbox/.tox/dist/pdf2textbox-0.3.1.zip
py36 installed: atomicwrites==1.1.5,attrs==18.1.0,certifi==2018.4.16,chardet==3.0.4,check-manifest==0.37,coverage==4.5.1,idna==2.7,more-itertools==4.2.0,pdf2textbox==0.3.1,pdfminer2==20151206,pluggy==0.6.0,py==1.5.4,pytest==3.6.3,pytest-cov==2.5.1,requests==2.19.1,six==1.11.0,urllib3==1.23
py36 runtests: PYTHONHASHSEED='2082557123'
py36 runtests: commands[0] | check-manifest
lists of files in version control and sdist match
py36 runtests: commands[1] | python --version
Python 3.6.3
py36 runtests: commands[2] | py.test --cov=pdf2textbox --duration=0 tests
================================= test session starts =================================
platform linux -- Python 3.6.3, pytest-3.6.3, py-1.5.4, pluggy-0.6.0
rootdir: /home/frodo/Python-Projekte/.virtualenvs/pdf2textbox, inifile:
plugins: cov-2.5.1
collected 15 items                                                                    

tests/test_cli_slice_option.py ...                                              [ 20%]
tests/test_convert_pdf.py ......                                                [ 60%]
tests/test_get_pdf.py ....                                                      [ 86%]
tests/test_types.py ..                                                          [100%]

----------- coverage: platform linux, python 3.6.3-final-0 -----------
Name                                   Stmts   Miss  Cover
pdf2textbox/pdf2textbox.py     506    201    60%


=============================== slowest test durations ================================
53.42s call     tests/test_cli_slice_option.py::test_pdf_to_text_slice
1.23s call     tests/test_types.py::test_pdf_to_text_all
1.22s call     tests/test_types.py::test_textbox_structure
0.46s call     tests/test_convert_pdf.py::test_convert_three_cols_and_header
============================ 15 passed in 58.94 seconds ==============================

py37 recreate: /home/frodo/Python-Projekte/.virtualenvs/pdf2textbox/.tox/py37
py37 installdeps: check-manifest, pytest, requests, pdfminer2, coverage, pytest-cov
py37 inst: /home/frodo/Python-Projekte/.virtualenvs/pdf2textbox/.tox/dist/pdf2textbox-0.3.1.zip
py37 installed: atomicwrites==1.1.5,attrs==18.1.0,certifi==2018.4.16,chardet==3.0.4,check-manifest==0.37,coverage==4.5.1,idna==2.7,more-itertools==4.2.0,pdf2textbox==0.3.1,pdfminer2==20151206,pluggy==0.6.0,py==1.5.4,pytest==3.6.3,pytest-cov==2.5.1,requests==2.19.1,six==1.11.0,urllib3==1.23
py37 runtests: PYTHONHASHSEED='2082557123'
py37 runtests: commands[0] | check-manifest
lists of files in version control and sdist match
py37 runtests: commands[1] | python --version
Python 3.7.0
py37 runtests: commands[2] | py.test --cov=pdf2textbox --duration=0 tests
================================= test session starts =================================
platform linux -- Python 3.7.0, pytest-3.6.3, py-1.5.4, pluggy-0.6.0
rootdir: /home/frodo/Python-Projekte/.virtualenvs/pdf2textbox, inifile:
plugins: cov-2.5.1
collected 15 items                                                                    

tests/test_cli_slice_option.py ...                                              [ 20%]
tests/test_convert_pdf.py ......                                                [ 60%]
tests/test_get_pdf.py ....                                                      [ 86%]
tests/test_types.py ..                                                          [100%]

----------- coverage: platform linux, python 3.7.0-final-0 -----------
Name                                   Stmts   Miss  Cover
pdf2textbox/pdf2textbox.py     506    201    60%


=============================== slowest test durations ================================
426.08s call     tests/test_cli_slice_option.py::test_pdf_to_text_slice
9.55s call     tests/test_types.py::test_pdf_to_text_all
9.50s call     tests/test_types.py::test_textbox_structure
3.48s call     tests/test_convert_pdf.py::test_convert_three_cols_and_header
3.42s call     tests/test_convert_pdf.py::test_convert_three_cols
3.15s call     tests/test_convert_pdf.py::test_convert_two_cols_and_header
3.09s call     tests/test_convert_pdf.py::test_convert_two_cols
2.81s call     tests/test_convert_pdf.py::test_convert_text_only
2.74s call     tests/test_convert_pdf.py::test_convert_one_col_and_header
0.38s call     tests/test_get_pdf.py::test_get_pdf_file_url
...
============================ 15 passed in 465.17 seconds =============================

__________________________________ summary _______________________________________
  py35: commands succeeded
  py36: commands succeeded
  py37: commands succeeded
  congratulations :)

[nedbat]

Thanks. Now can you add "coverage --version" to the tox.ini [commands] section, so that we can see the coverage version for each of the different Python versions?

[0LL13]

I did that and now as command[2] there is:

py36 runtests: commands[2] | coverage --version
Coverage.py, version 4.5.1 with C extension
Documentation at https://coverage.readthedocs.io

Otherwise no change.

[nedbat]

@0LL13 I don't know how else to say this: I need to see the version of coverage for each Python version. I suspect that Python 3.7 is running with the Python extension, not the C extension.

[0LL13]

Sorry for the delay. You were absolutely right:

ppy35 runtests: commands[0] | check-manifest
lists of files in version control and sdist match
py35 runtests: commands[1] | python --version
Python 3.5.5
py35 runtests: commands[2] | coverage --version
Coverage.py, version 4.5.1 with C extension
Documentation at https://coverage.readthedocs.io
py35 runtests: commands[3] | py.test --cov=pdf2textbox --duration=0 tests

py36 runtests: commands[0] | check-manifest
lists of files in version control and sdist match
py36 runtests: commands[1] | python --version
Python 3.6.3
py36 runtests: commands[2] | coverage --version
Coverage.py, version 4.5.1 with C extension
Documentation at https://coverage.readthedocs.io
py36 runtests: commands[3] | py.test --cov=pdf2textbox --duration=0 tests

py37 runtests: commands[0] | check-manifest
lists of files in version control and sdist match
py37 runtests: commands[1] | python --version
Python 3.7.0
py37 runtests: commands[2] | coverage --version
Coverage.py, version 4.5.1 without C extension
Documentation at https://coverage.readthedocs.io
py37 runtests: commands[3] | py.test --cov=pdf2textbox --duration=0 tests

Otherwise the same output as above.
I will be offline for a couple of days.

[nedbat]

Thanks. 4.5.1 doesn't have a wheel for 3.7, so it tries to install from source. Perhaps you don't have the needed python-dev packages installed? http://coverage.readthedocs.io/en/coverage-4.5.1a/install.html#c-extension

[0LL13]

frodo@theshire:~$ sudo apt-get install python3-dev gcc
[sudo] Passwort für frodo:
Paketlisten werden gelesen... Fertig
Abhängigkeitsbaum wird aufgebaut.
Statusinformationen werden eingelesen.... Fertig
gcc ist schon die neueste Version (4:7.3.0-3ubuntu2).
gcc wurde als manuell installiert festgelegt.
python3-dev ist schon die neueste Version (3.6.5-3ubuntu1).
0 aktualisiert, 0 neu installiert, 0 zu entfernen und 6 nicht aktualisiert.

(vEnv)frodo@theshire:pdf2textbox$ python -m coverage --version
Coverage.py, version 4.5.1 with C extension

Coverage is only within the virtualenv for my project.

[vst]

Confirming that pip install coverage comes without C extension under Python 3.7 which I have installed using pyenv. As a result, my pytest --cov took ~20 mins instead of usual ~5mins.

Solution for my case:

pip uninstall coverage
pip install --global-option=build_ext --global-option="$(python-config --includes)" coverage

[nedbat]

OK, I've uploaded wheels for v4.5.1 for Python 3.7 on mac and manylinux to pypi. Let me know if that solves the problem. Thanks for all the details.

[cardoe]

I believe this change has broken folks using pipenv and a Pipfile/Pipfile.lock as the hash for 4.5.1 has now changed.

Installing dependencies from Pipfile.lock (a03fec)...
An error occurred while installing coverage==4.5.1! Will try again.
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 25/25 — 00:00:01
Installing initially failed dependencies...
Looking in indexes: https://pypi.python.org/simple00
Collecting coverage==4.5.1
  Using cached https://files.pythonhosted.org/packages/c7/d0/337673c08f5b0cc7ada3dfe2a998ae8a97d482722045644be3d79bbcbe05/coverage-4.5.1-cp37-cp37m-macosx_10_13_x86_64.whl

THESE PACKAGES DO NOT MATCH THE HASHES FROM Pipfile.lock!. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.```

[0LL13]

(vEnv)$ pip uninstall coverage
(vEnv)$ pip install --no-cache-dir coverage
Collecting coverage
Downloading https://files.pythonhosted.org/packages/3d/a0/b12090c40e0b8196b973962ec71c1c541a6c04af58ba5ad85683b3de251a/coverage-4.5.1-cp36-cp36m-manylinux1_x86_64.whl (202kB)
100% |████████████████████████████████| 204kB 6.9MB/s
Installing collected packages: coverage
Successfully installed coverage-4.5.1

No change in py.test --cov time for Python 3.7

[nedbat]

@cardoe I don't understand how uploading a new file that never existed before could break an installation. Can you provide more details? What was the old hash computed from, and what is the new hash computed from?

@0LL13 Looks like that wheel you installed was for Python 3.6. Can you double-check that you are installing for 3.7?

[cardoe]

@nedbat It came from running pipenv install --dev coverage a few months back sometime after 4.5.1 was released. pipenv automatically generates a Pipfile.lock which contains all the hashes of that version on pypi. According to pypa folks its a feature. https://docs.pipenv.org/basics/#pipfile-lock-security-features From the rough looks of it they're just using pip hash which appears to have changed for a pip cache containing coverage from about 10 hours ago to a cache that was recently created. Ultimately our tests started failing because pipenv reported that coverage may have been tampered with. I came to the GitHub project and searched and landed on this issue. Our last successful test was 42 minutes before your comment about new wheels being uploaded. Our first failure was 23 minutes after that comment. I very infrequently write Python now days so I'm just trying to go with the tools I've got. 🤷‍♂️

[nedbat]

@cardoe I guess I understand what is happening. I'm not sure what I can do about it. Uploading new wheels for new Python versions seems like a net positive to me. Perhaps you should re-create your Pipfile.lock file? I'm not sure what the pipenv authors would recommend.

[vst]

@nedbat It seems that wheel works fine now for py37. Thanks a lot!

coverage comes with C extension via tox --recreate, too, as expected:

python runtests: commands[0] | coverage --version
Coverage.py, version 4.5.1 with C extension
Documentation at https://coverage.readthedocs.io
python runtests: commands[1] | pytest --cov
...

As for pipenv: I am not using it right now but I guess that the only solution is to recreate Pipfile.lock. Yet, some friends may need to make new releases for packaged applications if they are shipping Pipfile.lock (such as Docker etc...).

[nedbat]

I asked about pipenv in the pipenv repo: pypa/pipenv#2618

[abk-code]

I can confirm that recreation of Pipfile.lock followed by pipenv install --dev will solve. Thanks @cardoe for noticing this issue. Thank you!

[0LL13]

I guess you need these lines, if not I need advice how to do it:

(vEnv)$ tox
...
py37 inst-nodeps: /home/frodo/Python-Projekte/.virtualenvs/pdf2textbox/.tox/dist/pdf2textbox-0.3.1.zip
py37 installed: atomicwrites==1.1.5,attrs==18.1.0,certifi==2018.4.16,chardet==3.0.4,check-manifest==0.37,coverage==4.5.1,idna==2.7,more-itertools==4.2.0,pdf2textbox==0.3.1,pdfminer2==20151206,pluggy==0.6.0,py==1.5.4,pytest==3.6.3,pytest-cov==2.5.1,requests==2.19.1,six==1.11.0,urllib3==1.23
py37 runtests: PYTHONHASHSEED='513262920'
py37 runtests: commands[0] | check-manifest
lists of files in version control and sdist match
py37 runtests: commands[1] | python --version
Python 3.7.0
py37 runtests: commands[2] | coverage --version
Coverage.py, version 4.5.1 without C extension
Documentation at https://coverage.readthedocs.io
py37 runtests: commands[3] | py.test --cov=pdf2textbox --duration=0 tests
================================= test session starts =================================
platform linux -- Python 3.7.0, pytest-3.6.3, py-1.5.4, pluggy-0.6.0
rootdir: /home/frodo/Python-Projekte/.virtualenvs/pdf2textbox, inifile:
plugins: cov-2.5.1
collected 15 items

Times are as before.

[vst]

@0LL13 Can you try tox --recreate instead of just tox?

[0LL13]

Cool.

(vEnv)$ tox --recreate
...
py37 recreate: /home/frodo/Python-Projekte/.virtualenvs/pdf2textbox/.tox/py37
py37 installdeps: check-manifest, pytest, requests, pdfminer2, coverage, pytest-cov
py37 inst: /home/frodo/Python-Projekte/.virtualenvs/pdf2textbox/.tox/dist/pdf2textbox-0.3.1.zip
py37 installed: atomicwrites==1.1.5,attrs==18.1.0,certifi==2018.4.16,chardet==3.0.4,check-manifest==0.37,coverage==4.5.1,idna==2.7,more-itertools==4.2.0,pdf2textbox==0.3.1,pdfminer2==20151206,pluggy==0.6.0,py==1.5.4,pytest==3.6.3,pytest-cov==2.5.1,requests==2.19.1,six==1.11.0,urllib3==1.23
py37 runtests: PYTHONHASHSEED='3853648192'
py37 runtests: commands[0] | check-manifest
lists of files in version control and sdist match
py37 runtests: commands[1] | python --version
Python 3.7.0
py37 runtests: commands[2] | coverage --version
Coverage.py, version 4.5.1 with C extension
Documentation at https://coverage.readthedocs.io
py37 runtests: commands[3] | py.test --cov=pdf2textbox --duration=0 tests
================================= test session starts =================================
platform linux -- Python 3.7.0, pytest-3.6.3, py-1.5.4, pluggy-0.6.0
rootdir: /home/frodo/Python-Projekte/.virtualenvs/pdf2textbox, inifile:
plugins: cov-2.5.1
collected 15 items

tests/test_cli_slice_option.py ... [ 20%]
tests/test_convert_pdf.py ...... [ 60%]
tests/test_get_pdf.py .... [ 86%]
tests/test_types.py .. [100%]

----------- coverage: platform linux, python 3.7.0-final-0 -----------
Name Stmts Miss Cover
.....................................................................................
pdf2textbox/pdf2textbox.py 506 201 60%

=============================== slowest test durations ================================
52.36s call tests/test_cli_slice_option.py::test_pdf_to_text_slice
2.64s call tests/test_get_pdf.py::test_get_pdf_file_url
1.14s call tests/test_types.py::test_textbox_structure
1.14s call tests/test_types.py::test_pdf_to_text_all
...
========================= 15 passed in 59.84 seconds ==============================
__________________________________ summary _______________________________________
py35: commands succeeded
py36: commands succeeded
py37: commands succeeded
congratulations :)

[nedbat]

@0LL13 looks like it's all fixed. :)

[0LL13]

Yep :)

[coffeebenzene]

Hi, sorry for reviving this issue, but are there C extensions for Python 3.7 on windows?
I have:

python --version
Python 3.7.0
coverage --version
Coverage.py, version 4.5.1 without C extension

[nedbat]

@coffeebenzene Right, there are no windows 3.7 files at the moment. I can try to build them, but something is wrong at AppVeyor, and it is difficult to diagnose.

[nedbat]

@coffeebenzene ah, but they are here: https://www.lfd.uci.edu/~gohlke/pythonlibs/#coverage . Perhaps I will upload those.

[nedbat]

@coffeebenzene I uploaded those .whl to pypi. You should be all set.

[coffeebenzene]

Thanks!
I just reinstalled with pip and I got the C extensions now.