Skip to content

pytest local plugin not working: unknown hook 'pytest_disable_cov' #5169

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
3 tasks done
iamabigstone opened this issue Apr 25, 2019 · 20 comments
Closed
3 tasks done
Labels
type: question general question, might be closed after 2 weeks of inactivity

Comments

@iamabigstone
Copy link

iamabigstone commented Apr 25, 2019

  • a detailed description of the bug or suggestion
    I have a pytest plugin in test folder's conftest.py:
    for e.g.
...
def is_debugging():
    return True


def pytest_disable_cov(args):
    if is_debugging():
        args[:] = ['--no-cov'] + args
...

the dirs look like:

myrepo/
  ENV/
    ...
  mypackage/
    ...
  test/
    ...
    conftest.py

Now if I run below from virtual env:
cd myrepo; python -m pytest test/test_something.py

it outputs:

(ENV) MacBook-Pro-4:~ $ cd mypackage; python -m pytest test/test_cam_embedded_device.py
[2019-04-25 16:20:18,387] [mypackage.resources.macos.get_logger] [INFO] [35401]: mypackage run uuid is: b02fcd0c-67b0-11e9-a0bb-6c4008baa5e0
[2019-04-25 16:20:18,387] [mypackage.resources.macos.get_logger] [INFO] [35401]: logs will be written to /tmp/mypackage_logs/b02fcd0c-67b0-11e9-a0bb-6c4008baa5e0/
===================================================================== test session starts =====================================================================
platform darwin -- Python 2.7.15, pytest-4.4.1, py-1.8.0, pluggy-0.9.0 -- /Users/local/mypackage/ENV/bin/python
cachedir: .pytest_cache
rootdir: /Users/local/mypackage, inifile: pytest.ini
plugins: cov-2.6.1
collecting ... INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/Users/local/mypackage/ENV/lib/python2.7/site-packages/_pytest/main.py", line 209, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/Users/local/mypackage/ENV/lib/python2.7/site-packages/_pytest/main.py", line 248, in _main
INTERNALERROR>     config.hook.pytest_collection(session=session)
INTERNALERROR>   File "/Users/local/mypackage/ENV/lib/python2.7/site-packages/pluggy/hooks.py", line 289, in __call__
INTERNALERROR>     return self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR>   File "/Users/local/mypackage/ENV/lib/python2.7/site-packages/pluggy/manager.py", line 68, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/Users/local/mypackage/ENV/lib/python2.7/site-packages/pluggy/manager.py", line 62, in <lambda>
INTERNALERROR>     firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
INTERNALERROR>   File "/Users/local/mypackage/ENV/lib/python2.7/site-packages/pluggy/callers.py", line 208, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/Users/local/mypackage/ENV/lib/python2.7/site-packages/pluggy/callers.py", line 81, in get_result
INTERNALERROR>     _reraise(*ex)  # noqa
INTERNALERROR>   File "/Users/local/mypackage/ENV/lib/python2.7/site-packages/pluggy/callers.py", line 187, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/Users/local/mypackage/ENV/lib/python2.7/site-packages/_pytest/main.py", line 258, in pytest_collection
INTERNALERROR>     return session.perform_collect()
INTERNALERROR>   File "/Users/local/mypackage/ENV/lib/python2.7/site-packages/_pytest/main.py", line 495, in perform_collect
INTERNALERROR>     self.config.pluginmanager.check_pending()
INTERNALERROR>   File "/Users/local/mypackage/ENV/lib/python2.7/site-packages/pluggy/manager.py", line 251, in check_pending
INTERNALERROR>     % (name, hookimpl.plugin),
INTERNALERROR> PluginValidationError: unknown hook 'pytest_disable_cov' in plugin <module 'test.conftest' from '/Users/local/mypackage/test/conftest.py'>

================================================================ no tests ran in 0.40 seconds =================================================================
  • output of pip list from the virtual environment you are using
(ENV) locals-MacBook-Pro-4:mypackage local$ pip list
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Package                                 Version    
--------------------------------------- -----------
alabaster                               0.7.12     
astroid                                 1.6.6      
atomicwrites                            1.3.0      
attrs                                   19.1.0     
automationresultcode                    0.6.9      
autopep8                                1.4.3      
Babel                                   2.6.0      
backports.functools-lru-cache           1.5        
cachetools                              3.1.0      
certifi                                 2019.3.9   
chardet                                 3.0.4      
configparser                            3.7.4      
coverage                                4.5.3      
docutils                                0.14       
easy-dict-cache                         1.2.0      
enum34                                  1.1.6      
funcsigs                                1.0.2      
functools32                             3.2.3.post2
future                                  0.16.0     
futures                                 3.2.0      
idna                                    2.8        
imagesize                               1.1.0      
isort                                   4.3.17     
Jinja2                                  2.10.1     
jsonschema                              2.6.0      
lazy-object-proxy                       1.3.1      
MarkupSafe                              1.1.1      
mccabe                                  0.6.1      
mongoengine                             0.15.0     
more-itertools                          5.0.0      
numpy                                   1.16.3     
packaging                               19.0       
pandas                                  0.24.0     
pandas-validator                        0.5.0      
pathlib2                                2.3.3      
pip                                     19.1       
pluggy                                  0.9.0      
py                                      1.8.0      
pycodestyle                             2.4.0      
pycoreautomation                        0.4.5      
pycoreautomation-ext                    0.4.31     
Pygments                                2.3.1      
pylint                                  1.9.3      
pymongo                                 3.8.0      
pyobjc                                  5.1        
pyobjc-core                             5.1        
pyobjc-framework-Accounts               5.1        
pyobjc-framework-AddressBook            5.1        
pyobjc-framework-AdSupport              5.1        
pyobjc-framework-AppleScriptKit         5.1        
pyobjc-framework-AppleScriptObjC        5.1        
pyobjc-framework-ApplicationServices    5.1        
pyobjc-framework-Automator              5.1        
pyobjc-framework-AVFoundation           5.1        
pyobjc-framework-AVKit                  5.1        
pyobjc-framework-BusinessChat           5.1        
pyobjc-framework-CalendarStore          5.1        
pyobjc-framework-CFNetwork              5.1        
pyobjc-framework-CloudKit               5.1        
pyobjc-framework-Cocoa                  5.1        
pyobjc-framework-Collaboration          5.1        
pyobjc-framework-ColorSync              5.1        
pyobjc-framework-Contacts               5.1        
pyobjc-framework-ContactsUI             5.1        
pyobjc-framework-CoreAudio              5.1        
pyobjc-framework-CoreAudioKit           5.1        
pyobjc-framework-CoreBluetooth          5.1        
pyobjc-framework-CoreData               5.1        
pyobjc-framework-CoreLocation           5.1        
pyobjc-framework-CoreMedia              5.1        
pyobjc-framework-CoreMediaIO            5.1        
pyobjc-framework-CoreML                 5.1        
pyobjc-framework-CoreServices           5.1        
pyobjc-framework-CoreSpotlight          5.1        
pyobjc-framework-CoreText               5.1        
pyobjc-framework-CoreWLAN               5.1        
pyobjc-framework-CryptoTokenKit         5.1        
pyobjc-framework-DictionaryServices     5.1        
pyobjc-framework-DiscRecording          5.1        
pyobjc-framework-DiscRecordingUI        5.1        
pyobjc-framework-DiskArbitration        5.1        
pyobjc-framework-DVDPlayback            5.1        
pyobjc-framework-EventKit               5.1        
pyobjc-framework-ExceptionHandling      5.1        
pyobjc-framework-ExternalAccessory      5.1        
pyobjc-framework-FinderSync             5.1        
pyobjc-framework-FSEvents               5.1        
pyobjc-framework-GameCenter             5.1        
pyobjc-framework-GameController         5.1        
pyobjc-framework-GameKit                5.1        
pyobjc-framework-GameplayKit            5.1        
pyobjc-framework-ImageCaptureCore       5.1        
pyobjc-framework-IMServicePlugIn        5.1        
pyobjc-framework-InputMethodKit         5.1        
pyobjc-framework-InstallerPlugins       5.1        
pyobjc-framework-InstantMessage         5.1        
pyobjc-framework-Intents                5.1        
pyobjc-framework-IOSurface              5.1        
pyobjc-framework-iTunesLibrary          5.1        
pyobjc-framework-LatentSemanticMapping  5.1        
pyobjc-framework-LaunchServices         5.1        
pyobjc-framework-libdispatch            5.1        
pyobjc-framework-LocalAuthentication    5.1        
pyobjc-framework-MapKit                 5.1        
pyobjc-framework-MediaAccessibility     5.1        
pyobjc-framework-MediaLibrary           5.1        
pyobjc-framework-MediaPlayer            5.1        
pyobjc-framework-MediaToolbox           5.1        
pyobjc-framework-ModelIO                5.1        
pyobjc-framework-MultipeerConnectivity  5.1        
pyobjc-framework-NaturalLanguage        5.1        
pyobjc-framework-NetFS                  5.1        
pyobjc-framework-Network                5.1        
pyobjc-framework-NetworkExtension       5.1        
pyobjc-framework-NotificationCenter     5.1        
pyobjc-framework-OpenDirectory          5.1        
pyobjc-framework-OSAKit                 5.1        
pyobjc-framework-Photos                 5.1        
pyobjc-framework-PhotosUI               5.1        
pyobjc-framework-PreferencePanes        5.1        
pyobjc-framework-PubSub                 5.1        
pyobjc-framework-QTKit                  5.1        
pyobjc-framework-Quartz                 5.1        
pyobjc-framework-SafariServices         5.1        
pyobjc-framework-SceneKit               5.1        
pyobjc-framework-ScreenSaver            5.1        
pyobjc-framework-ScriptingBridge        5.1        
pyobjc-framework-SearchKit              5.1        
pyobjc-framework-Security               5.1        
pyobjc-framework-SecurityFoundation     5.1        
pyobjc-framework-SecurityInterface      5.1        
pyobjc-framework-ServiceManagement      5.1        
pyobjc-framework-Social                 5.1        
pyobjc-framework-SpriteKit              5.1        
pyobjc-framework-StoreKit               5.1        
pyobjc-framework-SyncServices           5.1        
pyobjc-framework-SystemConfiguration    5.1        
pyobjc-framework-UserNotifications      5.1        
pyobjc-framework-VideoSubscriberAccount 5.1        
pyobjc-framework-VideoToolbox           5.1        
pyobjc-framework-Vision                 5.1        
pyobjc-framework-WebKit                 5.1        
pyparsing                               2.4.0      
pytest                                  4.4.1      
pytest-cov                              2.6.1      
python-dateutil                         2.8.0      
pytoolkit                               0.6.26     
pytz                                    2019.1     
recertifi                               0.11.0     
requests                                2.21.0     
scandir                                 1.10.0     
setuptools                              41.0.1     
singledispatch                          3.4.0.3    
six                                     1.12.0     
snowballstemmer                         1.2.1      
speedtracer-client                      1.3.2      
Sphinx                                  1.8.0      
sphinxcontrib-websupport                1.1.0      
testautomationextras                    0.7.48     
testlogger                              0.4.3      
testlogs                                0.6.10     
testresults                             0.7.8      
typing                                  3.6.6      
unixcommand                             0.6.24     
urllib2-ext                             1.9.10     
urllib3                                 1.24.2     
wheel                                   0.33.1     
wrapt                                   1.11.1    
  • pytest and operating system versions
(ENV) locals-MacBook-Pro-4:mypackage local$ pip freeze | grep pytest
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
pytest==4.4.1
pytest-cov==2.6.1
(ENV) locals-MacBook-Pro-4:mypackage local$ 
(ENV) locals-MacBook-Pro-4:mypackage local$ plutil -p /System/Library/CoreServices/SystemVersion.plist
{
  "iOSSupportVersion" => "12.0"
  "ProductBuildVersion" => "18D25"
  "ProductCopyright" => "1983-2018 Apple Inc."
  "ProductName" => "Mac OS X"
  "ProductUserVisibleVersion" => "10.14.3"
  "ProductVersion" => "10.14.3"
}
@blueyed
Copy link
Contributor

blueyed commented Apr 26, 2019

The error is:

PluginValidationError: unknown hook 'pytest_disable_cov' in plugin <module 'test.conftest' from '/Users/local/mypackage/test/conftest.py'>

Looks like you want to use pytest_cmdline_preparse instead?!

@blueyed blueyed changed the title pytest local plugin not working pytest local plugin not working: unknown hook 'pytest_disable_cov' Apr 26, 2019
@blueyed blueyed added the type: question general question, might be closed after 2 weeks of inactivity label Apr 26, 2019
@iamabigstone
Copy link
Author

@blueyed
Ok, I have tried pytest_load_initial_conftests (pytest_load_initial_conftests is deprecated), but it does not work:

The test did run and pass, but --no-cov option does not take effect, here is output of pytest test/test_myrepo.py:

$ pytest test/test_myrepo.py 
['--no-cov', '-vvv', '--setup-show', '-s', '--cov=mypackage', 'test/test_myrepo.py']
=================================================== test session starts ====================================================
platform darwin -- Python 2.7.15, pytest-4.4.1, py-1.8.0, pluggy-0.9.0 -- /Users/local/myrepo/ENV/bin/python2.7
cachedir: .pytest_cache
rootdir: /Users/local/myrepo, inifile: pytest.ini
plugins: mypackage-0.0.0, cov-2.6.1
collected 1 item                                                                                                           

test/test_myrepo.py::test_x 
        test/test_myrepo.py::test_xPASSED

---------- coverage: platform darwin, python 2.7.15-final-0 ----------
Name                          Stmts   Miss  Cover
-------------------------------------------------
mypackage/__init__.py             0      0   100%
mypackage/custom_plugins.py       6      2    67%
mypackage/x.py                    2      0   100%
-------------------------------------------------
TOTAL                             8      2    75%


================================================= 1 passed in 0.02 seconds =================================================

my code attached:
myrepo.zip

notes: .envrc is used to automate shell env of the proj (ref: https://direnv.net/)
also I run pip install -e . to install the mypackage in edit mode

@blueyed
Copy link
Contributor

blueyed commented Apr 27, 2019

Please try pytest_cmdline_preparse.

@blueyed
Copy link
Contributor

blueyed commented Apr 27, 2019

See #5024 (comment).

@blueyed
Copy link
Contributor

blueyed commented Apr 27, 2019

Also: just use -p no:cov :)

@blueyed blueyed closed this as completed Apr 27, 2019
@iamabigstone
Copy link
Author

@blueyed

After made the changes you have suggested, it still does not work:

$ pytest test/test_myrepo.py 
['-p', 'no:cov', '-vvv', '--setup-show', '-s', '--cov=mypackage', 'test/test_myrepo.py']
=================================================== test session starts ====================================================
platform darwin -- Python 2.7.15, pytest-4.4.1, py-1.8.0, pluggy-0.9.0 -- /Users/local/myrepo/ENV/bin/python2.7
cachedir: .pytest_cache
rootdir: /Users/local/myrepo, inifile: pytest.ini
plugins: mypackage-0.0.0, cov-2.6.1
collected 1 item                                                                                                           

test/test_myrepo.py::test_x 
        test/test_myrepo.py::test_xPASSED

---------- coverage: platform darwin, python 2.7.15-final-0 ----------
Name                          Stmts   Miss  Cover
-------------------------------------------------
mypackage/__init__.py             0      0   100%
mypackage/custom_plugins.py       6      2    67%
mypackage/x.py                    2      0   100%
-------------------------------------------------
TOTAL                             8      2    75%


================================================= 1 passed in 0.02 seconds =================================================

updated code attached:
myrepo_updated.zip

@blueyed
Copy link
Contributor

blueyed commented Apr 27, 2019

Just run pytest -p no:cov test/test_myrepo.py - no need for a custom conftest.

@blueyed
Copy link
Contributor

blueyed commented Apr 27, 2019

Also conftest is not in the zip?!

@blueyed
Copy link
Contributor

blueyed commented Apr 27, 2019

Ah, you used mypackage/custom_plugins.py there.
Anyway.. -p no:cov does not work at that point anymore, and --no-cov could work, but would need to be appended likely also..

@iamabigstone
Copy link
Author

iamabigstone commented Apr 27, 2019

@blueyed

Nope... tried append --no-cov in custom_plugins.py, and still no luck...

The reason I create a plugin is to dynamically disable coverage if it is running in python debug mode - debugger does not work properly when pytest runs with coverage (ref: https://pytest-cov.readthedocs.io/en/latest/debuggers.html)

@blueyed
Copy link
Contributor

blueyed commented Apr 27, 2019

Doesn't pytest -p no:cov work?
Then pytest-cov should not show up in the "plugins" line at all.

Also consider maybe running with coverage only on demand, i.e. create a .coveragerc, so that then passing --cov is enough. Or try adding everything to addopts for pytest, with --no-cov there at the end already, and then hope/try that manually appending --cov on the command line triggers it then.

@iamabigstone
Copy link
Author

@blueyed

Doesn't pytest -p no:cov work?
Then pytest-cov should not show up in the "plugins" line at all.

If I directly pass -p no:cov or --no-cov from CLI, it does work, but not from pytest_cmdline_preparse plugin... So should I write a python script which calls pytest binary with the -p no:cov option from subprocess or alike? Is there a better way to handle this?

$ pytest -p no:cov
updated_args:['-p', 'pytest_cov', '-vvv', '-s', '--cov=mypackage', '-p', 'no:cov']
=================================================== test session starts ====================================================
platform darwin -- Python 2.7.15, pytest-4.4.1, py-1.8.0, pluggy-0.9.0 -- /Users/local/myrepo/ENV/bin/python2.7
cachedir: .pytest_cache
rootdir: /Users/local/myrepo, inifile: pytest.ini
plugins: cov-2.6.1, mypackage-0.0.0
collected 1 item                                                                                                           

test/test_myrepo.py::test_x PASSED

================================================= 1 passed in 0.01 seconds =================================================
$ pytest
updated_args:['-p', 'pytest_cov', '-vvv', '-s', '--cov=mypackage']
=================================================== test session starts ====================================================
platform darwin -- Python 2.7.15, pytest-4.4.1, py-1.8.0, pluggy-0.9.0 -- /Users/local/myrepo/ENV/bin/python2.7
cachedir: .pytest_cache
rootdir: /Users/local/myrepo, inifile: pytest.ini
plugins: cov-2.6.1, mypackage-0.0.0
collected 1 item                                                                                                           

test/test_myrepo.py::test_x PASSED

---------- coverage: platform darwin, python 2.7.15-final-0 ----------
Name                          Stmts   Miss  Cover
-------------------------------------------------
mypackage/__init__.py             0      0   100%
mypackage/custom_plugins.py       6      2    67%
mypackage/x.py                    2      0   100%
-------------------------------------------------
TOTAL                             8      2    75%


================================================= 1 passed in 0.03 seconds =================================================

Also tried all your other suggestions, none of them working... Could there be a bug in pytest? Maybe caused by #935?

@blueyed
Copy link
Contributor

blueyed commented Apr 29, 2019

I am not really sure what you're trying to achieve here: it sounds like you want to use pytest-cov not by default when developing or something like that?

If you run pytest -p no:cov you are still adding it to args via your plugin, no?

['-p', 'pytest_cov', '-vvv', '-s', '--cov=mypackage', '-p', 'no:cov']

I would just not try adjusting args in that regard from the beginning.

And then you could use e.g. -p no:cov in pytest's addopts config (or $PYTEST_ADDOPTS), and enable/use it on demand only.

@iamabigstone
Copy link
Author

@blueyed

I have mentioned the reason in one of my previous comments:

The reason I create a plugin is to dynamically disable coverage if it is running in python debug mode - debugger does not work properly when pytest runs with coverage (ref: https://pytest-cov.readthedocs.io/en/latest/debuggers.html)

And

If you run pytest -p no:cov you are still adding it to args via your plugin, no?
['-p', 'pytest_cov', '-vvv', '-s', '--cov=mypackage', '-p', 'no:cov']

No, in the plugin I just print out the args without changing a bit. Basically pytest.ini:

[pytest]
addopts = -p pytest_cov -vvv -s --cov=mypackage

In the CLI, I run pytest -p no:cov

@blueyed
Copy link
Contributor

blueyed commented Apr 30, 2019

Well.. you should decide if you want to see coverage all the time, or not - I'd rather only run it on CI or manually (i.e. via pytest --cov).
If you want to see it all the time, you could just ignore false reports after having used a debugger, no?

You are also mixing pytest_cov and cov here - so with your addopts (-p pytest_cov) you should also use -p no:pytest_cov then.

Just ensure that pytest's output is what you want ("plugins: cov-2.6.1, mypackage-0.0.0").

@iamabigstone
Copy link
Author

@blueyed

If you want to see it all the time, you could just ignore false reports after having used a debugger, no?

Nope, if I run pytest with coverage enabled through a debugger, I cannot hit breakpoint. The issue quoting from https://pytest-cov.readthedocs.io/en/latest/debuggers.html:

Coverage does not play well with other tracers simultaneously running. This manifests itself in behaviour that PyCharm might not hit a breakpoint no matter what the user does. Since it is common practice to have coverage configuration in the pytest.ini file and pytest does not support removeopts or similar the –no-cov flag can disable coverage completely.

@blueyed
Copy link
Contributor

blueyed commented May 3, 2019

Oh.. for me import pdb; pdb.set_trace() works (but not using PyCharm) - it just influences coverage then.

Still, why do you want to use --cov by default?

@blueyed
Copy link
Contributor

blueyed commented May 3, 2019

How does PyCharm invoke debugging, i.e. how does it run pytest then?
Might be an idea for pytest-cov to skip its setup then automatically, if this could be seen from some environment variable or similar. /cc @ionelmc

@ionelmc
Copy link
Member

ionelmc commented May 3, 2019

I suspect @iamabigstone doesn't want to have different run configuration when debugging.

Conftests will not work. With a custom plugin this will do the job:

import pytest


@pytest.mark.tryfirst
def pytest_load_initial_conftests(early_config):
    if is_debugging():
        early_config.known_args_namespace.no_cov = True

@ionelmc
Copy link
Member

ionelmc commented May 3, 2019

Actually, since pytest_cov bootstraps via a tryfirst pytest_load_initial_conftests, it may not work, depending on plugins ordering. You may need to use -p mypackage.custom_plugins to be 100% sure it always gets run before.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type: question general question, might be closed after 2 weeks of inactivity
Projects
None yet
Development

No branches or pull requests

3 participants