pytest local plugin not working: unknown hook 'pytest_disable_cov'

[iamabigstone]

• a detailed description of the bug or suggestion
  I have a pytest plugin in test folder's conftest.py:
  for e.g.

...
def is_debugging():
    return True


def pytest_disable_cov(args):
    if is_debugging():
        args[:] = ['--no-cov'] + args
...

the dirs look like:

myrepo/
  ENV/
    ...
  mypackage/
    ...
  test/
    ...
    conftest.py

Now if I run below from virtual env:
cd myrepo; python -m pytest test/test_something.py

it outputs:

(ENV) MacBook-Pro-4:~ $ cd mypackage; python -m pytest test/test_cam_embedded_device.py
[2019-04-25 16:20:18,387] [mypackage.resources.macos.get_logger] [INFO] [35401]: mypackage run uuid is: b02fcd0c-67b0-11e9-a0bb-6c4008baa5e0
[2019-04-25 16:20:18,387] [mypackage.resources.macos.get_logger] [INFO] [35401]: logs will be written to /tmp/mypackage_logs/b02fcd0c-67b0-11e9-a0bb-6c4008baa5e0/
===================================================================== test session starts =====================================================================
platform darwin -- Python 2.7.15, pytest-4.4.1, py-1.8.0, pluggy-0.9.0 -- /Users/local/mypackage/ENV/bin/python
cachedir: .pytest_cache
rootdir: /Users/local/mypackage, inifile: pytest.ini
plugins: cov-2.6.1
collecting ... INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/Users/local/mypackage/ENV/lib/python2.7/site-packages/_pytest/main.py", line 209, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/Users/local/mypackage/ENV/lib/python2.7/site-packages/_pytest/main.py", line 248, in _main
INTERNALERROR>     config.hook.pytest_collection(session=session)
INTERNALERROR>   File "/Users/local/mypackage/ENV/lib/python2.7/site-packages/pluggy/hooks.py", line 289, in __call__
INTERNALERROR>     return self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR>   File "/Users/local/mypackage/ENV/lib/python2.7/site-packages/pluggy/manager.py", line 68, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/Users/local/mypackage/ENV/lib/python2.7/site-packages/pluggy/manager.py", line 62, in <lambda>
INTERNALERROR>     firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
INTERNALERROR>   File "/Users/local/mypackage/ENV/lib/python2.7/site-packages/pluggy/callers.py", line 208, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/Users/local/mypackage/ENV/lib/python2.7/site-packages/pluggy/callers.py", line 81, in get_result
INTERNALERROR>     _reraise(*ex)  # noqa
INTERNALERROR>   File "/Users/local/mypackage/ENV/lib/python2.7/site-packages/pluggy/callers.py", line 187, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/Users/local/mypackage/ENV/lib/python2.7/site-packages/_pytest/main.py", line 258, in pytest_collection
INTERNALERROR>     return session.perform_collect()
INTERNALERROR>   File "/Users/local/mypackage/ENV/lib/python2.7/site-packages/_pytest/main.py", line 495, in perform_collect
INTERNALERROR>     self.config.pluginmanager.check_pending()
INTERNALERROR>   File "/Users/local/mypackage/ENV/lib/python2.7/site-packages/pluggy/manager.py", line 251, in check_pending
INTERNALERROR>     % (name, hookimpl.plugin),
INTERNALERROR> PluginValidationError: unknown hook 'pytest_disable_cov' in plugin <module 'test.conftest' from '/Users/local/mypackage/test/conftest.py'>

================================================================ no tests ran in 0.40 seconds =================================================================

• output of pip list from the virtual environment you are using

(ENV) locals-MacBook-Pro-4:mypackage local$ pip list
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Package                                 Version    
--------------------------------------- -----------
alabaster                               0.7.12     
astroid                                 1.6.6      
atomicwrites                            1.3.0      
attrs                                   19.1.0     
automationresultcode                    0.6.9      
autopep8                                1.4.3      
Babel                                   2.6.0      
backports.functools-lru-cache           1.5        
cachetools                              3.1.0      
certifi                                 2019.3.9   
chardet                                 3.0.4      
configparser                            3.7.4      
coverage                                4.5.3      
docutils                                0.14       
easy-dict-cache                         1.2.0      
enum34                                  1.1.6      
funcsigs                                1.0.2      
functools32                             3.2.3.post2
future                                  0.16.0     
futures                                 3.2.0      
idna                                    2.8        
imagesize                               1.1.0      
isort                                   4.3.17     
Jinja2                                  2.10.1     
jsonschema                              2.6.0      
lazy-object-proxy                       1.3.1      
MarkupSafe                              1.1.1      
mccabe                                  0.6.1      
mongoengine                             0.15.0     
more-itertools                          5.0.0      
numpy                                   1.16.3     
packaging                               19.0       
pandas                                  0.24.0     
pandas-validator                        0.5.0      
pathlib2                                2.3.3      
pip                                     19.1       
pluggy                                  0.9.0      
py                                      1.8.0      
pycodestyle                             2.4.0      
pycoreautomation                        0.4.5      
pycoreautomation-ext                    0.4.31     
Pygments                                2.3.1      
pylint                                  1.9.3      
pymongo                                 3.8.0      
pyobjc                                  5.1        
pyobjc-core                             5.1        
pyobjc-framework-Accounts               5.1        
pyobjc-framework-AddressBook            5.1        
pyobjc-framework-AdSupport              5.1        
pyobjc-framework-AppleScriptKit         5.1        
pyobjc-framework-AppleScriptObjC        5.1        
pyobjc-framework-ApplicationServices    5.1        
pyobjc-framework-Automator              5.1        
pyobjc-framework-AVFoundation           5.1        
pyobjc-framework-AVKit                  5.1        
pyobjc-framework-BusinessChat           5.1        
pyobjc-framework-CalendarStore          5.1        
pyobjc-framework-CFNetwork              5.1        
pyobjc-framework-CloudKit               5.1        
pyobjc-framework-Cocoa                  5.1        
pyobjc-framework-Collaboration          5.1        
pyobjc-framework-ColorSync              5.1        
pyobjc-framework-Contacts               5.1        
pyobjc-framework-ContactsUI             5.1        
pyobjc-framework-CoreAudio              5.1        
pyobjc-framework-CoreAudioKit           5.1        
pyobjc-framework-CoreBluetooth          5.1        
pyobjc-framework-CoreData               5.1        
pyobjc-framework-CoreLocation           5.1        
pyobjc-framework-CoreMedia              5.1        
pyobjc-framework-CoreMediaIO            5.1        
pyobjc-framework-CoreML                 5.1        
pyobjc-framework-CoreServices           5.1        
pyobjc-framework-CoreSpotlight          5.1        
pyobjc-framework-CoreText               5.1        
pyobjc-framework-CoreWLAN               5.1        
pyobjc-framework-CryptoTokenKit         5.1        
pyobjc-framework-DictionaryServices     5.1        
pyobjc-framework-DiscRecording          5.1        
pyobjc-framework-DiscRecordingUI        5.1        
pyobjc-framework-DiskArbitration        5.1        
pyobjc-framework-DVDPlayback            5.1        
pyobjc-framework-EventKit               5.1        
pyobjc-framework-ExceptionHandling      5.1        
pyobjc-framework-ExternalAccessory      5.1        
pyobjc-framework-FinderSync             5.1        
pyobjc-framework-FSEvents               5.1        
pyobjc-framework-GameCenter             5.1        
pyobjc-framework-GameController         5.1        
pyobjc-framework-GameKit                5.1        
pyobjc-framework-GameplayKit            5.1        
pyobjc-framework-ImageCaptureCore       5.1        
pyobjc-framework-IMServicePlugIn        5.1        
pyobjc-framework-InputMethodKit         5.1        
pyobjc-framework-InstallerPlugins       5.1        
pyobjc-framework-InstantMessage         5.1        
pyobjc-framework-Intents                5.1        
pyobjc-framework-IOSurface              5.1        
pyobjc-framework-iTunesLibrary          5.1        
pyobjc-framework-LatentSemanticMapping  5.1        
pyobjc-framework-LaunchServices         5.1        
pyobjc-framework-libdispatch            5.1        
pyobjc-framework-LocalAuthentication    5.1        
pyobjc-framework-MapKit                 5.1        
pyobjc-framework-MediaAccessibility     5.1        
pyobjc-framework-MediaLibrary           5.1        
pyobjc-framework-MediaPlayer            5.1        
pyobjc-framework-MediaToolbox           5.1        
pyobjc-framework-ModelIO                5.1        
pyobjc-framework-MultipeerConnectivity  5.1        
pyobjc-framework-NaturalLanguage        5.1        
pyobjc-framework-NetFS                  5.1        
pyobjc-framework-Network                5.1        
pyobjc-framework-NetworkExtension       5.1        
pyobjc-framework-NotificationCenter     5.1        
pyobjc-framework-OpenDirectory          5.1        
pyobjc-framework-OSAKit                 5.1        
pyobjc-framework-Photos                 5.1        
pyobjc-framework-PhotosUI               5.1        
pyobjc-framework-PreferencePanes        5.1        
pyobjc-framework-PubSub                 5.1        
pyobjc-framework-QTKit                  5.1        
pyobjc-framework-Quartz                 5.1        
pyobjc-framework-SafariServices         5.1        
pyobjc-framework-SceneKit               5.1        
pyobjc-framework-ScreenSaver            5.1        
pyobjc-framework-ScriptingBridge        5.1        
pyobjc-framework-SearchKit              5.1        
pyobjc-framework-Security               5.1        
pyobjc-framework-SecurityFoundation     5.1        
pyobjc-framework-SecurityInterface      5.1        
pyobjc-framework-ServiceManagement      5.1        
pyobjc-framework-Social                 5.1        
pyobjc-framework-SpriteKit              5.1        
pyobjc-framework-StoreKit               5.1        
pyobjc-framework-SyncServices           5.1        
pyobjc-framework-SystemConfiguration    5.1        
pyobjc-framework-UserNotifications      5.1        
pyobjc-framework-VideoSubscriberAccount 5.1        
pyobjc-framework-VideoToolbox           5.1        
pyobjc-framework-Vision                 5.1        
pyobjc-framework-WebKit                 5.1        
pyparsing                               2.4.0      
pytest                                  4.4.1      
pytest-cov                              2.6.1      
python-dateutil                         2.8.0      
pytoolkit                               0.6.26     
pytz                                    2019.1     
recertifi                               0.11.0     
requests                                2.21.0     
scandir                                 1.10.0     
setuptools                              41.0.1     
singledispatch                          3.4.0.3    
six                                     1.12.0     
snowballstemmer                         1.2.1      
speedtracer-client                      1.3.2      
Sphinx                                  1.8.0      
sphinxcontrib-websupport                1.1.0      
testautomationextras                    0.7.48     
testlogger                              0.4.3      
testlogs                                0.6.10     
testresults                             0.7.8      
typing                                  3.6.6      
unixcommand                             0.6.24     
urllib2-ext                             1.9.10     
urllib3                                 1.24.2     
wheel                                   0.33.1     
wrapt                                   1.11.1    

• pytest and operating system versions

(ENV) locals-MacBook-Pro-4:mypackage local$ pip freeze | grep pytest
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
pytest==4.4.1
pytest-cov==2.6.1
(ENV) locals-MacBook-Pro-4:mypackage local$ 

(ENV) locals-MacBook-Pro-4:mypackage local$ plutil -p /System/Library/CoreServices/SystemVersion.plist
{
  "iOSSupportVersion" => "12.0"
  "ProductBuildVersion" => "18D25"
  "ProductCopyright" => "1983-2018 Apple Inc."
  "ProductName" => "Mac OS X"
  "ProductUserVisibleVersion" => "10.14.3"
  "ProductVersion" => "10.14.3"
}

[blueyed]

The error is:

PluginValidationError: unknown hook 'pytest_disable_cov' in plugin <module 'test.conftest' from '/Users/local/mypackage/test/conftest.py'>

Looks like you want to use pytest_cmdline_preparse instead?!

[iamabigstone]

@blueyed
Ok, I have tried pytest_load_initial_conftests (pytest_load_initial_conftests is deprecated), but it does not work:

The test did run and pass, but --no-cov option does not take effect, here is output of pytest test/test_myrepo.py:

$ pytest test/test_myrepo.py 
['--no-cov', '-vvv', '--setup-show', '-s', '--cov=mypackage', 'test/test_myrepo.py']
=================================================== test session starts ====================================================
platform darwin -- Python 2.7.15, pytest-4.4.1, py-1.8.0, pluggy-0.9.0 -- /Users/local/myrepo/ENV/bin/python2.7
cachedir: .pytest_cache
rootdir: /Users/local/myrepo, inifile: pytest.ini
plugins: mypackage-0.0.0, cov-2.6.1
collected 1 item                                                                                                           

test/test_myrepo.py::test_x 
        test/test_myrepo.py::test_xPASSED

---------- coverage: platform darwin, python 2.7.15-final-0 ----------
Name                          Stmts   Miss  Cover
-------------------------------------------------
mypackage/__init__.py             0      0   100%
mypackage/custom_plugins.py       6      2    67%
mypackage/x.py                    2      0   100%
-------------------------------------------------
TOTAL                             8      2    75%


================================================= 1 passed in 0.02 seconds =================================================

my code attached:
myrepo.zip

notes: .envrc is used to automate shell env of the proj (ref: https://direnv.net/)
also I run pip install -e . to install the mypackage in edit mode

[blueyed]

Please try pytest_cmdline_preparse.

[blueyed]

See #5024 (comment).

[blueyed]

Also: just use -p no:cov :)

[iamabigstone]

@blueyed

After made the changes you have suggested, it still does not work:

$ pytest test/test_myrepo.py 
['-p', 'no:cov', '-vvv', '--setup-show', '-s', '--cov=mypackage', 'test/test_myrepo.py']
=================================================== test session starts ====================================================
platform darwin -- Python 2.7.15, pytest-4.4.1, py-1.8.0, pluggy-0.9.0 -- /Users/local/myrepo/ENV/bin/python2.7
cachedir: .pytest_cache
rootdir: /Users/local/myrepo, inifile: pytest.ini
plugins: mypackage-0.0.0, cov-2.6.1
collected 1 item                                                                                                           

test/test_myrepo.py::test_x 
        test/test_myrepo.py::test_xPASSED

---------- coverage: platform darwin, python 2.7.15-final-0 ----------
Name                          Stmts   Miss  Cover
-------------------------------------------------
mypackage/__init__.py             0      0   100%
mypackage/custom_plugins.py       6      2    67%
mypackage/x.py                    2      0   100%
-------------------------------------------------
TOTAL                             8      2    75%


================================================= 1 passed in 0.02 seconds =================================================

updated code attached:
myrepo_updated.zip

[blueyed]

Just run pytest -p no:cov test/test_myrepo.py - no need for a custom conftest.

[blueyed]

Also conftest is not in the zip?!

[blueyed]

Ah, you used mypackage/custom_plugins.py there.
Anyway.. -p no:cov does not work at that point anymore, and --no-cov could work, but would need to be appended likely also..

[iamabigstone]

@blueyed

Nope... tried append --no-cov in custom_plugins.py, and still no luck...

The reason I create a plugin is to dynamically disable coverage if it is running in python debug mode - debugger does not work properly when pytest runs with coverage (ref: https://pytest-cov.readthedocs.io/en/latest/debuggers.html)

[blueyed]

Doesn't pytest -p no:cov work?
Then pytest-cov should not show up in the "plugins" line at all.

Also consider maybe running with coverage only on demand, i.e. create a .coveragerc, so that then passing --cov is enough. Or try adding everything to addopts for pytest, with --no-cov there at the end already, and then hope/try that manually appending --cov on the command line triggers it then.

[iamabigstone]

@blueyed

Doesn't pytest -p no:cov work?
Then pytest-cov should not show up in the "plugins" line at all.

If I directly pass -p no:cov or --no-cov from CLI, it does work, but not from pytest_cmdline_preparse plugin... So should I write a python script which calls pytest binary with the -p no:cov option from subprocess or alike? Is there a better way to handle this?

$ pytest -p no:cov
updated_args:['-p', 'pytest_cov', '-vvv', '-s', '--cov=mypackage', '-p', 'no:cov']
=================================================== test session starts ====================================================
platform darwin -- Python 2.7.15, pytest-4.4.1, py-1.8.0, pluggy-0.9.0 -- /Users/local/myrepo/ENV/bin/python2.7
cachedir: .pytest_cache
rootdir: /Users/local/myrepo, inifile: pytest.ini
plugins: cov-2.6.1, mypackage-0.0.0
collected 1 item                                                                                                           

test/test_myrepo.py::test_x PASSED

================================================= 1 passed in 0.01 seconds =================================================
$ pytest
updated_args:['-p', 'pytest_cov', '-vvv', '-s', '--cov=mypackage']
=================================================== test session starts ====================================================
platform darwin -- Python 2.7.15, pytest-4.4.1, py-1.8.0, pluggy-0.9.0 -- /Users/local/myrepo/ENV/bin/python2.7
cachedir: .pytest_cache
rootdir: /Users/local/myrepo, inifile: pytest.ini
plugins: cov-2.6.1, mypackage-0.0.0
collected 1 item                                                                                                           

test/test_myrepo.py::test_x PASSED

---------- coverage: platform darwin, python 2.7.15-final-0 ----------
Name                          Stmts   Miss  Cover
-------------------------------------------------
mypackage/__init__.py             0      0   100%
mypackage/custom_plugins.py       6      2    67%
mypackage/x.py                    2      0   100%
-------------------------------------------------
TOTAL                             8      2    75%


================================================= 1 passed in 0.03 seconds =================================================

Also tried all your other suggestions, none of them working... Could there be a bug in pytest? Maybe caused by #935?