Nose Tests discovery fails when tests are split by folders and imported to init.py (wantModule and wantDirectory are not parsed) #7124

IBestuzhev · 2019-08-28T10:21:33Z

Environment data

VS Code version: 1.37.1
Extension version (available under the Extensions sidebar): 2019.8.30787
OS and version: Windows 10 running VS Code, remote plugin connects to Ubuntu 18.04.2 via SSH
Python version (& distribution if applicable, e.g. Anaconda): python 2.7.15rc1
Type of virtual environment used (N/A | venv | virtualenv | conda | ...): virtualenv
Relevant/affected Python packages and their versions: Django==1.11.23; django-nose==1.4.6; nose==1.3.7
Jedi or Language Server? (i.e. what is "python.jediEnabled" set to; more info How to update the language server to the latest stable version #3977): jedi

Expected behaviour

Test discovery for nose should scan for both wantFile and wantModule and/or wantDirectory log outputs

Actual behaviour

We have a django project with django-nose test runner. Tests are split into modules, and all tests are imported to __init__.py. I think these are some obsolete tests, from default django test runner. But nose runs them perfectly.

I create a simple script, that translates the nosetests call to ./manage.py test call.
Basically it just strips -vvv and replaces it with --verbosity

#!/bin/bash
function arg_remove ()
{
    local array=("${@:3}")

    for ((i=0; i<"${#array[@]}"; ++i)); do
        case ${array[i]} in
            "$2") unset array[i]; break ;;
        esac
    done

    # clean up unset array indexes
    for i in "${!array[@]}"; do
        new_array+=( "${array[i]}" )
    done
    array=("${new_array[@]}")
    unset new_array

    # assign array outside function scope
    local -g "$1=(  )"
        eval ${1}='("${array[@]}")'
}
arg_remove ARG_PASS "-vvv" "$@"
echo $@
direnv exec /home/igor/myproject/backend/ /home/igor/.virtualenvs/myproject/bin/python /home/igor/myproject/backend/manage.py test --settings=core.local_settings --verbosity=3 -l nose.selector --nologcapture "${ARG_PASS[@]}"

The structure of tests is (stripped)

File backend/apps/tests/__init__.py

from .action import ActionTestCase

File backend/apps/tests/action.py

from apps.myproj.tests.base import BaseTestCase


class ActionMSFTestCase(BaseTestCase):
    pass
    # stripped

The output of --collect-only command is different to what vscode-python expects.

Relevant output to tests above is:

nose.selector: DEBUG: wantDirectory /home/igor/ata_portal/backend/apps/api/tests? True
nose.selector: DEBUG: Test name /home/igor/ata_portal/backend/apps/api/tests resolved to file /home/igor/ata_portal/backend/apps/api/tests, module None, call None
nose.selector: DEBUG: Final resolution of test name /home/igor/ata_portal/backend/apps/api/tests: file /home/igor/ata_portal/backend/apps/api/tests module apps.api.tests call None
nose.selector: DEBUG: wantModule <module 'apps.api.tests' from '/home/igor/ata_portal/backend/apps/api/tests/__init__.pyc'>? True
nose.selector: DEBUG: wantClass <class 'apps.api.tests.action_msf.ActionMSFTestCase'>? True

parserService here scans only for wantFile output.

In case above it founds wantClass above any wantFile ... py? True line. That causes testFile.suites.push(testSuite); to fail

Steps to reproduce:

Create a package tests
Add a module action.py to package with some real test case
import action inside tests/__init__.py
Configure nose tests for workspace
Run Python: Discover Tests command

Logs

Output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)

Test Discovery failed: 
TypeError: Cannot read property 'suites' of undefined

Output from Console under the Developer Tools panel (toggle Developer Tools on under Help; turn on source maps to make any tracebacks be useful by running Enable source map support for extension debugging)

notificationsAlerts.ts:40 Test discovery error, please check the configuration settings for the tests.
onDidNotificationChange @ notificationsAlerts.ts:40
console.ts:137 [Extension Host] Python Extension: displayDiscoverStatus TypeError: Cannot read property 'suites' of undefined
	at t.forEach.t (/home/igor/.vscode-server/extensions/ms-python.python-2019.8.30787/out/client/extension.js:75:1015174)
	at Array.forEach (<anonymous>)
	at h.parseNoseTestModuleCollectionResult (/home/igor/.vscode-server/extensions/ms-python.python-2019.8.30787/out/client/extension.js:75:1014343)
	at e.split.forEach (/home/igor/.vscode-server/extensions/ms-python.python-2019.8.30787/out/client/extension.js:75:1013968)
	at Array.forEach (<anonymous>)
	at h.getTestFiles (/home/igor/.vscode-server/extensions/ms-python.python-2019.8.30787/out/client/extension.js:75:1013849)
	at h.parse (/home/igor/.vscode-server/extensions/ms-python.python-2019.8.30787/out/client/extension.js:75:1013661)
	at h.discoverTests (/home/igor/.vscode-server/extensions/ms-python.python-2019.8.30787/out/client/extension.js:75:1012881)
	at process._tickCallback (internal/process/next_tick.js:68:7)

The text was updated successfully, but these errors were encountered:

ericsnowcurrently · 2019-08-29T16:25:49Z

Thanks for letting us know about this, @IBestuzhev. This sounds similar to a problem with the latest version of pytest (#6990), where the XML output changed. We use JUnit XML files to get the output from nose. Was this working on an earlier version of nose? If so then we can look into that further.

Just in case that isn't the problem, I have some questions that may help us figure out what is going on:

what version of nose are you using? (Presumably it is nose2 since nose "1" is effectively EOL.)
what happens when you run the nose directly in the terminal?
what is the command (with args) that you use to do so?
how is the bash script you included above related to running the tests?
what is in your workspace settings.json?

Also note that our support for nose is fairly low, due to lack of demand. However, that doesn't mean we won't try to help you resolve this issue. :)

IBestuzhev · 2019-08-29T17:20:25Z

Thanks @ericsnowcurrently

Yes, I understand it's quite rare issue, the project is legacy and so it uses not recent versions.

Was this working on an earlier version of nose?

Never checked before.

what version of nose are you using?

nose==1.3.7

what happens when you run the nose directly in the terminal?
what is the command (with args) that you use to do so?

That is Django project, so I can't run nosetests command directly. It fails to import most of the tests and they require django.setup() to be called.

The error is AppRegistryNotReady: Apps aren't loaded yet. And I think it's not relevant to this issue.

But I can run python manage.py test, and this command does some django setup and then runs nosetests --verbosity=2 --with-id

So for test discovery I run ./manage.py test --settings=core.local_settings --verbosity=3 -l nose.selector --nologcapture --collect-only

how is the bash script you included above related to running the tests?

It translates nosetests call to python manage.py test command

what is in your workspace settings.json?

"python.testing.cwd": "/home/igor/project/backend",
"python.testing.nosetestPath": "/home/igor/custom_nose",  # custom bash script
"python.testing.promptToConfigure": false,

ericsnowcurrently · 2019-10-07T17:18:08Z

@IBestuzhev, we've added a fix for our JUnit XML parsing. Please see if that fixes the problem. It is available through the "insiders" build of the extension (set the "python.insidersChannel" in your user settings to "daily").

IBestuzhev · 2019-10-08T07:37:35Z

Hi @ericsnowcurrently
Thanks for letting me know.
I downloaded insiders build and run test discovery.

Nothing changed.

Also, my initial problem was with parsing of Nose output, not JUnit output.

ericsnowcurrently · 2019-10-08T17:20:45Z

@IBestuzhev, sorry for the confusion. I was able to reproduce the problem using the steps you originally provided. It looks like we expect a file to be picked if a class is picked, but that isn't the case here. In my nose output I see the following:

...
nose.loader: DEBUG: Load from module <module 'tests' from '/.../gh-7124/tests/__init__.py'>
nose.selector: DEBUG: wantClass <class 'tests.action.ActionTestCase'>? True
...
nose.selector: DEBUG: wantFile /.../gh-7124/tests/action.py? None
...

We'll investigate further.

ericsnowcurrently · 2019-10-08T17:25:37Z

My full nose output:

$ /usr/bin/python3.7 -m nose --collect-only -vvv tests
nose.config: INFO: Ignoring files matching ['^\\.', '^_', '^setup\\.py$']
nose.plugins.manager: DEBUG: Configuring plugins
nose.plugins.manager: DEBUG: Plugins enabled: [<nose.plugins.capture.Capture object at 0x7f09e6b18748>, <nose.plugins.logcapture.LogCapture object at 0x7f09e6bb30b8>, <nose.plugins.deprecated.Deprecated object at 0x7f09e6b4ef60>, <nose.plugins.skip.Skip object at 0x7f09e68ab4e0>, <nose.plugins.collect.CollectOnly object at 0x7f09e68ab5f8>]
nose.core: DEBUG: configured Config(addPaths=True, args=(), configSection='nosetests', debug=None, debugLog=None, env={}, exclude=None, files=[], firstPackageWins=False, getTestCaseNamesCompat=False, ignoreFiles=[re.compile('^\\.'), re.compile('^_'), re.compile('^setup\\.py$')], ignoreFilesDefaultStrings=['^\\.', '^_', '^setup\\.py$'], include=None, includeExe=False, logStream=<_io.TextIOWrapper name='<stderr>' mode='w' encoding='UTF-8'>, loggingConfig=None, options=<Values at 0x7f09e631a208: {'version': False, 'showPlugins': False, 'verbosity': 4, 'files': None, 'where': None, 'py3where': None, 'testMatch': '(?:^|[\\b_\\./-])[Tt]est', 'testNames': None, 'debug': None, 'debugLog': None, 'loggingConfig': None, 'ignoreFiles': [], 'exclude': [], 'include': [], 'stopOnError': False, 'addPaths': True, 'includeExe': False, 'traverseNamespace': False, 'firstPackageWins': False, 'byteCompile': True, 'attr': None, 'eval_attr': None, 'capture': True, 'logcapture': True, 'logcapture_format': '%(name)s: %(levelname)s: %(message)s', 'logcapture_datefmt': None, 'logcapture_filters': None, 'logcapture_clear': False, 'logcapture_level': 'NOTSET', 'enable_plugin_coverage': None, 'cover_packages': None, 'cover_erase': None, 'cover_tests': None, 'cover_min_percentage': None, 'cover_inclusive': None, 'cover_html': None, 'cover_html_dir': 'cover', 'cover_branches': None, 'cover_xml': None, 'cover_xml_file': 'coverage.xml', 'debugBoth': False, 'debugFailures': False, 'debugErrors': False, 'noDeprecated': False, 'enable_plugin_doctest': None, 'doctest_tests': None, 'doctestExtension': None, 'doctest_result_var': None, 'doctestFixtures': None, 'doctestOptions': None, 'enable_plugin_isolation': None, 'detailedErrors': None, 'noSkip': False, 'enable_plugin_id': None, 'testIdFile': '.noseids', 'failed': False, 'multiprocess_workers': 0, 'multiprocess_timeout': 10, 'multiprocess_restartworker': False, 'enable_plugin_xunit': None, 'xunit_file': 'nosetests.xml', 'xunit_testsuite_name': 'nosetests', 'enable_plugin_allmodules': None, 'collect_only': True}>, parser=<optparse.OptionParser object at 0x7f09e837e7b8>, parserClass=<class 'optparse.OptionParser'>, plugins=<nose.plugins.manager.DefaultPluginManager object at 0x7f09e837e3c8>, py3where=(), runOnInit=True, srcDirs=('lib', 'src'), stopOnError=False, stream=<_io.TextIOWrapper name='<stderr>' mode='w' encoding='UTF-8'>, testMatch=re.compile('(?:^|[\\b_\\./-])[Tt]est'), testMatchPat='(?:^|[\\b_\\./-])[Tt]est', testNames=['tests'], traverseNamespace=False, verbosity=4, where=(), worker=False, workingDir='/.../gh-7124')
nose.importer: DEBUG: Add path /.../gh-7124
nose.plugins.collect: DEBUG: Preparing test loader
nose.core: DEBUG: test loader is <nose.loader.TestLoader object at 0x7f09e631a358>
nose.core: DEBUG: defaultTest .
nose.core: DEBUG: Test names are ['tests']
nose.core: DEBUG: createTests called with None
nose.loader: DEBUG: load from tests (None)
nose.selector: DEBUG: Test name tests resolved to file tests, module None, call None
nose.selector: DEBUG: Final resolution of test name tests: file /.../gh-7124/tests module tests call None
nose.importer: DEBUG: Import tests from /.../gh-7124
nose.importer: DEBUG: Add path /.../gh-7124
nose.importer: DEBUG: find module part tests (tests) in ['/.../gh-7124']
nose.loader: DEBUG: Load from module <module 'tests' from '/.../gh-7124/tests/__init__.py'>
nose.selector: DEBUG: wantClass <class 'tests.action.ActionTestCase'>? True
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.addCleanup>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.addTypeEqualityFunc>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.assertAlmostEqual>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.deprecated_func>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.assertCountEqual>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.assertDictContainsSubset>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.assertDictEqual>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.assertEqual>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.deprecated_func>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.assertFalse>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.assertGreater>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.assertGreaterEqual>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.assertIn>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.assertIs>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.assertIsInstance>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.assertIsNone>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.assertIsNot>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.assertIsNotNone>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.assertLess>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.assertLessEqual>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.assertListEqual>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.assertLogs>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.assertMultiLineEqual>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.assertNotAlmostEqual>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.deprecated_func>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.assertNotEqual>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.deprecated_func>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.assertNotIn>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.assertNotIsInstance>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.assertNotRegex>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.deprecated_func>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.assertRaises>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.assertRaisesRegex>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.deprecated_func>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.assertRegex>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.deprecated_func>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.assertSequenceEqual>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.assertSetEqual>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.assertTrue>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.assertTupleEqual>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.assertWarns>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.assertWarnsRegex>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.deprecated_func>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.countTestCases>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.debug>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.defaultTestResult>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.doCleanups>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.fail>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.deprecated_func>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.deprecated_func>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.deprecated_func>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.deprecated_func>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.deprecated_func>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.deprecated_func>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.deprecated_func>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.id>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.run>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.setUp>? None
nose.selector: DEBUG: wantMethod <bound method TestCase.setUpClass of <class 'tests.transplant_class.<locals>.C'>>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.shortDescription>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.skipTest>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.subTest>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.tearDown>? None
nose.selector: DEBUG: wantMethod <bound method TestCase.tearDownClass of <class 'tests.transplant_class.<locals>.C'>>? None
nose.plugins.collect: DEBUG: TestSuite(<map object at 0x7f09e631a9b0>)
nose.loader: DEBUG: Load tests from module path /.../gh-7124/tests?
nose.loader: DEBUG: path: /.../gh-7124/tests os.path.realpath(/.../gh-7124/tests): /.../gh-7124/tests
nose.loader: DEBUG: load from dir /.../gh-7124/tests
nose.importer: DEBUG: Add path /.../gh-7124/tests
nose.importer: DEBUG: Add path /.../gh-7124
nose.selector: DEBUG: __init__.py matches ignoreFiles pattern; skipped
nose.selector: DEBUG: wantFile /.../gh-7124/tests/action.py? None
nose.plugins.collect: DEBUG: TestSuite(<nose.suite.ContextList object at 0x7f09e631a9b0>)
nose.plugins.collect: DEBUG: Add test <nose.plugins.collect.TestSuite tests=[]>
nose.plugins.collect: DEBUG: TestSuite([<nose.plugins.collect.TestSuite tests=[<nose.plugins.collect.TestSuite tests=[]>]>])
nose.plugins.collect: DEBUG: Add test <nose.plugins.collect.TestSuite tests=[<nose.plugins.collect.TestSuite tests=[]>]>
nose.core: DEBUG: runTests called

----------------------------------------------------------------------
Ran 0 tests in 0.000s

OK

kainjow · 2020-05-13T22:00:07Z

I'm experiencing a similar (or same) issue. Also using nose 1.3.7, but on Python 3.7, macOS 10.15, and VSCode 1.45.0.

@IBestuzhev your shell script idea worked for me, although I modified it to work on macOS. Thanks!

karthiknadig · 2021-07-29T21:24:47Z

Closing via #16371

IBestuzhev added triage-needed Needs assignment to the proper sub-team bug Issue identified by VS Code Team member as probable bug labels Aug 28, 2019

kimadeline assigned ericsnowcurrently Aug 28, 2019

kimadeline added the triage label Aug 28, 2019

ghost removed the triage-needed Needs assignment to the proper sub-team label Aug 28, 2019

ericsnowcurrently added the info-needed Issue requires more information from poster label Aug 29, 2019

ericsnowcurrently removed the info-needed Issue requires more information from poster label Aug 29, 2019

luabud mentioned this issue Sep 25, 2019

[meta] Use conda run for running tools #7425

Closed

ericsnowcurrently added the info-needed Issue requires more information from poster label Oct 7, 2019

ericsnowcurrently removed the info-needed Issue requires more information from poster label Oct 8, 2019

ericsnowcurrently removed their assignment Oct 8, 2019

ericsnowcurrently added area-testing needs PR and removed triage labels Oct 8, 2019

karthiknadig closed this as completed Jul 29, 2021

github-actions bot locked as resolved and limited conversation to collaborators Sep 17, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Nose Tests discovery fails when tests are split by folders and imported to init.py (wantModule and wantDirectory are not parsed) #7124

Nose Tests discovery fails when tests are split by folders and imported to init.py (wantModule and wantDirectory are not parsed) #7124

IBestuzhev commented Aug 28, 2019

ericsnowcurrently commented Aug 29, 2019

Uh oh!

IBestuzhev commented Aug 29, 2019

Uh oh!

ericsnowcurrently commented Oct 7, 2019

Uh oh!

IBestuzhev commented Oct 8, 2019

Uh oh!

ericsnowcurrently commented Oct 8, 2019 •

edited

Loading

Uh oh!

ericsnowcurrently commented Oct 8, 2019

Uh oh!

kainjow commented May 13, 2020

Uh oh!

karthiknadig commented Jul 29, 2021

Uh oh!

Nose Tests discovery fails when tests are split by folders and imported to __init__.py (wantModule and wantDirectory are not parsed) #7124

Nose Tests discovery fails when tests are split by folders and imported to __init__.py (wantModule and wantDirectory are not parsed) #7124

Comments

IBestuzhev commented Aug 28, 2019

Environment data

Expected behaviour

Actual behaviour

Steps to reproduce:

Logs

ericsnowcurrently commented Aug 29, 2019

Uh oh!

IBestuzhev commented Aug 29, 2019

Uh oh!

ericsnowcurrently commented Oct 7, 2019

Uh oh!

IBestuzhev commented Oct 8, 2019

Uh oh!

ericsnowcurrently commented Oct 8, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ericsnowcurrently commented Oct 8, 2019

Uh oh!

kainjow commented May 13, 2020

Uh oh!

karthiknadig commented Jul 29, 2021

Uh oh!

Nose Tests discovery fails when tests are split by folders and imported to init.py (wantModule and wantDirectory are not parsed) #7124

Nose Tests discovery fails when tests are split by folders and imported to init.py (wantModule and wantDirectory are not parsed) #7124

ericsnowcurrently commented Oct 8, 2019 •

edited

Loading