"Compilation" error does not report tests in test tree

[zackarydev]

Environment data

• VS Code version: 1.52.1
• Extension version (available under the Extensions sidebar): 2021.1.502429796
• OS and version: Darwin x64 20.3.0
• Python version (& distribution if applicable, e.g. Anaconda): 3.6
• Type of virtual environment used (N/A | venv | virtualenv | conda | ...): venv
• Relevant/affected Python-related VS Code extensions and their versions: Pylance (2021.1.2)
• Value of the python.languageServer setting: Pylance

Expected behaviour

When a test cannot be imported due to code failures (ex: invalid syntax, import errors) I expect the test to be reported in the test view. Its super important to make these errors visible... There are a bunch of tests that contain error and just failed
"silently".

Actual behaviour

The test fails silently and the test is not reported in the test treeview. (it is reported in the Python Test Log under output though)

Steps to reproduce:

Create a file:

# tests/helpers.py
def get_constants_from_class(class): # <-- invalid syntax here 'class' is a reserved keyword!
    # https://stackoverflow.com/questions/1398022/looping-over-all-member-variables-of-a-class-in-python
    get_consts = lambda x: [attr for attr in dir(x) if not callable(getattr(x, attr)) and not attr.startswith("__")]
    return get_consts(class)

Create a test that imports this function:

# tests/unit_tests/test_has_all_constants.py
import unittest

from tests.helpers import get_constants_from_class

Class TestHasAllConstants(unittest.TestCase):
    def test_has_a_const(self):
           get_constants_from_class(SomeClass)

Run tests using the test view:

Notice that the class is not identified.

Look at the Python logs:

======================================================================
ERROR: some.module (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: some.module
Traceback (most recent call last):
  File "/opt/python3.6/lib/python3.6/unittest/loader.py", line 428, in _find_test_path
    module = self._get_module_from_name(name)
  File "/opt/python3.6/lib/python3.6/unittest/loader.py", line 369, in _get_module_from_name
    __import__(name)
  File "/some/file.path.py", line 4, in <module>
    from tests.helpers import get_constants_from_class
  File "/home/vmagent/app/tests/helpers.py", line 17
    def get_constants_from_class(class: Any):
                                     ^
SyntaxError: invalid syntax

Logs

Output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)

======================================================================
ERROR: some.module (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: some.module
Traceback (most recent call last):
  File "/opt/python3.6/lib/python3.6/unittest/loader.py", line 428, in _find_test_path
    module = self._get_module_from_name(name)
  File "/opt/python3.6/lib/python3.6/unittest/loader.py", line 369, in _get_module_from_name
    __import__(name)
  File "/some/file.path.py", line 4, in <module>
    from tests.helpers import get_constants_from_class
  File "/home/vmagent/app/tests/helpers.py", line 17
    def get_constants_from_class(class: Any):
                                     ^
SyntaxError: invalid syntax

[zackarydev]

This might? be related: #15147

[karthiknadig]

The PR will partially address this #15129. To actually report this in the test tree, that is a new feature. We have marked this issue as "needs decision" to make sure we have a conversation about your idea. We plan to leave this feature request open for at least a month to see how many 👍 votes the opening comment gets to help us make our decision.

[eleanorjboyd]

Hello! Just fixed this in our new python rewrite. Error tolerant discovery is now out on insiders and will go out on stable in the next release. It is part of our testing rewrite and so if you would like to try it yourself, you need to add this setting to your users settings.json "python.experiments.optInto": ["pythonTestAdapter"] on insiders. Let me know if it doesn’t work for you and we can re-open this issue. Thanks!