Test discovery fails if package is not installed

[RobTheFiveNine]

Environment data

• VS Code version: 1.42.1
• Extension version (available under the Extensions sidebar): 2020.2.64397
• OS and version: Ubuntu 18.04
• Python version (& distribution if applicable, e.g. Anaconda): 3.6.7
• Type of virtual environment used (N/A | venv | virtualenv | conda | ...): venv
• Relevant/affected Python packages and their versions: N/A
• Relevant/affected Python-related VS Code extensions and their versions: N/A
• Jedi or Language Server? (i.e. what is "python.jediEnabled" set to; more info How to update the language server to the latest stable version #3977): false
• Value of the python.languageServer setting: Microsoft

Expected behaviour

When launching vscode, the tests from the tests/ directory should be discovered successfully; regardless of whether the project has been installed as a package.

Actual behaviour

Import errors are thrown out in the output tab and test discovery fails.

Example output:

python <path_to_home>/.vscode/extensions/ms-python.python-2020.2.64397/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir <path_to_project> -s --cache-clear -o junit_family=xunit1
Test Discovery failed: 
Error: ============================= test session starts ==============================
platform linux -- Python 3.6.7, pytest-5.3.5, py-1.8.1, pluggy-0.13.1
rootdir: <path_to_project>
collected 0 items / 4 errors

==================================== ERRORS ====================================
__________________ ERROR collecting <path_to_test_file> ___________________
ImportError while importing test module '<path_to_test_file> '.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
<path_to_test_file>:5: in <module>
    import <module_within_project>
E   ImportError: cannot import name '<module_name>'

Running pytest separately, with the same launch arguments works fine in a terminal window, but when executed using vscode's test discovery function, it fails to import unless the project is installed as a package first (i.e. by running python setup.py install --record record.txt)

Steps to reproduce:

1. Clone this sample repository: https://github.com/RobTheFiveNine/vscode-python-bug
2. Create a venv: virtualenv venv -p python3
3. Activate it: source venv/bin/activate
4. Install pytest: pip install pytest
5. In a terminal, to verify pytest works, run python -m pytest in the project directory
6. To verify it doesn't work in vscode, open vscode and configure the Python extension to use pytest, see that test discovery fails
7. Open the output tab and view the Python Test Log section to see the errors
8. After seeing it fail, run python setup.py install --record record.txt to install the project as a package in the venv, and then click on the tests label in the status bar or relaunch vscode, to find that vscode now discovers the tests successfully

Logs

python <path_to_home>/.vscode/extensions/ms-python.python-2020.2.64397/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir <path_to_project> -s --cache-clear -o junit_family=xunit1
Test Discovery failed: 
Error: ============================= test session starts ==============================
platform linux -- Python 3.6.7, pytest-5.3.5, py-1.8.1, pluggy-0.13.1
rootdir: <path_to_project>
collected 0 items / 4 errors

==================================== ERRORS ====================================
__________________ ERROR collecting <path_to_test_file> ___________________
ImportError while importing test module '<path_to_test_file> '.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
<path_to_test_file>:5: in <module>
    import <module_within_project>
E   ImportError: cannot import name '<module_name>'

[brettcannon]

Unfortunately this is a shortcoming/design of pytest as it imports code to do its test discovery and we need pytest to tell us what tests there are and where those tests are. So for us to do this would require pytest updating their test discovery to not require importing code to detect tests.

[RobTheFiveNine]

Running pytest on its own doesn't result in the same error, it's only when executed via the vscode extension. For example, if you run python -m pytest --co in the example directory, the output looks like this:

============================= test session starts ==============================
platform linux -- Python 3.6.9, pytest-5.3.5, py-1.8.1, pluggy-0.13.1
rootdir: <path_to_project>
collected 1 item                                                               
<Module tests/test_bar.py>
  <UnitTestCase TestFoo>
    <TestCaseFunction test_foo>

============================ no tests ran in 0.01s =============================

[brettcannon]

@RobTheFiveNine we will have a look!

[kimadeline]

Hi @RobTheFiveNine , thank you for reaching out.

I was able to replicate your issue with the sample repo and steps you provided, so thank you for that as well, they were very helpful!

👉 Instead of having to install the project as a package, adding an __init__.py file to the tests folder allowed me to run test discovery without issues, let me know if that works for you too.

📋 Notes

When running python ~/.vscode/extensions/ms-python.python-2020.2.64397/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir <path/to>/vscode-python-bug -s --cache-clear tests (the command run by PVSC), the equivalent command (that we display) is python -m pytest --collect-only --rootdir <path/to>/vscode-python-bug -s --cache-clear tests.

Running the PVSC command fails, but running the equivalent command works:

Adding __init__.py to the source folder didn't fix test discovery, but adding __init__.py to the test directory did.

@brettcannon something to do about how run_adapter.py sees the test directory as a top-level package?

[RobTheFiveNine]

@kimadeline just tried your suggestion of adding tests/__init__.py and it seemed to fix the discovery in vscode!

[brettcannon]

@kimadeline it possibly does, but it might also just be pytest since our discovery is run as a plug-in to pytest while also launching it. Question is whether we consider it a bug or more of a feature enhancement.

[kimadeline]

Fair enough, I'll mark it as an enhancement and "needs decision" for now.

[mahmoudajawad]

I'm facing the same issue. I have this structure:

root
|-__init__.py
...
|-tests
  |-__init__.py
  |-unit
    |-__init__.py
    |- test_X
       |-__init__.py
       |-test_X_Y.py
       |-test_X_Z.py
...

So, I have __init__ file across all the path. However, it seems vscode is getting pytest to collect tests at folder level, that's why top-level modules are not available for importing, thus failing. What I need to understand is if there's a way to update vscode discovery strategy to depend on regular pytest attempt to run all tests (even if some fails) and then collect the tests from that. Yes, this means tests discovery might take forever because it will run all the tests, but at least this can be a quick hack to test locally until your team comes with better strategy.

Another option might be capturing the output of pytest discovery command and listing the tests even if they fail to be imported at discovery time. Am I making sense? I have to ask as I'm not entirely aware of how you are reading pytest or any other test framework output.

[brettcannon]

@mahmoudajawad we aren't reading pytest output, actually. We inject a pytest plug-in that gets run as part of discovery and use that to do the recording of what tests exist.

[mahmoudajawad]

@brettcannon, I see. That makes sense. I'll try to look on what's going on and see if I can patch it to make it record tests even if pytest fail.

[mahmoudajawad]

UPDATE: nvm, I found at it:

https://github.com/microsoft/vscode-python/blob/649156a09ccdc51c0d20f7cd44540f1918f9347b/pythonFiles/testing_tools/adapter/pytest/_discovery.py

@brettcannon, can you direct me to where is the plugin that is being injected? I couldn't figure it out myself through the quick scan of the structure being offered in this extension.

[brettcannon]

@mahmoudajawad the Python code is in https://github.com/microsoft/vscode-python/tree/master/pythonFiles/testing_tools, the TS code is in https://github.com/microsoft/vscode-python/tree/master/src/client/testing/pytest

[eleanorjboyd]

dup of #13301 which has just been fixed and is insiders.