Test Discovery Fails - Reports as PyTest not installed

[drcrook1]

Testing was working yesterday; I signed off around 4:45pm EST, got up this morning and encountered issues discovering and importing the packages from vscode. To me it appears as though the root of the workspace is not being added to the path.

Example: executing python -m pytest ETC from c:/projects/customers/CUSTOMERX/PROJECTX/ adds that to the path, specifying ./tests/ searches ./tests for tests and I can import the high level package from there. I feel as if the break is likely in this area.

Environment data

• VS Code version: XXX
  Version: 1.35.1 (user setup)
  Commit: c7d83e57cd18f18026a8162d042843bda1bcf21f
  Date: 2019-06-12T14:30:02.622Z
  Electron: 3.1.8
  Chrome: 66.0.3359.181
  Node.js: 10.2.0
  V8: 6.6.346.32
  OS: Windows_NT x64 10.0.17134

• Extension version (available under the Extensions sidebar): 2019.6.22090 (25 June 2019)

• Python version (& distribution if applicable, e.g. Anaconda): Anaconda 2019.03 for Windows python version 3.6.8

• Type of virtual environment used (N/A | venv | virtualenv | conda | ...): N/A base

• Relevant/affected Python packages and their versions: pytest

• Jedi or Language Server? (i.e. what is "python.jediEnabled" set to; more info How to update the language server to the latest stable version #3977): Jedi enabled (output indicates it started jedi python language engine)

Expected behaviour

Discover Unit Tests - for example below works:

c:\projects\customers\CUSTOMERX\PROJECTX>python -m pytest --doctest-modules --junitxml=./tests/junit/test-results.xml --cov=./PACKAGEX/ ./tests/ --cov-report=xml:./tests/junit/coverage.xml --cov-report=html:./tests/junit/cov_html
================================================= test session starts =================================================
platform win32 -- Python 3.6.8, pytest-4.6.2, py-1.8.0, pluggy-0.12.0
rootdir: c:\projects\customers\CUSTOMERX\PACKAGEX, inifile: setup.cfg
plugins: arraydiff-0.3, cov-2.7.1, doctestplus-0.3.0, openfiles-0.3.2, remotedata-0.3.1
collected 14 items

tests\test_system_files.py ..                                                                                    [ 14%]

Actual behaviour

From the python test log (outputs) - Tried pytest 4.6.2, 4.6.3 and 4.4.1

python C:\Users\DrCrook\.vscode\extensions\ms-python.python-2019.6.22090\pythonFiles\testing_tools\run_adapter.py discover pytest -- -s --cache-clear c:\projects\customers\CUSTOMERX\PROJECTX
Test Discovery failed: 
Error: ============================= test session starts =============================
platform win32 -- Python 3.6.8, pytest-4.4.1, py-1.8.0, pluggy-0.12.0
rootdir: c:\projects\customers\CUSTOMERX\PROJECTX, inifile: setup.cfg
plugins: arraydiff-0.3, cov-2.7.1, doctestplus-0.3.0, openfiles-0.3.2, remotedata-0.3.1
collected 0 items / 9 errors

=================================== ERRORS ====================================
_________________ ERROR collecting tests/test_system_files.py _________________
ImportError while importing test module 'c:\projects\customers\CUSTOMERX\PROJECTX\tests\test_system_files.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
tests\test_system_files.py:6: in <module>
    from PACKAGEX.result import Result
E   ModuleNotFoundError: No module named 'PACKAGEX'

My folder structure:

/root
   /PACKAGEX
        /submoduleX
        /submoduleY
   /tests
        /helpers
        /test_submoduleX
        /test_submoduleY

.vscode/settings.json

{
    "python.testing.pytestArgs": [
    "--verbose",
    "--rootdir",
    "${workspaceFolder}",
    "${workspaceFolder}/tests"
    ],
    "python.testing.cwd": "${workspaceFolder}",
    "python.testing.unittestEnabled": false,
    "python.testing.nosetestsEnabled": false,
    "python.testing.pytestEnabled": true,
    "python.pythonPath": "C:\\ProgramData\\Anaconda3\\python.exe"
   }

[drcrook1]

Additionally, this is new and maybe related. Ctrl + Shift + P -> Python: Configure Tests yeilds:

could be related:
#5619

[drcrook1]

Verified my alternate work computer has the same problem. All machines I have across all python projects are failing on this as of today.

[bchamand]

I'm in the same situation of you except I'm in MacOS 10.14.5. I directly downgrade the version of the extension to go back to the previous version.

[jahan01]

+1, I am on Mac 10.14, presumably started after last update of python extension

[lukasz-migas]

Same problem here. Downgrading to the previous version (2019.5.18875) fixes the problem.

[drcrook1]

downgrading the extension works for me as well. version (2019.5.18875)

[kimadeline]

Hello @drcrook1 , @bchamand , @jahan01 and @lukasz-migas 👋 thank you for reporting this issue and confirming that downgrading the extension works. If you could verify the following while I comb through the extension that'd be great 🔎:

• you said that python -m pytest ./tests works, can you try running pytest ./tests/?
• are there any __init__.py files in your test folders? Something that would look like:

root/
   mypackage/
      __init__.py
      submoduleA/
         __init__.py
         fileA.py
      submoduleB/
         __init__.py
         fileB.py
   tests/
      __init__.py
      test_subA/
         __init__.py
         test_A.py
      test_subB/
         __init__.py
         test_B.py

[lukasz-migas]

@kimadeline python -m pytest works fine but pytest ./tests does not. My project is structured mostly like that although I keep the tests folders within each submodule. Each folder and subfolder has its own __init__.py.

[kimadeline]

The failing discovery command is pytest -s --cache-clear --collect-only /path/to/root/folder being run in the root folder.

We changed how pytest discovers tests in 2019.6.22090:
✋ We used to parse some of pytest's output to know which files contained tests, and then loop through these files one by one. However, it was unreliable and discouraged.
👉 Now we leave the entire discovery up to pytest.

Assuming you have this structure (notice the absence of __init__.py files in tests/ and its subfolders):

root/
   mypackage/
      __init__.py
      submoduleA/
         __init__.py
         fileA.py
      submoduleB/
         __init__.py
         fileB.py
   tests/
      test_subA/
         test_A.py
      test_subB/
         test_B.py

There is a difference in running python -m pytest [...] and just pytest [...], because the former will add the current directory to sys.path, which in return will make mypackage importable, while the latter doesn't. Because we are now executing a script to call pytest with our own discovery plug-in, we are no longer able to use python -m pytest for that implicit change to sys.path.

By adding __init__.py files to your tests folders, pytest will import them as packages, add the root directory to sys.path, and make mypackage visible.

In other words because we are doing the "right" thing by directly plugging into pytest to do discovery, pytest's sys.path is different and so it's resulting in pytest finding things differently from how we used to do it (although it's the same as if you had specified python.testing.pytestPath). If people specifically would like us to consider adding '.' to sys.path, please feel free to open a new issue for the feature request (and people can 👍 it to show that they support the idea).

[ericsnowcurrently]

FWIW, it is a bit more correct to say the command is <python> -m pytest ... (for whatever Python binary your workspace is set to use).

[ericsnowcurrently]

Hmm, maybe not so much. We should be setting $PYTHONPATH for the script exec. The old code was invoking pytest with python -m pytest ..., but now we are running the adapter script as python run_adapter.py.

[ericsnowcurrently]

The solution will be to fix run_adapter.py to not replace sys.path[0], since that blows away the value set by the interpreter: the workspace folder. However, we still need that new value in sys.path, so we should instead insert it at position 1.

[KsenoJiva]

I noticed that the TestExplorer shows all Paths as lowercase. Therefore it could not find folders and modules within. In your case PACKAGEX might be uppercase too. Might there be a connection?

[DonJayamanne]

Dup of #5919