Pytest Adapter not compatible with xdist_group marker

[skilkis]

Issue Type: Bug

Behaviour

Thanks for your continued efforts on developing VS-Code Python! I recently ran into a bug with the pytest-xdist package with the VS Code test runner. Test result display fails when the xdist_group marker is used to control grouping of tests into a specific worker. See screenshots below for expected vs. actual behavior. Tests are collected and run successfully as per the Python Test Log:

============================= test session starts =============================
platform win32 -- Python 3.9.13, pytest-7.1.2, pluggy-1.0.0
rootdir: c:\Users\MDRyzenPC\Python\vscode_pytest_bug
plugins: forked-1.4.0, xdist-2.5.0
gw0 I / gw1 I / gw2 I / gw3 I / gw4 I / gw5 I / gw6 I / gw7 I / gw8 I / gw9 I / gw10 I / gw11 I / gw12 I / gw13 I / gw14 I / gw15 I
gw0 [3] / gw1 [3] / gw2 [3] / gw3 [3] / gw4 [3] / gw5 [3] / gw6 [3] / gw7 [3] / gw8 [3] / gw9 [3] / gw10 [3] / gw11 [3] / gw12 [3] / gw13 [3] / gw14 [3] / gw15 [3]

...                                                                      [100%]
- generated xml file: C:\Users\MDRYZE~1\AppData\Local\Temp\tmp-19948B3djAfVroxql.xml -
============================== 3 passed in 1.53s ==============================

Expected vs. Actual

Steps to reproduce:

1. Create the test following test file:

import pytest

def test_foo():
    pass

@pytest.mark.xdist_group("groupA")
def test_bar():
    pass

@pytest.mark.xdist_group("groupB")
def test_spam():
    pass

2. Run tests from Test Explorer using the following settings.json to observe expected behavior:

{
  "python.testing.pytestArgs": ["-n", "auto", "--dist", "load"]
}

2. Run tests from Test Explorer using the following settings.json to observe actual behavior:

{
  "python.testing.pytestArgs": ["-n", "auto", "--dist", "loadgroup"]
}

Diagnostic data

• Python version (& distribution if applicable, e.g. Anaconda): 3.9.13
• Type of virtual environment used (e.g. conda, venv, virtualenv, etc.): Venv
• Value of the python.languageServer setting: Pylance
• pytest-xdist: 2.5.0
• pytest: 7.1.2

Output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)

> .\.env\Scripts\python.exe ~\.vscode\extensions\ms-python.python-2022.9.11781004\pythonFiles\testing_tools\run_adapter.py discover pytest -- --rootdir . -s --cache-clear -n auto --dist loadgroup
cwd: .
> .\.env\Scripts\python.exe -m pytest --rootdir . --override-ini junit_family=xunit1 --junit-xml=C:\Users\MDRYZE~1\AppData\Local\Temp\tmp-19948tlPbJ3ncE8yY.xml -n auto --dist loadgroup
cwd: .

User Settings

languageServer: "Pylance"

linting
• flake8Enabled: true
• mypyEnabled: true

formatting
• provider: "black"

testing
• pytestEnabled: true

Extension version: 2022.9.11781004
VS Code version: Code 1.68.1 (30d9c6cd9483b2cc586687151bcbcd635f373630, 2022-06-14T12:48:58.283Z)
OS version: Windows_NT x64 10.0.19043
Restricted Mode: No

System Info

Item	Value
CPUs	AMD Ryzen 7 3700X 8-Core Processor (16 x 3593)
GPU Status	2d_canvas: enabled canvas_oop_rasterization: disabled_off direct_rendering_display_compositor: disabled_off_ok gpu_compositing: enabled multiple_raster_threads: enabled_on oop_rasterization: enabled opengl: enabled_on rasterization: enabled raw_draw: disabled_off_ok skia_renderer: enabled_on video_decode: enabled video_encode: enabled vulkan: disabled_off webgl: enabled webgl2: enabled
Load (avg)	undefined
Memory (System)	31.93GB (21.69GB free)
Process Argv	--folder-uri file:///c%3A/users/mdryzenpc/python/parapy --crash-reporter-id 67dc880c-0ae9-4a1f-a9ba-8e9e66182f1d
Screen Reader	no
VM	0%

A/B Experiments

vsliv368cf:30146710
vsreu685:30147344
python383cf:30185419
vspor879:30202332
vspor708:30202333
vspor363:30204092
vswsl492cf:30256860
vslsvsres303:30308271
pythonvspyl392:30443607
vserr242:30382549
pythontb:30283811
vsjup518:30340749
pythonvspyt551:30345470
pythonptprofiler:30281270
vshan820:30294714
vstes263:30335439
pythondataviewer:30285071
vscod805cf:30301675
binariesv615:30325510
bridge0708:30335490
bridge0723:30353136
vsaa593:30376534
vsc1dst:30438360
pythonvs932:30410667
wslgetstarted:30449410
vscscmwlcmt:30465135
cppdebug:30492333
vsclangdc:30486549

[karthiknadig]

Duplicate of #16852

The way we work with pytest currently makes it incompatible with several plugins. We are currently looking into a generic solution so that our controller does not get in the way of pytest doing its thing.

[skilkis]

@karthiknadig great to hear that there is on-going work to bring a general solution for plugins! Just to be clear since the description I gave above may be ambiguous, this issue is not about test-discovery which the posts in #16852 seems to be focused on. The tests discover and run just fine in this case but the collection of test-results is where the bug occurs 😊 Thanks again for your hard work!