Tests from mypy/test/teststubtest.py fail without pytest-xdist or with v2.3.0

[sbraz]

Bug Report

Hi,
I have noticed that tests frommypy/test/teststubtest.py fail with pytest-xdist 2.3.0. This is the faulty commit:
pytest-dev/pytest-xdist@56e631b

It is especially problematic because they also fail without pytest-xdist, so I am not sure the bug is actually pytest-xdist's fault.

I understand that you have pinned an older version of pytest-xdist but some day you might still to have to deal with this issue.

To Reproduce

Here is an example Dockerfile which you can edit to test the faulty commit and its working parent. Getting rid of pytest-xdist does not fix the issue either.

FROM python:3.9
RUN git clone --depth 1 https://github.com/python/mypy
WORKDIR mypy
RUN pip install -r test-requirements.txt

# This commit is fine
#RUN pip install -U git+https://github.com/pytest-dev/pytest-xdist@7f2426b11d2c7890e5637fdaa3aa1c75b4f31758

# This one is broken
RUN pip install -U git+https://github.com/pytest-dev/pytest-xdist@56e631b4bbeedf7ef6da94d3c5eab00c20a07d12

# Disabling pytest-xdist makes tests fail too
#RUN pip uninstall -y pytest-xdist
#RUN pytest -o addopts= -vv mypy/test/teststubtest.py

RUN pytest -vv mypy/test/teststubtest.py

Expected Behavior

Tests should pass.

Actual Behavior

Tests fail because of a sys.path problem, e.g.:

_________________________ StubtestMiscUnit.test_output _________________________ 
[gw2] linux -- Python 3.9.6 /usr/local/bin/python                                                        
                                                    
self = <mypy.test.teststubtest.StubtestMiscUnit testMethod=test_output>
                                                    
    def test_output(self) -> None:
        output = run_stubtest(
            stub="def bad(number: int, text: str) -> None: ...",
            runtime="def bad(num, text): pass",
            options=[],
        )    
        expected = (
            'error: {0}.bad is inconsistent, stub argument "number" differs from runtime '
            'argument "num"\nStub: at line 1\ndef (number: builtins.int, text: builtins.str)\n'
            "Runtime: at line 1 in file {0}.py\ndef (num, text)\n\n".format(TEST_MODULE_NAME)
        )                                                                                                
>       assert remove_color_code(output) == expected                            
E       AssertionError: assert ("error: test_module failed to import: No module named 'test_module'\n"\n 'Stub: at line 1\n'\n 'MypyFile:1(\n'\n '  test_module.pyi)\n'\n 'Runtime:\n'\n 'MISSING\n'\n '\n') == ('e
rror: test_module.bad is inconsistent, stub argument "number" differs from '\n 'runtime argument "num"\n'\n 'Stub: at line 1\n'\n 'def (number: builtins.int, text: builtins.str)\n'\n 'Runtime: at line 1 in file 
test_module.py\n'\n 'def (num, text)\n'\n '\n')                                                          
E         - error: test_module.bad is inconsistent, stub argument "number" differs from runtime argument "num"
E         + error: test_module failed to import: No module named 'test_module'  
E           Stub: at line 1                                                                              
E         - def (number: builtins.int, text: builtins.str)                      
E         - Runtime: at line 1 in file test_module.py                           
E         - def (num, text)                                                                              
E         + MypyFile:1(                                                                                  
E         +   test_module.pyi)                                                                           
E         + Runtime:                                                                                     
E         + MISSING                                                                                      
                                                                                                         
mypy/test/teststubtest.py:754: AssertionError                                                            

Your Environment

• Mypy version used: git master
• Mypy command-line flags: N/A
• Mypy configuration options from mypy.ini (and other config files): N/A
• Python version used: 3.9.6
• Operating system and version: Debian Bullseye