xdist race condition on node restart

[RonnyPfannschmidt]

it seems we have a order invariant that very rarely happens on node restart

to even see the issue creep up i have to do

while tox --skip-missing-interpreters  -- -k test_each_multiple
do
  sleep .1
done

the result is an action with something we don't have a accounted for yet

py27-pytest27-pexpect runtests: commands[1] | py.test -k test_each_multiple
============================================================================ test session starts ============================================================================
platform linux2 -- Python 2.7.10 -- py-1.4.30 -- pytest-2.7.2
rootdir: /home/private/Projects/pytest-dev/pytest-xdist, inifile: tox.ini
plugins: xdist, xdist, xdist
collected 98 items 

acceptance_test.py F

================================================================================= FAILURES ==================================================================================
____________________________________________________________________ TestNodeFailure.test_each_multiple _____________________________________________________________________

self = <acceptance_test.TestNodeFailure instance at 0x7f59903c0cb0>, testdir = <TmpTestdir local('/tmp/pytest-774/testdir/test_each_multiple0')>

    def test_each_multiple(self, testdir):
        f = testdir.makepyfile("""
                import os
                def test_a(): os._exit(1)
                def test_b(): pass
            """)
        res = testdir.runpytest(f, '--dist=each', '--tx=2*popen')
        res.stdout.fnmatch_lines([
            "*Replacing crashed slave*",
            "*Slave*crashed while running*",
>           "*2 failed*2 passed*",
        ])
E       Failed: remains unmatched: '*Slave*crashed while running*', see stderr

/home/private/Projects/pytest-dev/pytest-xdist/testing/acceptance_test.py:600: Failed
--------------------------------------------------------------------------- Captured stdout call ----------------------------------------------------------------------------
running ['/home/private/Projects/pytest-dev/pytest-xdist/.tox/py27-pytest27-pexpect/bin/python2.7', '/home/private/Projects/pytest-dev/pytest-xdist/.tox/py27-pytest27-pexpect/lib/python2.7/site-packages/pytest.py', '--basetemp=/tmp/pytest-774/testdir/test_each_multiple0/runpytest-0', '/tmp/pytest-774/testdir/test_each_multiple0/test_each_multiple.py', '--dist=each', '--tx=2*popen'] curdir= /tmp/pytest-774/testdir/test_each_multiple0
============================= test session starts ==============================
platform linux2 -- Python 2.7.10 -- py-1.4.30 -- pytest-2.7.2
rootdir: /tmp/pytest-774/testdir/test_each_multiple0, inifile: 
plugins: xdist, xdist, xdist
gw0 I / gw1 I
gw0 [2] / gw1 [2]

scheduling tests via EachScheduling
[gw0] node down: Not properly terminated
fReplacing crashed slave gw0
[gw1] node down: Not properly terminated
fReplacing crashed slave gw1
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/home/private/Projects/pytest-dev/pytest-xdist/.tox/py27-pytest27-pexpect/lib/python2.7/site-packages/_pytest/main.py", line 84, in wrap_session
INTERNALERROR>     doit(config, session)
INTERNALERROR>   File "/home/private/Projects/pytest-dev/pytest-xdist/.tox/py27-pytest27-pexpect/lib/python2.7/site-packages/_pytest/main.py", line 122, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/home/private/Projects/pytest-dev/pytest-xdist/.tox/py27-pytest27-pexpect/lib/python2.7/site-packages/_pytest/core.py", line 521, in __call__
INTERNALERROR>     return self._docall(self.methods, kwargs)
INTERNALERROR>   File "/home/private/Projects/pytest-dev/pytest-xdist/.tox/py27-pytest27-pexpect/lib/python2.7/site-packages/_pytest/core.py", line 528, in _docall
INTERNALERROR>     firstresult=self.firstresult).execute()
INTERNALERROR>   File "/home/private/Projects/pytest-dev/pytest-xdist/.tox/py27-pytest27-pexpect/lib/python2.7/site-packages/_pytest/core.py", line 394, in execute
INTERNALERROR>     res = method(*args)
INTERNALERROR>   File "/home/private/Projects/pytest-dev/pytest-xdist/.tox/py27-pytest27-pexpect/lib/python2.7/site-packages/xdist/dsession.py", line 521, in pytest_runtestloop
INTERNALERROR>     self.loop_once()
INTERNALERROR>   File "/home/private/Projects/pytest-dev/pytest-xdist/.tox/py27-pytest27-pexpect/lib/python2.7/site-packages/xdist/dsession.py", line 539, in loop_once
INTERNALERROR>     call(**kwargs)
INTERNALERROR>   File "/home/private/Projects/pytest-dev/pytest-xdist/.tox/py27-pytest27-pexpect/lib/python2.7/site-packages/xdist/dsession.py", line 629, in slave_collectionfinish
INTERNALERROR>     self.sched.init_distribute()
INTERNALERROR>   File "/home/private/Projects/pytest-dev/pytest-xdist/.tox/py27-pytest27-pexpect/lib/python2.7/site-packages/xdist/dsession.py", line 130, in init_distribute
INTERNALERROR>     pending[:] = range(len(self.node2collection[node]))
INTERNALERROR> KeyError: <SlaveController gw3>

=========================== 2 failed in 0.71 seconds ===========================
--------------------------------------------------------------------------- Captured stderr call ----------------------------------------------------------------------------
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/home/private/Projects/pytest-dev/pytest-xdist/.tox/py27-pytest27-pexpect/lib/python2.7/site-packages/_pytest/main.py", line 84, in wrap_session
INTERNALERROR>     doit(config, session)
INTERNALERROR>   File "/home/private/Projects/pytest-dev/pytest-xdist/.tox/py27-pytest27-pexpect/lib/python2.7/site-packages/_pytest/main.py", line 84, in wrap_session
INTERNALERROR>   File "/home/private/Projects/pytest-dev/pytest-xdist/.tox/py27-pytest27-pexpect/lib/python2.7/site-packages/_pytest/main.py", line 122, in _main
INTERNALERROR>     doit(config, session)
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/home/private/Projects/pytest-dev/pytest-xdist/.tox/py27-pytest27-pexpect/lib/python2.7/site-packages/_pytest/main.py", line 122, in _main
INTERNALERROR>   File "/home/private/Projects/pytest-dev/pytest-xdist/.tox/py27-pytest27-pexpect/lib/python2.7/site-packages/_pytest/core.py", line 521, in __call__
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>     return self._docall(self.methods, kwargs)
INTERNALERROR>   File "/home/private/Projects/pytest-dev/pytest-xdist/.tox/py27-pytest27-pexpect/lib/python2.7/site-packages/_pytest/core.py", line 521, in __call__
INTERNALERROR>   File "/home/private/Projects/pytest-dev/pytest-xdist/.tox/py27-pytest27-pexpect/lib/python2.7/site-packages/_pytest/core.py", line 528, in _docall
INTERNALERROR>     return self._docall(self.methods, kwargs)
INTERNALERROR>     firstresult=self.firstresult).execute()
INTERNALERROR>   File "/home/private/Projects/pytest-dev/pytest-xdist/.tox/py27-pytest27-pexpect/lib/python2.7/site-packages/_pytest/core.py", line 528, in _docall
INTERNALERROR>   File "/home/private/Projects/pytest-dev/pytest-xdist/.tox/py27-pytest27-pexpect/lib/python2.7/site-packages/_pytest/core.py", line 394, in execute
INTERNALERROR>     firstresult=self.firstresult).execute()
INTERNALERROR>     res = method(*args)
INTERNALERROR>   File "/home/private/Projects/pytest-dev/pytest-xdist/.tox/py27-pytest27-pexpect/lib/python2.7/site-packages/_pytest/core.py", line 394, in execute
INTERNALERROR>   File "<remote exec>", line 47, in pytest_runtestloop
INTERNALERROR>     res = method(*args)
INTERNALERROR>   File "/home/private/Projects/pytest-dev/pytest-xdist/.tox/py27-pytest27-pexpect/lib/python2.7/site-packages/execnet/gateway_base.py", line 711, in receive
INTERNALERROR>   File "<remote exec>", line 47, in pytest_runtestloop
INTERNALERROR>     raise self._getremoteerror() or EOFError()
INTERNALERROR>   File "/home/private/Projects/pytest-dev/pytest-xdist/.tox/py27-pytest27-pexpect/lib/python2.7/site-packages/execnet/gateway_base.py", line 711, in receive
INTERNALERROR> EOFError
INTERNALERROR>     raise self._getremoteerror() or EOFError()
INTERNALERROR> EOFError
nomatch: '*Replacing crashed slave*'
    and: u'============================= test session starts =============================='
    and: u'platform linux2 -- Python 2.7.10 -- py-1.4.30 -- pytest-2.7.2'
    and: u'rootdir: /tmp/pytest-774/testdir/test_each_multiple0, inifile: '
    and: u'plugins: xdist, xdist, xdist'
    and: u'gw0 I / gw1 I'
    and: u'gw0 [2] / gw1 [2]'
    and: u''
    and: u'scheduling tests via EachScheduling'
    and: u'[gw0] node down: Not properly terminated'
fnmatch: '*Replacing crashed slave*'
   with: u'fReplacing crashed slave gw0'
nomatch: '*Slave*crashed while running*'
    and: u'[gw1] node down: Not properly terminated'
    and: u'fReplacing crashed slave gw1'
    and: u'INTERNALERROR> Traceback (most recent call last):'
    and: u'INTERNALERROR>   File "/home/private/Projects/pytest-dev/pytest-xdist/.tox/py27-pytest27-pexpect/lib/python2.7/site-packages/_pytest/main.py", line 84, in wrap_session'
    and: u'INTERNALERROR>     doit(config, session)'
    and: u'INTERNALERROR>   File "/home/private/Projects/pytest-dev/pytest-xdist/.tox/py27-pytest27-pexpect/lib/python2.7/site-packages/_pytest/main.py", line 122, in _main'
    and: u'INTERNALERROR>     config.hook.pytest_runtestloop(session=session)'
    and: u'INTERNALERROR>   File "/home/private/Projects/pytest-dev/pytest-xdist/.tox/py27-pytest27-pexpect/lib/python2.7/site-packages/_pytest/core.py", line 521, in __call__'
    and: u'INTERNALERROR>     return self._docall(self.methods, kwargs)'
    and: u'INTERNALERROR>   File "/home/private/Projects/pytest-dev/pytest-xdist/.tox/py27-pytest27-pexpect/lib/python2.7/site-packages/_pytest/core.py", line 528, in _docall'
    and: u'INTERNALERROR>     firstresult=self.firstresult).execute()'
    and: u'INTERNALERROR>   File "/home/private/Projects/pytest-dev/pytest-xdist/.tox/py27-pytest27-pexpect/lib/python2.7/site-packages/_pytest/core.py", line 394, in execute'
    and: u'INTERNALERROR>     res = method(*args)'
    and: u'INTERNALERROR>   File "/home/private/Projects/pytest-dev/pytest-xdist/.tox/py27-pytest27-pexpect/lib/python2.7/site-packages/xdist/dsession.py", line 521, in pytest_runtestloop'
    and: u'INTERNALERROR>     self.loop_once()'
    and: u'INTERNALERROR>   File "/home/private/Projects/pytest-dev/pytest-xdist/.tox/py27-pytest27-pexpect/lib/python2.7/site-packages/xdist/dsession.py", line 539, in loop_once'
    and: u'INTERNALERROR>     call(**kwargs)'
    and: u'INTERNALERROR>   File "/home/private/Projects/pytest-dev/pytest-xdist/.tox/py27-pytest27-pexpect/lib/python2.7/site-packages/xdist/dsession.py", line 629, in slave_collectionfinish'
    and: u'INTERNALERROR>     self.sched.init_distribute()'
    and: u'INTERNALERROR>   File "/home/private/Projects/pytest-dev/pytest-xdist/.tox/py27-pytest27-pexpect/lib/python2.7/site-packages/xdist/dsession.py", line 130, in init_distribute'
    and: u'INTERNALERROR>     pending[:] = range(len(self.node2collection[node]))'
    and: u'INTERNALERROR> KeyError: <SlaveController gw3>'
    and: u''
    and: u'=========================== 2 failed in 0.71 seconds ==========================='
========================================================================== short test summary info ==========================================================================
FAIL acceptance_test.py::TestNodeFailure::()::test_each_multiple
=============================================================== 97 tests deselected by '-ktest_each_multiple' ===============================================================
================================================================== 1 failed, 97 deselected in 1.37 seconds ==================================================================
ERROR: InvocationError: '/home/private/Projects/pytest-dev/pytest-xdist/.tox/py27-pytest27-pexpect/bin/py.test -k test_each_multiple'

[wagnerluis1982]

This issue is fixed?

[flub]

Looking at the traceback I can't even find the method in which shows up in your tracebacks. I also can't reproduce this either currently. Can you still reproduce it @RonnyPfannschmidt ? If not maybe we should close it.

[RonnyPfannschmidt]

the condition no longer occurs, i'm reluctantly closing this as its no longer a issue, i'd prefer to know why its no longer a issue, but i cant spare the time