Skip to content

xdist race condition on node restart #927

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
RonnyPfannschmidt opened this issue Aug 8, 2015 · 3 comments
Closed

xdist race condition on node restart #927

RonnyPfannschmidt opened this issue Aug 8, 2015 · 3 comments
Assignees
@flub
Labels
plugin: xdist related to the xdist external plugin status: help wanted developers would like help from experts on this topic type: bug problem that needs to be addressed

Comments

@RonnyPfannschmidt
Copy link
Member

it seems we have a order invariant that very rarely happens on node restart

to even see the issue creep up i have to do

while tox --skip-missing-interpreters  -- -k test_each_multiple
do
  sleep .1
done

the result is an action with something we don't have a accounted for yet

py27-pytest27-pexpect runtests: commands[1] | py.test -k test_each_multiple
============================================================================ test session starts ============================================================================
platform linux2 -- Python 2.7.10 -- py-1.4.30 -- pytest-2.7.2
rootdir: /home/private/Projects/pytest-dev/pytest-xdist, inifile: tox.ini
plugins: xdist, xdist, xdist
collected 98 items 

acceptance_test.py F

================================================================================= FAILURES ==================================================================================
____________________________________________________________________ TestNodeFailure.test_each_multiple _____________________________________________________________________

self = <acceptance_test.TestNodeFailure instance at 0x7f59903c0cb0>, testdir = <TmpTestdir local('/tmp/pytest-774/testdir/test_each_multiple0')>

    def test_each_multiple(self, testdir):
        f = testdir.makepyfile("""
                import os
                def test_a(): os._exit(1)
                def test_b(): pass
            """)
        res = testdir.runpytest(f, '--dist=each', '--tx=2*popen')
        res.stdout.fnmatch_lines([
            "*Replacing crashed slave*",
            "*Slave*crashed while running*",
>           "*2 failed*2 passed*",
        ])
E       Failed: remains unmatched: '*Slave*crashed while running*', see stderr

/home/private/Projects/pytest-dev/pytest-xdist/testing/acceptance_test.py:600: Failed
--------------------------------------------------------------------------- Captured stdout call ----------------------------------------------------------------------------
running ['/home/private/Projects/pytest-dev/pytest-xdist/.tox/py27-pytest27-pexpect/bin/python2.7', '/home/private/Projects/pytest-dev/pytest-xdist/.tox/py27-pytest27-pexpect/lib/python2.7/site-packages/pytest.py', '--basetemp=/tmp/pytest-774/testdir/test_each_multiple0/runpytest-0', '/tmp/pytest-774/testdir/test_each_multiple0/test_each_multiple.py', '--dist=each', '--tx=2*popen'] curdir= /tmp/pytest-774/testdir/test_each_multiple0
============================= test session starts ==============================
platform linux2 -- Python 2.7.10 -- py-1.4.30 -- pytest-2.7.2
rootdir: /tmp/pytest-774/testdir/test_each_multiple0, inifile: 
plugins: xdist, xdist, xdist
gw0 I / gw1 I
gw0 [2] / gw1 [2]

scheduling tests via EachScheduling
[gw0] node down: Not properly terminated
fReplacing crashed slave gw0
[gw1] node down: Not properly terminated
fReplacing crashed slave gw1
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/home/private/Projects/pytest-dev/pytest-xdist/.tox/py27-pytest27-pexpect/lib/python2.7/site-packages/_pytest/main.py", line 84, in wrap_session
INTERNALERROR>     doit(config, session)
INTERNALERROR>   File "/home/private/Projects/pytest-dev/pytest-xdist/.tox/py27-pytest27-pexpect/lib/python2.7/site-packages/_pytest/main.py", line 122, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/home/private/Projects/pytest-dev/pytest-xdist/.tox/py27-pytest27-pexpect/lib/python2.7/site-packages/_pytest/core.py", line 521, in __call__
INTERNALERROR>     return self._docall(self.methods, kwargs)
INTERNALERROR>   File "/home/private/Projects/pytest-dev/pytest-xdist/.tox/py27-pytest27-pexpect/lib/python2.7/site-packages/_pytest/core.py", line 528, in _docall
INTERNALERROR>     firstresult=self.firstresult).execute()
INTERNALERROR>   File "/home/private/Projects/pytest-dev/pytest-xdist/.tox/py27-pytest27-pexpect/lib/python2.7/site-packages/_pytest/core.py", line 394, in execute
INTERNALERROR>     res = method(*args)
INTERNALERROR>   File "/home/private/Projects/pytest-dev/pytest-xdist/.tox/py27-pytest27-pexpect/lib/python2.7/site-packages/xdist/dsession.py", line 521, in pytest_runtestloop
INTERNALERROR>     self.loop_once()
INTERNALERROR>   File "/home/private/Projects/pytest-dev/pytest-xdist/.tox/py27-pytest27-pexpect/lib/python2.7/site-packages/xdist/dsession.py", line 539, in loop_once
INTERNALERROR>     call(**kwargs)
INTERNALERROR>   File "/home/private/Projects/pytest-dev/pytest-xdist/.tox/py27-pytest27-pexpect/lib/python2.7/site-packages/xdist/dsession.py", line 629, in slave_collectionfinish
INTERNALERROR>     self.sched.init_distribute()
INTERNALERROR>   File "/home/private/Projects/pytest-dev/pytest-xdist/.tox/py27-pytest27-pexpect/lib/python2.7/site-packages/xdist/dsession.py", line 130, in init_distribute
INTERNALERROR>     pending[:] = range(len(self.node2collection[node]))
INTERNALERROR> KeyError: <SlaveController gw3>

=========================== 2 failed in 0.71 seconds ===========================
--------------------------------------------------------------------------- Captured stderr call ----------------------------------------------------------------------------
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/home/private/Projects/pytest-dev/pytest-xdist/.tox/py27-pytest27-pexpect/lib/python2.7/site-packages/_pytest/main.py", line 84, in wrap_session
INTERNALERROR>     doit(config, session)
INTERNALERROR>   File "/home/private/Projects/pytest-dev/pytest-xdist/.tox/py27-pytest27-pexpect/lib/python2.7/site-packages/_pytest/main.py", line 84, in wrap_session
INTERNALERROR>   File "/home/private/Projects/pytest-dev/pytest-xdist/.tox/py27-pytest27-pexpect/lib/python2.7/site-packages/_pytest/main.py", line 122, in _main
INTERNALERROR>     doit(config, session)
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/home/private/Projects/pytest-dev/pytest-xdist/.tox/py27-pytest27-pexpect/lib/python2.7/site-packages/_pytest/main.py", line 122, in _main
INTERNALERROR>   File "/home/private/Projects/pytest-dev/pytest-xdist/.tox/py27-pytest27-pexpect/lib/python2.7/site-packages/_pytest/core.py", line 521, in __call__
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>     return self._docall(self.methods, kwargs)
INTERNALERROR>   File "/home/private/Projects/pytest-dev/pytest-xdist/.tox/py27-pytest27-pexpect/lib/python2.7/site-packages/_pytest/core.py", line 521, in __call__
INTERNALERROR>   File "/home/private/Projects/pytest-dev/pytest-xdist/.tox/py27-pytest27-pexpect/lib/python2.7/site-packages/_pytest/core.py", line 528, in _docall
INTERNALERROR>     return self._docall(self.methods, kwargs)
INTERNALERROR>     firstresult=self.firstresult).execute()
INTERNALERROR>   File "/home/private/Projects/pytest-dev/pytest-xdist/.tox/py27-pytest27-pexpect/lib/python2.7/site-packages/_pytest/core.py", line 528, in _docall
INTERNALERROR>   File "/home/private/Projects/pytest-dev/pytest-xdist/.tox/py27-pytest27-pexpect/lib/python2.7/site-packages/_pytest/core.py", line 394, in execute
INTERNALERROR>     firstresult=self.firstresult).execute()
INTERNALERROR>     res = method(*args)
INTERNALERROR>   File "/home/private/Projects/pytest-dev/pytest-xdist/.tox/py27-pytest27-pexpect/lib/python2.7/site-packages/_pytest/core.py", line 394, in execute
INTERNALERROR>   File "<remote exec>", line 47, in pytest_runtestloop
INTERNALERROR>     res = method(*args)
INTERNALERROR>   File "/home/private/Projects/pytest-dev/pytest-xdist/.tox/py27-pytest27-pexpect/lib/python2.7/site-packages/execnet/gateway_base.py", line 711, in receive
INTERNALERROR>   File "<remote exec>", line 47, in pytest_runtestloop
INTERNALERROR>     raise self._getremoteerror() or EOFError()
INTERNALERROR>   File "/home/private/Projects/pytest-dev/pytest-xdist/.tox/py27-pytest27-pexpect/lib/python2.7/site-packages/execnet/gateway_base.py", line 711, in receive
INTERNALERROR> EOFError
INTERNALERROR>     raise self._getremoteerror() or EOFError()
INTERNALERROR> EOFError
nomatch: '*Replacing crashed slave*'
    and: u'============================= test session starts =============================='
    and: u'platform linux2 -- Python 2.7.10 -- py-1.4.30 -- pytest-2.7.2'
    and: u'rootdir: /tmp/pytest-774/testdir/test_each_multiple0, inifile: '
    and: u'plugins: xdist, xdist, xdist'
    and: u'gw0 I / gw1 I'
    and: u'gw0 [2] / gw1 [2]'
    and: u''
    and: u'scheduling tests via EachScheduling'
    and: u'[gw0] node down: Not properly terminated'
fnmatch: '*Replacing crashed slave*'
   with: u'fReplacing crashed slave gw0'
nomatch: '*Slave*crashed while running*'
    and: u'[gw1] node down: Not properly terminated'
    and: u'fReplacing crashed slave gw1'
    and: u'INTERNALERROR> Traceback (most recent call last):'
    and: u'INTERNALERROR>   File "/home/private/Projects/pytest-dev/pytest-xdist/.tox/py27-pytest27-pexpect/lib/python2.7/site-packages/_pytest/main.py", line 84, in wrap_session'
    and: u'INTERNALERROR>     doit(config, session)'
    and: u'INTERNALERROR>   File "/home/private/Projects/pytest-dev/pytest-xdist/.tox/py27-pytest27-pexpect/lib/python2.7/site-packages/_pytest/main.py", line 122, in _main'
    and: u'INTERNALERROR>     config.hook.pytest_runtestloop(session=session)'
    and: u'INTERNALERROR>   File "/home/private/Projects/pytest-dev/pytest-xdist/.tox/py27-pytest27-pexpect/lib/python2.7/site-packages/_pytest/core.py", line 521, in __call__'
    and: u'INTERNALERROR>     return self._docall(self.methods, kwargs)'
    and: u'INTERNALERROR>   File "/home/private/Projects/pytest-dev/pytest-xdist/.tox/py27-pytest27-pexpect/lib/python2.7/site-packages/_pytest/core.py", line 528, in _docall'
    and: u'INTERNALERROR>     firstresult=self.firstresult).execute()'
    and: u'INTERNALERROR>   File "/home/private/Projects/pytest-dev/pytest-xdist/.tox/py27-pytest27-pexpect/lib/python2.7/site-packages/_pytest/core.py", line 394, in execute'
    and: u'INTERNALERROR>     res = method(*args)'
    and: u'INTERNALERROR>   File "/home/private/Projects/pytest-dev/pytest-xdist/.tox/py27-pytest27-pexpect/lib/python2.7/site-packages/xdist/dsession.py", line 521, in pytest_runtestloop'
    and: u'INTERNALERROR>     self.loop_once()'
    and: u'INTERNALERROR>   File "/home/private/Projects/pytest-dev/pytest-xdist/.tox/py27-pytest27-pexpect/lib/python2.7/site-packages/xdist/dsession.py", line 539, in loop_once'
    and: u'INTERNALERROR>     call(**kwargs)'
    and: u'INTERNALERROR>   File "/home/private/Projects/pytest-dev/pytest-xdist/.tox/py27-pytest27-pexpect/lib/python2.7/site-packages/xdist/dsession.py", line 629, in slave_collectionfinish'
    and: u'INTERNALERROR>     self.sched.init_distribute()'
    and: u'INTERNALERROR>   File "/home/private/Projects/pytest-dev/pytest-xdist/.tox/py27-pytest27-pexpect/lib/python2.7/site-packages/xdist/dsession.py", line 130, in init_distribute'
    and: u'INTERNALERROR>     pending[:] = range(len(self.node2collection[node]))'
    and: u'INTERNALERROR> KeyError: <SlaveController gw3>'
    and: u''
    and: u'=========================== 2 failed in 0.71 seconds ==========================='
========================================================================== short test summary info ==========================================================================
FAIL acceptance_test.py::TestNodeFailure::()::test_each_multiple
=============================================================== 97 tests deselected by '-ktest_each_multiple' ===============================================================
================================================================== 1 failed, 97 deselected in 1.37 seconds ==================================================================
ERROR: InvocationError: '/home/private/Projects/pytest-dev/pytest-xdist/.tox/py27-pytest27-pexpect/bin/py.test -k test_each_multiple'
@RonnyPfannschmidt RonnyPfannschmidt added type: bug problem that needs to be addressed status: help wanted developers would like help from experts on this topic plugin: xdist related to the xdist external plugin threads labels Aug 8, 2015
@davehunt davehunt mentioned this issue Nov 25, 2015
Closed
This was referenced Dec 4, 2015
Merged
Open
@davehunt davehunt mentioned this issue Aug 7, 2017
Closed
@wagnerluis1982
Copy link

This issue is fixed?

@flub
Copy link
Member

flub commented Feb 20, 2018

Looking at the traceback I can't even find the method in which shows up in your tracebacks. I also can't reproduce this either currently. Can you still reproduce it @RonnyPfannschmidt ? If not maybe we should close it.

@RonnyPfannschmidt
Copy link
Member Author

the condition no longer occurs, i'm reluctantly closing this as its no longer a issue, i'd prefer to know why its no longer a issue, but i cant spare the time

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@flub flub

Labels
plugin: xdist related to the xdist external plugin status: help wanted developers would like help from experts on this topic type: bug problem that needs to be addressed
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@flub @RonnyPfannschmidt @wagnerluis1982 @nicoddemus