Allow tests to be logged multiple times with pytest-xdist

[davehunt]

From pytest created by davehunt : pytest-dev/pytest#1193

Note that this is not related to pytest-dev/pytest#927

In pytest-rerunfailures the test protocol is run multiple times in the event of a failure. An enhancement will log each iteration of the test with an outcome of 'rerun' until the maximum number of reruns is reached, at which point the genuine outcome is preserved. This works fine unless running the tests without pytest-xdist, which causes issues when the first rerun of a test is attempted to be removed from the scheduler:

INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/Users/dhunt/.virtualenvs/tmp-aae67d87c3b61cd1/lib/python2.7/site-packages/pytest-2.8.3-py2.7.egg/_pytest/main.py", line 90, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/Users/dhunt/.virtualenvs/tmp-aae67d87c3b61cd1/lib/python2.7/site-packages/pytest-2.8.3-py2.7.egg/_pytest/main.py", line 121, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/Users/dhunt/.virtualenvs/tmp-aae67d87c3b61cd1/lib/python2.7/site-packages/pytest-2.8.3-py2.7.egg/_pytest/vendored_packages/pluggy.py", line 724, in __call__
INTERNALERROR>     return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
INTERNALERROR>   File "/Users/dhunt/.virtualenvs/tmp-aae67d87c3b61cd1/lib/python2.7/site-packages/pytest-2.8.3-py2.7.egg/_pytest/vendored_packages/pluggy.py", line 338, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/Users/dhunt/.virtualenvs/tmp-aae67d87c3b61cd1/lib/python2.7/site-packages/pytest-2.8.3-py2.7.egg/_pytest/vendored_packages/pluggy.py", line 333, in <lambda>
INTERNALERROR>     _MultiCall(methods, kwargs, hook.spec_opts).execute()
INTERNALERROR>   File "/Users/dhunt/.virtualenvs/tmp-aae67d87c3b61cd1/lib/python2.7/site-packages/pytest-2.8.3-py2.7.egg/_pytest/vendored_packages/pluggy.py", line 596, in execute
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/Users/dhunt/.virtualenvs/tmp-aae67d87c3b61cd1/lib/python2.7/site-packages/xdist/dsession.py", line 521, in pytest_runtestloop
INTERNALERROR>     self.loop_once()
INTERNALERROR>   File "/Users/dhunt/.virtualenvs/tmp-aae67d87c3b61cd1/lib/python2.7/site-packages/xdist/dsession.py", line 539, in loop_once
INTERNALERROR>     call(**kwargs)
INTERNALERROR>   File "/Users/dhunt/.virtualenvs/tmp-aae67d87c3b61cd1/lib/python2.7/site-packages/xdist/dsession.py", line 643, in slave_testreport
INTERNALERROR>     self.sched.remove_item(node, rep.item_index, rep.duration)
INTERNALERROR>   File "/Users/dhunt/.virtualenvs/tmp-aae67d87c3b61cd1/lib/python2.7/site-packages/xdist/dsession.py", line 280, in remove_item
INTERNALERROR>     self.node2pending[node].remove(item_index)
INTERNALERROR> ValueError: list.remove(x): x not in list

For now I can avoid logging multiple reports if we're running tests in parallel, but that does mean that the only indication of multiple tests running will be the multiple logstart calls. Ideally there should be no difference between running tests in serial or parallel.

[davehunt]

@RonnyPfannschmidt do you have some suggestions on where to start with this?

[RonnyPfannschmidt]

given the way tests and nextitem interact its not clear, i'll defer to @hpk42 since he invented nextitem despite protest