serializing collection process (per host) on xdist to avoid conflicts/collection errors

[pytestbot]

Originally reported by: Oleg Sinyavskiy (BitBucket: otognan, GitHub: otognan)

---

Hi guys,

I have a strange internal error that appears sometimes (20% of runs) on certain machines (Ubuntu), but not on others (Mac and CentOS). xdist 1.8. Any ideas? Thanks in advance!

#!python



============================= test session starts ==============================
platform linux2 -- Python 2.7.3 -- pytest-2.3.4 -- /mnt/jenkins-workspace/workspace/my_stuff/venv/bin/python
cachedir: /mnt/jenkins-workspace/workspace/my_stuff/.cache
plugins: cache, pep8, cov, capturelog, xdist
gw0 I / gw1 I / gw2 I / gw3 I / gw4 I / gw5 I / gw6 I / gw7 I

[gw0] linux2 Python 2.7.3 cwd: /mnt/jenkins-workspace/workspace/my_stuff

[gw1] linux2 Python 2.7.3 cwd: /mnt/jenkins-workspace/workspace/my_stuff

[gw2] linux2 Python 2.7.3 cwd: /mnt/jenkins-workspace/workspace/my_stuff

[gw3] linux2 Python 2.7.3 cwd: /mnt/jenkins-workspace/workspace/my_stuff

[gw4] linux2 Python 2.7.3 cwd: /mnt/jenkins-workspace/workspace/my_stuff

[gw5] linux2 Python 2.7.3 cwd: /mnt/jenkins-workspace/workspace/my_stuff

[gw6] linux2 Python 2.7.3 cwd: /mnt/jenkins-workspace/workspace/my_stuff

[gw7] linux2 Python 2.7.3 cwd: /mnt/jenkins-workspace/workspace/my_stuff

[gw1] Python 2.7.3 (default, Aug  1 2012, 05:14:39)  -- [GCC 4.6.3]

[gw4] Python 2.7.3 (default, Aug  1 2012, 05:14:39)  -- [GCC 4.6.3]

[gw0] Python 2.7.3 (default, Aug  1 2012, 05:14:39)  -- [GCC 4.6.3]

[gw3] Python 2.7.3 (default, Aug  1 2012, 05:14:39)  -- [GCC 4.6.3]

[gw2] Python 2.7.3 (default, Aug  1 2012, 05:14:39)  -- [GCC 4.6.3]

[gw5] Python 2.7.3 (default, Aug  1 2012, 05:14:39)  -- [GCC 4.6.3]

[gw6] Python 2.7.3 (default, Aug  1 2012, 05:14:39)  -- [GCC 4.6.3]

[gw7] Python 2.7.3 (default, Aug  1 2012, 05:14:39)  -- [GCC 4.6.3]
gw0 [496] / gw1 [496] / gw2 [496] / gw3 [496] / gw4 [496] / gw5 [357] / gw6 [496] / gw7 [496]

scheduling tests via LoadScheduling
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/mnt/jenkins-workspace/workspace/my_stuff/venv/local/lib/python2.7/site-packages/_pytest/main.py", line 81, in wrap_session
INTERNALERROR>     doit(config, session)
INTERNALERROR>   File "/mnt/jenkins-workspace/workspace/my_stuff/venv/local/lib/python2.7/site-packages/_pytest/main.py", line 113, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/mnt/jenkins-workspace/workspace/my_stuff/venv/local/lib/python2.7/site-packages/_pytest/core.py", line 422, in __call__
INTERNALERROR>     return self._docall(methods, kwargs)
INTERNALERROR>   File "/mnt/jenkins-workspace/workspace/my_stuff/venv/local/lib/python2.7/site-packages/_pytest/core.py", line 433, in _docall
INTERNALERROR>     res = mc.execute()
INTERNALERROR>   File "/mnt/jenkins-workspace/workspace/my_stuff/venv/local/lib/python2.7/site-packages/_pytest/core.py", line 351, in execute
INTERNALERROR>     res = method(**kwargs)
INTERNALERROR>   File "/mnt/jenkins-workspace/workspace/my_stuff/venv/local/lib/python2.7/site-packages/xdist/dsession.py", line 201, in pytest_runtestloop
INTERNALERROR>     self.loop_once()
INTERNALERROR>   File "/mnt/jenkins-workspace/workspace/my_stuff/venv/local/lib/python2.7/site-packages/xdist/dsession.py", line 219, in loop_once
INTERNALERROR>     call(**kwargs)
INTERNALERROR>   File "/mnt/jenkins-workspace/workspace/my_stuff/venv/local/lib/python2.7/site-packages/xdist/dsession.py", line 272, in slave_collectionfinish
INTERNALERROR>     self.sched.init_distribute()
INTERNALERROR>   File "/mnt/jenkins-workspace/workspace/my_stuff/venv/local/lib/python2.7/site-packages/xdist/dsession.py", line 132, in init_distribute
INTERNALERROR>     assert collection == col
INTERNALERROR> AssertionError: assert ['conftest.py', 'my_stuff/__init__.py', ...] == ['conftest.py', 'my_stuff/__init__.py', ...]

---

• Bitbucket: https://bitbucket.org/pytest-dev/pytest/issue/301

[pytestbot]

Original comment by holger krekel (BitBucket: hpk42, GitHub: hpk42):

---

On the face of it, it appears that what the different subprocesses collect is different. The assertion tries to make sure that all collections are the same. Is the above really the precise message you got or some hand-edited version? The assertion representation looks abbreviated.

[pytestbot]

Original comment by Oleg Sinyavskiy (BitBucket: otognan, GitHub: otognan):

---

Thanks for a quick response!
yep, its hand edited. There was a bunch of test names, I shortened it into 'my stuff'.
if it helps, I see this behavior only on amazon cloud testing nodes

[pytestbot]

Original comment by holger krekel (BitBucket: hpk42, GitHub: hpk42):

---

could you post the full assert failure?

[pytestbot]

Original comment by Oleg Sinyavskiy (BitBucket: otognan, GitHub: otognan):

---

For some reason I can not reproduce it these days..
But I swear, it doesn't look more informative than the one I've sent. It just bunch of files, then '...' on both sides. And you do not see the difference - it is probably only one file that is hidden inside '...'.
P.s. google gives me more examples of this error from other people:
https://gist.github.com/baijum/3432544
http://lists.idyll.org/pipermail/testing-in-python/2012-August/005142.html

[pytestbot]

Original comment by holger krekel (BitBucket: hpk42, GitHub: hpk42):

---

Instead of the assert we should probably normalize the collections we receive from the slaves. Or at least do an order-independent comparison using sets or so. It would be good to have a reproducable example problem which would be fixed by such a change.

[pytestbot]

Original comment by Michael Droettboom (BitBucket: mdboom, GitHub: mdboom):

---

I haven't been able to create a minimal thing that reproduces the problem, but I see this every time when running tests for astropy (see below) in parallel on Python 3.3. Changing the assertion to

assert set(collection) == set(col)

resolves it.

To reproduce: Use the "testing/parallel" branch in my astropy fork at http://github.com/mdboom/astropy. After installing Numpy and pytest-xdist, run "python setup.py test --parallel=-1". There is also a Travis failure log here:

Travis Failure

[pytestbot]

Original comment by Oleg Sinyavskiy (BitBucket: otognan, GitHub: otognan):

---

Hi,
We created a fix here (https://bitbucket.org/AutomatedTester/pytest-xdist/pull-request/1/there-is-an-internal-error-during-test/diff),
but what I see now is this error:

#!python

platform linux2 -- Python 2.7.3 -- pytest-2.2.4 -- /local/mnt/workspace/jenkins/workspace/neural_library-fp_periodic_pattern_generation/venv/bin/python27
cachedir: /local/mnt/workspace/jenkins/workspace/neural_library-fp_periodic_pattern_generation/neural_library/.cache
gw0 I / gw1 I / gw2 I / gw3 I / gw4 I / gw5 I / gw6 I / gw7 I

[gw0] linux2 Python 2.7.3 cwd: /local/mnt/workspace/jenkins/workspace/neural_library-fp_periodic_pattern_generation/neural_library

[gw1] linux2 Python 2.7.3 cwd: /local/mnt/workspace/jenkins/workspace/neural_library-fp_periodic_pattern_generation/neural_library

[gw2] linux2 Python 2.7.3 cwd: /local/mnt/workspace/jenkins/workspace/neural_library-fp_periodic_pattern_generation/neural_library

[gw3] linux2 Python 2.7.3 cwd: /local/mnt/workspace/jenkins/workspace/neural_library-fp_periodic_pattern_generation/neural_library

[gw4] linux2 Python 2.7.3 cwd: /local/mnt/workspace/jenkins/workspace/neural_library-fp_periodic_pattern_generation/neural_library

[gw5] linux2 Python 2.7.3 cwd: /local/mnt/workspace/jenkins/workspace/neural_library-fp_periodic_pattern_generation/neural_library

[gw6] linux2 Python 2.7.3 cwd: /local/mnt/workspace/jenkins/workspace/neural_library-fp_periodic_pattern_generation/neural_library

[gw7] linux2 Python 2.7.3 cwd: /local/mnt/workspace/jenkins/workspace/neural_library-fp_periodic_pattern_generation/neural_library
[gw7] node down: Traceback (most recent call last):
  File "/local/mnt/workspace/jenkins/workspace/neural_library-fp_periodic_pattern_generation/venv/lib/python2.7/site-packages/execnet/gateway_base.py", line 800, in executetask
    do_exec(co, loc)
  File "<string>", line 1, in do_exec
  File "", line 134, in <module>
  File "", line 19, in __init__
  File "/local/mnt/workspace/jenkins/workspace/neural_library-fp_periodic_pattern_generation/venv/lib/python2.7/site-packages/_pytest/core.py", line 89, in register
    self.hook.pytest_plugin_registered(manager=self, plugin=plugin)
  File "/local/mnt/workspace/jenkins/workspace/neural_library-fp_periodic_pattern_generation/venv/lib/python2.7/site-packages/_pytest/core.py", line 421, in __call__
    return self._docall(methods, kwargs)
  File "/local/mnt/workspace/jenkins/workspace/neural_library-fp_periodic_pattern_generation/venv/lib/python2.7/site-packages/_pytest/core.py", line 432, in _docall
    res = mc.execute()
  File "/local/mnt/workspace/jenkins/workspace/neural_library-fp_periodic_pattern_generation/venv/lib/python2.7/site-packages/_pytest/core.py", line 350, in execute
    res = method(**kwargs)
  File "/local/mnt/workspace/jenkins/workspace/neural_library-fp_periodic_pattern_generation/venv/lib/python2.7/site-packages/_pytest/helpconfig.py", line 176, in pytest_plugin_registered
    raise PluginValidationError("%s:\n%s" % (name, stringio.getvalue()))
PluginValidationError: <__channelexec__.SlaveInteractor instance at 0x2f11bd8>:
found unknown hook: pytest__teardown_final_logerror

[pytestbot]

Original comment by Oleg Sinyavskiy (BitBucket: otognan, GitHub: otognan):

---

We apologies for making a pull request into forked repo:) we closed it.
PluginValidationError was probably caused by cloning wrong repo.. Anyway, I still have assert error event with set:

#!python
INTERNALERROR>     assert set(collection) == set(col)
INTERNALERROR> AssertionError: assert set(['conftest.py', 'neural_library/__init__.py', 'neural_library/analog_to_spike/__init__.py', 'neural_library/analog...'neural_library/analog_to_spike/performance_to_rate.py', 'neural_library/analog_to_spike/receptive_fields_1d.py', ...]) == set(['conftest.py', 'neural_library/__init__.py', 'neural_library/analog_to_spike/__init__.py', 'neural_library/analog...'neural_library/analog_to_spike/performance_to_rate.py', 'neural_library/analog_to_spike/receptive_fields_1d.py', ...])
INTERNALERROR>  +  where set(['conftest.py', 'neural_library/__init__.py', 'neural_library/analog_to_spike/__init__.py', 'neural_library/analog...'neural_library/analog_to_spike/performance_to_rate.py', 'neural_library/analog_to_spike/receptive_fields_1d.py', ...]) = set(['conftest.py', 'neural_library/__init__.py', 'neural_library/docdemo.py', 'neural_library/analog_to_spike/__init__.py', 'neural_library/analog_to_spike/float_to_latency.py', 'neural_library/analog_to_spike/performance_to_rate.py', ...])
INTERNALERROR>  +  and   set(['conftest.py', 'neural_library/__init__.py', 'neural_library/analog_to_spike/__init__.py', 'neural_library/analog...'neural_library/analog_to_spike/performance_to_rate.py', 'neural_library/analog_to_spike/receptive_fields_1d.py', ...]) = set(['conftest.py', 'neural_library/__init__.py', 'neural_library/docdemo.py', 'neural_library/analog_to_spike/__init__.py', 'neural_library/analog_to_spike/float_to_latency.py', 'neural_library/analog_to_spike/performance_to_rate.py', ...])

[pytestbot]

Original comment by Oleg Sinyavskiy (BitBucket: otognan, GitHub: otognan):

---

I figured out how to have robust reproducibility! Looking into it..

[pytestbot]

Original comment by holger krekel (BitBucket: hpk42, GitHub: hpk42):

---

to be sure, could you try using pytest-2.3.5 and check that subprocesses are using that version as well?

[pytestbot]

Original comment by Oleg Sinyavskiy (BitBucket: otognan, GitHub: otognan):

---

I found what is wrong but I'm not sure how to patch it without changing logic considerably.

Here is what happened:
We have several hundreds of files to test and use xdist locally on a servers.
Recently we introduced a cython file into library and we used pyximport to import it invisibly. Right now every xdist worker collects all test files independently of each other. During collection import statements were called for every test including pyximport one. pyximport starts compiler and compiles cython on import. If number of processors is more than ~6 compilation collisions started to happen 90% of time because xdist workers follow the same collection algorithm and wanted to import this file approximately at the same time. When compilation collision happens, import error occurs during collection for some of the workers but not for others. Whats why collection sizes were not equal and assertion was raised. Recent pull request (https://bitbucket.org/hpk42/pytest-xdist/pull-request/2/patch-for-issue-180/diff) was not able to inform about this import error because assert is located before xdist reports this error to pytest.

What works for us is just to remove this assert statement. Since import error will be reported after assert, the whole test suite will fail. However as I know xdist has multinode remote processing features and I'm not sure how this patch will play with those (no time to try this).
Also I see this assertion error to be a part of design chosen - each worker relies on pytest collection mechanism. After collection is done, scheduler independently may assign test that failed to import to a worker that does not have this test. This will cause crash in a worker - test will not be found in his dictionaries. (example: worker 1 imported testA successfully, worker 2 failed to import testA. Assert is removed, scheduler relies on worker 1 tests collection and for some reason decides to put testA on worker 2).

Any suggestions?

[pytestbot]

Original comment by holger krekel (BitBucket: hpk42, GitHub: hpk42):

---

if we had a semaphore-like mechanism around the offending/concurrent import would that help your case? something like a xdist_semaphored_import=path/to/test_file.py ini-style configuration. This would guarantee that only one worker imports this file at a time. Or maybe simpler/coarser-grained a xdist_oneworker_collects_first = True which would make xdist run the collection in one worker and then spawn off all others in parallel. As the first one creates pyc files etc. it might not even be that much slower.
what do you think?

[pytestbot]

Original comment by Oleg Sinyavskiy (BitBucket: otognan, GitHub: otognan):

---

xdist_oneworker_collects_first = True - this was my first guess when I realized the problem. But I was afraid it will take me quite some time to rewrite the logic without breaking stuff since I do not know all the internals of pytest.. The second issue is multimachine tests execution which, as I said, I've never tried. Who would be the first worker in this case? Master node?

I think that collection should not be job of the first worker but of scheduler. Consider a scheduler to collect all the stuff (in remote case, it runs on a master node). Then it divides tests into chunks and sends them to completely identical workers. Import error will be caught before running and only once. Execution will be separated from collection - a nice design property.

xdist_semaphored_import - is not helpful here because first you need to figure out which file causes a problem and for that you need to catch import error first.

[pytestbot]

Original comment by holger krekel (BitBucket: hpk42, GitHub: hpk42):

---

pytest used to have the scheduler/master do the collections - that provided all kinds of problems (because some people do a lot of things during imports).

If we restrict ourselves to a subprocesses on the same machine, i think xdist_oneworker_collects_first is not too hard to do. Basically the DSession object would first trigger collection on one host, and once that is completed, trigger all the rest's collections like now.

[pytestbot]

Original comment by Dmitrijs Milajevs (BitBucket: dimazest, GitHub: dimazest):

---

we also run into a similar issue (slaves collected different tests). the real reason is not really clear to me, it has something to do with the way we install packages into a virtualenv, which we rsynch together with the code and activate on each worker.

but it would be nice to show more details: what are the different tests and what are the nodes.