Flaky TestArgComplete.test_compare_with_compgen

[nicoddemus]

Example failure:

__________________ TestArgComplete.test_compare_with_compgen ___________________
[gw3] linux2 -- Python 2.7.6 /home/travis/build/pytest-dev/pytest/.tox/py27-nobyte/bin/python
self = <test_argcomplete.TestArgComplete object at 0x7f6da5d3a7d0>
    @pytest.mark.skipif("sys.platform in ('win32', 'darwin')")
    def test_compare_with_compgen(self):
        from _pytest._argcomplete import FastFilesCompleter
    
        ffc = FastFilesCompleter()
        fc = FilesCompleter()
        for x in ["/", "/d", "/data", "qqq", ""]:
>           assert equal_with_bash(x, ffc, fc, out=sys.stdout)
E           AssertionError: assert False
E            +  where False = equal_with_bash('', <_pytest._argcomplete.FastFilesCompleter object at 0x7f6da5d881d0>, <test_argcomplete.FilesCompleter object at 0x7f6da5d88150>, out=<_pytest.capture.EncodedFile object at 0x7f6da6f9cbd0>)
E            +    where <_pytest.capture.EncodedFile object at 0x7f6da6f9cbd0> = sys.stdout
testing/test_argcomplete.py:96: AssertionError

Always on py27.

Seen on #4161 and #4153.

[nicoddemus]

cc @asottile

[nicoddemus]

It happened on the py27-nobyte job and this is suspicious:

gw0 ok / gw1 ok / gw2 ok / gw3 ok / gw4 ok / gw5 ok / gw6 ok / gw7 ok / gw8 ok / gw9 ok / gw10 ok / gw11 ok / gw12 ok / gw13 ok / gw14 ok / gw15 ok / gw16 ok / gw17 ok / gw18 ok / gw19 ok / gw20 ok / gw21 ok / gw22 ok / gw23 ok / gw24 C / gw25 C / gw26 C / gw27 C / gw28 C / gw29 C / gw30 C / gw31 C

32 cores? That might as well be the problem...

[asottile]

testing filesystem state of the current working directory in a multiprocess system is asking for sadness 😆

[nicoddemus]

py27-xdist on the same build (#4161) uses 48 cores:

gw0 [2281] / gw1 [2281] / gw2 [2281] / gw3 [2281] / gw4 [2281] / gw5 [2281] / gw6 [2281] / gw7 [2281] / gw8 [2281] / gw9 [2281] / gw10 [2281] / gw11 [2281] / gw12 [2281] / gw13 [2281] / gw14 [2281] / gw15 [2281] / gw16 [2281] / gw17 [2281] / gw18 [2281] / gw19 [2281] / gw20 [2281] / gw21 [2281] / gw22 [2281] / gw23 [2281] / gw24 [2281] / gw25 [2281] / gw26 [2281] / gw27 [2281] / gw28 [2281] / gw29 [2281] / gw30 [2281] / gw31 [2281] / gw32 [2281] / gw33 [2281] / gw34 [2281] / gw35 [2281] / gw36 [2281] / gw37 [2281] / gw38 [2281] / gw39 [2281] / gw40 [2281] / gw41 [2281] / gw42 [2281] / gw43 [2281] / gw44 [2281] / gw45 [2281] / gw46 [2281] / gw47 [2281]

[nicoddemus]

We have a workaround on pytest-xdist to detect Travis explicitly:

https://github.com/pytest-dev/pytest-xdist/blob/86e2fb5a5a3193578237c17cc91a6045d7fc58ff/xdist/plugin.py#L11

Hmm...

[asottile]

we could probably work around it by disabling coverage for this build? no idea why we don't hit it on the others as well though

[nicoddemus]

Not sure, I'm not getting why xdist is not applying the workaround on Travis...

[asottile]

hmmm probably tox is blanking the environ? passenv = TRAVIS needed?

[nicoddemus]

Argh that's exactly it. I was looking for that problem, but I was looking at an old features branch (from my fork) which has passenv = TRAVIS. 😓

[nicoddemus]

It was removed by accident in 7c380b1.

[nicoddemus]

Opening a PR...

[asottile]

still.... even with parallelism of 2 this is still hittable, but exceedingly rare ;)

[nicoddemus]

I didn't the time to understand the test, but I believe in you. 😁

[asottile]

I'm ok paving over it though ;)

[blueyed]

It failed again here: https://travis-ci.org/pytest-dev/pytest/jobs/452673676

Should we a) skip it with xdist, or b) ensure that it gets to the same worker (if that's the issue)?

[asottile]

I think we can fix this by giving this test the old:

with tmpdir.as_cwd():
   ...

treatment

[blueyed]

@asottile
Yeah, that would also work.

It happens because some .coverage* file might appear with prefix="" I assume (unfortunately it/x is not displayed as local in the traceback?!).

See #4356 for my attempt.
I think as_cwd is better in general, but then should also maybe create some file(s)?

[asottile]

The per-test temp directory always starts empty AFAIK 🤷‍♂️

[blueyed]

The failing/flaky test uses arbitrary dirs for its test, where "" appears
to be flaky (when files get added).

Maybe only this one could be replaced by a clean tmpdir?

But also for the others (e.g. "/") it might make sense to test against either
a fixed list of (generated) files, or just e.g. pytest's own root (assuming
that .coverage files are not created there - maybe testing/ or something else otherwise).