Tests no longer test local behavior

[jaraco]

I've observed that the tests for six aren't actually testing new behavior... because they install the released version of six. This is because tox uses pip to install . and because six has manual versioning so the version in the repo is 1.10.0 (current release) which is already satisfied in a new virtualenv with setuptools installed. Worse yet, if one bumps the version to try to force the repo version to be installed, it triggers pypa/pip#4264.

Adding usedevelop=True to the test environment seems to help with the first issue, causing the second issue to be triggered immediately.

[jaraco]

This issue probably emerged with Setuptools 34 due to pypa/setuptools#581.

I've filed tox-dev/tox#467 to capture the unexpected behavior in tox.

[jaraco]

I've pushed changes to a branch demonstrating that my assertions about usedevelop and fixed pip versions correct the issue.

[benjaminp]

Thanks a lot for noticing and looking into this.

[leorochael]

I have found a simpler workaround in tox.ini that doesn't require the experimental pip:

[testenv]
deps= pytest
setenv=
    PYTHONPATH=.
commands= py.test -rfsxX {posargs}

[leorochael]

BTW, without either the workaround above or @jaraco's, I get errors like this:

==================================== FAILURES =====================================
_________________________________ test_getoutput __________________________________

    def test_getoutput():
>       from six.moves import getoutput
E       ImportError: cannot import name getoutput

test_six.py:236: ImportError
______________________________ test_getstatusoutput _______________________________

    def test_getstatusoutput():
>       from six.moves import getstatusoutput
E       ImportError: cannot import name getstatusoutput

test_six.py:242: ImportError
================= 2 failed, 185 passed, 1 skipped in 0.23 seconds =================

[jaraco]

This issue no longer exists, due to 0d89d58, which because it's invoking python and not some script, causes sys.path[0] == '.', leading to the effect suggested here, and causes ./six.py to be tested instead of .tox/python/lib/python3.7/site-packages/six.py. In fact, the project could also skip the install altogether

diff --git a/tox.ini b/tox.ini
index fe56730..fa9c1a2 100644
--- a/tox.ini
+++ b/tox.ini
@@ -7,6 +7,7 @@ indexserver=
 [testenv]
 deps= pytest
 commands= python -m pytest -rfsxX {posargs}
+skip_install = True

 [testenv:flake8]
 basepython=python

And tests still pass.

Moreover, late virtualenvs no longer get six by default, so the behavior that instigated this issue is no longer present as well.

There's still an issue there are two different copies of six around when running the tox tests (./six.py and the one in site-packages), but because tox copies/installs the file on every run, I guess that's okay.