tox breaks with latest pytest (py.io.TerminalWriter())

[hloeung]

I first reported this against pytest in pytest-dev/pytest#10505

In pytest-dev/pytest@19dda7c, pytest now includes py.py. This causes tox to break as follows:

$ tox -e unittest
Traceback (most recent call last):
  File "/usr/bin/tox", line 33, in <module>
    sys.exit(load_entry_point('tox==3.21.4', 'console_scripts', 'tox')())
  File "/usr/bin/tox", line 25, in importlib_load_entry_point
    return next(matches).load()
  File "/usr/lib/python3.10/importlib/metadata/__init__.py", line 171, in load
    module = import_module(match.group('module'))
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/usr/lib/python3/dist-packages/tox/__init__.py", line 32, in <module>
    from .session import cmdline  # isort:skip
  File "/usr/lib/python3/dist-packages/tox/session/__init__.py", line 20, in <module>
    from tox import reporter
  File "/usr/lib/python3/dist-packages/tox/reporter.py", line 127, in <module>
    _INSTANCE = Reporter()
  File "/usr/lib/python3/dist-packages/tox/reporter.py", line 32, in __init__
    self._reset(**kwargs)
  File "/usr/lib/python3/dist-packages/tox/reporter.py", line 38, in _reset
    self.tw = py.io.TerminalWriter()
AttributeError: module 'py' has no attribute 'io'

Removing ~/.local/lib/python3.10/site-packages/py.py allows it to run successfully.

My steps to reproduce this is to have requirements.txt as follows:

pycountry-convert

Then:

pip install -r requirements.txt

Now check out ~/.local/lib/python3.10/site-packages/ which now has py.py. Then try run tox -e ....

This is with an Ubuntu Jammy/22.04 VM.

This bug reported against pytest has been closed referencing pytest-dev/pytest#10420 (comment) where it mentions "depending on py without declaring it as a dependency".

Is this something that should be fixed in tox itself?

Thanks in advance.

[hloeung]

Oh, actually. I think this should be in my requirements.txt having py. Apologies for the noise.

[rgildein]

Same problem here and installing py<1.10.0 will fix it.

Ubuntu 22.10
Python 3.10.7 
py==1.9.0

[jugmac00]

Hey @hloeung,
is this bug report still valid? You both closed(and re-opened) the issue, and you suggested a solution / workaround?

Anyway... I tried to reproduce it...

• start a container with jammy
• install tox via apt
• run tox with pytest -> no issues

I then had another look at your description, and noticed that you mentioned ~/.local/lib/python3.10/site-packages/.

AFAIK ~/.local/lib/ only gets created when you do a pip install --user <package>.

I then did a pip install --user pytest and boom - I see your traceback.

I assume that - instead of using py as required and installed as a dep of tox now the py module from the user's .local dir was used. I do not think tox can do anything about that.

I would strongly suggest to not use pip install --user <package> to install packages, but rather e.g. https://github.com/pypa/pipx or manually creating virtual envs and then symlinking the binary to the path.

As mentioned, I do not think tox can do anything to prevent this issue... well except dropping the dependency on py completely, which will happen with the upcoming version 4 of tox (no ETA).

This would not help you anytime soon, as you obviously use tox via system packages.

I was about to close this issue, but then just seconds before doing so another report by @rgildein popped up.

@asottile As you worked on the removing py as a dep from pyest, maybe you could have a look whether my analysis is correct or whether we could do anything to prevent breakage for our users?

[jugmac00]

@rgildein Could you please do a ...

$ python3
>>> import py
>>>> py.__file__

... and show us your output?

[rgildein]

I tired this in jammy container as well and it works fine, so I uninstall tox (apt remove tox) from my machine + py=1.9.0 and install tox again. Now it's working fine.

Not sure why it was not working before, because same version of py is used now.

[asottile]

the error indicates you need to install py (probably pip install py --ignore-installed) -- idk what the <1.10 bit is, that doesn't seem needed

[hloeung]

Thanks all for looking into this.

I first closed it off thinking my project can just include py as a requirement and be done with it. But I'm wondering if it's something tox itself can handle and why I re-opened it. Though in hindsight, I should have commented.

This is a recent change only something like 29 days ago per the pytest commit - pytest-dev/pytest@19dda7c#diff-994f6af5db2b5c8a72f4551358eebaa81827cd0fc1a60ac1b60658ee00626182

I'm wondering if tox could try import py.io.TerminalWriter() and fallback to something else if that import fails?

[jugmac00]

My understanding is...

tox depends on and requires the py package for installation - so it should be there. What I understand is that pytest vendored py as a module, and now when e.g. pytest gets installed via pip install --user pytest the py module lands in ~/.local/lib/python3.10/site-packages/ and this py shadows the installed py package, as the vendored py module does not ship the subpackages/modul available in the py package. Now when tox runs in this constellation it goes boom.

[hloeung]

This was actually from a pip install -r requirements.txt which the python package, pycountry-convert (though old and now unmaintained), pulled in pytest & py.

[gaborbernat]

If anything this is a pytest bug, not tox.

[rgildein]

the error indicates you need to install py (probably pip install py --ignore-installed) -- idk what the <1.10 bit is, that doesn't seem needed

What is indicating that py is not installed? There is no message like this ModuleNotFoundError: No module named 'py', but AttributeError: module 'py' has no attribute 'io'. I checked and I had py==1.10.0 installed, so I tried just to downgrade it and that worked.

I think that @hloeung is right and the problem is caused by these changes in pytest. I was able to reproduced this inside container.

$ lxc launch ubuntu-daily:jammy tox
$ lxc shell tox
root@tox:~# su ubuntu
ubuntu@tox:/root$ cd ~
ubuntu@tox:~$ sudo apt update && sudo apt install tox -y
ubuntu@tox:~$ tee -a tox.ini <<EOF
[tox]
skipsdist=True
envlist = test

[testenv:test]
deps = pytest
commands = pytest --version
EOF
ubuntu@tox:~$ tox
tox 
test create: /home/ubuntu/.tox/test
test installdeps: pytest
test installed: attrs==22.1.0,exceptiongroup==1.0.4,iniconfig==1.1.1,packaging==21.3,pluggy==1.0.0,pyparsing==3.0.9,pytest==7.2.0,tomli==2.0.1
test run-test-pre: PYTHONHASHSEED='3983464967'
test run-test: commands[0] | pytest --version
pytest 7.2.0
_________________________________________________________________ summary __________________________________________________________________
  test: commands succeeded
  congratulations :)
ubuntu@tox:~$ pip3 install pytest
Defaulting to user installation because normal site-packages is not writeable
Collecting pytest
  Using cached pytest-7.2.0-py3-none-any.whl (316 kB)
Requirement already satisfied: tomli>=1.0.0 in ./.local/lib/python3.10/site-packages (from pytest) (2.0.1)
Requirement already satisfied: exceptiongroup>=1.0.0rc8 in ./.local/lib/python3.10/site-packages (from pytest) (1.0.4)
Requirement already satisfied: iniconfig in ./.local/lib/python3.10/site-packages (from pytest) (1.1.1)
Requirement already satisfied: pluggy<2.0,>=0.12 in /usr/lib/python3/dist-packages (from pytest) (0.13.0)
Requirement already satisfied: packaging in /usr/lib/python3/dist-packages (from pytest) (21.3)
Requirement already satisfied: attrs>=19.2.0 in /usr/lib/python3/dist-packages (from pytest) (21.2.0)
Installing collected packages: pytest
  WARNING: The scripts py.test and pytest are installed in '/home/ubuntu/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed pytest-7.2.0
ubuntu@tox:~$ tox
Traceback (most recent call last):
  File "/usr/bin/tox", line 33, in <module>
    sys.exit(load_entry_point('tox==3.21.4', 'console_scripts', 'tox')())
  File "/usr/bin/tox", line 25, in importlib_load_entry_point
    return next(matches).load()
  File "/usr/lib/python3.10/importlib/metadata/__init__.py", line 171, in load
    module = import_module(match.group('module'))
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/usr/lib/python3/dist-packages/tox/__init__.py", line 32, in <module>
    from .session import cmdline  # isort:skip
  File "/usr/lib/python3/dist-packages/tox/session/__init__.py", line 20, in <module>
    from tox import reporter
  File "/usr/lib/python3/dist-packages/tox/reporter.py", line 127, in <module>
    _INSTANCE = Reporter()
  File "/usr/lib/python3/dist-packages/tox/reporter.py", line 32, in __init__
    self._reset(**kwargs)
  File "/usr/lib/python3/dist-packages/tox/reporter.py", line 38, in _reset
    self.tw = py.io.TerminalWriter()
AttributeError: module 'py' has no attribute 'io'

I don't think this is an issue that should be solved by tox. I'll report a bug for pytest.