Misleading pytest exception description

[pytestbot]

Originally reported by: Jurko Gospodnetić (BitBucket: jurko, GitHub: jurko)

---

Today, while mentoring a new colleague in Python and TDD, she ran into a pytest exception description which confused her, and with her being a novice programmer, was completely out of her league to debug.

Here is a reduced script demonstrating the issue:

InvalidLabSize = -1

class Lab:
    def __init__(self, size):
        if size <= 0:
            raise Exception(InvalidLabSize)

import pytest

def test_invalid_lab_size():
    with pytest.raises(InvalidLabSize):
        Lab(0)

The given test code has a bug - pytest.raises() should be given Exception class as its parameter and not an integer InvalidLabSize. However, pytest
reports this in a most misleading way:

#!text
D:\Workplace>py3 -m pytest aaa.py
============================= test session starts =============================
platform win32 -- Python 3.3.3 -- py-1.4.20 -- pytest-2.5.2
collected 1 items

aaa.py F

================================== FAILURES ===================================
____________________________ test_invalid_lab_size ____________________________

    def test_invalid_lab_size():
        with pytest.raises(InvalidLabSize):
>           Lab(0)
E           TypeError: issubclass() arg 2 must be a class or tuple of classes

aaa.py:15: TypeError
========================== 1 failed in 0.05 seconds ===========================

First, it points to an incorrect code line - it points to the code line Lab(0) as causing the error when in fact it was the line before that caused it.
Second, it complains about some 'argument number 2' to some issubclass() call having to be a class or a tuple of classes, when neither the user code nor anything displayed in the given exception stack information contains any issubclass() calls.

It should in fact say that the first parameter to pyteset.raises() should be a class or a tuple of classes.

---

As a side-note, I tried rewriting the test using the older pytest.raises() calling style (without using a with context manager):

#!python
InvalidLabSize = -1

class Lab:
    def __init__(self, size):
        if size <= 0:
            raise Exception(InvalidLabSize)

import pytest

def test_invalid_lab_size():
    pytest.raises(InvalidLabSize, Lab, 0)

and this reports much more relevant information when run under Python 3.x:

D:\Workplace>py3 -m pytest aaa.py
============================= test session starts =============================
platform win32 -- Python 3.3.3 -- py-1.4.20 -- pytest-2.5.2
collected 1 items

aaa.py F

================================== FAILURES ===================================
____________________________ test_invalid_lab_size ____________________________

    def test_invalid_lab_size():
>       pytest.raises(InvalidLabSize, Lab, 0)
E       TypeError: catching classes that do not inherit from BaseException is not allowed

aaa.py:31: TypeError
========================== 1 failed in 0.04 seconds ===========================

Running it under Python 2.x on the other hand displays a bit less but still misleading information:

D:\Workplace>py2 -m pytest aaa.py
============================= test session starts =============================
platform win32 -- Python 2.7.6 -- py-1.4.20 -- pytest-2.5.2
collected 1 items

aaa.py F

================================== FAILURES ===================================
____________________________ test_invalid_lab_size ____________________________

    def test_invalid_lab_size():
>       pytest.raises(InvalidLabSize, Lab, 0)

aaa.py:31:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <aaa.Lab instance at 0x0000000003043B88>, size = 0

    def __init__(self, size):
        if size <= 0:
>           raise Exception(InvalidLabSize)
E           Exception: -1

aaa.py:24: Exception
========================== 1 failed in 0.12 seconds ===========================

It says that an Exception got raised when it should in fact say that an Exception got raised but an integer -1 was expected.

Hope this helps.

Best regards,
Jurko Gospodnetić

---

• Bitbucket: https://bitbucket.org/pytest-dev/pytest/issue/475

[pytestbot]

Original comment by Floris Bruynooghe (BitBucket: flub, GitHub: flub):

---

So it seems that python 3 only allows subclasses from BaseException, we could enforece this in raises? This is likely too strict though for backwards compatibility so maybe simply checking it is a class or tuple as suggested is better.

[pytestbot]

Original comment by Jurko Gospodnetić (BitBucket: jurko, GitHub: jurko):

---

In both py2 & py3 pytest.raises() should take a class or a tuple of classes as a parameter. Otherwise an internal issubclass() check will fail as expected (on the passed class or the items in the passed tuple) and display a confusing/misleading error message.

On py3 it would also be nice to check that the passed classes are subclasses of BaseException, but that's a separate and I guess an unrelated issue.

Best regards,
Jurko Gospodnetić

[pytestbot]

Original comment by holger krekel (BitBucket: hpk42, GitHub: hpk42):

---

anyone up for a PR?

[pytestbot]

Original comment by holger krekel (BitBucket: hpk42, GitHub: hpk42):

---

@flub has just fixed this but forgot to reference from the commit :)