self._outcome.result is not a unittest.TestResult when subclassing unittest.TestCase on Python 3.11

[johnthacker]

There's a reasonably common idiom to get results from a test that subclass unittest.TestCase inside a tearDown() method by accessing self._outcome.errors (see #7000). In Python 3.11, unittest has changed so that self._outcome.errors doesn't exist (the _Outcome object does not have the errors attribute), but a popular workaround suggested on StackOverflow involves accessing self._outcome.result on Python 3.11 (and later), which is a unittest.TestResult that has members errors and failures (and also skipped) and can be processed in the same way.

However, when running tests inside pytest, self._outcome.result inside the tearDown() method is not a unittest.TestResult, but a pytest._unittest.TestCaseFunction, which means that the above snipped from StackOverflow that works, e.g., with CTest, will produce AttributeErrors and report test failures:

        if self._outcome:
            if hasattr(self._outcome, 'errors'):
                # Python 3.4 - 3.10
                result = self.defaultTestResult()
                self._feedErrorsToResult(result, self._outcome.errors)
            else:
                # Python 3.11+
                result = self._outcome.result
>           for test_case, exc_info in (result.errors + result.failures):
E           AttributeError: 'TestCaseFunction' object has no attribute 'errors'

Wireshark uses this idiom for our testing, and while our standard CI/CD tests use CTest (with a test target inside cmake), some developers appreciate being able to run the tests within pytest, which worked up until Python 3.11.

Installed versions

pytest 7.1.3, Fedora 37, Python 3.11.1

$ pip list
Package              Version
-------------------- -------------------------
aiodns               3.0.0
aiohttp              3.8.3
aiosignal            1.3.1
apipkg               3.0.1
appdirs              1.4.4
argcomplete          2.0.0
asttokens            2.0.8
async-timeout        4.0.2
atomicwrites         1.4.0
attrs                22.1.0
Automat              20.2.0
Babel                2.10.3
backcall             0.1.0
Beaker               1.10.0
beautifulsoup4       4.11.0
black                22.8.0
blivet               3.5.0
blivet-gui           2.4.0
Brlapi               0.8.4
Brotli               1.0.9
cached-property      1.5.2
certifi              2021.10.8
cffi                 1.15.1
chardet              5.1.0
charset-normalizer   2.1.0
click                8.1.3
constantly           15.1.0
construct            2.10.68
contourpy            1.0.6
cryptography         37.0.2
cupshelpers          1.0
cycler               0.11.0
dasbus               1.6
dbus-python          1.3.2
decorator            5.1.1
deluge               2.1.1
devedeng             4.17.0
discid               1.2.0
distro               1.7.0
dnspython            2.2.1
execnet              1.9.0
executing            1.1.0
fail2ban             1.0.2
fasteners            0.18
fedora-third-party   0.10
feedparser           6.0.10
file-magic           0.4.0
fonttools            4.37.2
fros                 1.1
frozenlist           1.3.3
fs                   2.4.11
future               0.18.2
GeoIP                1.3.2
git-review           2.3.1
gpg                  1.17.0
gssapi               1.7.3
html2text            2020.1.16
humanize             3.13.1
hyperlink            21.0.0
idna                 3.3
incremental          21.3.0
iniconfig            1.1.1
ipython              8.5.0
ipython-genutils     0.2.0
jedi                 0.18.1
Jinja2               3.0.3
kiwisolver           1.4.4
koji                 1.31.0
langtable            0.0.61
libcomps             0.1.18
libtorrent           2.0.8
lxml                 4.9.1
maildirproc          1.0.1
Mako                 1.1.4
Markdown             3.4.1
MarkupSafe           2.1.1
matplotlib           3.6.2
matplotlib-inline    0.1.2
more-itertools       8.12.0
multidict            6.0.2
munkres              1.1.2
musicbrainzngs       0.7.1
mutagen              1.45.1
mypy-extensions      0.4.3
mysqlclient          2.1.1
MythTV               32.0.-1
nftables             0.1
ntpsec               1.2.1
numpy                1.22.0
offlineimap          8.0.0
olefile              0.46
packaging            21.3
parso                0.8.3
Paste                3.5.0
pathspec             0.10.1
pexpect              4.8.0
picard               2.8.5
pickleshare          0.7.5
pid                  2.2.3
Pillow               9.2.0
pip                  22.2.2
platformdirs         2.5.2
pluggy               1.0.0
ply                  3.11
productmd            1.33
progressbar2         3.53.2
prompt-toolkit       3.0.29
ptyprocess           0.6.0
pure-eval            0.2.2
pwquality            1.4.4
py                   1.11.0
pyasn1               0.4.8
pyasn1-modules       0.2.8
pybeam               0.7
pycairo              1.21.0
pycares              4.3.0
pycparser            2.20
pycrypto             2.6.1
pycryptodomex        3.15.0
pycups               2.0.1
pycurl               7.45.1
pyenchant            3.2.2
pygame               2.1.2
pygit2               1.7.1
Pygments             2.12.0
PyGObject            3.42.2
pyinotify            0.9.6
PyJWT                2.4.0
pykickstart          3.41
pyOpenSSL            21.0.0
pyparsing            3.0.9
pyparted             3.12.0
PyQt5                5.15.6
PyQt5-sip            12.11.0
pyroute2             0.6.12
pyroute2.core        0.6.12
pyroute2.ethtool     0.6.12
pyroute2.ipdb        0.6.12
pyroute2.ipset       0.6.12
pyroute2.ndb         0.6.12
pyroute2.nftables    0.6.12
pyroute2.nslink      0.6.12
PySocks              1.7.1
pytest               7.1.3
pytest-forked        1.4.0
pytest-xdist         2.5.0
python-augeas        1.1.0
python-dateutil      2.8.2
python-libdiscid     2.0.1
python-meh           0.50
python-utils         3.1.0
pytz                 2022.7
pyudev               0.23.2
PyX                  0.15
pyxdg                0.27
PyYAML               6.0
quodlibet            4.5.0
regex                2022.10.31
rencode              1.0.6
reportlab            3.6.11
requests             2.28.1
requests-cache       0.5.1
requests-file        1.5.1
requests-ftp         0.3.1
requests-gssapi      1.2.3
rpm                  4.18.0
rpmautospec          0.3.0
rpmconf              1.1.7
rpmlint              2.4.0
rss2email            3.14
scapy                git-archive.dev8b63d73a17
SciPy                1.8.1
selinux              3.4
sepolicy             3.4
service-identity     21.1.0
setools              4.4.0
setproctitle         1.2.3
setuptools           62.6.0
sgmllib3k            1.0.0
simpleaudio          1.0.4
simpleline           1.9.0
six                  1.16.0
slip                 0.6.4
slip.dbus            0.6.4
sos                  4.4
soundconverter       4.0.3
soupsieve            2.3.2.post1
stack-data           0.3.0
systemd-python       235
Tempita              0.5.2
templated-dictionary 1.1
toml                 0.10.2
tomli                2.0.1
tomli_w              1.0.0
traitlets            5.2.1.post0
Twisted              22.4.0
typing_extensions    4.2.0
urllib3              1.26.12
wcwidth              0.2.5
websockets           10.3
yarl                 1.7.2
youtube-dl           2021.12.17
yt-dlp               2022.11.11
zope.event           4.5.0
zope.interface       5.4.0
zstandard            0.19.0
zstd                 1.4.5.1

[Zac-HD]

The leading underscore in ._outcome indicates that this is a private attribute, and I'm really not keen to add special-case support for something which is explicitly unsupported upstream. Fundamentally, we just don't store errors in the same way as unittest.

[DiegoF90]

@Zac-HD please can you let us know if there is an alternative to get the list of errors, failures, etc we where getting using that private attribute?

[RonnyPfannschmidt]

there currently is no alternative - based on the code in unittest and pytests unittest plugin it seems possible to create a custom result class that reports to pytest, however currently nobody is putting dedicated work into the unittest plugin

i'd recommend creating a mr as experiment to see if it can be hoisted - it would be a good idea to split the result class from the unittest item anyway,

based on the runners in unittest, management of cleanup is unclear tho