ExitCode: support custom codes

[blueyed]

Ref: #5420

[blueyed]

Fails on py35:

Traceback (most recent call last):
  File "d:\a\1\s\.tox\py35-xdist\lib\site-packages\pytest.py", line 17, in <module>
    from _pytest.main import ExitCode
  File "d:\a\1\s\.tox\py35-xdist\lib\site-packages\_pytest\main.py", line 21, in <module>
    class ExitCode(enum.IntEnum):
  File "d:\a\1\s\.tox\py35-xdist\lib\site-packages\_pytest\main.py", line 43, in ExitCode
    @classmethod
  File "C:\hostedtoolcache\windows\Python\3.5.4\x64\Lib\enum.py", line 61, in __setitem__
    raise ValueError('_names_ are reserved for future Enum use')
ValueError: _names_ are reserved for future Enum use

[blueyed]

Do we want to monkeypatch py35, or only have it for py36+ then?
It affects plugins using non-standard exit codes for example.

[RonnyPfannschmidt]

The refed pr intentionally locked down exit codes

[blueyed]

Update the doc then?
https://github.com/pytest-dev/pytest/pull/6022/files#diff-7191acb1a6aaab2f77b6e53ce837e348L27

But I think we should not prevent custom exit codes, should we?

[RonnyPfannschmidt]

What actual value do they add that justifies enabling people to break the pytest cli api

[blueyed]

I rather think that we should not break plugins doing it already (for whatever reason).

btw: came to it via #4901.

[bluetech]

Didn't know _missing_ existed - interesting.

If you decide that ExitCode should be an open enum, then the question is how exit code variables/arguments are intended to be type-annotated.

If annotated as ExitCode, then this is a good approach. Users will need to make sure to wrap custom exit codes with ExitCode(123).

If annotated as int, then this will confuse things somewhat. ExitCode is already an IntEnum, so it will be better to keep it just for well-known codes.

@RonnyPfannschmidt wrote in #5420 that "User defined exit codes are still valid, but should be used with caution.", so it sounds like it should be an int for now, no?

[nicoddemus]

@RonnyPfannschmidt wrote in #5420 that "User defined exit codes are still valid, but should be used with caution.", so it sounds like it should be an int for now, no?

I agree.

About #4901, we just need to update the representation to use str(retcode):

>>> from enum import IntEnum
>>> class X(IntEnum):
...     A = 1
...     B = 2
...
>>> str(X.A)
'X.A'

?

EDIT: of course that doesn't work because retcode is obtained from the system.

[nicoddemus]

Submitted a different proposal at #6043

[blueyed]

Cool, closing then.

[blueyed]

It's a bit bad though having to use it in try … except ValueError though, e.g. with pytester: https://github.com/pytest-dev/pytest/pull/6025/files