xfail behaving more like skip

[mentaal]

Hi there,
In the docs for xfail it says:

def test_function2():
    import slow_module

    if slow_module.slow_function():
        pytest.xfail("slow_module taking too long")

These two examples illustrate situations where you don’t want to check for a condition at the module level, which is when a condition would otherwise be evaluated for marks.

This will make test_function XFAIL. Note that no other code is executed after the pytest.xfail call, differently from the marker. That’s because it is implemented internally by raising a known exception.

The behavior describes doesn't really sound like xfail to me. It's more like skip. The testcase isn't run. If I put the pytest.xfail call at the end of the test, an exception will be thrown before I get to it.

Am I misunderstanding. What's the intended usage for this?
The reason what I want a dynamic xfail is because I have to read information during the test and use that to determine if the test can be marked as xfail.

Thank you

[RonnyPfannschmidt]

as documented jsut below the example, the imperative xfail indeed behaves like the imperative skip
as its implemented in terms of a specific exception

as far as i understood your use-case is triggering a declared xfail from within the test
(whic would mean still running the code expected to fail)

we currently don't have such a concept available, but it is a desired feature as this is a case where a surprise XPASS is valuable information

i vaguely recall we have a issue about this but i don't remember the details, so i wont try to search it for now

[mentaal]

Ok let me try and explain with an example:

def test_xfail_1():
    test_should_fail_today = True
    if test_should_fail_today:
        pytest.xfail("I'm gonna fail but I still expect to execute")
    print("I'm doing stuff..")
    raise ValueError("Boooo")

Results in:
..\..\..\temp\test_xfail.py::test_xfail_1 XFAIL

This isn't an xfail... it's a skip.
Try again:

def test_xfail_2():
    test_should_fail_today = True
    print("I'm doing stuff..")
    raise ValueError("Boooo")
    if test_should_fail_today:
        pytest.xfail("I'm gonna fail but I still expect to execute")

Results in:
E ValueError: Boooo
Not an xfail at all, though obviously the reason is understood.

I did read the documentation I quoted. The reason is clear and I understand it. My point is if the usage in my first example here is how it is intended to be used, this is semantically not xfail at all and the user may as well just use pytest.skip.

There is one way around this which does yield semantically correct behavior:

def test_xfail():
    test_should_fail_today = True
    try:
        print("I'm doing stuff..")
        raise ValueError("Boooo")
    except ValueError:
        if test_should_fail_today:
            pytest.xfail("I'm gonna fail but I still expect to execute")

Yields:

..\..\..\temp\test_xfail.py::test_xfail I'm doing stuff..
XFAIL

If this is the intention usage of imperative xfail, then perhaps the docs should be adjusted to reflect this.

[RonnyPfannschmidt]

That's not the intended usage, thats a bad workaround for the missing imperative marking

[mentaal]

Can you please elaborate?

[RonnyPfannschmidt]

imperative xfail is intended to exit a test right at the point where it happens, indicating a known hopeless case

we do not have a delayed xfail which is applied dynamically and then is evaluated after a failure/pass

[mentaal]

Which in my head at least is an imperative skip for the reasons described. To refer to it otherwise is confusing because the semantics are different to the declarative variant. I'll beg to disagree on the bad workaround as it matches more closely the semantics of xfail. I'm not stuck or anything so I'll close this ticket.
Thanks for taking the time.