Dynamically generated test methods not distinguishable on failure

[volans-]

I'm dynamically generating test methods in a test class with different parameters, setting the __name__ and __doc__ of the methods to be able to distinguish them. When I was running the tests with nose, on failure the name and the docstring of the failed method were printed and I could easily recognize which of the generated test was failing.

I'm moving to pytest but I'm facing the problem that only the name of the function is printed and not the docstring, that is the one where I was saving all the parameters with which the function was generated, that allowed to clearly distinguish them. I cannot embed all the parameters in the test name of course.
Regarding the code that pytest prints on failure, is the generic one so it doesn't help at all in this case.
I have hence no easy way of distinguish which test failed without the use of some hack like printing to stdout/err the info I need, but de facto altering the tests given that those specific ones are testing a command line application and it's output.

One easy solution would be to implement the feature request in #445, printing the docstring of the failed method along with it's name.
The other possibility would be to show the actually generated code instad of the generating one.

pytest version 3.1.2
pytest-xdist-1.17.1
pytest-cov-2.5.1
Python 2.7.10

[nicoddemus]

Thanks for writing @volans-.

One easy solution would be to implement the feature request in #445, printing the docstring of the failed method along with it's name.

Hmm so your suggestion would be to show the docstring somewhere with the failure? As mentioned in #445, in general the docstring already appears in the traceback (excuse the dummy example):

================================== FAILURES ===================================
_______________________________ test_protocol_A _______________________________

    def test_protocol_A():
        """
        Ensure communication follows protocol RCF A1029.
    
        The protocol blah blah
        """
>       communicate()

foo.py:13: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

    def communicate():
>       assert 0, 'connection failed'
E       AssertionError: connection failed
E       assert 0

foo.py:4: AssertionError
========================== 1 failed in 0.04 seconds ===========================

So in general adding the docstring would be noise for users in general.

Perhaps displaying the docstring when --tb=short?

================================== FAILURES ===================================
_______________________________ test_protocol_A _______________________________
              """Ensure communication follows protocol RCF A1029."""
foo.py:13: in test_protocol_A
    communicate()
foo.py:4: in communicate
    assert 0, 'connection failed'
E   AssertionError: connection failed
E   assert 0
========================== 1 failed in 0.03 seconds ===========================

That actually looks nice, but I don't think it will solve your particular problem (I assume you use the default traceback option).

The other possibility would be to show the actually generated code instad of the generating one.

I believe you are generating the code programatically using function wrappers and such right? Unfortunately that's not trivial to do (in fact I don't think it is possible, but I might be wrong).

---

One solution I can suggest is to add another section to the test report, like pytest does already for "Captured stdout" and "Capture stderr".

Suppose this example which I think reproduces your use case:

import pytest


def check_greater_than(x, y):
    assert x > y


def make_test(x, y, extra):
    @pytest.mark.custom_params(x=x, y=y, extra=extra)
    def test():
        check_greater_than(x, y)
    return test


test_1 = make_test(6, 4, 'foo')
test_2 = make_test(1, 4, 'bar')
test_3 = make_test(9, 1, 'fizz')
test_4 = make_test(2, 9, 'foobar')

Note that I'm marking the test function with the custom parameters used.

Add this to your conftest.py file at the root test directory:

import pytest

@pytest.hookimpl(hookwrapper=True)
def pytest_runtest_makereport(item, call):
    outcome = yield
    m = item.get_marker('custom_params')
    if m:
        rep = outcome.get_result()
        rep.sections.append(('Custom params', sorted(m.kwargs.items())))

Here's how the test run looks like:

================================== FAILURES ===================================
___________________________________ test_2 ____________________________________
foo.py:11: in test
    check_greater_than(x, y)
foo.py:5: in check_greater_than
    assert x > y
E   assert 1 > 4
-------------------------------- Custom params --------------------------------
[('extra', 'bar'), ('x', 1), ('y', 4)]
___________________________________ test_4 ____________________________________
foo.py:11: in test
    check_greater_than(x, y)
foo.py:5: in check_greater_than
    assert x > y
E   assert 2 > 9
-------------------------------- Custom params --------------------------------
[('extra', 'foobar'), ('x', 2), ('y', 9)]
===================== 2 failed, 2 passed in 9.70 seconds ======================

Note that you can add any amount of text as the second value of the tuple in sections.append, I just went with the shorter route of displaying the bare parameters dictionary.

[volans-]

@nicoddemus thanks a lot for your quick and detailed reply!

Regarding #445 I was thinking maybe to print the docstring only if the name of the function in the report (like test_2 in your last example) is different from the name of the function in the code (def test(): for example) for the general case with default traceback.
For the --tb=short case it might be useful to show it all the time, but I'm sure everyone has a different opinion on this 😉
Also while this would cover my use case, there may be different use cases I'm not aware for which this would not be a good solution anyway.

Regarding your suggested solution is very interesting, I didn't know I could add output sections (my bad). I've played a bit with it and it works great and definitely solve my issue, again, thanks a lot!
(for the record I've inserted it in first position insert(0, ...), to have the parameters right below the traceback, before the stdout/err sections.

Resolving this as there is a fair workaround to it and I guess this is not the best place to continue the discussion about #445.