Fix flaky test_rerun test on Windows

[gnikonorov]

test_rerun is flaky only for windows environments. The root cause should be identified and fixed.

I have access to a windows machine, so I'll try and take a look soon. In the meantime, please rerun pipelines if this test fails for windows environments only, as it's most likely due to this issue and not a real problem

FYI: @BeyondEvil @ssbarnea

[BeyondEvil]

Could you describe how it fails and maybe link to a test run where it failed (if it happens in GHA)? @gnikonorov

[gnikonorov]

Sure @BeyondEvil.

This is the output:

_____________________________ TestHTML.test_rerun _____________________________

self = <test_pytest_html.TestHTML object at 0x00000147DD60FA60>
testdir = <Testdir local('C:\\Users\\runneradmin\\AppData\\Local\\Temp\\pytest-of-unknown\\pytest-0\\test_rerun0')>

    def test_rerun(self, testdir):
        testdir.makeconftest(
            """
            import pytest
    
            @pytest.hookimpl(hookwrapper=True)
            def pytest_runtest_makereport(item, call):
                pytest_html = item.config.pluginmanager.getplugin("html")
                outcome = yield
                report = outcome.get_result()
    
                extra = getattr(report, "extra", [])
                if report.when == "call":
                    extra.append(pytest_html.extras.url("http://www.example.com/"))
                report.extra = extra
        """
        )
    
        testdir.makepyfile(
            """
            import pytest
            import time
    
            @pytest.mark.flaky(reruns=2)
            def test_example():
                time.sleep(1)
                assert False
        """
        )
    
        result, html = run(testdir)
        assert result.ret
        assert_results(html, passed=0, failed=1, rerun=2)
    
        expected_report_durations = r'<td class="col-duration">1.\d{2}</td>'
>       assert len(re.findall(expected_report_durations, html)) == 3
E       assert 2 == 3
E         +2
E         -3

D:\a\pytest-html\pytest-html\testing\test_pytest_html.py:224: AssertionError

Here is a link to a sample failure: https://github.com/pytest-dev/pytest-html/pull/405/checks?check_run_id=1524749825

[BeyondEvil]

Such a weird error... did we ever add what the report actually looks like somewhere, or did we only discuss it? @ssbarnea

Does GHA support storing test artifacts?

[BeyondEvil]

Could this be leveraged https://github.com/actions/upload-artifact ?

[gnikonorov]

As per #410 (comment) the current proposal is to rerun the test, since it's flaky.

Personally I'm in favor of this. What do you think @ssbarnea ?

[BeyondEvil]

My reasoning is that rerunning this flaky test is OK until a real user reports it as a bug/problem. Until then, I don't really have the time to chase down a heisenbug that no user has reported (yet).

Hope that makes sense. I'm aiming to draw up a PR for this later tonight once the family is asleep.

[BeyondEvil]

#414 is merged. But I want to keep this issue open. At least for a while.