Skip to content

Ensure all test reports are processed regardless of prior report status #124

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 3 commits into from
Closed

Ensure all test reports are processed regardless of prior report status #124

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
6d07db3
ensure that we call pytest_runtest_logreport for all reports of tests
gnikonorov Aug 2, 2020
2685d80
Add changelog entry
gnikonorov Aug 2, 2020
97806ab
fixup some comments
gnikonorov Aug 2, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CHANGES.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,10 @@ Changelog
(`#117 <https://github.com/pytest-dev/pytest-rerunfailures/pull/117>`_)
(PR from `@gnikonorov`_)

- Ensure ``pytest_runtest_logreport`` is called for all reports in a test.
(`#108 <https://github.com/pytest-dev/pytest-rerunfailures/issues/108>`_)
(PR from `@gnikonorov`_)

.. _@gnikonorov: https://github.com/gnikonorov


Expand Down
27 changes: 16 additions & 11 deletions pytest_rerunfailures.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -226,40 +226,45 @@ def pytest_runtest_protocol(item, nextitem):
parallel = hasattr(item.config, "slaveinput")
item.execution_count = 0

need_to_run = True
while need_to_run:
need_to_rerun = True
while need_to_rerun:
item.execution_count += 1
item.ihook.pytest_runtest_logstart(nodeid=item.nodeid, location=item.location)
reports = runtestprotocol(item, nextitem=nextitem, log=False)

need_to_rerun = False
for report in reports: # 3 reports: setup, call, teardown
is_terminal_error = _should_hard_fail_on_error(item.session.config, report)
out_of_runs = item.execution_count > reruns
report.rerun = item.execution_count - 1
xfail = hasattr(report, "wasxfail")
if (
item.execution_count > reruns
out_of_runs
or not report.failed
or xfail
or is_terminal_error
or need_to_rerun
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With this way of writing we have 5 possible cases in this branch, which will never be properly addressed by any branch coverage. I would prefer a way, so that we need a testcase to make the coverage happy. And I would prefer a testcase.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This hook was not called for reports of a test run beyond the first failure, which this PR fixes. So, before this PR if a test fails in phase setup, its call and teardown reports aren't processed ( in this case all we do for processing is call pytest_runtest_logreport

Now that I look at my change again I have a general question. Why do we not call pytest_runtest_logreport for failed test phases when xdist or parellel?

):
# last run or no failure detected, log normally
# out of rerun attempts, no errors detected, or we're finishing off
# the reports of a failed test, log normally
item.ihook.pytest_runtest_logreport(report=report)
else:
# failure detected and reruns not exhausted, since i < reruns
need_to_rerun = True
report.outcome = "rerun"
time.sleep(delay)

if not parallel or works_with_current_xdist():
# will rerun test, log intermediate result
item.ihook.pytest_runtest_logreport(report=report)

# cleanin item's cashed results from any level of setups
_remove_cached_results_from_failed_fixtures(item)
_remove_failed_setup_state_from_session(item)
if need_to_rerun:
# at least one report failed and we have reruns left
# prepare to rerun the test
time.sleep(delay)

break # trigger rerun
else:
need_to_run = False
# cleanin item's cashed results from any level of setups
_remove_cached_results_from_failed_fixtures(item)
_remove_failed_setup_state_from_session(item)

item.ihook.pytest_runtest_logfinish(nodeid=item.nodeid, location=item.location)

Expand Down