Skip to content

regrtest: add --bisect option to run test.bisect_cmd on failed tests #115122

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
vstinner opened this issue Feb 7, 2024 · 2 comments
Closed

regrtest: add --bisect option to run test.bisect_cmd on failed tests #115122

vstinner opened this issue Feb 7, 2024 · 2 comments
Labels
tests Tests in the Lib/test dir type-feature A feature request or enhancement

Comments

@vstinner
Copy link
Member

vstinner commented Feb 7, 2024 •
edited by bedevere-app bot
Loading

When running tests with -R 3:3 to check for a reference leak, it's not always obvious which tests lead to the leak. I propose to add a --bisect option which runs the -m test.bisect_cmd command to bisect failing tests.

Linked PRs
vstinner added a commit to vstinner/cpython that referenced this issue Feb 7, 2024
@vstinner
pythongh-115122: Add --bisect option to regrtest
09efe89 
* test.bisect_cmd now exit with code 0 on success, and code 1 on
  failure. Before, it was the opposite.
* --fast-ci and --slow-ci options set --bisect option.
* Add create_python_cmd() and bisect_cmd() methods to RunTests.
vstinner added a commit to vstinner/cpython that referenced this issue Feb 7, 2024
@vstinner
pythongh-115122: Add --bisect option to regrtest
d2d345b 
* test.bisect_cmd now exit with code 0 on success, and code 1 on
  failure. Before, it was the opposite.
* --fast-ci and --slow-ci options set --bisect option.
* Add create_python_cmd() and bisect_cmd() methods to RunTests.
@bedevere-app bedevere-app bot mentioned this issue Feb 7, 2024
Merged
@Eclips4 Eclips4 added tests Tests in the Lib/test dir type-feature A feature request or enhancement labels Feb 7, 2024
@vstinner
Copy link
Member Author

vstinner commented Feb 7, 2024

I wrote this issue to debug issue gh-115124: AMD64 Windows11 Bigmem 3.x: test_bigmem failed with !_Py_IsImmortal(FROM_GC(gc)) assertion error.

@vstinner vstinner mentioned this issue Feb 7, 2024
Closed
vstinner added a commit to vstinner/cpython that referenced this issue Feb 7, 2024
@vstinner
pythongh-115122: Add --bisect option to regrtest
4421e42 
* test.bisect_cmd now exit with code 0 on success, and code 1 on
  failure. Before, it was the opposite.
* --fast-ci and --slow-ci options set --bisect option.
* Add create_python_cmd() and bisect_cmd() methods to RunTests.
vstinner added a commit to vstinner/cpython that referenced this issue Feb 18, 2024
@vstinner
pythongh-115122: Add --bisect option to regrtest
244bd5f 
* test.bisect_cmd now exit with code 0 on success, and code 1 on
  failure. Before, it was the opposite.
* --fast-ci and --slow-ci options set --bisect option.
* Add create_python_cmd() and bisect_cmd() methods to RunTests.
vstinner added a commit to vstinner/cpython that referenced this issue Feb 18, 2024
@vstinner
pythongh-115122: Add --bisect option to regrtest
167707c 
* test.bisect_cmd now exit with code 0 on success, and code 1 on
  failure. Before, it was the opposite.
* test.bisect_cmd now runs the test worker process with
  -X faulthandler.
* regrtest RunTests: Add create_python_cmd() and bisect_cmd()
  methods.
vstinner added a commit to vstinner/cpython that referenced this issue Feb 18, 2024
@vstinner
pythongh-115122: Add --bisect option to regrtest
7cbb66d 
* test.bisect_cmd now exit with code 0 on success, and code 1 on
  failure. Before, it was the opposite.
* test.bisect_cmd now runs the test worker process with
  -X faulthandler.
* regrtest RunTests: Add create_python_cmd() and bisect_cmd()
  methods.
vstinner added a commit to vstinner/cpython that referenced this issue Feb 18, 2024
@vstinner
pythongh-115122: Add --bisect option to regrtest
e87af95 
* test.bisect_cmd now exit with code 0 on success, and code 1 on
  failure. Before, it was the opposite.
* test.bisect_cmd now runs the test worker process with
  -X faulthandler.
* regrtest RunTests: Add create_python_cmd() and bisect_cmd()
  methods.
vstinner added a commit to vstinner/cpython that referenced this issue Feb 18, 2024
@vstinner
pythongh-115122: Add --bisect option to regrtest
156297d 
* test.bisect_cmd now exit with code 0 on success, and code 1 on
  failure. Before, it was the opposite.
* test.bisect_cmd now runs the test worker process with
  -X faulthandler.
* regrtest RunTests: Add create_python_cmd() and bisect_cmd()
  methods.
vstinner added a commit that referenced this issue Feb 18, 2024
@vstinner
gh-115122: Add --bisect option to regrtest (#115123)
1e5719a 
* test.bisect_cmd now exit with code 0 on success, and code 1 on
  failure. Before, it was the opposite.
* test.bisect_cmd now runs the test worker process with
  -X faulthandler.
* regrtest RunTests: Add create_python_cmd() and bisect_cmd()
  methods.
woodruffw pushed a commit to woodruffw-forks/cpython that referenced this issue Mar 4, 2024
@vstinner @woodruffw
pythongh-115122: Add --bisect option to regrtest (python#115123)
3cbf4d8 
* test.bisect_cmd now exit with code 0 on success, and code 1 on
  failure. Before, it was the opposite.
* test.bisect_cmd now runs the test worker process with
  -X faulthandler.
* regrtest RunTests: Add create_python_cmd() and bisect_cmd()
  methods.
@hugovk
Copy link
Member

hugovk commented Mar 15, 2024

Closing because the PR has been merged. Please re-open if there's more needed here. Thanks!

@hugovk hugovk closed this as completed Mar 15, 2024
vstinner added a commit to vstinner/cpython that referenced this issue Mar 26, 2024
@vstinner
pythongh-115122: Add --bisect option to regrtest (python#115123)
a3c5f91 
* test.bisect_cmd now exit with code 0 on success, and code 1 on
  failure. Before, it was the opposite.
* test.bisect_cmd now runs the test worker process with
  -X faulthandler.
* regrtest RunTests: Add create_python_cmd() and bisect_cmd()
  methods.

(cherry picked from commit 1e5719a)
vstinner added a commit to vstinner/cpython that referenced this issue Mar 26, 2024
@vstinner
pythongh-115122: Add --bisect option to regrtest (python#115123)
efc1ea7 
* test.bisect_cmd now exit with code 0 on success, and code 1 on
  failure. Before, it was the opposite.
* test.bisect_cmd now runs the test worker process with
  -X faulthandler.
* regrtest RunTests: Add create_python_cmd() and bisect_cmd()
  methods.

(cherry picked from commit 1e5719a)
@vstinner vstinner mentioned this issue Mar 26, 2024
Merged
vstinner added a commit that referenced this issue Mar 26, 2024
@vstinner @encukou
[3.12] gh-83434: Sync libregrtest and test_regrtest with the main bra…
477ef90 
…nch (#117250)

* gh-115122: Add --bisect option to regrtest (#115123)

* test.bisect_cmd now exit with code 0 on success, and code 1 on
  failure. Before, it was the opposite.
* test.bisect_cmd now runs the test worker process with
  -X faulthandler.
* regrtest RunTests: Add create_python_cmd() and bisect_cmd()
  methods.

(cherry picked from commit 1e5719a)

* gh-115720: Show number of leaks in huntrleaks progress reports (GH-115726)

Instead of showing a dot for each iteration, show:
- '.' for zero (on negative) leaks
- number of leaks for 1-9
- 'X' if there are more leaks

This allows more rapid iteration: when bisecting, I don't need
to wait for the final report to see if the test still leaks.

Also, show the full result if there are any non-zero entries.
This shows negative entries, for the unfortunate cases where
a reference is created and cleaned up in different runs.

Test *failure* is still determined by the existing heuristic.

(cherry picked from commit af5f9d6)

* gh-83434: Disable XML in regrtest when -R option is used (#117232)

(cherry picked from commit d52bdfb)

---------

Co-authored-by: Petr Viktorin <[email protected]>
@miss-islington miss-islington mentioned this issue Mar 26, 2024
Merged
miss-islington pushed a commit to miss-islington/cpython that referenced this issue Mar 26, 2024
@vstinner @encukou @miss-islington
[3.12] pythongh-83434: Sync libregrtest and test_regrtest with the ma…
23a5deb 
…in branch (pythonGH-117250)

* pythongh-115122: Add --bisect option to regrtest (pythonGH-115123)

* test.bisect_cmd now exit with code 0 on success, and code 1 on
  failure. Before, it was the opposite.
* test.bisect_cmd now runs the test worker process with
  -X faulthandler.
* regrtest RunTests: Add create_python_cmd() and bisect_cmd()
  methods.

(cherry picked from commit 1e5719a)

* pythongh-115720: Show number of leaks in huntrleaks progress reports (pythonGH-115726)

Instead of showing a dot for each iteration, show:
- '.' for zero (on negative) leaks
- number of leaks for 1-9
- 'X' if there are more leaks

This allows more rapid iteration: when bisecting, I don't need
to wait for the final report to see if the test still leaks.

Also, show the full result if there are any non-zero entries.
This shows negative entries, for the unfortunate cases where
a reference is created and cleaned up in different runs.

Test *failure* is still determined by the existing heuristic.

(cherry picked from commit af5f9d6)

* pythongh-83434: Disable XML in regrtest when -R option is used (pythonGH-117232)

(cherry picked from commit d52bdfb)

---------

(cherry picked from commit 477ef90)

Co-authored-by: Victor Stinner <[email protected]>
Co-authored-by: Petr Viktorin <[email protected]>
vstinner added a commit that referenced this issue Mar 26, 2024
@miss-islington @vstinner @encukou
[3.11] [3.12] gh-83434: Sync libregrtest and test_regrtest with the m…
04b26ee 
…ain branch (GH-117250) (#117251)

[3.12] gh-83434: Sync libregrtest and test_regrtest with the main branch (GH-117250)

* gh-115122: Add --bisect option to regrtest (GH-115123)

* test.bisect_cmd now exit with code 0 on success, and code 1 on
  failure. Before, it was the opposite.
* test.bisect_cmd now runs the test worker process with
  -X faulthandler.
* regrtest RunTests: Add create_python_cmd() and bisect_cmd()
  methods.

(cherry picked from commit 1e5719a)

* gh-115720: Show number of leaks in huntrleaks progress reports (GH-115726)

Instead of showing a dot for each iteration, show:
- '.' for zero (on negative) leaks
- number of leaks for 1-9
- 'X' if there are more leaks

This allows more rapid iteration: when bisecting, I don't need
to wait for the final report to see if the test still leaks.

Also, show the full result if there are any non-zero entries.
This shows negative entries, for the unfortunate cases where
a reference is created and cleaned up in different runs.

Test *failure* is still determined by the existing heuristic.

(cherry picked from commit af5f9d6)

* gh-83434: Disable XML in regrtest when -R option is used (GH-117232)

(cherry picked from commit d52bdfb)

---------

(cherry picked from commit 477ef90)

Co-authored-by: Victor Stinner <[email protected]>
Co-authored-by: Petr Viktorin <[email protected]>
diegorusso pushed a commit to diegorusso/cpython that referenced this issue Apr 17, 2024
@vstinner @diegorusso
pythongh-115122: Add --bisect option to regrtest (python#115123)
48ba5db 
* test.bisect_cmd now exit with code 0 on success, and code 1 on
  failure. Before, it was the opposite.
* test.bisect_cmd now runs the test worker process with
  -X faulthandler.
* regrtest RunTests: Add create_python_cmd() and bisect_cmd()
  methods.
LukasWoodtli pushed a commit to LukasWoodtli/cpython that referenced this issue Jan 22, 2025
@vstinner @LukasWoodtli
pythongh-115122: Add --bisect option to regrtest (python#115123)
b9adb79 
* test.bisect_cmd now exit with code 0 on success, and code 1 on
  failure. Before, it was the opposite.
* test.bisect_cmd now runs the test worker process with
  -X faulthandler.
* regrtest RunTests: Add create_python_cmd() and bisect_cmd()
  methods.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
tests Tests in the Lib/test dir type-feature A feature request or enhancement
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@vstinner @hugovk @Eclips4