Skip to content

Simple provision to help dealing with flakey CI jobs (repeat_command.py) #3106

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 12 commits into from

Conversation

rwgk
Copy link
Collaborator

@rwgk rwgk commented Jul 13, 2021

Experiments trying to understand the MSVC test_iostream.py flakiness.

@rwgk rwgk force-pushed the test_iostream_num_runs branch 3 times, most recently from fa56549 to a0584bc Compare July 15, 2021 18:51
@rwgk rwgk force-pushed the test_iostream_num_runs branch from a0584bc to 4369781 Compare July 16, 2021 00:37
@rwgk rwgk changed the title Experiments: test_iostream.py NUM_RUNS Simple provision to help dealing with flakey CI jobs (repeat_command.py) Jul 16, 2021
@Skylion007
Copy link
Collaborator

There is also pytest-flake which could be useful here.

@rwgk
Copy link
Collaborator Author

rwgk commented Aug 7, 2021

If anyone wants to use my trick to deflake tests in the future, PR #3180 should make it straightforward to pin-point where the tests are run, and repeat_command.py now lives here: https://github.com/rwgk/rwgk_config/tree/master/bin

Permalink: https://github.com/rwgk/rwgk_config/blob/e8396e6126741b8916dae91b3024fa0d713e2cea/bin/repeat_command.py

The only slight wrinkle is that one ci.yml job runs the Python tests outside the pybind11 source directory. This is what I did for that one special case:

    - name: Copy tools/repeat_command.py to /tmp
      run: cp -a tools/repeat_command.py /tmp
...
      run: python3 /tmp/repeat_command.py 1 make pytest -j 2

@rwgk rwgk closed this Aug 7, 2021
@rwgk rwgk deleted the test_iostream_num_runs branch August 7, 2021 14:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants