tests: Require run-fail ui tests to have an exit code (SIGABRT not ok)
#143002
+127
−81
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
rustbot has assigned @petrochenkov. Use |
rustbot
added
A-compiletest
|
Some changes occurred in src/tools/compiletest cc @jieyouxu |
Enselic
added
S-waiting-on-author
rustbot
added
the
T-infra
This comment has been minimized.
This comment has been minimized.
|
why not simply also would accept |
|
@bors2 delegate=try |
|
@Enselic can now perform try builds on this pull request |
Enselic
force-pushed
the
tests-ui-run-fail-exit-vs-signal
branch
from
6ad1973 to
17be091
Compare
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
|
@bors2 try jobs=x86_64-msvc-1,x86_64-msvc-2 |
rust-bors bot
added a commit
that referenced
this pull request
Jun 25, 2025
…try> tests: Require `run-fail` ui tests to have an exit code (`SIGABRT` not ok) Normally a `run-fail` ui test shall not be terminated by a signal like `SIGABRT`. So begin having that as a hard requirement. Some of our current tests do terminate by a signal however. Introduce and use `run-fail-without-exit-code` for those tests. This adds further (cc #142304, #142886) protection against the regression in #123733 since that bug also manifested as `SIGABRT` in `tests/ui/panics/panic-main.rs` (shown as `Aborted (core dumped)` in the logs attached to that issue, and I have also been able to reproduce this locally). ### TODO - [ ] what about on Windows? - [ ] also update docs at https://rustc-dev-guide.rust-lang.org/tests/directives.html#controlling-outcome-expectations - [ ] clean up the code ### Zulip discussion See https://rust-lang.zulipchat.com/#narrow/channel/122651-general/topic/compiletest.3A.20terminate.20by.20signal.20vs.20exit.20with.20error/with/525611235 try-job: x86_64-msvc-1 try-job: x86_64-msvc-2
This comment was marked as resolved.
This comment was marked as resolved.
Enselic
force-pushed
the
tests-ui-run-fail-exit-vs-signal
branch
from
17be091 to
ad4e082
Compare
|
An only-windows test that was run-fail now must be run-crash. Fixed now. Trying again: @bors2 try jobs=x86_64-msvc-1,x86_64-msvc-2 |
rust-bors bot
added a commit
that referenced
this pull request
Jun 25, 2025
…try> tests: Require `run-fail` ui tests to have an exit code (`SIGABRT` not ok) Normally a `run-fail` ui test shall not be terminated by a signal like `SIGABRT`. So begin having that as a hard requirement. Some of our current tests do terminate by a signal however. Introduce and use `run-fail-without-exit-code` for those tests. This adds further (cc #142304, #142886) protection against the regression in #123733 since that bug also manifested as `SIGABRT` in `tests/ui/panics/panic-main.rs` (shown as `Aborted (core dumped)` in the logs attached to that issue, and I have also been able to reproduce this locally). ### TODO - [ ] **Q:** what about on Windows? **A:** we'll treat any exit code outside of 1 - 127 as "crashed", and we'll do the same on unix. - [ ] also update docs at https://rustc-dev-guide.rust-lang.org/tests/directives.html#controlling-outcome-expectations - [ ] clean up the code - [ ] test all permutations of actual vs expected ### Zulip discussion See https://rust-lang.zulipchat.com/#narrow/channel/122651-general/topic/compiletest.3A.20terminate.20by.20signal.20vs.20exit.20with.20error/with/525611235 try-job: x86_64-msvc-1 try-job: x86_64-msvc-2
This comment was marked as resolved.
This comment was marked as resolved.
Enselic
force-pushed
the
tests-ui-run-fail-exit-vs-signal
branch
from
ad4e082 to
2499dac
Compare
|
Here is the updated version of #143002 (comment) using this commit. I tested what When When when when when when |
Enselic
force-pushed
the
tests-ui-run-fail-exit-vs-signal
branch
from
f7a54fe to
64c076e
Compare
|
I'm happy with the code and how it works so this is ready for review. Summary of where we stand:
@rustbot ready |
rustbot
added
S-waiting-on-review
|
r=me with the cleanup from #143002 (comment). |
petrochenkov
added
S-waiting-on-author
…t ok) Normally a `run-fail` ui test shall not be terminated by a signal like `SIGABRT`. So begin having that as a hard requirement. Some of our current tests do terminate by a signal however. Introduce and use `run-crash` for those tests.
Enselic
force-pushed
the
tests-ui-run-fail-exit-vs-signal
branch
from
64c076e to
4573c84
Compare
|
I'm only a member of wg-triage so I'm pretty sure I can't @bors r=petrochenkov |
|
@Enselic: 🔑 Insufficient privileges: Not in reviewers |
|
@bors r+ |
bors
added
S-waiting-on-bors
compiler-errors
added a commit
to compiler-errors/rust
that referenced
this pull request
Jun 27, 2025
…signal, r=petrochenkov tests: Require `run-fail` ui tests to have an exit code (`SIGABRT` not ok) Normally a `run-fail` ui test shall not be terminated by a signal like `SIGABRT`. So begin having that as a hard requirement. Some of our current tests do terminate by a signal however. Introduce and use ~`run-fail-without-exit-code`~ `run-crash` for those tests. This adds further (cc rust-lang#142304, rust-lang#142886) protection against the regression in rust-lang#123733 since that bug also manifested as `SIGABRT` in `tests/ui/panics/panic-main.rs` (shown as `Aborted (core dumped)` in the logs attached to that issue, and I have also been able to reproduce this locally). ### TODO - [x] **Q:** what about on Windows? **A:** we'll treat any exit code outside of 1 - 127 as "crashed", and we'll do the same on unix. - [x] test all permutations of actual vs expected **Done:** See rust-lang#143002 (comment). ### Zulip discussion See https://rust-lang.zulipchat.com/#narrow/channel/122651-general/topic/compiletest.3A.20terminate.20by.20signal.20vs.20exit.20with.20error/with/525611235
bors
added a commit
that referenced
this pull request
Jun 27, 2025
…rors Rollup of 8 pull requests Successful merges: - #139858 (New const traits syntax) - #140809 (Reduce special casing for the panic runtime) - #142963 (Skip unnecessary components in x64 try builds) - #142974 (Update stage0 to 1.89.0-beta.1) - #142987 (rustdoc: show attributes on enum variants) - #143002 (tests: Require `run-fail` ui tests to have an exit code (`SIGABRT` not ok)) - #143092 (const checks for lifetime-extended temporaries: avoid 'top-level scope' terminology) - #143096 (tag_for_variant: properly pass TypingEnv) r? `@ghost` `@rustbot` modify labels: rollup
|
@bors r- |
bors
added
S-waiting-on-author
|
@bors try |
bors
added a commit
that referenced
this pull request
Jun 27, 2025
…try> tests: Require `run-fail` ui tests to have an exit code (`SIGABRT` not ok) Normally a `run-fail` ui test shall not be terminated by a signal like `SIGABRT`. So begin having that as a hard requirement. Some of our current tests do terminate by a signal however. Introduce and use ~`run-fail-without-exit-code`~ `run-crash` for those tests. This adds further (cc #142304, #142886) protection against the regression in #123733 since that bug also manifested as `SIGABRT` in `tests/ui/panics/panic-main.rs` (shown as `Aborted (core dumped)` in the logs attached to that issue, and I have also been able to reproduce this locally). ### TODO - [x] **Q:** what about on Windows? **A:** we'll treat any exit code outside of 1 - 127 as "crashed", and we'll do the same on unix. - [x] test all permutations of actual vs expected **Done:** See #143002 (comment). ### Zulip discussion See https://rust-lang.zulipchat.com/#narrow/channel/122651-general/topic/compiletest.3A.20terminate.20by.20signal.20vs.20exit.20with.20error/with/525611235 // try-job: aarch64-apple try-job: x86_64-msvc-1 try-job: x86_64-gnu try-job: dist-i586-gnu-i586-i686-musl try-job: test-various
|
Yeah, the test that failed in that rollup should?? probably be marked run-crash too |
|
The job Click to see the possible cause of the failure (guessed by this bot) |
|
💔 Test failed - checks-actions |
|
For the test-various job the problem seems to be that platforms without unwind support will abort instead, but aborting means "crashing", so Let's take for example The test will pass on $ ./x test tests/ui/async-await/issues/issue-65419/issue-65419-coroutine-resume-after-completion.rs
...
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 19329 filtered out; finished in 70.26msbut fail with exit code 134 (128 + 6 for SIGABRT) with $ PATH="/home/martin/opt/wasmtime-v19/wasmtime-v19.0.0-x86_64-linux:$PATH" WASI_SDK_PATH=/home/martin/opt/wasi-sdk-25/wasi-sdk-25.0-x86_64-linux ./x test --set rust.lld=true --target wasm32-wasip1 tests/ui/async-await/issues/issue-65419/issue-65419-coroutine-resume-after-completion.rs
...
error: test did not exit with failure! code=Some(134)
...
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 19329 filtered out; finished in 90.22msSolution proposalI think the best way to handle this is to adjust compiletest to allow crashes with |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Normally a
run-failui test shall not be terminated by a signal likeSIGABRT. So begin having that as a hard requirement.Some of our current tests do terminate by a signal however. Introduce and use
run-fail-without-exit-coderun-crashfor those tests.This adds further (cc #142304, #142886) protection against the regression in #123733 since that bug also manifested as
SIGABRTintests/ui/panics/panic-main.rs(shown asAborted (core dumped)in the logs attached to that issue, and I have also been able to reproduce this locally).TODO
A: we'll treat any exit code outside of 1 - 127 as "crashed", and we'll do the same on unix.
Done: See tests: Require
run-failui tests to have an exit code (SIGABRTnot ok) #143002 (comment).Zulip discussion
See https://rust-lang.zulipchat.com/#narrow/channel/122651-general/topic/compiletest.3A.20terminate.20by.20signal.20vs.20exit.20with.20error/with/525611235
// try-job: aarch64-apple
try-job: x86_64-msvc-1
try-job: x86_64-gnu
try-job: dist-i586-gnu-i586-i686-musl
try-job: test-various