Brave new ui tests

[killercup]

This will

1. take the files from the new clippy_tests' crate's examples dir
2. use cargo check (with clippy-driver as rustc) to compile them
3. apply the suggested fixes
4. compile the fixed file

By default, it only logs which files failed one of the steps. Set RUST_LOG=clippy_test=info (or debug) to get more information.

You can also set RUSTFIX_PLS_FIX_AND_RECORD_MY_CODE=true to have it write the recored fixes to examples/$originalName--recorded.rs. (Needs to be separated with -- instead of . because it needs to be a valid crate name for cargo to compile it as an example.)

Running cargo run in clippy_test/ will now output:

test result: ok. 0 passed; 0 failed; 174 ignored;

This is expected. There are no --fixed.rs files, so we don't try to make any assertions on the applied suggestions. Furthermore, we haven't implemented tests for the human-readable diagnostics, so we don't care about those .stderr files at all for now.

PS: Because we have a lot of examples, I took the liberty of using rayon to have as many cargos as possible running at the same time. I'm not sure how much of a speedup this is going to get us, but it was definitely faster than without it, and was quite easy to do.

[killercup]

Obviously WIP right now :)

r? @oli-obk

[oli-obk]

You can do an intermediate thing by pointing the compiletest test to the clippy_tests/examples directory.

I'll have a closer look later, but skimming through it, it already looks good.

[killercup]

Okay! UI tests are on their way! (Give RECORD_NEW_DIAGNOSTICS=true while you test this!)

Currently, the test script generates some rather ugly diffs with the existing UI tests, but I hope fixing those is just a combination of passing the correct RUSTC_FLAGS and s/examples/$DIR/g:

diff --git a/clippy_tests/examples/absurd-extreme-comparisons.stderr b/clippy_tests/examples/absurd-extreme-comparisons.stderr
index 2b1e9ad6..bed9cddf 100644
--- a/clippy_tests/examples/absurd-extreme-comparisons.stderr
+++ b/clippy_tests/examples/absurd-extreme-comparisons.stderr
@@ -1,147 +1,149 @@
-error: this comparison involving the minimum or maximum element for this type contains a case that is always true or always false
-  --> $DIR/absurd-extreme-comparisons.rs:10:5
+warning: this comparison involving the minimum or maximum element for this type contains a case that is always true or always false
+  --> examples/absurd-extreme-comparisons.rs:10:5
    |
 10 |     u <= 0;
    |     ^^^^^^
    |
-   = note: `-D absurd-extreme-comparisons` implied by `-D warnings`
+note: lint level defined here
+  --> examples/absurd-extreme-comparisons.rs:4:9
+   |
+4  | #![warn(absurd_extreme_comparisons)]
+   |         ^^^^^^^^^^^^^^^^^^^^^^^^^^
    = help: because 0 is the minimum value for this type, the case where the two sides are not equal never occurs, consider using u == 0 instead

[phansch]

@killercup Would you like to continue with this PR?

[phansch]

I talked with @killercup on Discord and I will continue this by trying to add the // run-rustfix ui test annotation to compiletest-rs. Going to close this and will open a new PR once there's something to show 👍