Default output_conflict_handling ignores test failures

[RalfJung]

This seems like a bad default:

ui_test/src/config.rs

Line 141 in 266e25d

output_conflict_handling: OutputConflictHandling::Bless,

[RalfJung]

In fact it's even worse... I have set output_conflict_handling to Error, but run_tests by default overwrites it back to Bless!

[oli-obk]

The first part is #176

The second part should only happen if --bless is passed as an argument. I did a quick skim of the code and I don't see how else the behaviour you're describing could happen

[RalfJung]

The second part should only happen if --bless is passed as an argument. I did a quick skim of the code and I don't see how else the behaviour you're describing could happen

This may have changed in master. In 0.21.1 we have

ui_test/src/lib.rs

Line 129 in 3d262a2

config.with_args(&args, true);

where the true sets default_bless.

[RalfJung]

However even in the last commit, something seems odd here -- args.check seems to be off-by-default?

Maybe there should be a test using run_tests with an all-default configuration and just making sure that this actually complains about output differences.

[oli-obk]

-- args.check seems to be off-by-default?

That's irrelevant, if it isn't set the default isn't changed.

[RalfJung]

That's irrelevant, if it isn't set the default isn't changed.

Well it is relevant insofar as it makes the default set in fn rustc matter.

But yeah, the actual issue is that default IMO, not the behavior of with_args. (A different default would make --check redundant though, it should be --no-check then.)

[oli-obk]

A different default would make --check redundant though, it should be --no-check then

you can still change the default in your Config and then parse the command line flags