Report all assertion failures

[jamestalmage]

With #259, we have moved from only printing the last assertion failure, to only printing the first.

Once #243 (chainable methods) happens, I think we should allow printing of every failed assertion to give the most information possible.

test.verbose('title', t => {
  t.fail('a');
  t.fail('b');
});

Perhaps only mode should also just imply verbose mode (you should have way less noise with a single test).

[sindresorhus]

Not sure we should have a verbose modifier. I don't see why would want to augment just some tests. I'd rather have a --verbose flag for all tests.

Perhaps only mode should also just imply verbose mode (you should have way less noise with a single test).

I think I like the idea of only triggering verbose mode for those tests.

[Qix-]

ava -v would be wonderful. Not sure about individual tests, though I see the point being made.

[vadimdemedes]

As for me, I can't find the use-case for a verbose mode in my mind. If you could help me with that, that'd be great :) If I have 10 assertions with 1st assertion causing the rest to fail, why do we need to see the other 9? Are there any examples in some projects or other test runners, where this functionality is needed/implemented?

[jamestalmage]

tap/tape shows you every assertion failure.

Usually, yes - you only need to see the first assertion. Though sometimes additional information can be helpful:

var a = sum(2, 2); // => 4 (correct)
var b = sum(4, 4); // => 7 (wrong - for some weird reason)
var c = sum(a, b); // => 11 (correct for the wrong input).

t.is(c, 12); // this will fail (telling us there is something wrong - but not what the root of the problem is)
t.is(b, 8);  // this will also fail and in this case is the more helpful information

Granted, the solution above is pretty obvious - reorder the assertions. But that is not so easy to see in every problem.

[jamestalmage]

Here is an example where I wish I could have seen every failure:
https://github.com/jamestalmage/babel-plugin-detective/blob/master/test/babel-6-test.js#L165

I could not use deepEquals, and using assert which throws at the first error was annoying because you can not know ahead of time which failure will highlight your mistake.

[vadimdemedes]

tap/tape shows you every assertion failure.

This shouldn't really bother us, since AVA is not trying to replicate tap/tape, but trying to stand out.

Though sometimes additional information can be helpful

In that case, verbose mode should definitely be optional and dead-clear to enable. I'm not a fan of enabling verbose mode automatically for test.only().

[sindresorhus]

Been thinking hard about this one today. I've changed my mind about implicit verbose when using test.only(). That's just too magic. As for a --verbose flag, I guess that could be useful, but I've honestly never needed it. I think we should punt it for now, but leave this open for additional comments. We can make a decisions later.

[novemberborn]

Just changed the title on this issue to "Report all assertion failures" given that we now have a --verbose reporter.

---

If we were to run all assertions and print subsequent failures how would that work with features like #493? In that case if const err = t.throws(fn) fails because fn does not throw an error then any subsequent assertions or even code is guaranteed to blow up. It's the same in @jamestalmage's example where if t.same(expressions, []) then code like expressions[0].type will blow up.

How does tap/tape handle these scenarios?

[vadimdemedes]

As previously, no matter how hard I try I don't see a point in continuing test execution, when test had already failed. And it will for sure confuse users, when tests continue to run after failed assertions. I think we should close this.