Make it easier to write reliable cleanup tasks

[jamestalmage]

See: #918 (comment)

after.always has some issues when used as a cleanup task. Specifically, it won't run if:

• There are test failures and --fail-fast is used.
• There are uncaught exceptions thrown.

I've advocated using .before or .beforeEach to ensure state is clean before running, but that means state is left on disk after the test run. It's easy enough to get around that:

function cleanup() {
  if (temDirExists()) {
    removeTempDir();
  }
}

test.before(cleanup);
test.after.always(cleanup);

Still, it might be nicer if we had a modifier that allowed you to do it a little cleaner:

// runs as before and after
test.cleanup(cleanupFn);

// runs as beforeEach and after (not sure afterEach makes much sense?)
test.cleanupEach(cleanupFn);

Or maybe we introduce a .and modifier:

test.before.and.after(cleanupFn);
test.beforeEach.and.after(cleanupFn);
test.beforeEach.and.afterEach(cleanupFn);

I think the second gives you a little more flexibility and is clearer without reading the docs. The first is probably simpler to implement (though I don't think the second would be very hard)

There is a $82.00 open bounty on this issue. Add more on Issuehunt.

• Checkout the Issuehunt explorer to discover more funded issues.
• Need some help from other developers? Add your repositories on Issuehunt to raise funds.

[novemberborn]

cleanup better communicates intended use. .and. feels overly verbose.

Agreed that afterEach doesn't make sense for .cleanupEach.

[sotojuan]

Another vote for cleanup. .and. reminds me of Mocha/Chai—cleanup just looks for AVA-ish if that makes sense.

[sindresorhus]

I prefer cleanup too.

Can we make it cleanup even on uncaughtException? I'd like cleanup to work no matter what happens.

[jamestalmage]

I lean toward and, though I see problems with it as well.

The only reason is that I think it makes it more obvious what is going on without reading the AVA docs. The notion that cleanup is going to happen both before and after is going to surprise people. They are going to write rmdir code without checking to make sure the directory exists, and be surprised when it fails. I think and better optimizes for test readability and being a self documenting API. cleanup seems only to optimize for characters typed.

My concern with and is that it might suggest combinations we don't want to support.

Can we make it cleanup even on uncaughtException? I'd like cleanup to work no matter what happens.

We can try. There is no guarantee it would work.

The only way to guarantee cleanup works would be to kill the first process immediately, then relaunch and run just the cleanup method. That falls apart if you use something like unique-temp-dir and store a reference for cleanup (not a good example - no need to clean up temp).

[vadimdemedes]

Hmm, wasn't .always supposed to serve this exact same purpose? To always run the clean up task, regardless of previous failure? Perhaps it's worth fixing the existing modifier, instead of introducing a new one. How would we explain the difference between the two to the user?

[sindresorhus]

@vdemedes I would prefer that too, but I remember there being some good points about why .always can't actually be always. Can't remember where we discussed it. @jamestalmage @novemberborn ?

[sindresorhus]

My concern with and is that it might suggest combinations we don't want to support.

Not just that, but now we're introducing combinatory methods, making the test syntax harder to understand. Starting to get bit too DSL'y.

[novemberborn]

Hmm, wasn't .always supposed to serve this exact same purpose? To always run the clean up task, regardless of previous failure?

That was our assumption, yes. The question though is how --fail-fast is supposed to behave. The fastest way to fail is to forcibly exit the process. Trying to run just the always hook of the failing test whilst other asynchronous tests are active is tricky. For debugging purposes it can be useful if test state remains after a failure, --fail-fast would provide that behavior.

After an uncaught exception there are no guarantees as to what code may still be able to run. Here too we end up forcibly exiting the process.

Note that in both cases we first do IPC with the main process before actually exiting. I'm not quite sure how much code still has a chance to run. It's possible always.after is entered but if it does anything asynchronous then that gets interrupted. This needs some research.

We should decide what guarantees we want from --fail-fast and uncaughtException.

Also, our understanding of "test cleanup" has evolved to the point where we know see the best strategy of ensuring a clean test environment is to clean up before you run your test, and then clean up after to avoid leaving unnecessary files and test databases lying around. Hence the proposal for a .cleanup hook.

Note that .always.after is still useful in other scenarios, e.g. #840.

[sindresorhus]

After an uncaught exception there are no guarantees as to what code may still be able to run. Here too we end up forcibly exiting the process.

Maybe we could rerun the process and only run the .always.after hooks.

[jamestalmage]

See #928 (comment)

That falls apart if you use something like unique-temp-dir and store a reference for cleanup (not a good example - no need to clean up temp

[catdad]

Just my two cents, since I was welcomed to comment. I have only used ava for one project so far, but I immediately ran into this issue and it confused me very much as a user and avid code tester.

My expectation, when splitting my code up into a before, test, and after, is that I am doing build-up and tear-down that needs to happen in order to test a small piece of functionality, but does not otherwise relate to the test code itself. In order for my tests to be predictable, I expect that these blocks always run, no matter what. From an end-user perspective, having an after, after.always, and cleanup is just confusing and redundant. Of course I want to clean up, and of course I want that to always happen. What are the use cases for anything else? (Yes, I understand technical debt and backwards compatibility, but they should only be considerations in design, not driving forces.)

Also, I see some bad advice and rhetoric happening here and in #918, suggesting that cleanup should happen before and after, allowing you to re-run tests even in a dirty environment. While I don't disagree that it is a good idea to check that your environment is exactly as desired before a test, there has to be a stick in the ground saying that unit tests must not permanently alter the environment, even if there are other tools in the toolchain (such as .gitignore) which will handle that for you. And to the extent that that is the fault of the test framework, it should be treated as an egregious and urgent bug. As an end-user, I should not have to choose between speed and sane, repeatable tests.

With that said, I would support the option of a .cleanup hook that ava transparently runs both before and after (and always run it after), provided that the use cases are fully considered. There might potentially be issues with running unexpected cleanup code before the tests, as that is rather unorthodox among the other test frameworks.

[novemberborn]

@catdad

While I don't disagree that it is a good idea to check that your environment is exactly as desired before a test, there has to be a stick in the ground saying that unit tests must not permanently alter the environment, even if there are other tools in the toolchain (such as .gitignore) which will handle that for you. And to the extent that that is the fault of the test framework, it should be treated as an egregious and urgent bug.

A crash due to a bug in AVA would be something we'd fix, yes. But what if the crash is caused by the code being tested? There is no 100% reliable way to recover from that and undo changes to the environment. Similarly the --fail-fast feature is designed to leave the environment in the state in which the failure occurred.

In other words, unless your tests always pass, there will be moments where a test run leaves the environment in a different state. AVA can't know that's the case since it doesn't understand before/after code. At least the cleanup hook makes it easier to sanitize your environment before test runs (in case it was left in a dirty state), and after test runs (because cleaning up is nice).

There might potentially be issues with running unexpected cleanup code before the tests, as that is rather unorthodox among the other test frameworks.

We should clearly document the intent of the hook. But AVA is not afraid of being unorthodox 😉