customized testing

[PavelVozenilek]

Right now tests are compiled in and executed only if test is specified on command line. Testing mechanism is in zig/std/special/test_runner.zig and its use is probably hardcoded.

This arrangement is not the only possible one. For example, one may prefer to always compile tests in (in debug mode) and to run tests from recently modified files automatically, whenever the executable runs.

I propose:

1. Expand builtin.__zig_test_fn_slice structure. Add source file name and line for every test. Add how "old" every test is (e.g. how many minutes passed since the source file was last modified).

2. Add "currently running test file/line/name" members into the same structure. These values would be set/unset by the test runner. assert then could inspect them and show more detailed error message. This is above stack tracing feature: stack trace is rather clumsy to parse.

3. Document how can one replace the default test runner. (Modifying stdlib is IMHO valid choice.)

This is related to #531 (comment). If one can discover at runtime whether tests were enabled or not, he may then invoke/not try to invoke the most recent ones.

---

Yet more ambitious testing mechanism may ensure that tests do not exceed certain time limit. (If they do there's grave performance regression.) It may look like:

test "test with default timeout, say 20 milliseconds"
{
  ...
}

test(100) "test which is expected take no more than 100 milliseconds"
{
  ...
}

If a test exceeds the timeout test runner may execute it few more times, to eliminate influence of imprecise time measuring (could be ~15 milliseconds on Windows) and influence of OS activities.

[PavelVozenilek]

Since Zig allows mixed code with C it may be also desirable to allow to register/call tests written in C.

GCC/clang allows to use __attribute__(constructor) to implement testing mechanism e.g. as:

#ifdef TESTS
TEST()
{
  assert(2 + 2 == 4);
}
#endif

(where the TEST() is macro trickery.)

If the suggestion here ( #531 (comment) ) is implemented the Zig compilation mode flags (debug/release/tests/...) may be also "exported" to C as preprocessor definitions.