[Idea]: test macros

[jamestalmage]

From: #78 (comment)

Occasionally it's useful to create a function that calls test for you:

// fictional module that analyzes math expressions in strings:
import calculator from 'calculator-parser';
import ava from 'ava';

function test(input, expected, only) {
  const method = only ? ava.only : ava;
  method(input + ' === ' + expected, t => {
    const actual = calculator.calculate(input);
    t.is(actual, expected);
  });
}

test('3 + 4', 7);
test('5 - 2', 3);
test('6 * 3', 18, true); // causes this to be interpreted as an exclusive test
// ...

Unfortunately, such helpers create a situation where it becomes basically impossible to do static analysis on the code (static analysis may be necessary to solve #78).

I think we could solve some of this with a macro command:

import test from `ava`;

const myMacro = test.macro(
  (t, input, expected) => { // additional arguments to the macro are passed in. 
    const actual = calculator.calculate(input);
    t.is(actual, expected);
  },
  (input, expected) => input + ' === ' + expected // optional function to compute the test tile from the input.
);

myMacro('3 + 4', 7);

// macros get some of the same modifiers as a normal test function:
myMacro.skip('3 * 6', 18);
myMacro.only('9 / 3', 3);

[jamestalmage]

@jfmengels points out another potential benefit of this in our linter: avajs/eslint-plugin-ava#56 (comment)

[jfmengels]

I love the idea for test macros, but I noticed that it will also make linting some stuff harder (finding out that .only, .skip is used etc.). It's a trade-off.

[jamestalmage]

but I noticed that it will also make linting some stuff harder

I am not sure if Eslint has an equivalent to Babel's bindings. It would make linting macros pretty easy. I have not seen anything in the docs though.

[jamestalmage]

@jfmengels - What if, instead of storing a reference, you had to pass a string identifier, and it became just another macro.

test.macro('myMacro', (t, input, expected) => ...);

test.myMacro('input', 'expected');
test.only.myMacro('input', 'expected');

I think that makes static analysis a little easier (especially if you define macros in helpers).

[jfmengels]

Interesting idea, but then you'd have

1. tampering of the AVA import, which will not affect other tests but is still pretty ugly
2. to import a file who'd have a side-effect of defining a macro, which seems counter to AVA's explicit imports (compared to Mocha where describe, it that are injected for instance). I like explicit imports.
3. Odd behavior when definining a macro called only for instance.

An alternative would be

// Proposal 1
// Defining
test.defineMacro('myMacro', (t, input, expected) => ...);
// Using
test.macro('myMacro', 'input', 'expected');

But you'd still have problem 1 above.
An even better alternative:

// Proposal 2
// Defining (call it createMacro, defineMacro, whatever)
const myMacro = test.defineMacro((t, input, expected) => ...);
// Using the macro
test.macro('title', myMacro, 'input', 'expected');

// Proposal 3
// or even alternatively, by re-using `test`, that adapts its behavior 
const myMacro = test.macro((t, input, expected) => ...);
// Using
test('title', myMacro, 'input', 'expected');

I think proposal 2 is the best implementation. You get

• static analysis capability
• test creators in helpers that will not be linted as errors
• Explicit imports of macros
• No tampering of the AVA/test object
• Explicit use of a macro (which is neutral, but worth pointing out)

[jamestalmage]

My only concern with your proposals is the verbosity. I was already concerned about that with my second proposal here, your proposals just make it worse.

Ideally, we could use my first proposal, as it is the least verbose. My second one was just to address the static analysis problem.

IMO, most of your concerns aren't that big a deal:

1. tampering of the AVA import, which will not affect other tests but is still pretty ugly

Exactly - it won't affect other tests (because we fork), so no surprising behavior. You define which macros you load in each test (so no trouble figuring out what is happening).

2. to import a file who'd have a side-effect of defining a macro, which seems counter to AVA's explicit imports (compared to Mocha where describe, it that are injected for instance). I like explicit imports.

I think in the case of a helper defining macros - the user has decided to do that themselves. AVA's not doing that to them. They won't be surprised by it. In general, I agree with your sentiment, but this seems like a reasonable exception.

3. Odd behavior when definining a macro called only for instance.

We could protect against macros that try to overwrite built in properties.

[jfmengels]

I think in the case of a helper defining macros - the user has decided to do that themselves. AVA's not doing that to them

Sure, but still, I myself as a user don't like it. And if AVA doesn't allow macros to be used without a variable, AVA will kind of force me to do that, should I want to create a macro used in multiple files.

It's more verbose, but not that much, but that's just my opinion. If you're that concerned about verbosity, my proposal 3 is pretty short. It can be even be shortened by simply having macro be a simple function and injecting the arguments

// Normal tests
test(t => { var a = 2, b = 3; ... });
test(t => { var a = 2, b = 4; ... });

// Proposal 4
// Macro/reusable test
var macro = (t, a, b) => { ... };
test(macro, 2, 3);
test(macro, 2, 4);

Importing would be easy, as it's a special function. No naming, no AVA tampering. Alternatively, args could be added to the context: t.context.args === [2, 3]. I like this version even more.

[jamestalmage]

Hmm. Proposal 4 looks pretty good, and it should be really simple to implement.

I like it.

[jamestalmage]

Alternatively, args could be added to the context: t.context.args === [2, 3].

👎 You can completely replace t.context in a beforeEach. t.context = something.

[jfmengels]

👎 You can completely replace t.context in a beforeEach. t.context = something.

Proposed it just in case proposal 4 could not be done because test already uses further arguments or so. Much prefer the args in function option too.

[jamestalmage]

Rather than spreading the arguments ourselves, we should pass additional args as an array:

function macro(t, additionalArgs) {
  if (additionalArgs.length === 2) {
    // ...
  } else {
    // ...
  }
}

test(macro, 'foo', 'bar');
test(macro, 'foo');

This allows us to pass additional args in the future breaking changes. With destructuring, defining your macro isn't really any more verbose:

function macro(t, [foo, bar]) {
  // ...
}

[novemberborn]

👍 on proposal 4.

Perhaps we could expose the macro creating functionality through an ava/macro module. That way helper files won't accidentally become AVA runners.

[jfmengels]

@novemberborn With proposal 4 macros are simple functions, and there's no need for a special macro creator.