Augmentations, non-redirecting generative constructors and instance variable initializer expressions.

[lrhn]

TL;DR: All instance variables with initializer expressions of a class definition are initialized first thing, before invoking any non-redirecting generative constructor code. The the non-redirecting generative constructor code is executed in stack order, with the parameter list and initializer list of augmenting constructors executed before recursing on the augmenting definition, and the body executed after coming back. When reaching the base declaration it recurses on the superclass constructor as an outside call, which means it too initializes instance variables when firt reaching a non-redirecting generative constructor, but possibly after redirecting generative constructors. Super-parameters are accumulated along the way, put into pre-computed positions.

There is entirely too much backgound and details below, and it may be slightly more general than necessary, but I think it is consistent and usable.

This defines when initializer expressions for instance variable declarations are evaluated (first) and in which order (base declaration source order, with multi-file ordering being preorder depth-first on parts).

The biggest thing here is that an augmenting constructor (effectively) prepends its initializer list to the existing list.
This ensures that the scope used to evaluate the initializer list can be stack allocated.

We need to have recursed to the base definition at the end of the initialzier list, where we call the superclass constructor.
If we append new initializer lists, then we need the augmenting declaration's initializer list entries between the initialzier list and the super-constructor invocation of the base declaration, at which point we don't want to unwind the stack to the augmenting declaration's parameter scope. If we prepend, it just works.

Motivation

We have agreed that an augmenting non-redirecting generative constructor declaration with a body will execute its body after the execution of the body (or transitive bodies) of the constructor it is augmenting.

We also need to specify when instance variable initializer expressions are evaluated. Those are currently evaluated at the beginning of the invocation of a non-redirecting generative constructor. (Or just prior to the invocation, depending on perspective. Since every generative constructor performs the same initialization, it may be preferable to see it as existing outside of the constructors.)

With augmentations, we can now have augmenting constructors and augmenting variable initializer expressions. We need to place the variable initialization in the invocation hierarchy, preferably in a consistent and predictable place.

Terminology

A base (non-augmenting) declaration introduces a definition which mirrors the declaration, but at a slightly higher abstraction level. It’s possible for two differently written declarations to introduce indistinguishable definitions, if the difference is entirely syntactic. (For example abstract final int x; and int get x; both introduce a single abstract getter definition named x with type int.)

The kind of definition determines which properties it has.

An augmenting declaration is applied to an existing definition and introduces a new definition of the same kind. The new definition may refer to the prior definition, so that its implementation may refer to the augmented implementation.

Proposed augmented generative constructor behavior

When we invoke a generative constructor named g (for example C.id or C) on an class definition C with type parameters X1,..,Xn, instantiated with type arguments T1,…,Tn, with an argument list L to initialize an object o (with a runtime type that is known to extend C and implement C<T1,..,Tn>):

• Let d be the constructor definition named g of C.
• If d is a redirecting generative constructor definition, where g’ is name of the constructor it redirects to (this.id denotes a target of C.id in a class named C):
  • Bind actual arguments L to the formal parameters of G, creating the parameter scope.
  • Evaluate the arguments of the redirection clause in the parameter scope to an argument list A.
  • Then invoke the generative constructor named g’ on C instantiated with type arguments T1,…,Tn, with argument list A to initilaize o.
• If d is a non-redirecting generative constructor definition:
  • For each instance variable definition of C in base declaration order (the order of the base declaration in the total traversal ordering of a library), where the variable definition has an initializer expression, and is not declared late:
    • Evaluate the initializer expression of that definition to a value v.
      • Evaluating the initializer expression of a variable definition is defined recursively on the stack of augmenting definitions. The stack of definitions is traversed until a definition is found which introduces an initializing expression, then that expression is evaluated. If the definition is an augmenting definition and its augmented definition also has an initializing expression, then the expression is evaluated in a context where augmented may be referenced, and doing so evalautes the initializer expression of the augmented definition. That is: Having an initializer expression is a property we can know directly for each variable definition, but how to evaluate it depends on the entire stack of definitions.
    • Initialize the corresponding instance variable of o to the value v.
  • Let S be an uninitialized argument list with the number of positional arguments and names of named arguments of superParameters of d (which is the shape of the argument list of the super-constructor invocation including all explicit arguments and super parameters of the entire stack of definitions).
  • Invoke the constructor definition d of C<T1,…,Tn> with argument list L and super parameters S to initialize the value o.

We then define the invocation of a constructor definition as follows, allowing an augmenting constructor’s definition to delegate to the definition it augments.

Invocation of a non-redirecting generative constructor definition d on an instantiated class definition C<T1,…,Tn> with argument list L and super parameters S to initialize an object o proceeds as follows (at least for any class other than Object, which just completes immediately):

• Bind actual arguments to formal parameters. For each parameter in the parameterList of d (in sequence order):
  • If the argument list L has a value for the corresponding positional position or named argument name, let v be that value.
  • Otherwise, the parameter must be optional.
    • If the parameter has a default value expression, let v be the value of that expression.
    • Otherwise let v be the null value.
• If the parameter is an initializing formal with name n,
  • Initialize the variable of o corresponding to the instance variable named n of C to the value v.
  • Bind the name n to v in the initializer-list scope.
• Otherwise, if the parameter is a super-parameter.
  • If the parameter is named, set the value of the argument with name n of S to v.
  • Otherwise the parameter is positional:
    • Let i be one plus the number of prior positional super parameters in the parameter list of d.
    • If d has a superDefinition d’, add the count from superParameters of d’.
    • Otherwise if d has a superConstructorInvocation, add the number of positional arguments in the argument list of that invoaction.
    • Set the positional argument value with position i in S to v.
  • Bind the name of the parameter to v in the initializer-list scope.
• Otherwise the parameter is a normal parameter. Bind the name of the parameter to v in the parameter scope.
• For each entry in the initializer list of d, in sequence order:
  • If assert entry, evaluate the first operand in the initializer list scope. If evaluate to false, evaluate the second operand to a message value m, if there is a second operand, otherwise let m be the null value. Throw an AssertionError with m as message.
  • If variable intializer entry, evaluate expression in initializer list scope to value v. Initialize the variable of o corresponding to variable with the initializer entry name in C to the value v.
• If d has an augmentedDefinition d’
  • Invoke d’ on C<T1,…,Tn> with arguments L and super-arguments S to initialize o.
  • This recurses on the augmented definition.
• Otherwise:
  • Let U be be the instantiated superclass definition of C<T1,…,Tn>.
    • This may be a synthetic definition (for an anonymous mixin application) computed from the declared superclass and declared mixins of C. _(Or we can do the short-circuiting here since mixin applications can only)
  • If d has a superConstructorInvocation:
    • Evaluate each argument of the argument list of the super-constructor invocation in the initializer list scope, in source order, and set the entry of S with the same position or name to the resulting value.
    • Let g be the name of superclass constructor of U targeted by the superConstructorInvocation (class-name of U plus .id if referenced as super.id).
  • Otherwise let g be the name of the unnamed constructor of S.
  • Invoke the constructor named g on U with arguments S to initialize o.
  • This recurses on the superclass constructor.
• When this has completed, execute the body of d, if any, in a scope which has the parameter scope of the invocation of d as parent scope.
• Then invocation of d completes normally.

The consequence of this definition is an execution order for initialization of an instance of a class of:

• Field initializers of class, from all augmentations, in “base declaration source order”.
• Augmenting declaration parameter lists and initializer lists, for augmentations in last-to-first order.
• Base declaration parameter list, initializer list
• Base declaration super-constructor invocation, recurses to this entire list on superclass.
• Body of base declaration.
• Bodies of augmenting declarations in first-to-last order.

It ensures that we only recurse in one place, keeping a stack discipline. The parameter scope of the invocation can be stack-allocated, and be on top of the stack when the scope is used. (If we execute initializer list entries after the ones of an augmented definition, we fail to maintain that stack order.)

Details of the definitions

This focuses only on the properties of definitions that are relevant here.

A variable definition has (among others) the following properties:

• hasInitializerExpression (Boolean, derivable, true if the definition itself has an initializerExpression, or if it has an augmented definition which hasInitializerExpression, false otherwise)
• initializerExpression (optional source code, post type inference, may contain augmented reserved words if there is an augmentedDefinition that hasInitializerExpression.)
• augmentedDefinition (optional, set for definitions introduced by applying augmenting declarations, to the augmented definition, not for the definition of a base declaration).

A (non-abstract, non-external) variable declaration introduces a variable definition. This is the definition that implies a backing storage. A variable declaration also introduces a getter definition, and maybe a setter definition. Those are not relevant here.

These properties are updated when applying an augmenting variable declaration A to a variable definition d as follows:

• The augmentedDefinition of the result is always d.

• If A has an initializer expression, then that is the resulting definition’s initializerExpression and hasInitializerExpression is true.

• Otherwise the resulting definition has no initializerExpression and its hasInitializerExpression has teh same value as the hasInitializerExpression of d.

• (Any other change an augmenting variable declaration can do, which is just adding metadata. Name and types are inherited as-is.)

You evaluate the initializer expression of a variable definition d which hasInitialierExpression to a value v in a scope S as follows:

• If d does not have an initializerExpression,

  • Then d has an augmentedDefinition, d’, which hasInitializerExpression, so evaluate the initializer expression of d’ to a value v in scope S. Then evaluating the initialier expression of d evaluates to v too.

• Otherwise let e be the initializerExpression of d:

  • If d does not have an augmentedDefinition (it’s the definition of a base declaration, so augmented is not reserved in e), then evaluate the expression e to a value v in scope S. Then evaluating the initializer expression of d evaluates to v too.

  • Otherwise:

    • Let d’ be the augmentedDefinition of d.

    • Then augmented is a reserved word inside e. If augmented occurs inside e, then the hasInitializerExpression of d’ must be true.

    • Evaluate the expression e in an “augmented-allowing” scope extending S to a value v, where evaluating the identifier expression augmented performs the following operation:

      • Evaluate the initializer expression of d’ to value v’.
      • The expression augmented evalautes to v’.

    • Then evaluating the initializer expression of d evaluates to v.

A non-redirecting generative constructor definition has (among others) the following properties:

• augmentedDefintion: Optional, set if introduced by an augmentation application, the non-redirecting genreative constructor defintion that was augmented.

• parameterList: Sequence of constructor parameter definitions. Each has a name and type, can be optional or required, positional or named, have a default value expression if optional, and be either an initializing formal, a super-parameter or a “normal” parameter.

  The parameters must always match the types and names of parameterList of augmentedDefinition, if any. The “position” of a positional parameter in such a list is defined as one plus the count of positional parameters earlier in the sequence.

• initializerList: Sequence of initializer entries, either variable initializer or assert.

• InitializedVariables: Set of identifiers. The instance variables initialized by this definition stack.

• superConstructorInvocation: Optional, only set on base declaration’s definition.

• superParameters: integer count and set of identifier names, the super-parameter positions and names already filled by the initializer lists and super-constructor invocation of this definition, positional and named.

• body: Optional code block.

The definition of a non-augmenting non-redirecting generative constructor declaration G has the following properties:

• No augmentedDefinition.
• parameterList directly derived from the declaration
• initializerList directly derived from the declaration
• The superConstructorInvocation of the declaration, if it has one.
• initializedVariables: Set of names of variables initialized by initializing formals in the parameter list, and names of variable initializer entries of the initializer list. Static error if any name is initialized more than once.
• superParameters: Count of positional arguments of super-constructor invocation (zero if none) plus count of positional super parameters in parameter list, and set of names of named arguments of super-constructor invocation (if any) and names of all named super parameters of parameter list. Static error if same name occurs more than once.
• body: Constructor body, if any.
• (Everything else)

Applying an augmenting non-redirecting generative constructor declaration A to such a definition, d produces a result with the following properties:

• augmentedDefinition is d
• parameterList is a sequence of parameters derived from the parameter list of A and the parameterList of d.
  • The parameter list of A must have the same number of positional, and optional positional, parameters as the parameterList of d and the same names and optionality of named parameters as the as the parameterList of d.
  • The parameterList of the resulting definition has one entry per parameter declaration in the parameter list of A, in source order. The entry is an initializing formal or super parameter if the parameter of A is. The parameter is named or positional, and required or optional, as the parameter of A, and has the same default value expression, if any.
  • If the parameter list of A omits a type variable from a parameter declaration, the resulting parameterList’s type for that parameter is the same type as that of the corresponding parameter in d’s parameterList.
  • It’s a compile-time error if A contains an initializing formal parameter with a name that is in the initializedVariables of d.
  • It’s a compile-time error if A contains a super-parameter with a name that is in the set of names of the superParameters of d.
• initializerList is a sequence pf the entries of the initializer list of A.
  • It’s a compile-time error if the initializer list of A contains an initializer for a variable with the same name as an initializing formal of A or with a name in the initializedVariables of d.
• initialziedVariables: The (disjoint) union of the initializedVariables of d and the set of names of initializing formals of A and the variable names of variable initialziers in the initializer list of A.
• superParameters: Count of positional super-parameters of d plus number of positional super-parameters in the parameter list of A, and set of super-parameter names of d (disjoint) union with the set of names of named super-parameters in the parameter list of A.
• body: The body of A, if any.

Existing object creation and generative constructor behavior

For generative constructors, the existing pre-augmentation specified behavior is:

When you evaluate a constructor-based object creation expression (expression which invokes a constructor of a class, C, with or without an explicit new, as opposed to, for example, an object creation expression which is a list literal), the following happens (which ignores cases that would have been rejected as compile-time errors):

• A new object instance of the requested class C is created. If the class declaration for C is generic, the type arguments given to C are stored on the object, and type arguments to superclasses are stored too, somehow, so that they can be accessed in instance members.
• The denoted constructor of the mentioned class is invoked with the given argument list to initialize the new object. (“Invoke … with argument list … to initialize” is what you do when to an existing object, just “Invoke” of a constructor is a short way to refer to an constructor-based object creation expression.)

Invoking a generative constructor G of a class C with an argument list L to initialize an object o (phew) is performed as follows:

• If the constructor G is a redirecting generative constructor:

  • Bind the argument list L to its formal parameters to create a parameter scope (as usual, absent arguments imply optional parameters which then get their default value, or a default default value of null). The parent scope of the parameter scope is the member scope of the surrounding class.
  • Evaluates the arguments of the redirection clause in the parameter scope to an argument list A.
  • Invoke the target generative constructor (always of the samme class) with the new argument list A to initialize the same object o.

• If G is a non-redirecting generative constructor:

  • For each instance variable declaration in the surrounding class declaration, in source order, if the instance variable declaration has an initializer expression e:

    • Evaluate e in the body scope of the class C to a value v. As usual, if anything throws, then so does the constructor invocation to initialize.
    • Initialize the corresponding instance variable of o to v.

  • Let A be an uninitialized argument list corresponding to the super parameters of the constructor and the arguments of the super-constructor invocation, if any: One positional entry per positional super parameter or positional argument in the super-constructor invocation, one named entry per named super parameter or named argument in the super-constructor invocation, with no values set yet.)

  • Bind arguments to formals. This has more cases than normal function invocations due to initializing formals and super-parameters.

    • For each parameter of the constructor declaration, in source order:

      • If the argument list has a corresponding value, let v be that value.

        Otherwise the parameter must be optional.

        • If the parameter has a default value expression, let v be the value of that expression.

        • Otherwise let v be the null value.

      • If the parameter is an initializing formal for an instance variable:

        • Initialize the cooresponding variable of o to the value v.
        • Bind the name of the parameter to the value v in the initializer list scope.

      • If the parameter is an super parameter.

        • If the parameter is positional, let i be the number of prior positional super-parameters in the constructor parameter list plus the number of positional arguments in a super-constructor invocation, if any, plus one. Set v as the (one-based) ith positional argument of A.
        • If the parameter is named with name n, set the named argument n of A to v.
        • Bind the name of the parameter to the value v in the initializer list scope.

      • Otherwise the parameter is a normal paramter.

        • Bind the name of the parameter to the value v in the parameter list scope.

    • For each non-super-constructor entry in the initializer list of the constructor declaration:

      • If an assert entry with test expression e:
        • Evaluate e to a value v of type bool in the initializer list scope.
        • If e is false:
          • If the assert has as second expression, evaluate that expression to a value m, then throw an AssertionError containing the value m as message.
          • Otherwise throw an AssertionError with no message.
      • If an initializing entry for an instance variable, with initializer expression e (id = e or this.id = e).
        • Evaluate e in the initializer list scope to a value v.
        • Initialize the corresponding variable of o to the value v.

    • For each instance variable of the class declaration, if the corresponding instance variable of o has not yet been assigned a value, initialize that instance variable to the null value.

    • Let S be the superclass of the current class. (Not class declaration, but instantiated class, which can be an anonymous mixin application class with no explicit declaration.)

    • If the initializer list ends with a super-constructor invocation (super(args) or super.name(args)).

      • evaluate the argument expressions of that invocation in source order, in the initializer list scope. If the expression evaluates to a value v.
      • If the argument is positional, set the value at position i of A to v, where i is one plus the number of positional arguments prior to this argument in the super-constructor invocation arguments.
      • If the arugment is named with name n, set the value of the named argument n of A to v.
      • Let T be the constructor of class S targeted by the super (unnamed) or super.name (named)

    • Otherwise, if there is no super-constructor invocation, let T be the unnamed constructor of class S.

    • Invoke the constructor T of class S with argument list A to initialize o.

    • When this completes, execute the body of C with this bound to o. The parent scope of the body block’s scope is the parameter scope of the invocation.

[lrhn]

@dart-lang/language-team

As TL;DR says, I suggest doing instance variable initializer expression evaluation for a class before starting invocation of the (agumentation stack of) the non-redirecting generative constructor,
AND to execute augmenting initializer expressions first, before the ones of the augmented definition, because that keeps a stack-like execution order.

[jakemac53]

The only real piece of pushback I have here is that this proposal removes the ability for an augmentation to add an explicit super constructor invocation. This was allowed in the previous proposal.

It does complicate things, because now it isn't a strict pre-pend operation for initializers, but it seems like if augmentations can add an extends clause, they will also need the ability to make an explicit super constructor call.

[stereotype441]

The only real piece of pushback I have here is that this proposal removes the ability for an augmentation to add an explicit super constructor invocation. This was allowed in the previous proposal.

It does complicate things, because now it isn't a strict pre-pend operation for initializers, but it seems like if augmentations can add an extends clause, they will also need the ability to make an explicit super constructor call.

Interesting! I had not realized that we were allowing augmentations to add an extends clause. Do we also allow augmentations to change an existing extends clause? I'm assuming we don't, because the semantics of that seem... weird.

The reason I ask is because in my mental model all classes (except Object) as having an extends clause, but as a piece of convenient syntactic sugar, we allow extends Object to be elided. Similarly, I think of all constructors (except Object()) as having a super constructor invocation, but as a piece of convenient syntactic sugar, we allow : super() to be elided.

So if we're allowing an extends clause to be added, but we're not allowing an existing one to be changed, then in this mental model, what we'd actually be allowing is for the user to replace an existing extends clause (but only if the clause they're replacing is an implicit extends Object), and allowing them to replace an existing super constructor invocation (but only if the invocation they're replacing is an implicit : super()). Which seems oddly arbitrary IMHO.

Related question: if a class is declared with extends Object, do we allow an augmentation to "add" an extends clause (and thereby replace 'extends Object')? Similarly, if we want to allow an augmentation to "add" an explicit super constructor call, do we allow this to happen if there's an existing : super() (thereby replacing it)?

• If yes, then again, that seems oddly arbitrary IMHO.
• If no, then that means that class C extends Object {} no longer has precisely the same behavior as class C {}, and the constructor C() : super(); no longer has precisely the same behavior as C();, which seems unfortunate.

[bwilkerson]

The reason I ask is because in my mental model all classes (except Object) as having an extends clause ...

I agree, all classes have an extends clause, whether explicit or implicit. The way I'd work around that in my head is to distinguish between the class that's being defined, and the fragment that's contributing to that definition.

So in my head, a fragment isn't replacing an extends clause, it's contributing one to the merge operation. And the merge operation doesn't allow multiple extends clauses to be contributed (maybe unless they're identical? I haven't looked at the current proposal in a while).

[jakemac53]

What @bwilkerson said basically - the implicit extends Object, default constructor, etc are not there until all the augmentations have been merged into a complete class definition. Then, if there is no extends clause or default constructor they are added.

[lrhn]

Adding a super constructor invocation in an augmentation makes sense.

If we do that, then I think it should be an error to have super parameters in declarations prior to adding the super constructor invocation. Or rather, having a super parameter with no declared super constructor invocation implies one of the form super() as usual.
Which means that declarations until that have filled in zero positional and named arguments in the super constructor argument list too, so the first augmentation to declare a super invocation is free to start from scratch.

Until the first super parameter or explicit super constructor invocation, the super invocation remains undetermined, and open for later augmentations to determine. And if they don't, it was super() all along.

The invocation chain will then carry with it the partially or fully filled out super constructor invocation (the argument list and the target constructor) until reaching the base declaration. That contains only values and a static reference, so it doesn't need access to the scope of the later augmentations when it's invoked.

Can still work.

[lrhn]

I agree, all classes have an extends clause, whether explicit or implicit.

Let me disagree then. Pedantically and semantically only, I agree with the conclusion. I'm just not-picking terminology.

Every class has a superclass (except Object).
Not every class declaration has an extends clause, defining a declared superclass. Some don't, and until now that has meant that the class they define has the Object class as effective declared superclass (which is the actual superclass of the class if there are no declared mixins).

So I do agree that every class declaration introduces a class definition, and
Only the fully augmented class definition is used to introduce the actual class. The class must have s superclass.

And if that still had no declared superclass, mixins or constructors, the actual class gets Object as superclass, and gets a default constructor.

(I call these "definitions" because I want a name for the stack of declarations, and have a way to talk about its cumulative properties. We could achieve that entirely with helper functions, asking "what is the declared superclass of this stack of declarations", and just change the source to say "stack of declarations" where it currently says "declaration".
I like to give things names, a stack of declarations is a definition.)

There is a difference between class C {} and class C extends Object {} syntactically, and augmentations live somewhere between syntax and semantics. The two class declarations are different wrt. modifications. You can add extends Foo only to one of them. Directly or through augmentations.

The lesson to give users is that for augmentations, saying nothing actually means nothing. Defaults come after augmentations.

[jakemac53]

If we do that, then I think it should be an error to have super parameters in declarations prior to adding the super constructor invocation. Or rather, having a super parameter with no declared super constructor invocation implies one of the form super() as usual.

I think this is probably a reasonable restriction.

[lrhn]

I think we could safely allow an augmenting constructor to have an initializer list that ends in augmented(args), as a way to pass different arguments to the augmented constructor. If you don't have one, the implicit default is to call augmented with the same argument list that the augmenting constructor was called with, which is alway a valid argument list for something with the guaranteed same signature.

It could even be useful, fx, to allow an augmenting constructor to wrap an argument:

class ZonedTask {
  final Zone zone;
  final void Function() task;
  ZonedTask(this.zone, this.task);
  void run() { zone.run(task); }
}
augment class ZonedTask {
  augment ZonedTask(Zone zone, void Function() task) : augmented(
      zone.fork(zoneValues: {#_log: Logger.instance}),
      () { 
        (Zone.current[#_log] as Logger).log("task");
        task();
      },
  );
}

(OK, not a very good example.)

I think that should give all the benefits of being able to control the augment-super-invocation, without actually being able to run more or less than once.

[jakemac53]

I think we could safely allow an augmenting constructor to have an initializer list that ends in augmented(args), as a way to pass different arguments to the augmented constructor.

I would rather just not allow it. I am much more interested in allowing you to control the order in which the bodies run, than changing the arguments passed to them, which I think is an anti feature in this case. It is just too weird to see a field formal parameter, pass a value to it, and not have that be the value the field is initialized with.

[lrhn]

We can easily not allow an augmented(...) "super call".

If we allow it, it would be consistent with how we use augmented in other methods: the same way over could use super if the augmented method had been overridden instead of augment. (Does not apply to initializer expressions, can't super-call those.)

The end of the initializer list is the point where a constructor invokes another constructor.
If we ever do want to control or intercept the arguments passed to the augmented constructor, that would be the place, not anywhere in the body.