Completer.complete() signature is not null-safe-friendly

[tvolkert]

Consider the following definition of Completer.complete():

abstract class Completer<T> {
  void complete([FutureOr<T>? value]);
}

Now look at the following code:

void main() {
  Completer<int> completer = Completer<int>();
  completer.future.then((int value) {
    print(value);
  });
  completer.complete(null);
}

When running in sound null-safety, that code will produce no static analysis errors, compile cleanly, and throw a runtime error when it hits the call to completer.complete(null). This is highly counterintuitive and confusing to developers (at least it was to me when I ran into it).

Upon inspection, it's clear that this is clean to analysis because the completion value argument is optional. While this signature made sense in a pre-null-safety world, it's just going to be a source of errors when running with null safety.

I think we should consider a language change to fix this.

[leafpetersen]

cc @natebosch @lrhn @munificent this is pretty painful. Unfortunately, I imagine this would be massively breaking to fix by making the argument required.

What if we made the argument type FutureOr<T> (no ?) and used a default sentinel value to detect when nothing was passed?

class _FutureNever implements Future<Never> {
  const _FutureNever();
  noSuchMethod(_) => throw "Badness";
}

const Future<Never> sentinel = _FutureNever();

[natebosch]

I imagine this would be massively breaking to fix by making the argument required.

Yes it would be.

What if we made the argument type FutureOr<T> (no ?) and used a default sentinel value to detect when nothing was passed?

This would at least warn about completer.complete(null), but it wouldn't warn about completer.complete() which I think is more common. It would at least make it easier to catch things like completer.complete(someNullableVar) which could be helpful.

[tvolkert]

It would at least make it easier to catch things like completer.complete(someNullableVar) which could be helpful.

fwiw this was the actual case I ran into. The example above was a distillation of that.

[lrhn]

We have a number of functions which allow you to omit the argument and default to null, and which cause a run-time error if omitted when the type variable is non-nullable. They all suck as APIs, but would be breaking to change.

I'll look into whether using a default sentinel value makes sense in some cases. The completer example here seems to work because the sentinel can implement Future, but generally we can't have a default value of a generic type.

[mraleph]

What if this:

abstract class Completer<T> {
  void complete([FutureOr<T> value]);
}

actually behaved differently depending on whether T is nullable or not? e.g. if T is non-nullable then value became required rather than optional if no default value is provided?

[lrhn]

@mraleph That would be a nice language feature. We have the problem that this function is perfectly valid already because it's abstract. It doesn't say how it plans to implement that (likely a default value, but you never know). We can't see from the abstract function signature whether the invocation is allowed or not because we can't see whether it has a default value.

I suggested (somewhere) to allow a (generic or in a generic interface) function type to declare a default value type, and allow potentially unsafe default values and default values types, so you'd write:

 void complete([FutureOr<T> value = Null]); // abstract functions have default value types
/// and, having that type,
void Complete([FutureOr<T> value = null]) { ... } // concrete functions have concrete default values

and then an invocation of a generic function with an unsafe default value type for an omitted argument would only be valid if the type argument makes the default value valid.
(Non-generic functions can't really use this for anything, they're either always valid or always invalid).

That also allows you to have unsound defaults other than null. I think it's too specialized to only hang this on nullability.
You can also do something like: void sort([int compare(E a, E b) = Comparable.compare]) { ... } which can be called without an argument only if E implements Comparable.

[lrhn]

This particular case is actually simpler than it looks. We can just change the signature of complete to void complete([FutureOr<T> value]); and not bother finding a default value. The actual implementation classes can then implement it as void complete([FutureOr<T>? value]). Since those implementations are private, callers must still pass a non-nullable value, or not pass the argument at all, in which case the implementation uses null as the sentinel for non-nullable types and as the value for nullable types.

Even if you tear off the function, you can't tell the difference because it's already covariant by generic, so the run-time type is void Function(Object?) anyway.

Looking at this actually suggests to me that there is a use-case for having an optional argument with a tighter bound than the what the default value requires. So, a foo([int x = null]) where the internal type of x is int?, but the function doesn't accept null as argument. I can do that with overriding

abstract class C {
  factory C() = D;
  void foo([int x]);
}
class _D implements C {
  void foo([int? x = null]) { ... }
}

Maybe we should be able to do that in a single declaration as well. Say, if you can set the internal type of a parameter using as type:

class C {
  void foo([int x as int? = null]) { ... }
}

(Then we could allow the as to be both looser and stricter than the public type, to allow the use-case futureOfIterable.then((x as List) { ... I know it's a list ... }); which has been requested a few times already.

[tvolkert]

void complete([required? FutureOr value]);

Yes, it would be handy to be able to express "this argument is optional, but if it is specified, it needs be type X (e.g. FutureOr)"

[lrhn]

@tvolkert You can have optional and non-nullable parameters now, but only if you can provide a default value within that type. It's not possible to have a concrete method with an optional parameter with a restricted type, and a default value outside of that type. (And if that type happens to be a type variable, there is no way to create a sentinel value within the type).
That's why nullable optional parameters are so easy to work with, they have a default default value which is always valid. Non-nullable optional parameters need a specific default value to be provided (and a constant one, just to make it even harder).

In some situations, you can cheat.
An instance method can have an abstract superclass type with void foo([int x]); and a private implementation subclass with void foo([int? x]) { ... }. A factory constructor can be Foo([int x]) = _Foo; with _Foo([int? x]) => ...;. Those are the only two cases where you can write a stricter type in the interface than in the implementation, without having to provide an actual default value in the interface.
It would be nice if we could get the same effect for concrete methods, without going through hoops. For example, I can't make the Iterable interface stricter because it's also a skeleton implementation class.

[tvolkert]

Yep. I was saying that a way in the language to differentiate between an argument being specified vs unspecified could be handy, but the cognitive overhead it would cause is probably not worth it.

[lrhn]

About detecting whether an optional parameter was passed as an argument or not ... we tried that, and it made forwarding arguments more complicated. It needs to be combined with a way to specify "no argument" conditionally, so you can do

foo({int x}) => bar(x: x::wasPassed ? x : __nothing__);

(or some much better syntax) in order to be able to wrap functions and forward arguments without having an exponential number of cases in the number of optional parameters).

My idea of allowing bar({int x as int? = null}) to add a sentinel value outside of the outside parameter type has the same issue.
You can't forward that to another function without doing => x == null ? foo() : foo(x: x); and then we're back to being exponential in the independent (named) optional parameters if you want to optionally forward all arguments.

We could take a hint from list literals and allow: bar({int x as int? = null}) => foo(x: if (x != null) x); which will pass nothing if the condition is false. (Only works for named arguments, unless we start allowing omitting non-trailing positional arguments too, which we can: qux(2, 4, , , 5) would omit the third and fourth positional argument and still pass the fifth. It's definitely doable.

[rrousselGit]

As a comeback to this, would it be possible to instead deprecate complete in favor of a null-safe variant?

Especially now that we have dart fix, a fix could be implemented for this to automatically migrate.

[lrhn]

It's such a good name!

I'd rather have the option to "deprecate optionality", maybe just @deprecateOptional, meaning that the parameter will become required in the future. You can start passing arguments now nd get rid of the early warnings, but it won't break before the parameter actually becomes required.

Maybe we should have other deprecation-for-change markers, instead of just the current "will be removed" variant.

[rrousselGit]

maybe just @deprecateOptional,

That sounds like a good addition. And it could be applied to many things, like Future.value() or Iterable.firstElse(orElse:)

[leafpetersen]

This just bit someone internally via Future.value. Is there something we should do here? There are some suggestions in this thread about ways to change this without changing the language. Some other options:

• Just make a breaking change to make the parameter required.
• Add a lint for when passing a nullable value but the inferred type is non-nullable (cc @pq)?
• @munificent has a proposal somewhere to enable detecting when an argument is passed + conditional argument passing. This would enable optional non-nullable parameters, which could be used here.

[lrhn]

(Change the parameter and argument passing system so that nullable parameters are optional, and vice versa, and change complete to void complete(FutureOr<T> result). Then Completer.complete() can only be called without an argument if the future value type is actually nullable.)

[natebosch]

• Add a lint for when passing a nullable value but the inferred type is non-nullable

Outside of language level changes, this is my preference. If we have the same foot-gun in other APIs we could wrap them up in one lint.

Maybe we could add a lint for now, and if we get new language features to express this directly we could switch to that. If we do think it's plausible we'll get new language features in this area I'd prefer not to make the breaking change.

[leafpetersen]

it's plausible we'll get new language features in this area I'd prefer not to make the breaking change.

I think it will always be in principle breaking, since the goal is to turn current (potential) runtime errors into static errors. So for example, making the parameter type optional but non-nullable would cause a static error in cases that are currently potential (but not actual) runtime errors, because the value passed is never null.

[natebosch]

I think it will always be in principle breaking

True. I should have used a more loose definition like non-breaking for code which didn't trigger the proposed lint.

What I hope to avoid if we can, is requiring that all completer.complete() migrate to completer.complete(null).

[pq]

Add a lint for when passing a nullable value but the inferred type is non-nullable

@leafpetersen: are you proposing this for completer.complete exclusively or more broadly?

Looking way up-thread, I really resonate w/ @tvolkert's observation:

Yes, it would be handy to be able to express "this argument is optional, but if it is specified, it needs be type X (e.g. FutureOr)"

Not being able to express this has bitten us a lot.

If the proposal is for completer.complete specifically, I guess I might prefer an annotation (or pragma or whatever) to signal "this argument is optional, but if it is specified, it needs be type X" and then enforce this in analyzer as a hint.

[eernstg]

Unfolding an idea which has nearly been explored above (maybe I just missed it): Conditional default values.

Any optional formal parameter can have a conditional default value, ?= <expression>. The static type of the expression is unconstrained, and the expression does not have a context type.

An invocation where a formal parameter with an optional default value is omitted is an error, unless the static type of the default value is assignable to the type of the parameter. In the case where the enclosing function declaration is generic, this means that some actual type arguments will cause the parameter to have a default value, others will cause it to not have a default value (and in that case it is an error to omit the parameter).

A conditional default value is associated with a dynamic check (such that there is no doubt about the soundness), generally of the form x as T at the beginning of the method body. This also enables dynamic invocations where the soundness of executing the body is ensured.

For example:

void f<X>([X x ?= null]) {}

X g<X extends num>([X x ?= 1.5]) => x;

// Conditional default values have no context type, so we may need to specify type arguments.
X h<X>(List<X> xs ?= const <String>['Hello!']) => xs[0];

void main() {
  f(); // `f<dynamic>(null)`, OK.
  f<int>(); // Compile-time error, `Null` is not assignable to `int`.
  f<int?>(); // OK.
  (f as Function)<int>(); // Run-time error.
  (f as Function)<int?>(); // OK.

  g(5); // OK.
  g(); // OK, `g<num>(1.5)`.
  int i = g(); // Compile-time error.

  void local<S extends num>() {
    S s = g(); // Compile-time error, `double` not assignable to `S`.
  }

  h(); // OK, `h<String>(const <String>['Hello!'])`.
  List<Object> os = h(); // OK, `h<Object>(const <String>['Hello!'])`.
}

Of course, it is tempting to say that we can have multiple conditional default values, and then we'll select the first one that has an assignable type for statically checked invocations; but it is not obvious how to deal with dynamic invocations, and it's probably too twisted to carry its own weight.

A named optional parameter with a conditional default value just turns into a required named parameter when the typing does not match up. A positional optional parameter with a conditional default value will force all earlier positional parameters to be required when the typing does not match up.

I think it could work.

[lrhn]

@eernstg I remember specifying something like this too, so I definitely think it can work. You can drop the ? and just use =, the default value is implicitly conditional if it isn't always assignable to the parameter type. (But requiring the ? avoids accidentally choosing a too narrow default value, and not be warned about it.)

It requires including the default value's type in the function type as well, so something like;

void Function<X extend Foo>([X x ?= Bar])

is a function type for a function which takes a type argument, and which has a default value of type Bar which applies only when Bar <: X.
You can omit default value type when it always applies (like [X? x = Null] or [int x = int], meaning it's a subtype of the parameter type). If it always applies, it's not needed as part of the function type.
(It only really makes sense when a type variable is involved.)

A void Function([X = num]) is a subtype of void Function([X = int]), because the argument can be omitted in (at least) the same situations, so it's substitutable. A void Function([X]) is a subtype of void Function([X = whatever]).

Since the feature affects the type system, it's probably a "big" feature.

[eernstg]

Yes, {T t ?= S} in a function type means that the named parameter t is required unless S <: [T1/X1..Tk/Xk]T, where X1..Xk are the type parameters of the function type, and T1..Tk are the actual type arguments passed in the given function invocation. A similar rule applies for positional optionals, except that they make all parameters up to the one in focus required.

If we decide that this feature is really, really desirable, it would be useful to take a look at it, in order to see how big it actually is.

[leafpetersen]

If we decide that this feature is really, really desirable, it would be useful to take a look at it, in order to see how big it actually is.

Are there any really compelling use cases for anything other than null as the conditional default value? If not (and I can't think of any off hand), then you could simplify it somewhat I think, but the complexity still remains for function types (since you end up with functions with conditionally required parameters, and you need some way to record that in the function type).

[leafpetersen]

@leafpetersen: are you proposing this for completer.complete exclusively or more broadly?

More broadly, for some definition of broadly. The example that came up internally was with Future.value, which has the same issue. I imagine there are others. I'm not sure what the right lint criteria would be, this would require some design discussion and work to figure out the right approach.