Custom generators (like `async`, `async*` and `sync*`)

[TekExplorer]

I think it could be really interesting if we could create our own generators like async, async*, and sync*

I noticed that really, a lot of the features present in each kind already exists in normal code, ala Completer, StreamController, and custom iterables

I looked at yield and yield* and equated it to StreamController.add and StreamController.addAll
I recognized that async* returned an Iterable<Future<T>>, which Stream implements, not dissimilar to sync*
I looked at await, and recognized how it flattened the callback hell of .thens by just giving us the value and early returning the Completer.future

I looked at these and realized they all return before a single line of code ever runs.

And I looked at #2567 flutter/flutter#25280 and flutter/flutter#51752 and remembered how @rrousselGit mentioned that "hooks" would be a language feature akin to doing final x = use thing;

So the feature that comes to mind is, what if we could make our own functional generators? extend the dart language "directly"?

What if we could do something like:

// Effect, Hook, whatever
// myGenerator could register disposals and such into the resulting object, to be handled elsewhere
Effect<...> makeThing() myGenerator { 
  // primitives would directly implement Effect (similar to flutter_hook's Hook class)
  final (get, set) = use state(0);
 // additional code

  return (get(), set);
}

// elsewhere:
// something like
final (value, setValue) = handler.use(makeThing()) // receives Effect<T>

// where:
T handler.use<T>(Effect<T> effect) {
  registerDispose(effect.dispose);
  // more logic
  return readEffect(effect);
}

and of course, that logic can be anything.

a naive example:

// pseudocode, may not be the best api
generator stringbuilder returns String {
  final buffer = StringBuffer();

  operator yield(String str) => buffer.write(str);
  operator yield*(Iterable<String> strs) => buffer.writeAll(str);

  String operator return() => buffer.toString();
}

Iterable<String> makeManyStrings(int count, String str) sync* {
  for (final i = 0, i < count; i++) {
    yield str;
  }
}
String makeString() stringbuilder {
  yield 'Hello ';
  yield 'World!';
  yield '\n';
  yield* makeManyStrings(3, 'Hehe');
}

final String result = makeString(); // Hello World!\nHeheHeheHehe

simple, but it shows how simple it could be, yet gives a ton of flexibility in terms of power and complexity
isn't declarative code better?

• well okay, not always, but this can make code that would be better declarative, to be declarative

I think this can be a super powerful feature, allowing custom(ish) syntax to exist in packages (or sdks, hint hint) which could possibly simplify a lot.

It would make algebraic effects unnecessary since you'd have to provide them in the returned object before you can use any data.

the presence of yield would also provide "free" lazy loading for certain cases, like it does with stream and iterable

existing generators like async, async* and sync* could even be implemented in this feature, which would have the benefits of being able to look at its implementation (for better understanding and discovery) and also make it possible to add documentation to it.

Sometimes I try to hover over await or yield(*) to see what they would say and... nothing.

we can even reuse some existing syntax - await, yield etc would be definable operators

There's a lot of potential, and this would be a pretty major feature, so I'd like to hear some thoughts!

[hydro63]

I'm not really a fan of this feature, mostly because i don't really see a problem it solves, that a normal function doesn't. Features are not added just because they are interesting. Another problem i see, is that it feels very "meta", sort of like macro, but instead of generating boilerplate code, it changes how the entire language behaves. It's also quite difficult to read, and possibly ambiguous in some cases.

That being said, here are my suggestions:

1. Make it a class - there is no need to add a completely new construct to the language, instead just do the same thing macros did, and make it a special class

// small syntax change
generator class StringBuilder{
  final buffer = StringBuffer();

  operator yield(String str) => buffer.write(str);
  operator yield*(Iterable<String> strs) => buffer.writeAll(str);

  String operator return() => buffer.toString();
}

2. It's possible to make a general function class that would work as a framework on which to build a custom generator

// yield and yield* come from GeneratorFunction
function class StringBuilder<T> implements GeneratorFunction {
  final buffer = StringBuffer();

  @override
  operator yield(String str) => buffer.write(str);
  @override
  operator yield*(Iterable<String> strs) => buffer.writeAll(str);

  @override
  String operator return(T retValue) => buffer.toString();
}

And the reason i would do this is because of it giving it a greater use case, for example having a go-like defer, or to wrap the result.

function class DeferredStatement<T> implements GeneratorFunction {
  List<Function> deferredFuncs = [];
  @override
  operator yield(void Function() func) =>deferredFuncs.add(func);

  @override
  T operator return(T retValue){
    for(final func in deferredFuncs){
      func();
    }
    return retValue;
  }
}

String doSomething() DeferredStatement {
  // i KNOW that in Dart the files are automatically closed, but it's just demonstration
  File f = File("text.txt");
  yield () => f.close();
  return f.readAsStringSync(); //after return runs the yielded function
}

It's also possible to wrap the result in somehting like this

// CatcherFunction -> interface for catching exceptions that occured in function
function class ResultWrapper<T> implements CatcherFunction {
  @override
  Result<T> catch(T err) => Failure(err);

  @override
  Result<T> operator return(T retValue) => Success(retValue);
}

String toThrowOrNotToThrow() ResultWrapper {
  String out = funcCanThrow(); // if throws -> Failure  
  return out;  // success
} // actual return type is Result<String> bcs of ResultWrapper

Possible problems of my proposal, is that it obfuscates the return type of the function, which is not ideal. A possible solution is to have the returns hint that was in your proposed syntax.

That being said, it still is very meta, so i'm not really sure if it's good to have this before we have macros. Also, for such a big change in the language (both yours, and my proposed syntaxes), extensive discussion is needed to make it work. As such, @lrhn, could you please provide feedback.

My standpoint on either of the proposals is that it would be good to have, but the benefit of having this is not really proportional to the difficulty of implementing it correctly

[lrhn]

This sounds like monads, which means it's probably something that won't work well with the Dart type system if you try to abstract over it. That requires a higher-order type system which Dart doesn't have.

If you don't try to generalize, then it's more like normal functional programming.

String stringBuilder(
  void Function({required void Function(String) yield, required void Function(Iterable<String>) yieldAll}) body) {
  var buffer = StringBuffer();
  body(yield: buffer.write, yieldAll: buffer.writeAll);
  return buffer.toString();
}

String writer(Input input) => stringBuilder(({yield, yieldAll}) {
  while (imput.something) {
      yield(input.value);
  }
  yieldAll(input.rest);
});

You can even abstract over writers:

typedef Writer<R, T> = R Function(
    void Function({required void Function(T) yield, required void Function(Iterable<T>) yieldAll}));

but not over "body modifiers" in general, which means that it's likely not a good general language feature.

[TekExplorer]

I'm not particularly hung up on any particular syntax, so sure, make it a special class.

I'm not sure why you're obfuscating return types - existing generators require the "actual return type" be specified.

I'm not sure why it has to be overly meta - frankly the only meta thing about it is that it seems to have is the added operators and hidden controllers, which is already true of existing generators.

But then... That's why you can Ctrl+click to see it's definition.

I really like that deferred statement example, though a fully featured version of this would allow you to create a defer operator like any function, again visible through documentation and source code.

And of course we can use functions - but if that's the case, why do we bother having async await? Why bother having stream and iterable generators?

There's no "thing you can't do" here, as far as I'm aware, but it does mean that you're forced to do it manually.

What I'm asking for is an existing language feature to be generalized, so that we could make our own syntax in a way that can be visually indicative of particular usage, make certain code more ergonomic to read and use, and make some things that require boilerplat-y classes or added-code-padding cleaner and more readable.

Actually, the only particularly strange thing would perhaps be being able to pause execution at yield/await - but that doesn't necessarily have to be implemented here.

[lrhn]

The reason we bother with async and await is that it does something you can't just do with functions: capturing a continuation, without having to linearize it manually.

The async* and await for functionality is more convenience than necessity, it can be desugared to await in several ways, for example using StreamIterator.

You're asking for a generalization of async*, but I'm simply not seeing the benefit being worth the effort. The Iterable and Stream types are low-level abstractions for providing multiple values, either synchronously or asynchronously, pull or push.

Having a way to run code looking like that so that yield does something else than emit a value, that sounds like something you can do by first providing the value and then consuming it.

String stringbuilder(Iterable<String> elements) =>
    (StringBuffer()..writeAll(elements)).to string();

Iterable<String> makeManyStrings(int count, String str) sync* {
  for (final i = 0, i < count; i++) {
    yield str;
  }
}

String makeString() => stringbuilder(() sync* {
  yield 'Hello ';
  yield 'World!';
  yield '\n';
  yield* makeManyStrings(3, 'Hehe');
}());

The over thing I could get behind is a shorter "Iterable literal". Something like

  sync* ('Hello', 'World', '\n', ... makeManyStrings(e, 'Hehe'))

for creating an iterable of those values.
It still is a function body, not a nested expression, it can't use await, so it's not a perfect fit syntactically.

We have low level functionality for generating a sequence of values, for consuming them, for asynchrony, and for abstraction (functions). You can get very far by combining those.
Adding specialized syntax for some combinations of abstractions and producing/consuming values is possible, but it risks being too specific, so lots of similar use-cases do not fit, or to general, so the added benefit is smaller in each specific use-case.

Why abstract over yield? Why not abstract await instead? I can see very interesting things you can do with captured continuations, like proper co-routines, or other effect monads. Abstracting over how elements are consumed isn't that useful, we can already do that because the entire point of sync*/async* is to provide those values for consumption.

So, not seeing a big value-add for this feature, as I understand the proposal.
Not suction that's worth the complexity it adds.