IteratorResult<T> definition conflicts with the JS spec

[ldantonov]

TypeScript Version: 3.8.3

Search Terms: iterator return type

Code

const list = ['v1', 'v2', 'v3']

const f: () => Iterator<string> = () => {
  let index = 0
  return {
    next: (): IteratorResult<string> => {
      console.log(list)
      if (index < list.length) {
        return { value: list[index++]}
      } else {
        return { done: true }
      }
    }
  }
}

Expected behavior:
The code should compile without errors.
Actual behavior:
The code compiles with the error: "Type '{ done: true; }' is not assignable to type 'IteratorResult<string, any>'.
Property 'value' is missing in type '{ done: true; }' but required in type 'IteratorReturnResult'.(2322)"

The error can be bypassed by changing { done: true } to { value: undefined, done: true }. However, the specification allows omitting the value property. See Iteration Protocols

A suggested fix would be to change the definition of the type IteratorReturnResult to:

interface IteratorReturnResult<TReturn> {
    done: true;
    value?: TReturn;
}

Playground Link: https://www.typescriptlang.org/play?#code/MYewdgzgLgBANgS2jAvDA2gcgG4EZMA0MOATIcdgMyYC6AUHaJLAGYBcMAFAJSoB8MAJJQApgCcAhlBBiAPNDEIwAcwFoe-GAG86MeCNhKAJiIAeqGAAZdMMQYCuYsNpt6wZqBx4dh4qTIAlEQh7OCh5KEUVNQEdPXiYJggQOBEAOjgQZU5EaG5XeIQWLmMzGFl4JCgMkRUoAAteOIT4uyhHZy0YbAk4exEOXKh0UtMAajGaAF8CvSmYETgIERcW1ocnbRgjcAGYSP6YGbXj+NOZqaA

Related Issues:
#29982

[IllusionMH]

Suggested fix won't work as described here https://github.com/microsoft/TypeScript/pull/30790/files#r282595851

[ldantonov]

I confess I don't fully understand what the feared failure is, based on reading issue 30790. Is there a specific example of a type deduction failure that can illustrate that?

In any case, an alternative fix could be as follows, although without fully understanding the objection, I don't know if this is any better. This would see IteratorReturnResult tun into a type alias, and default TResult to match T to improve type inference:

type IteratorReturnResult<TResult> = { done: true, value?: TResult } | { done?: false, value: TResult }
type IteratorResult<T, TReturn = T> = IteratorYieldResult<T> | IteratorReturnResult<TReturn>

[IllusionMH]

I think there 2 kind of propblems will arise with value?: TReturn:

1. won't allow to use return value without checking for undefined first
2. won't catch cases when I forget to return a value.

const list = ['v1', 'v2', 'v3']

const f: () => Iterator<string, number> = () => {
  let index = 0
  return {
    next: (): IteratorResult<string, number> => {
      console.log(list)
      if (index < list.length) {
        return { value: list[index++]}
      } else {
        return { done: true, value: index } // Ooops, I can forget to return value here
      }
    }
  }
}

const iter = f();
const res = iter.next();

if (res.done) {
  console.log(`I wish I had ${res.value * 10} values instead`);
  //                          ~~~~~~~~~ Object is possibly 'undefined'. ts(2532)
}

[ldantonov]

Both points are inconveniences, but this is what the spec requires - value is optional when done is true, and therefore it can be undefined. Forcing a different protocol here could create an issue when interfacing with an iterator obtained from a JS library that follows the spec - TS asserts that value is always defined, yet the JS iterator could return { done: true }.

[fatcerberus]

One potential solution would be to define IteratorResult as a discriminated union (with done as the discriminant):

type IteratorResult<T> =
    | { done: false, value: T }
    | { done: true, value?: T };

Although it’s not clear what to do in the case where done is dynamically determined and therefore not statically known to be true or false (I’ve actually written code where this is the case). Should value be optional or mandatory in that case?

[rbuckton]

One potential solution would be to define IteratorResult as a discriminated union (with done as the discriminant):

type IteratorResult<T> =
    | { done: false, value: T }
    | { done: true, value?: T };

Although it’s not clear what to do in the case where done is dynamically determined and therefore not statically known to be true or false (I’ve actually written code where this is the case). Should value be optional or mandatory in that case?

IteratorResult is currently defined as a discriminated union. The problem with letting value be optional is as @IllusionMH describes:

won't allow to use return value without checking for undefined first

The only time it would be type-safe to allow value to be optional would be when TReturn is either undefined or void. We can't change the definition of IteratorReturnResult to make value optional only to serve that specific case. It might be worth considering whether we should consider a void property to be optional, just as we currently consider a void parameter "optional": Playground example.

@DanielRosenwasser - Any thoughts on extending the same treatment of a void parameter to a void property and treating it as an optional property?

[fatcerberus]

@rbuckton Maybe I'm missing something but the discriminated union I described would have value be mandatory if done is false - which means you only need to check for nullish values if done is true, and it's been my experience that in most cases code doesn't tend to return a meaningful value for done: true anyway (I'm open to being proven wrong on this, though). A typical loop would likely be:

const iter = obj[Symbol.iterator]();
while (!iter.done) {
    iter.value;  // never undefined, no need to check
}

No need to check for undefined inside the loop since the discriminated union in combination with CFA takes care of that. The only time you'd need to check is if done is true, which you probably should do anyway, or else use a non-nullish assertion ! if you're absolutely sure.

edit: I screwed up the loop, but it gets the point across and I'm too lazy to fix it. 😛

[ldantonov]

To reiterate, the problem is that, as written, IteratorReturnResult conflicts with the Iterator spec. @fatcerberus and I have suggested fixes. The argument against fixing it, so far, seems to be that it would make it less convenient to use, by requiring an undefined check. Two points:

1. The definition of the Iterator spec is what's driving the need for an 'undefined' check. I understand that one may wish it didn't, but that's a complaint about the spec itself. It doesn't work to have a type that reflects how we wish the spec to be and not how it actually is.
2. The inconvenience of checking for undefined seems extremely minor, especially considering the optional chaining and non-null assertion operators. Moreover, and this bears repeating, the Iterator spec is defined in a way that makes the 'undefined' check unavoidable.

[rbuckton]

To reiterate, the problem is that, as written, IteratorReturnResult conflicts with the Iterator spec. @fatcerberus and I have suggested fixes. The argument against fixing it, so far, seems to be that it would make it less convenient to use, by requiring an undefined check. Two points:

1. The definition of the Iterator spec is what's driving the need for an 'undefined' check. I understand that one may wish it didn't, but that's a complaint about the spec itself. It doesn't work to have a type that reflects how we wish the spec to be and not how it actually is.

I agree that value should be allowed to be removed when done is true, but only when the iterator is typed as IteratorResult<T, void>, which is why I'm investigating treating a property of type void as optional. However if your result is typed IteratorResult<T, number>, then the only correct type for the return result is { done: true, value: number }. Making value optional for that case would remove type safety. However, if we allow properties of type void to be optional as we do for parameters, then { done: true } would be assignable to { done: true, value: void }.

2. The inconvenience of checking for undefined seems extremely minor, especially considering the optional chaining and non-null assertion operators. Moreover, and this bears repeating, the Iterator spec is defined in a way that makes the 'undefined' check unavoidable.

We made it such that calling generator.return() also requires a value if your generator is typed with a non-void return value, which ensures type safety. The only way I can see to accurately address this issue is to allow void properties to be considered optional.

[zaach]

fwiw I ran into this in practice when attempting to use a ReadableStream as an async interator. E.g. this example from Jake Archibald's article doesn't type correctly:

function streamAsyncIterator(stream) {
  // Get a lock on the stream:
  const reader = stream.getReader();

  return {
    next() {
      // Stream reads already resolve with {done, value}, so
      // we can just call read:
      return reader.read();
    },
    return() {
      // Release the lock if the iterator terminates.
      reader.releaseLock();
      return {};
    },
    // for-await calls this on whatever it's passed, so
    // iterators tend to return themselves.
    [Symbol.asyncIterator]() {
      return this;
    }
  };
}

[Amebus]

Since, I, too, am having compilation problems regarding the IteratorReturnResult I add to this issue by adding a point in favor of @ldantonov.

The following is the current definition of the Iterator iterface and the IteratorReturnResult type provided by typescript and it contains a kind of paradox

interface IteratorYieldResult<TYield> {
    done?: false;
    value: TYield;
}

interface IteratorReturnResult<TReturn> {
    done: true;
    value: TReturn; // here the value can't be undefined
}

type IteratorResult<T, TReturn = any> = IteratorYieldResult<T> | IteratorReturnResult<TReturn>;

interface Iterator<T, TReturn = any, TNext = undefined> {
	next(...args: [] | [TNext]): IteratorResult<T, TReturn>;
	return?(value?: TReturn): IteratorResult<T, TReturn>; // here the value provided by the user can be undefined
	throw?(e?: any): IteratorResult<T, TReturn>;
}

If i try to implement the class MyIterator as follow:

class MyIterator<T, TReturn, TN> implements Iterator<T, TReturn, TN> {
	next(...args: [] | [TN]): IteratorResult<T, TReturn> {
		throw new Error("Method not implemented.");
	}
	return?(value?: TReturn): IteratorResult<T, TReturn> {
		return { done: true, value };  
		// what should I use as value if the user set the value parameter to undefined by calling myIterator.return() with empty paranthesis?
		// from MDN https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Iteration_protocols#the_iterator_protocol
		/*
 				return(value) Optional

    A function that accepts zero or one argument and returns an object conforming to the IteratorResult interface, 
    typically with value equal to the value passed in and done equal to true. 
    Calling this method tells the iterator that the caller does not intend to make any more next() calls and can perform any cleanup actions.

		*/
	}
	throw?(e?: any): IteratorResult<T, TReturn> {
		throw new Error("Method not implemented.");
	}
}

I get the following error:

(property) IteratorReturnResult<TReturn>.value: TReturn
Type 'TReturn | undefined' is not assignable to type 'TReturn | T'.
  Type 'undefined' is not assignable to type 'TReturn | T'.ts(2322)

So why the Iterator iterface allows to pass undefined to the Iterator.return function if the IteratorReturnResult don't allows to have undefined as a valid value for the value property?
There is no way to provide a default value for the TResult generic since it can be of any type and the done property typically is true (as stated in the MDN).
To have it working the current return signature should be return(value: TReturn) but then it will rises issues when the done property is set to false devs will be forced to pass a value to the return function.

As they are right now the Iterator interfaces are not consistent with the Iterator protocol and neither with themselves.

I don't know what is the best solution but i upvote the one proposed by @ldantonov since is the most closed to the iterator protocol: the iterator protocol itself tells you to check for undefined values

[rbuckton]

The shape for IteratorReturnResult is as close to the specification as would be allowed by type safety given the current capabilities of TypeScript without introducing significant complexity in the library typings. The only case in which a value could be safely elided for IteratorReturnResult is when TResult includes either void or undefined as a constituent, thus simply making value optional is not type safe.

We previously investigated whether we could make properties optional if they are instantiated with the type void, but ultimately decided against that as it would add more complexity to void.

For now, I am closing this as a design limitation.