Adding CallableParametersVariable to typing_extensions

[mrkmndz]

Corresponding typeshed update for python/typing#636.

Explanation for context from that PR copied below:

At previous typing meetups and at the summit we have discussed plans for typing *args and **kwargs using variadic type variables.
I still think that that is a worthwhile project, but have encountered a limitation of that approach.

If we were to try to type a decorator that transforms a function's return type while leaving the parameters alone, it would be reasonable to try to define a callback protocol that could capture the *args and **kwargs like so.

Treturn = typing.TypeVar(“Treturn”)
Tpositionals = ....
Tkeywords = ...
class BetterCallable(typing.Protocol[Tpositionals, Tkeywords, Treturn]):
  def __call__(*args: Tpositionals, **kwargs: Tkeywords) -> Treturn: …

However there are some problems with trying to come up with a consistent solution for those type variables for a given callable. This problem comes up with even the simplest of callables:

def simple(x: int) -> None: ...
simple <: BetterCallable[[int], [], None]
simple <: BetterCallable[[], {“x”: int}, None]
BetterCallable[[int], [], None] </: BetterCallable[[], {“x”: int}, None]

Any time where a type can implement a protocol in more than one way that aren’t mutually compatible, we can run into situations where we lose information. If we were to make a decorator using this protocol, we have to pick one calling convention to prefer.

def decorator(
  f: BetterCallable[[Ts], [Tmap], int],
) -> BetterCallable[[Ts], [Tmap], str]:
    def decorated(*args: Ts, **kwargs: Tmap) -> str:
       x = f(*args, **kwargs) 
       return int_to_str(x)
    return decorated
@decorator
def foo(x: int) -> int:
    return x
reveal_type(foo) # Option A: BetterCallable[[int], {}, str]
                 # Option B: BetterCallable[[], {x: int}, str]
foo(7)   # fails under option B
foo(x=7) # fails under option A

The core problem here is that by default, parameters in Python can either be passed in positionally or as a keyword parameter. This means we really have three categories (positional-only, positional-or-keyword, keyword-only) we’re trying to jam into two categories.

This strongly suggests the need for a higher-level primitive for capturing all three classes of parameters. I propose this syntax:

from typing import Callable
from typing_extensions import CallableParametersVariable
Tparams = CallableParametersVariable(“Tparams”)
def decorator(f: Callable[Tparams, int]) -> Callable[Tparams, str]: …
@decorator
def foo(x: int) -> int:
    return x
reveal_type(foo) # Callable[[Named(x, int)], str]
foo(7)   # succeeds!
foo(x=7) # also succeeds!

This syntax prioritizes the experience of the consumers of decorators as it directly supports transporting all of the parameter information. However, this comes at the cost of type checking the body of a decorator function:

def decorator(f: Callable[Tparams, int]) -> Callable[Tparams, str]:
    def decorated(*args: object, **kwargs: object) -> str:
       x = f(*args, **kwargs) # error: expected Tparams, got objects
       return int_to_str(x)
    return decorated # error: expected Callable[Tparams, str]

Without separate variables for the positional and keyword arguments, there is no safe way to call a function with CallableParametersVariable parameters. However, this is a small surface of errors/unsoundness as compared to the gains we are getting from the caller-side. This also meshes with the proposal at the meetup to not require body-checking of functions involving variadics at all.

However, with some additional extensions we could type check those calls if we would like to go down that road. We would need to define operators on these variables that would look something like this:

def decorator(f: Callable[Tparams, int]) -> Callable[Tparams, str]:
    def decorated(*args: Positionals[TParams], **kwargs: Keywords[TParams]) -> str:
       x = f(*args, **kwargs) # special case on calling Tparams functions with these special types
       return int_to_str(x)
    return decorated # special case functions defined like this to be subtypes of Callable[Tparams, str]

If you were to want to type a decorator that was doing some kind of mutation on the parameter set, you would likely need to accept the restrictions imposed by the other kinds of proposed variadics, or we would need to define some sort of operation on these CallableParameterVariables.

I have already implemented this in Pyre and would appreciate it's inclusion in typing_extensions.

[mrkmndz]

We're going to be going through a PEP instead