From 2756e1996fe6c8a71b214064681a92ef75bebac7 Mon Sep 17 00:00:00 2001 From: Till Varoquaux Date: Fri, 26 Apr 2019 17:00:37 -0400 Subject: [PATCH 1/3] Initial import of PEP 592 --- pep-0592.rst | 295 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 295 insertions(+) create mode 100644 pep-0592.rst diff --git a/pep-0592.rst b/pep-0592.rst new file mode 100644 index 00000000000..bd76e47f63a --- /dev/null +++ b/pep-0592.rst @@ -0,0 +1,295 @@ +PEP: 592 +Title: External annotations in the typing module +Author: Till Varoquaux , Konstantin Kashin +Sponsor: Ivan Levkivskyi +Discussions-To: typing-sig@python.org +Status: Draft +Type: Standards Track +Content-Type: text/x-rst +Created: 26-April-2019 +Python-Version: +Post-History: + +Abstract +-------- + +This PEP introduces a mechanism to extend the type annotations from PEP +484 with arbitrary metadata. + +Motivation +---------- + +PEP 484 provides a standard semantic for the annotations introduced in +PEP 3107. PEP 484 is prescriptive but it is the de-facto standard +for most of the consumers of annotations; in many statically checked +code bases, where type annotations are widely used, they have +effectively crowded out any other form of annotation. Some of the use +cases for annotations described in PEP 3107 (database mapping, +foreign languages bridge) are not currently realistic given the +prevalence of type annotations. Furthermore the standardisation of type +annotations rules out advanced features only supported by specific type +checkers. + +Rationale +--------- + +We propose adding an ``Annotated`` type to the typing module to decorate +existing types with context-specific metadata. Specifically, a type +``T`` can be annotated with metadata ``x`` via the typehint +``Annotated[T, x]``. This metadata can be used for either static +analysis or at runtime. If a library (or tool) encounters a typehint +``Annotated[T, x]`` and has no special logic for metadata ``x``, it +should ignore it and simply treat the type as ``T``. Unlike the +``no_type_check`` functionality that current exists in the ``typing`` +module which completely disables typechecking annotations on a function +or a class, the ``Annotated`` type allows for both static typechecking +of ``T`` (e.g., via mypy [mypy]_ or Pyre [pyre]_, which can safely ignore ``x``) +together with runtime access to ``x`` within a specific application. We +believe that the introduction of this type would address a diverse set +of use cases of interest to the broader Python community. + +This was originally brought up as issue 600 [issue-600]_ in the typing github +and then discussed in Python ideas [python-ideas]_. + +Motivating examples +------------------- + +Combining runtime and static uses of annotations +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +There's an emerging trend of libraries leveraging the typing annotations at +runtime (e.g.: dataclasses); having the ability to extend the typing annotations +with external data would be a great boon for those libraries. + +Example:: + + UnsignedShort = Annotated[int, cstruct.ctype('H')] + SignedChar = Annotated[int, cstruct.ctype('b')] + + class Student(cstruct.Packed): + # mypy typechecks 'name' field as 'str' + name: Annotated[str, cstruct.ctype("<10s")] + serialnum: UnsignedShort + school: SignedChar + gradelevel: SignedChar + + # 'unpack' only uses the metadata within the type annotations + Student.unpack(record) + # Student(name=b'raymond ', serialnum=4658, school=264, gradelevel=8) + +Lowering barriers to developing new typing constructs +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Typically when adding a new type, we need to upstream that type to the +typing module and change mypy, PyCharm [pycharm]_, Pyre, +pytype [pytype]_, etc... +This is particularly important when working on open-source code that +makes use of our new types, seeing as the code would not be immediately +transportable to other developers' tools without additional logic. As a result, +there is a high cost to developing and trying out new types in a codebase. +Ideally, we should be able to introduce new types in a manner that allows for +graceful degradation when clients do not have a custom mypy plugin +[mypy-plugin]_, which would lower the barrier to development and ensure some +degree of backward compatibility. + +For example, suppose that we wanted to add support for tagged unions +[tagged-union]_ to Python. One way to accomplish would be to annotate +``TypedDict`` [typed-dict]_ in Python such that only one field is allowed to be +set:: + + Currency = Annotated[ + TypedDict('Currency', {'dollars': float, 'pounds': float}, total=False), + TaggedUnion, + ] + +This is a somewhat cumbersome syntax but it allows us to iterate on this +proof-of-concept and have people with type checkers (or other tools) that don't +yet support this feature work in a codebase with tagged unions. We could easily +test this proposal and iron out the kinks before trying to upstream tagged union +to ``typing``, mypy, etc. Moreover, tools that do not have support for parsing +the ``TaggedUnion`` annotation would still be able able to treat ``Currency`` as +a ``TypedDict``, which is still a close approximation (slightly less strict). + +Specification +------------- + +Syntax +~~~~~~ + +``Annotated`` is parameterized with a type and an arbitrary list of +Python values that represent the annotations. Here are the specific +details of the syntax: + +* The first argument to ``Annotated`` must be a valid type + +* Multiple type annotations are supported (``Annotated`` supports variadic + arguments):: + + Annotated[int, ValueRange(3, 10), ctype("char")] + +* ``Annotated`` must be called with at least two arguments ( + ``Annotated[int]`` is not valid) + +* The order of the annotations is preserved and matters for equality + checks:: + + Annotated[int, ValueRange(3, 10), ctype("char")] != Annotated[ + int, ctype("char"), ValueRange(3, 10) + ] + +* Nested ``Annotated`` types are flattened, with metadata ordered + starting with the innermost annotation:: + + Annotated[Annotated[int, ValueRange(3, 10)], ctype("char")] == Annotated[ + int, ValueRange(3, 10), ctype("char") + ] + +* Duplicated annotations are not removed:: + + Annotated[int, ValueRange(3, 10)] != Annotated[ + int, ValueRange(3, 10), ValueRange(3, 10) + ] + +* ``Annotated`` can be used as a higher order aliases:: + + Typevar T = ... + Vec = Annotated[List[Tuple[T, T]], MaxLen(10)] + V = Vec[int] + + V == Annotated[List[Tuple[int, int]], MaxLen(10)] + +Consuming annotations +~~~~~~~~~~~~~~~~~~~~~ + +Ultimately, the responsibility of how to interpret the annotations (if +at all) is the responsibility of the tool or library encountering the +``Annotated`` type. A tool or library encountering an ``Annotated`` type +can scan through the annotations to determine if they are of interest +(e.g., using ``isinstance()``). + +**Unknown annotations:** When a tool or a library does not support +annotations or encounters an unknown annotation it should just ignore it +and treat annotated type as the underlying type. For example, if we were +to add an annotation that is not an instance of ``new_struct.ctype`` to the +annotation for name (e.g., +``Annotated[str, 'foo', new_struct.ctype("<10s")]``), the unpack method +should ignore it. + +**Namespacing annotations:** We do not need namespaces for annotations +since the class used by the annotations acts as a namespace. + +**Multiple annotations:** It's up to the tool consuming the annotations +to decide whether the client is allowed to have several annotations on +one type and how to merge those annotations. + +Since the ``Annotated`` type allows you to put several annotations of +the same (or different) type(s) on any node, the tools or libraries +consuming those annotations are in charge of dealing with potential +duplicates. For example, if you are doing value range analysis you might +allow this:: + + T1 = Annotated[int, ValueRange(-10, 5)] + T2 = Annotated[T1, ValueRange(-20, 3)] + +Flattening nested annotations, this translates to:: + + T2 = Annotated[int, ValueRange(-10, 5), ValueRange(-20, 3)] + +Interaction with ``get_type_hints()`` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +``typing.get_type_hints()`` will take a new argument ``include_extras`` that +defaults to ``False`` to preserve backward compatibility. When +``include_extras`` is ``False``, the extra annotations will be stripped +out of the returned value. Otherwise, the annotations will be returned +unchanged:: + + @struct.packedclass Student(NamedTuple): + name: Annotated[str, struct.ctype("<10s")] + + get_type_hints(Student) == {'name': str} + get_type_hints(Student, include_extras=False) == {'name': str} + get_type_hints(Student, include_extras=True) == { + 'name': Annotated[str, struct.ctype("<10s")] + } + +Aliases & Concerns over verbosity +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Writing ``typing.Annotated`` everywhere can be quite verbose; +fortunately, the ability to alias annotations means that in practice we +don't expect clients to have to write lots of boilerplate code:: + + T = TypeVar('T') + Const = Annotated[T, my_annotations.CONST] + + Class C: + def const_method(self: Const[List[int]]) -> int: + ... + +Rejected ideas +-------------- + +Some of the proposed ideas were rejected from this PEP because they would +cause ``Annotated`` to not integrate cleanly with the other typing annotations: + +* ``Annotated`` cannot infer the decorated type. You could imagine that + ``Annotated[..., Immutable]`` could be used to mark a value as immutable + while still infering its type. Typing does not support support using the + inferred type anywhere else [issue-276]_; it's best to not add this as a + special case. + +* We could use ``(Type, Ann1, Ann2, ...)`` instead of + ``Annotated[Type, Ann1, Ann2, ...]``. This would cause confusion when + annotations appear in nested positions (``Callable[[A, B], C]`` is too similar + to ``Callable[[(A, B)], C]``) and would make it impossible for constructors to + be passthrough (``T(5) == C(5)`` when ``C = Annotation[T, Ann]``). + +This feature was left out to keep the design simple: + +* ``Annotated`` cannot be called with a single argument. Annotated could support + returning the underlying value when called with a single argument (e.g.: + ``Annotated[int] == int``). This complicates the specifications and adds + little benefit. + + +References +---------- + +.. [issue-600] + https://github.com/python/typing/issues/600 + +.. [python-ideas] + https://mail.python.org/pipermail/python-ideas/2019-January/054908.html + +.. [struct-doc] + https://docs.python.org/3/library/struct.html#examples + +.. [mypy] + http://www.mypy-lang.org/ + +.. [pyre] + https://pyre-check.org/ + +.. [pycharm] + https://www.jetbrains.com/pycharm/ + +.. [pytype] + https://github.com/google/pytype + +.. [mypy-plugin] + https://github.com/python/mypy_extensions + +.. [tagged-union] + https://en.wikipedia.org/wiki/Tagged_union + +.. [typed-dict] + https://mypy.readthedocs.io/en/latest/more_types.html#typeddict + +.. [issue-276] + https://github.com/python/typing/issues/276 + +Copyright +--------- + +This document has been placed in the public domain. From 9eb4698c7bfa39243518b02fd6688c467ed41280 Mon Sep 17 00:00:00 2001 From: Till Varoquaux Date: Thu, 16 May 2019 00:36:54 -0400 Subject: [PATCH 2/3] renamed to pep-593, clarified a lot of "we"s and addressed comments... --- pep-0592.rst => pep-0593.rst | 75 ++++++++++++++++++------------------ 1 file changed, 38 insertions(+), 37 deletions(-) rename pep-0592.rst => pep-0593.rst (80%) diff --git a/pep-0592.rst b/pep-0593.rst similarity index 80% rename from pep-0592.rst rename to pep-0593.rst index bd76e47f63a..24998a290bb 100644 --- a/pep-0592.rst +++ b/pep-0593.rst @@ -1,5 +1,5 @@ -PEP: 592 -Title: External annotations in the typing module +PEP: 593 +Title: Flexible function and variable annotations Author: Till Varoquaux , Konstantin Kashin Sponsor: Ivan Levkivskyi Discussions-To: typing-sig@python.org @@ -33,20 +33,20 @@ checkers. Rationale --------- -We propose adding an ``Annotated`` type to the typing module to decorate +This PEP adds an ``Annotated`` type to the typing module to decorate existing types with context-specific metadata. Specifically, a type ``T`` can be annotated with metadata ``x`` via the typehint ``Annotated[T, x]``. This metadata can be used for either static analysis or at runtime. If a library (or tool) encounters a typehint ``Annotated[T, x]`` and has no special logic for metadata ``x``, it should ignore it and simply treat the type as ``T``. Unlike the -``no_type_check`` functionality that current exists in the ``typing`` +``no_type_check`` functionality that currently exists in the ``typing`` module which completely disables typechecking annotations on a function or a class, the ``Annotated`` type allows for both static typechecking of ``T`` (e.g., via mypy [mypy]_ or Pyre [pyre]_, which can safely ignore ``x``) -together with runtime access to ``x`` within a specific application. We -believe that the introduction of this type would address a diverse set -of use cases of interest to the broader Python community. +together with runtime access to ``x`` within a specific application. The +introduction of this type would address a diverse set of use cases of interest +to the broader Python community. This was originally brought up as issue 600 [issue-600]_ in the typing github and then discussed in Python ideas [python-ideas]_. @@ -61,38 +61,38 @@ There's an emerging trend of libraries leveraging the typing annotations at runtime (e.g.: dataclasses); having the ability to extend the typing annotations with external data would be a great boon for those libraries. -Example:: +Here's an example of how a hypothetical module could leverage annotations to +read c structs:: - UnsignedShort = Annotated[int, cstruct.ctype('H')] - SignedChar = Annotated[int, cstruct.ctype('b')] + UnsignedShort = Annotated[int, struct2.ctype('H')] + SignedChar = Annotated[int, struct2.ctype('b')] - class Student(cstruct.Packed): + class Student(struct2.Packed): # mypy typechecks 'name' field as 'str' - name: Annotated[str, cstruct.ctype("<10s")] + name: Annotated[str, struct2.ctype("<10s")] serialnum: UnsignedShort school: SignedChar - gradelevel: SignedChar # 'unpack' only uses the metadata within the type annotations Student.unpack(record) - # Student(name=b'raymond ', serialnum=4658, school=264, gradelevel=8) + # Student(name=b'raymond ', serialnum=4658, school=264) Lowering barriers to developing new typing constructs ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Typically when adding a new type, we need to upstream that type to the -typing module and change mypy, PyCharm [pycharm]_, Pyre, -pytype [pytype]_, etc... +Typically when adding a new type, a developer need to upstream that type to the +typing module and change mypy, PyCharm [pycharm]_, Pyre, pytype [pytype]_, +etc... This is particularly important when working on open-source code that -makes use of our new types, seeing as the code would not be immediately +makes use of these types, seeing as the code would not be immediately transportable to other developers' tools without additional logic. As a result, there is a high cost to developing and trying out new types in a codebase. -Ideally, we should be able to introduce new types in a manner that allows for -graceful degradation when clients do not have a custom mypy plugin -[mypy-plugin]_, which would lower the barrier to development and ensure some +Ideally, authors should be able to introduce new types in a manner that allows +for graceful degradation (e.g.: when clients do not have a custom mypy plugin +[mypy-plugin]_), which would lower the barrier to development and ensure some degree of backward compatibility. -For example, suppose that we wanted to add support for tagged unions +For example, suppose that an author wanted to add support for tagged unions [tagged-union]_ to Python. One way to accomplish would be to annotate ``TypedDict`` [typed-dict]_ in Python such that only one field is allowed to be set:: @@ -104,11 +104,12 @@ set:: This is a somewhat cumbersome syntax but it allows us to iterate on this proof-of-concept and have people with type checkers (or other tools) that don't -yet support this feature work in a codebase with tagged unions. We could easily -test this proposal and iron out the kinks before trying to upstream tagged union -to ``typing``, mypy, etc. Moreover, tools that do not have support for parsing -the ``TaggedUnion`` annotation would still be able able to treat ``Currency`` as -a ``TypedDict``, which is still a close approximation (slightly less strict). +yet support this feature work in a codebase with tagged unions. The author could +easily test this proposal and iron out the kinks before trying to upstream tagged +union to ``typing``, mypy, etc. Moreover, tools that do not have support for +parsing the ``TaggedUnion`` annotation would still be able able to treat +``Currency`` as a ``TypedDict``, which is still a close approximation (slightly +less strict). Specification ------------- @@ -150,7 +151,7 @@ details of the syntax: int, ValueRange(3, 10), ValueRange(3, 10) ] -* ``Annotated`` can be used as a higher order aliases:: +* ``Annotated`` can be used with nested and generic aliases:: Typevar T = ... Vec = Annotated[List[Tuple[T, T]], MaxLen(10)] @@ -169,14 +170,13 @@ can scan through the annotations to determine if they are of interest **Unknown annotations:** When a tool or a library does not support annotations or encounters an unknown annotation it should just ignore it -and treat annotated type as the underlying type. For example, if we were -to add an annotation that is not an instance of ``new_struct.ctype`` to the -annotation for name (e.g., -``Annotated[str, 'foo', new_struct.ctype("<10s")]``), the unpack method -should ignore it. +and treat annotated type as the underlying type. For example, when encountering +an annotation that is not an instance of ``struct2.ctype`` to the annotations +for name (e.g., ``Annotated[str, 'foo', struct2.ctype("<10s")]``), the unpack +method should ignore it. -**Namespacing annotations:** We do not need namespaces for annotations -since the class used by the annotations acts as a namespace. +**Namespacing annotations:** Namespaces are not needed for annotations since +the class used by the annotations acts as a namespace. **Multiple annotations:** It's up to the tool consuming the annotations to decide whether the client is allowed to have several annotations on @@ -204,7 +204,8 @@ defaults to ``False`` to preserve backward compatibility. When out of the returned value. Otherwise, the annotations will be returned unchanged:: - @struct.packedclass Student(NamedTuple): + @struct.packed + class Student(NamedTuple): name: Annotated[str, struct.ctype("<10s")] get_type_hints(Student) == {'name': str} @@ -239,7 +240,7 @@ cause ``Annotated`` to not integrate cleanly with the other typing annotations: inferred type anywhere else [issue-276]_; it's best to not add this as a special case. -* We could use ``(Type, Ann1, Ann2, ...)`` instead of +* Using ``(Type, Ann1, Ann2, ...)`` instead of ``Annotated[Type, Ann1, Ann2, ...]``. This would cause confusion when annotations appear in nested positions (``Callable[[A, B], C]`` is too similar to ``Callable[[(A, B)], C]``) and would make it impossible for constructors to From 4cb15021057ab22e6618370252f8cdcd56011079 Mon Sep 17 00:00:00 2001 From: Ivan Levkivskyi Date: Sun, 19 May 2019 20:01:14 +0100 Subject: [PATCH 3/3] Typo --- pep-0593.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pep-0593.rst b/pep-0593.rst index 24998a290bb..4d61d62a09b 100644 --- a/pep-0593.rst +++ b/pep-0593.rst @@ -204,7 +204,7 @@ defaults to ``False`` to preserve backward compatibility. When out of the returned value. Otherwise, the annotations will be returned unchanged:: - @struct.packed + @struct2.packed class Student(NamedTuple): name: Annotated[str, struct.ctype("<10s")]