From 53c3d1a703e4f3208c59699f9ad7e313919327cf Mon Sep 17 00:00:00 2001 From: Jelle Zijlstra Date: Sat, 10 Feb 2024 20:14:42 -0800 Subject: [PATCH] PEP 742: TypeNarrower (#3649) Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com> --- .github/CODEOWNERS | 1 + peps/pep-0483.rst | 1 + peps/pep-0742.rst | 336 +++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 338 insertions(+) create mode 100644 peps/pep-0742.rst diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index e17e0940f..cbdf45b6d 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -619,6 +619,7 @@ peps/pep-0736.rst @gvanrossum @Rosuav peps/pep-0737.rst @vstinner peps/pep-0738.rst @encukou peps/pep-0740.rst @dstufft +peps/pep-0742.rst @JelleZijlstra # ... # peps/pep-0754.rst # ... diff --git a/peps/pep-0483.rst b/peps/pep-0483.rst index 0ae2749fe..72e67681e 100644 --- a/peps/pep-0483.rst +++ b/peps/pep-0483.rst @@ -165,6 +165,7 @@ structural subtyping is considered more flexible. We strive to provide support for both approaches, so that structural information can be used in addition to nominal subtyping. +.. _pep-483-gradual-typing: Summary of gradual typing ========================= diff --git a/peps/pep-0742.rst b/peps/pep-0742.rst new file mode 100644 index 000000000..f6260c87f --- /dev/null +++ b/peps/pep-0742.rst @@ -0,0 +1,336 @@ +PEP: 742 +Title: Narrowing types with TypeNarrower +Author: Jelle Zijlstra +Status: Draft +Type: Standards Track +Topic: Typing +Created: 07-Feb-2024 +Python-Version: 3.13 +Replaces: 724 + + +Abstract +======== + +This PEP proposes a new special form, ``TypeNarrower``, to allow annotating functions that can be used +to narrow the type of a value, similar to the builtin :py:func:`isinstance`. Unlike the existing +:py:data:`typing.TypeGuard` special form, ``TypeNarrower`` can narrow the type in both the ``if`` +and ``else`` branches of a conditional. + + +Motivation +========== + +Typed Python code often requires users to narrow the type of a variable based on a conditional. +For example, if a function accepts a union of two types, it may use an :py:func:`isinstance` check +to discriminate between the two types. Type checkers commonly support type narrowing based on various +builtin function and operations, but occasionally, it is useful to use a user-defined function to +perform type narrowing. + +To support such use cases, :pep:`647` introduced the :py:data:`typing.TypeGuard` special form, which +allows users to define type guards:: + + from typing import assert_type, TypeGuard + + def is_str(x: object) -> TypeGuard[str]: + return isinstance(x, str) + + def f(x: object) -> None: + if is_str(x): + assert_type(x, str) + else: + assert_type(x, object) + +Unfortunately, the behavior of :py:data:`typing.TypeGuard` has some limitations that make it +less useful for many common use cases, as explained also in the "Motivation" section of :pep:`724`. +In particular: + +* Type checkers must use exactly the ``TypeGuard`` return type as the narrowed type if the + type guard returns ``True``. They cannot use pre-existing knowledge about the type of the + variable. +* In the case where the type guard returns ``False``, the type checker cannot apply any + additional narrowing. + +The standard library function :py:func:`inspect.isawaitable` may serve as an example. It +returns whether the argument is an awaitable object, and +`typeshed `__ +currently annotates it as:: + + def isawaitable(object: object) -> TypeGuard[Awaitable[Any]]: ... + +A user `reported `__ an issue to mypy about +the behavior of this function. They observed the following behavior:: + + import inspect + from collections.abc import Awaitable + from typing import reveal_type + + async def f(t: Awaitable[int] | int) -> None: + if inspect.isawaitable(t): + reveal_type(t) # Awaitable[Any] + else: + reveal_type(t) # Awaitable[int] | int + +This behavior is consistent with :pep:`647`, but it did not match the user's expectations. +Instead, they would expect the type of ``t`` to be narrowed to ``Awaitable[int]`` in the ``if`` +branch, and to ``int`` in the ``else`` branch. This PEP proposes a new construct that does +exactly that. + +Other examples of issues that arose out of the current behavior of ``TypeGuard`` include: + +* `Python typing issue `__ (``numpy.isscalar``) +* `Python typing issue `__ (:py:func:`dataclasses.is_dataclass`) +* `Pyright issue `__ (expecting :py:data:`typing.TypeGuard` to work like :py:func:`isinstance`) +* `Pyright issue `__ (expecting narrowing in the ``else`` branch) +* `Mypy issue `__ (expecting narrowing in the ``else`` branch) +* `Mypy issue `__ (combining multiple TypeGuards) +* `Mypy issue `__ (expecting narrowing in the ``else`` branch) +* `Mypy issue `__ (user-defined function similar to :py:func:`inspect.isawaitable`) +* `Typeshed issue `__ (``asyncio.iscoroutinefunction``) + +Rationale +========= + +The problems with the current behavior of :py:data:`typing.TypeGuard` compel us to improve +the type system to allow a different type narrowing behavior. :pep:`724` proposed to change +the behavior of the existing :py:data:`typing.TypeGuard` construct, but we :ref:`believe ` +that the backwards compatibility implications of that change are too severe. Instead, we propose +adding a new special form with the desired semantics. + +We acknowledge that this leads to an unfortunate situation where there are two constructs with +a similar purpose and similar semantics. We believe that users are more likely to want the behavior +of ``TypeNarrower``, the new form proposed in this PEP, and therefore we recommend that documentation +emphasize ``TypeNarrower`` over ``TypeGuard`` as a more commonly applicable tool. However, the semantics of +``TypeGuard`` are occasionally useful, and we do not propose to deprecate or remove it. In the long +run, most users should use ``TypeNarrower``, and ``TypeGuard`` should be reserved for rare cases +where its behavior is specifically desired. + + +Specification +============= + +A new special form, ``TypeNarrower``, is added to the :py:mod:`typing` +module. Its usage, behavior, and runtime implementation are similar to +those of :py:data:`typing.TypeGuard`. + +It accepts a single +argument and can be used as the return type of a function. A function annotated as returning a +``TypeNarrower`` is called a type narrowing function. Type narrowing functions must return ``bool`` +values, and the type checker should verify that all return paths return +``bool``. + +Type narrowing functions must accept at least one positional argument. The type +narrowing behavior is applied to the first positional argument passed to +the function. The function may accept additional arguments, but they are +not affected by type narrowing. If a type narrowing function is implemented as +an instance method or class method, the first positional argument maps +to the second parameter (after ``self`` or ``cls``). + +Type narrowing behavior +----------------------- + +To specify the behavior of ``TypeNarrower``, we use the following terminology: + +* I = ``TypeNarrower`` input type +* R = ``TypeNarrower`` return type +* A = Type of argument passed to type narrowing function (pre-narrowed) +* NP = Narrowed type (positive; used when ``TypeNarrower`` returned ``True``) +* NN = Narrowed type (negative; used when ``TypeNarrower`` returned ``False``) + +.. code-block:: python + + def narrower(x: I) -> TypeNarrower[R]: ... + + def func1(val: A): + if narrower(val): + assert_type(val, NP) + else: + assert_type(val, NN) + +The return type ``R`` must be :ref:`consistent with ` ``I``. The type checker should +emit an error if this condition is not met. + +Formally, type *NP* should be narrowed to :math:`A \land R`, +the intersection of *A* and *R*, and type *NN* should be narrowed to +:math:`A \land \neg R`, the intersection of *A* and the complement of *R*. +In practice, the theoretic types for strict type guards cannot be expressed +precisely in the Python type system. Type checkers should fall back on +practical approximations of these types. As a rule of thumb, a type checker +should use the same type narrowing logic -- and get results that are consistent +with -- its handling of :py:func:`isinstance`. This guidance allows for changes and +improvements if the type system is extended in the future. + +Examples +-------- + +Type narrowing is applied in both the positive and negative case:: + + from typing import TypeNarrower, assert_type + + def is_str(x: object) -> TypeNarrower[str]: + return isinstance(x, str) + + def f(x: str | int) -> None: + if is_str(x): + assert_type(x, str) + else: + assert_type(x, int) + +The final narrowed type may be narrower than **R**, due to the constraints of the +argument's previously-known type:: + + from collections.abc import Awaitable + from typing import Any, TypeNarrower, assert_type + import inspect + + def isawaitable(x: object) -> TypeNarrower[Awaitable[Any]]: + return inspect.isawaitable(x) + + def f(x: Awaitable[int] | int) -> None: + if isawaitable(x): + assert_type(x, Awaitable[int]) + else: + assert_type(x, int) + +It is an error to narrow to a type that is not consistent with the input type:: + + from typing import TypeNarrower + + def is_str(x: int) -> TypeNarrower[str]: # Type checker error + ... + +Subtyping +--------- + +``TypeNarrower`` is not a subtype of ``bool``. +The type ``Callable[..., TypeNarrower[int]]`` is not assignable to +``Callable[..., bool]`` or ``Callable[..., TypeGuard[int]]``, and vice versa. +This restriction is carried over from :pep:`647`. It may be possible to relax +it in the future, but that is outside the scope of this PEP. + +Unlike ``TypeGuard``, ``TypeNarrower`` is invariant in its argument type: +``TypeNarrower[B]`` is not a subtype of ``TypeNarrower[A]``, +even if ``B`` is a subtype of ``A``. +To see why, consider the following example:: + + def takes_narrower(x: int | str, narrower: Callable[[object], TypeNarrower[int]]): + if narrower(x): + print(x + 1) # x is an int + else: + print("Hello " + x) # x is a str + + def is_bool(x: object) -> TypeNarrower[bool]: + return isinstance(x, bool) + + takes_narrower(1, is_bool) # Error: is_bool is not a TypeNarrower[int] + +(Note that ``bool`` is a subtype of ``int``.) +This code fails at runtime, because the narrower returns ``False`` (1 is not a ``bool``) +and the ``else`` branch is taken in ``takes_narrower()``. +If the call ``takes_narrower(1, is_bool)`` was allowed, type checkers would fail to +detect this error. + +Backwards Compatibility +======================= + +As this PEP only proposes a new special form, there are no implications on +backwards compatibility. + + +Security Implications +===================== + +None known. + + +How to Teach This +================= + +Introductions to typing should cover ``TypeNarrower`` when discussing how to narrow types, +along with discussion of other narrowing constructs such as :py:func:`isinstance`. The +documentation should emphasize ``TypeNarrower`` over :py:data:`typing.TypeGuard`; while the +latter is not being deprecated and its behavior is occasionally useful, we expect that the +behavior of ``TypeNarrower`` is usually more intuitive, and most users should reach for +``TypeNarrower`` first. + + +Reference Implementation +======================== + +A draft implementation for mypy `is available `__. + + +Rejected Ideas +============== + +.. _pep-742-change-typeguard: + +Change the behavior of ``TypeGuard`` +------------------------------------ + +:pep:`724` previously proposed changing the specified behavior of :py:data:`typing.TypeGuard` so +that if the return type of the guard is consistent with the input type, the behavior proposed +here for ``TypeNarrower`` would apply. This proposal has some important advantages: because it +does not require any runtime changes, it requires changes only in type checkers, making it easier +for users to take advantage of the new, usually more intuitive behavior. + +However, this approach has some major problems. Users who have written ``TypeGuard`` functions +expecting the existing semantics specified in :pep:`647` would see subtle and potentially breaking +changes in how type checkers interpret their code. The split behavior of ``TypeGuard``, where it +works one way if the return type is consistent with the input type and another way if it is not, +could be confusing for users. The Typing Council was unable to come to an agreement in favor of +:pep:`724`; as a result, we are proposing this alternative PEP. + +Do nothing +---------- + +Both this PEP and the alternative proposed in :pep:`724` have shortcomings. The latter are +discussed above. As for this PEP, it introduces two special forms with very similar semantics, +and it potentially creates a long migration path for users currently using ``TypeGuard`` +who would be better off with different narrowing semantics. + +One way forward, then, is to do nothing and live with the current limitations of the type system. +However, we believe that the limitations of the current ``TypeGuard``, as outlined in the "Motivation" +section, are significant enough that it is worthwhile to change the type system to address them. +If we do not make any change, users will continue to encounter the same unintuitive behaviors from +``TypeGuard``, and the type system will be unable to properly represent common type narrowing functions +like ``inspect.isawaitable``. + +Open Issues +=========== + +Naming +------ + +This PEP currently proposes the name ``TypeNarrower``, emphasizing that the special form narrows +the type of its argument. However, other names have been suggested, and we are open to using a +different name. + +Options include: + +* ``IsInstance`` (`post by Paul Moore `__): + emphasizes that the new construct behaves similarly to the builtin :py:func:`isinstance`. +* ``Narrowed`` or ``NarrowedTo``: shorter than ``TypeNarrower`` but keeps the connection to "type narrowing" + (suggested by Eric Traut). +* ``Predicate`` or ``TypePredicate``: mirrors TypeScript's name for the feature, "type predicates". +* ``StrictTypeGuard`` (earlier drafts of :pep:`724`): emphasizes that the new construct performs a stricter + version of type narrowing than :py:data:`typing.TypeGuard`. +* ``TypeCheck`` (`post by Nicolas Tessore `__): + emphasizes the binary nature of the check. +* ``TypeIs``: emphasizes that the function returns whether the argument is of that type; mirrors + `TypeScript's syntax `__. + +Acknowledgments +=============== + +Much of the motivation and specification for this PEP derives from :pep:`724`. While +this PEP proposes a different solution for the problem at hand, the authors of :pep:`724`, Eric Traut, Rich +Chiodo, and Erik De Bonte, made a strong case for their proposal and this proposal +would not have been possible without their work. + + +Copyright +========= + +This document is placed in the public domain or under the +CC0-1.0-Universal license, whichever is more permissive.