PEP 742: TypeNarrower (#3649)

Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>

.github/CODEOWNERS

@@ -619,6 +619,7 @@ peps/pep-0736.rst  @gvanrossum @Rosuav
 peps/pep-0737.rst  @vstinner
 peps/pep-0738.rst  @encukou
 peps/pep-0740.rst  @dstufft
+peps/pep-0742.rst  @JelleZijlstra
 # ...
 # peps/pep-0754.rst
 # ...

peps/pep-0483.rst

@@ -165,6 +165,7 @@ structural subtyping is considered more flexible.
 We strive to provide support for both approaches, so that
 structural information can be used in addition to nominal subtyping.
 
+.. _pep-483-gradual-typing:
 
 Summary of gradual typing
 =========================

peps/pep-0742.rst

@@ -0,0 +1,336 @@
+PEP: 742
+Title: Narrowing types with TypeNarrower
+Author: Jelle Zijlstra <jelle.zijlstra@gmail.com>
+Status: Draft
+Type: Standards Track
+Topic: Typing
+Created: 07-Feb-2024
+Python-Version: 3.13
+Replaces: 724
+
+
+Abstract
+========
+
+This PEP proposes a new special form, ``TypeNarrower``, to allow annotating functions that can be used
+to narrow the type of a value, similar to the builtin :py:func:`isinstance`. Unlike the existing
+:py:data:`typing.TypeGuard` special form, ``TypeNarrower`` can narrow the type in both the ``if``
+and ``else`` branches of a conditional.
+
+
+Motivation
+==========
+
+Typed Python code often requires users to narrow the type of a variable based on a conditional.
+For example, if a function accepts a union of two types, it may use an :py:func:`isinstance` check
+to discriminate between the two types. Type checkers commonly support type narrowing based on various
+builtin function and operations, but occasionally, it is useful to use a user-defined function to
+perform type narrowing.
+
+To support such use cases, :pep:`647` introduced the :py:data:`typing.TypeGuard` special form, which
+allows users to define type guards::
+
+    from typing import assert_type, TypeGuard
+
+    def is_str(x: object) -> TypeGuard[str]:
+        return isinstance(x, str)
+
+    def f(x: object) -> None:
+        if is_str(x):
+            assert_type(x, str)
+        else:
+            assert_type(x, object)
+
+Unfortunately, the behavior of :py:data:`typing.TypeGuard` has some limitations that make it
+less useful for many common use cases, as explained also in the "Motivation" section of :pep:`724`.
+In particular:
+
+* Type checkers must use exactly the ``TypeGuard`` return type as the narrowed type if the
+  type guard returns ``True``. They cannot use pre-existing knowledge about the type of the
+  variable.
+* In the case where the type guard returns ``False``, the type checker cannot apply any
+  additional narrowing.
+
+The standard library function :py:func:`inspect.isawaitable` may serve as an example. It
+returns whether the argument is an awaitable object, and
+`typeshed <https://github.com/python/typeshed/blob/a4f81a67a07c18dd184dd068c459b02e71bcac22/stdlib/inspect.pyi#L219>`__
+currently annotates it as::
+
+    def isawaitable(object: object) -> TypeGuard[Awaitable[Any]]: ...
+
+A user `reported <https://github.com/python/mypy/issues/15520>`__ an issue to mypy about
+the behavior of this function. They observed the following behavior::
+
+    import inspect
+    from collections.abc import Awaitable
+    from typing import reveal_type
+
+    async def f(t: Awaitable[int] | int) -> None:
+        if inspect.isawaitable(t):
+            reveal_type(t)  # Awaitable[Any]
+        else:
+            reveal_type(t)  # Awaitable[int] | int
+
+This behavior is consistent with :pep:`647`, but it did not match the user's expectations.
+Instead, they would expect the type of ``t`` to be narrowed to ``Awaitable[int]`` in the ``if``
+branch, and to ``int`` in the ``else`` branch. This PEP proposes a new construct that does
+exactly that.
+
+Other examples of issues that arose out of the current behavior of ``TypeGuard`` include:
+
+* `Python typing issue <https://github.com/python/typing/issues/996>`__ (``numpy.isscalar``)
+* `Python typing issue <https://github.com/python/typing/issues/1351>`__ (:py:func:`dataclasses.is_dataclass`)
+* `Pyright issue <https://github.com/microsoft/pyright/issues/3450>`__ (expecting :py:data:`typing.TypeGuard` to work like :py:func:`isinstance`)
+* `Pyright issue <https://github.com/microsoft/pyright/issues/3466>`__ (expecting narrowing in the ``else`` branch)
+* `Mypy issue <https://github.com/python/mypy/issues/13957>`__ (expecting narrowing in the ``else`` branch)
+* `Mypy issue <https://github.com/python/mypy/issues/14434>`__ (combining multiple TypeGuards)
+* `Mypy issue <https://github.com/python/mypy/issues/15305>`__ (expecting narrowing in the ``else`` branch)
+* `Mypy issue <https://github.com/python/mypy/issues/11907>`__ (user-defined function similar to :py:func:`inspect.isawaitable`)
+* `Typeshed issue <https://github.com/python/typeshed/issues/8009>`__ (``asyncio.iscoroutinefunction``)
+
+Rationale
+=========
+
+The problems with the current behavior of :py:data:`typing.TypeGuard` compel us to improve
+the type system to allow a different type narrowing behavior. :pep:`724` proposed to change
+the behavior of the existing :py:data:`typing.TypeGuard` construct, but we :ref:`believe <pep-742-change-typeguard>`
+that the backwards compatibility implications of that change are too severe. Instead, we propose
+adding a new special form with the desired semantics.
+
+We acknowledge that this leads to an unfortunate situation where there are two constructs with
+a similar purpose and similar semantics. We believe that users are more likely to want the behavior
+of ``TypeNarrower``, the new form proposed in this PEP, and therefore we recommend that documentation
+emphasize ``TypeNarrower`` over ``TypeGuard`` as a more commonly applicable tool. However, the semantics of
+``TypeGuard`` are occasionally useful, and we do not propose to deprecate or remove it. In the long
+run, most users should use ``TypeNarrower``, and ``TypeGuard`` should be reserved for rare cases
+where its behavior is specifically desired.
+
+
+Specification
+=============
+
+A new special form, ``TypeNarrower``, is added to the :py:mod:`typing`
+module. Its usage, behavior, and runtime implementation are similar to
+those of :py:data:`typing.TypeGuard`.
+
+It accepts a single
+argument and can be used as the return type of a function. A function annotated as returning a
+``TypeNarrower`` is called a type narrowing function. Type narrowing functions must return ``bool``
+values, and the type checker should verify that all return paths return
+``bool``.
+
+Type narrowing functions must accept at least one positional argument. The type
+narrowing behavior is applied to the first positional argument passed to
+the function. The function may accept additional arguments, but they are
+not affected by type narrowing. If a type narrowing function is implemented as
+an instance method or class method, the first positional argument maps
+to the second parameter (after ``self`` or ``cls``).
+
+Type narrowing behavior
+-----------------------
+
+To specify the behavior of ``TypeNarrower``, we use the following terminology:
+
+* I = ``TypeNarrower`` input type
+* R = ``TypeNarrower`` return type
+* A = Type of argument passed to type narrowing function (pre-narrowed)
+* NP = Narrowed type (positive; used when ``TypeNarrower`` returned ``True``)
+* NN = Narrowed type (negative; used when ``TypeNarrower`` returned ``False``)
+
+.. code-block:: python
+
+    def narrower(x: I) -> TypeNarrower[R]: ...
+
+    def func1(val: A):
+        if narrower(val):
+            assert_type(val, NP)
+        else:
+            assert_type(val, NN)
+
+The return type ``R`` must be :ref:`consistent with <pep-483-gradual-typing>` ``I``. The type checker should
+emit an error if this condition is not met.
+
+Formally, type *NP* should be narrowed to :math:`A \land R`,
+the intersection of *A* and *R*, and type *NN* should be narrowed to
+:math:`A \land \neg R`, the intersection of *A* and the complement of *R*.
+In practice, the theoretic types for strict type guards cannot be expressed
+precisely in the Python type system. Type checkers should fall back on
+practical approximations of these types. As a rule of thumb, a type checker
+should use the same type narrowing logic -- and get results that are consistent
+with -- its handling of :py:func:`isinstance`. This guidance allows for changes and
+improvements if the type system is extended in the future.
+
+Examples
+--------
+
+Type narrowing is applied in both the positive and negative case::
+
+    from typing import TypeNarrower, assert_type
+
+    def is_str(x: object) -> TypeNarrower[str]:
+        return isinstance(x, str)
+
+    def f(x: str | int) -> None:
+        if is_str(x):
+            assert_type(x, str)
+        else:
+            assert_type(x, int)
+
+The final narrowed type may be narrower than **R**, due to the constraints of the
+argument's previously-known type::
+
+    from collections.abc import Awaitable
+    from typing import Any, TypeNarrower, assert_type
+    import inspect
+
+    def isawaitable(x: object) -> TypeNarrower[Awaitable[Any]]:
+        return inspect.isawaitable(x)
+
+    def f(x: Awaitable[int] | int) -> None:
+        if isawaitable(x):
+            assert_type(x, Awaitable[int])
+        else:
+            assert_type(x, int)
+
+It is an error to narrow to a type that is not consistent with the input type::
+
+    from typing import TypeNarrower
+
+    def is_str(x: int) -> TypeNarrower[str]:  # Type checker error
+        ...
+
+Subtyping
+---------
+
+``TypeNarrower`` is not a subtype of ``bool``.
+The type ``Callable[..., TypeNarrower[int]]`` is not assignable to
+``Callable[..., bool]`` or ``Callable[..., TypeGuard[int]]``, and vice versa.
+This restriction is carried over from :pep:`647`. It may be possible to relax
+it in the future, but that is outside the scope of this PEP.
+
+Unlike ``TypeGuard``, ``TypeNarrower`` is invariant in its argument type:
+``TypeNarrower[B]`` is not a subtype of ``TypeNarrower[A]``,
+even if ``B`` is a subtype of ``A``.
+To see why, consider the following example::
+
+    def takes_narrower(x: int | str, narrower: Callable[[object], TypeNarrower[int]]):
+        if narrower(x):
+            print(x + 1)  # x is an int
+        else:
+            print("Hello " + x)  # x is a str
+
+    def is_bool(x: object) -> TypeNarrower[bool]:
+        return isinstance(x, bool)
+
+    takes_narrower(1, is_bool)  # Error: is_bool is not a TypeNarrower[int]
+
+(Note that ``bool`` is a subtype of ``int``.)
+This code fails at runtime, because the narrower returns ``False`` (1 is not a ``bool``)
+and the ``else`` branch is taken in ``takes_narrower()``.
+If the call ``takes_narrower(1, is_bool)`` was allowed, type checkers would fail to
+detect this error.
+
+Backwards Compatibility
+=======================
+
+As this PEP only proposes a new special form, there are no implications on
+backwards compatibility.
+
+
+Security Implications
+=====================
+
+None known.
+
+
+How to Teach This
+=================
+
+Introductions to typing should cover ``TypeNarrower`` when discussing how to narrow types,
+along with discussion of other narrowing constructs such as :py:func:`isinstance`. The
+documentation should emphasize ``TypeNarrower`` over :py:data:`typing.TypeGuard`; while the
+latter is not being deprecated and its behavior is occasionally useful, we expect that the
+behavior of ``TypeNarrower`` is usually more intuitive, and most users should reach for
+``TypeNarrower`` first.
+
+
+Reference Implementation
+========================
+
+A draft implementation for mypy `is available <https://github.com/python/mypy/pull/16898>`__.
+
+
+Rejected Ideas
+==============
+
+.. _pep-742-change-typeguard:
+
+Change the behavior of ``TypeGuard``
+------------------------------------
+
+:pep:`724` previously proposed changing the specified behavior of :py:data:`typing.TypeGuard` so
+that if the return type of the guard is consistent with the input type, the behavior proposed
+here for ``TypeNarrower`` would apply. This proposal has some important advantages: because it
+does not require any runtime changes, it requires changes only in type checkers, making it easier
+for users to take advantage of the new, usually more intuitive behavior.
+
+However, this approach has some major problems. Users who have written ``TypeGuard`` functions
+expecting the existing semantics specified in :pep:`647` would see subtle and potentially breaking
+changes in how type checkers interpret their code. The split behavior of ``TypeGuard``, where it
+works one way if the return type is consistent with the input type and another way if it is not,
+could be confusing for users. The Typing Council was unable to come to an agreement in favor of
+:pep:`724`; as a result, we are proposing this alternative PEP.
+
+Do nothing
+----------
+
+Both this PEP and the alternative proposed in :pep:`724` have shortcomings. The latter are
+discussed above. As for this PEP, it introduces two special forms with very similar semantics,
+and it potentially creates a long migration path for users currently using ``TypeGuard``
+who would be better off with different narrowing semantics.
+
+One way forward, then, is to do nothing and live with the current limitations of the type system.
+However, we believe that the limitations of the current ``TypeGuard``, as outlined in the "Motivation"
+section, are significant enough that it is worthwhile to change the type system to address them.
+If we do not make any change, users will continue to encounter the same unintuitive behaviors from
+``TypeGuard``, and the type system will be unable to properly represent common type narrowing functions
+like ``inspect.isawaitable``.
+
+Open Issues
+===========
+
+Naming
+------
+
+This PEP currently proposes the name ``TypeNarrower``, emphasizing that the special form narrows
+the type of its argument. However, other names have been suggested, and we are open to using a
+different name.
+
+Options include:
+
+* ``IsInstance`` (`post by Paul Moore <https://discuss.python.org/t/pep-724-stricter-type-guards/34124/60>`__):
+  emphasizes that the new construct behaves similarly to the builtin :py:func:`isinstance`.
+* ``Narrowed`` or ``NarrowedTo``: shorter than ``TypeNarrower`` but keeps the connection to "type narrowing"
+  (suggested by Eric Traut).
+* ``Predicate`` or ``TypePredicate``: mirrors TypeScript's name for the feature, "type predicates".
+* ``StrictTypeGuard`` (earlier drafts of :pep:`724`): emphasizes that the new construct performs a stricter
+  version of type narrowing than :py:data:`typing.TypeGuard`.
+* ``TypeCheck`` (`post by Nicolas Tessore <https://discuss.python.org/t/pep-724-stricter-type-guards/34124/59>`__):
+  emphasizes the binary nature of the check.
+* ``TypeIs``: emphasizes that the function returns whether the argument is of that type; mirrors
+  `TypeScript's syntax <https://www.typescriptlang.org/docs/handbook/2/narrowing.html#using-type-predicates>`__.
+
+Acknowledgments
+===============
+
+Much of the motivation and specification for this PEP derives from :pep:`724`. While
+this PEP proposes a different solution for the problem at hand, the authors of :pep:`724`, Eric Traut, Rich
+Chiodo, and Erik De Bonte, made a strong case for their proposal and this proposal
+would not have been possible without their work.
+
+
+Copyright
+=========
+
+This document is placed in the public domain or under the
+CC0-1.0-Universal license, whichever is more permissive.