From 3e200c644bb82d34f3368ba5dd952d51bbe82d90 Mon Sep 17 00:00:00 2001 From: David Foster Date: Sun, 16 Jun 2024 15:42:44 -0700 Subject: [PATCH] PEP 747: TypeExpr: Type Hint for a Type Expression (#3798) Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com> Co-authored-by: Alex Waygood Co-authored-by: Jelle Zijlstra --- .github/CODEOWNERS | 2 + peps/pep-0655.rst | 2 +- peps/pep-0747.rst | 977 +++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 980 insertions(+), 1 deletion(-) create mode 100644 peps/pep-0747.rst diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index f78adabb0..67f038535 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -627,6 +627,8 @@ peps/pep-0745.rst @hugovk peps/pep-0746.rst @JelleZijlstra peps/pep-0749.rst @JelleZijlstra # ... +peps/pep-0747.rst @JelleZijlstra +# ... # peps/pep-0754.rst # ... peps/pep-0789.rst @njsmith diff --git a/peps/pep-0655.rst b/peps/pep-0655.rst index 263cdb096..587e1fad9 100644 --- a/peps/pep-0655.rst +++ b/peps/pep-0655.rst @@ -461,7 +461,7 @@ Such operators could be implemented on ``type`` via the ``__pos__``, grammar. It was decided that it would be prudent to introduce long-form notation -(i.e. ``Required[]`` and ``NotRequired[]``) before introducing +(i.e. ``Required[]`` and ``NotRequired[]``) before introducing any short-form notation. Future PEPs may reconsider introducing this or other short-form notation options. diff --git a/peps/pep-0747.rst b/peps/pep-0747.rst new file mode 100644 index 000000000..3c8b7da32 --- /dev/null +++ b/peps/pep-0747.rst @@ -0,0 +1,977 @@ +PEP: 747 +Title: TypeExpr: Type Hint for a Type Expression +Author: David Foster +Sponsor: Jelle Zijlstra +Status: Draft +Type: Standards Track +Topic: Typing +Created: 27-May-2024 +Python-Version: 3.14 +Post-History: `19-Apr-2024 `__, `04-May-2024 `__ + + +Abstract +======== + +:pep:`484` defines the notation ``type[C]`` where ``C`` is a class, to +refer to a class object that is a subtype of ``C``. It explicitly does +not allow ``type[C]`` to refer to arbitrary +:ref:`type expression ` objects such +as the runtime object ``str | None``, even if ``C`` is an unbounded +``TypeVar``. [#type_c]_ In cases where that restriction is unwanted, this +PEP proposes a new notation ``TypeExpr[T]`` where ``T`` is a type, to +refer to a either a class object or some other type expression object +that is a subtype of ``T``, allowing any kind of type to be referenced. + +This PEP makes no Python grammar changes. Correct usage of +``TypeExpr[]`` is intended to be enforced only by static and runtime +type checkers and need not be enforced by Python itself at runtime. + + +.. _motivation: + +Motivation +========== + +The introduction of ``TypeExpr`` allows new kinds of metaprogramming +functions that operate on type expressions to be type-annotated and +understood by type checkers. + +For example, here is a function that checks whether a value is +assignable to a variable of a particular type, and if so returns the +original value: + +:: + + def trycast[T](typx: TypeExpr[T], value: object) -> T | None: ... + +The use of ``TypeExpr[]`` and the type variable ``T`` enables the return +type of this function to be influenced by a ``typx`` value passed at +runtime, which is quite powerful. + +Here is another function that checks whether a value is assignable to a +variable of a particular type, and if so returns ``True`` (as a special +``TypeIs[]`` bool [#TypeIsPep]_): + +:: + + def isassignable[T](value: object, typx: TypeExpr[T]) -> TypeIs[T]: ... + +The use of ``TypeExpr[]`` and ``TypeIs[]`` together enables type +checkers to narrow the return type appropriately depending on what type +expression is passed in: + +:: + + request_json: object = ... + if isassignable(request_json, MyTypedDict): + assert_type(request_json, MyTypedDict) # type is narrowed! + +That ``isassignable`` function enables a kind of enhanced ``isinstance`` +check which is useful for `checking whether a value decoded from JSON +conforms to a particular structure`_ of nested ``TypedDict``\ s, +lists, unions, ``Literal``\ s, and other types. This kind +of check was alluded to in :pep:`PEP 589 <589#using-typeddict-types>` but could +not be implemented at the time without a notation similar to +``TypeExpr[]``. + +.. _checking whether a value decoded from JSON conforms to a particular structure: https://mail.python.org/archives/list/typing-sig@python.org/thread/I5ZOQICTJCENTCDPHLZR7NT42QJ43GP4/ + + +Why can’t ``type[]`` be used? +----------------------------- + +One might think you could define the example functions above to take a +``type[C]`` - which is syntax that already exists - rather than a +``TypeExpr[T]``. However if you were to do that then certain type +expressions like ``str | None`` - which are not class objects and +therefore not ``type``\ s at runtime - would be rejected: + +:: + + # NOTE: Uses a type[C] parameter rather than a TypeExpr[T] + def trycast_type[C](typ: type[C], value: object) -> T | None: ... + + trycast_type(str, 'hi') # ok; str is a type + trycast_type(Optional[str], 'hi') # ERROR; Optional[str] is not a type + trycast_type(str | int, 'hi') # ERROR; (str | int) is not a type + trycast_type(MyTypedDict, dict(value='hi')) # questionable; accepted by mypy 1.9.0 + +To solve that problem, ``type[]`` could be widened to include the +additional values allowed by ``TypeExpr``. However doing so would lose +``type[]``\ ’s current ability to spell a class object which always +supports instantiation and ``isinstance`` checks, unlike arbitrary type +expression objects. Therefore ``TypeExpr`` is proposed as new notation +instead. + +For a longer explanation of why we don’t just widen ``type[T]`` to +accept all type expressions, see +:ref:`widen_type_C_to_support_all_type_expressions`. + + +Common kinds of functions that would benefit from TypeExpr +---------------------------------------------------------- + +`A survey of various Python libraries`_ revealed a few kinds of commonly +defined functions which would benefit from ``TypeExpr[]``: + +.. _A survey of various Python libraries: https://github.com/python/mypy/issues/9773#issuecomment-2017998886 + +- Assignability checkers: + + - Returns whether a value is assignable to a type expression. If so + then also narrows the type of the value to match the type + expression. + - Pattern 1: + ``def isassignable[T](value: object, typx: TypeExpr[T]) -> TypeIs[T]`` + - Pattern 2: + ``def ismatch[T](value: object, typx: TypeExpr[T]) -> TypeGuard[T]`` + - Examples: beartype.\ `is_bearable`_, trycast.\ `isassignable`_, + typeguard.\ `check_type`_, xdsl.\ `isa`_ + +.. _is_bearable: https://github.com/beartype/beartype/issues/255 +.. _isassignable: https://github.com/davidfstr/trycast?tab=readme-ov-file#isassignable-api +.. _check_type: https://typeguard.readthedocs.io/en/latest/api.html#typeguard.check_type +.. _isa: https://github.com/xdslproject/xdsl/blob/ac12c9ab0d64618475efb98d1d197bdd79f593c3/xdsl/utils/hints.py#L23 + +- Converters: + + - If a value is assignable to (or coercible to) a type expression, + a *converter* returns the value narrowed to (or coerced to) that type + expression. Otherwise, it raises an exception. + - Pattern 1: + ``def convert[T](value: object, typx: TypeExpr[T]) -> T`` + + - Examples: cattrs.BaseConverter.\ `structure`_, trycast.\ `checkcast`_, + typedload.\ `load`_ + + - Pattern 2: + + :: + + class Converter[T]: + def __init__(self, typx: TypeExpr[T]) -> None: ... + def convert(self, value: object) -> T: ... + + - Examples: pydantic.\ `TypeAdapter(T).validate_python`_, + mashumaro.\ `JSONDecoder(T).decode`_ + +.. _structure: https://github.com/python-attrs/cattrs/blob/5f5c11627a7f67a23d6212bc7df9f96243c62dc5/src/cattrs/converters.py#L332-L334 +.. _checkcast: https://github.com/davidfstr/trycast#checkcast-api +.. _load: https://ltworf.github.io/typedload/ +.. _TypeAdapter(T).validate_python: https://stackoverflow.com/a/61021183/604063 +.. _JSONDecoder(T).decode: https://github.com/Fatal1ty/mashumaro?tab=readme-ov-file#usage-example + +- Typed field definitions: + + - Pattern: + + :: + + class Field[T]: + value_type: TypeExpr[T] + + - Examples: attrs.\ `make_class`_, + dataclasses.\ `make_dataclass`_ [#DataclassInitVar]_, `openapify`_ + +.. _make_class: https://www.attrs.org/en/stable/api.html#attrs.make_class +.. _make_dataclass: https://github.com/python/typeshed/issues/11653 +.. _openapify: https://github.com/Fatal1ty/openapify/blob/c8d968c7c9c8fd7d4888bd2ddbe18ffd1469f3ca/openapify/core/models.py#L16 + +The survey also identified some introspection functions that take +annotation expressions as input using plain ``object``\ s which would +*not* gain functionality by marking those inputs as ``TypeExpr[]``: + +- General introspection operations: + + - Pattern: ``def get_annotation_info(maybe_annx: object) -> object`` + - Examples: typing.{`get_origin`_, `get_args`_}, + `typing_inspect`_.{is_*_type, get_origin, get_parameters} + +.. _get_origin: https://docs.python.org/3/library/typing.html#typing.get_origin +.. _get_args: https://docs.python.org/3/library/typing.html#typing.get_args +.. _typing_inspect: https://github.com/ilevkivskyi/typing_inspect?tab=readme-ov-file#readme + + +Rationale +========= + +Before this PEP existed there were already a few definitions in use to describe +different kinds of type annotations: + +:: + + +----------------------------------+ + | +------------------------------+ | + | | +-------------------------+ | | + | | | +---------------------+ | | | + | | | | Class object | | | | = type[C] + | | | +---------------------+ | | | + | | | Type expression object | | | = TypeExpr[T] <-- new! + | | +-------------------------+ | | + | | Annotation expression object | | + | +------------------------------+ | + | Object | = object + +----------------------------------+ + +- :ref:`Class objects `, + spelled as ``type[C]``, support ``isinstance`` checks and are callable. + + - Examples: ``int``, ``str``, ``MyClass`` + +- :ref:`Type expressions ` + include any type annotation which describes a type. + + - Examples: ``list[int]``, ``MyTypedDict``, ``int | str``, + ``Literal['square']``, any class object + +- :ref:`Annotation expressions ` + include any type annotation, including those only valid in specific contexts. + + - Examples: ``Final[int]``, ``Required[str]``, ``ClassVar[str]``, + any type expression + +``TypeExpr`` aligns with an existing definition from the above list - +*type expression* - to avoid introducing yet another subset of type annotations +that users of Python typing need to think about. + +``TypeExpr`` aligns with *type expression* specifically +because a type expression is already used to parameterize type variables, +which are used in combination with ``TypeIs`` and ``TypeGuard`` to enable +the compelling examples mentioned in :ref:`Motivation `. + + +Specification +============= + +A ``TypeExpr`` value represents a :ref:`type expression ` +such as ``str | None``, ``dict[str, int]``, or ``MyTypedDict``. +A ``TypeExpr`` type is written as +``TypeExpr[T]`` where ``T`` is a type or a type variable. It can also be +written without brackets as just ``TypeExpr``, in which case a type +checker should apply its usual type inference mechanisms to determine +the type of its argument, possibly ``Any``. + + +Using TypeExprs +--------------- + +A ``TypeExpr`` is a new kind of type expression, usable in any context where a +type expression is valid, as a function parameter type, a return type, +or a variable type: + +:: + + def is_union_type(typx: TypeExpr) -> bool: ... # parameter type + +:: + + def union_of[S, T](s: TypeExpr[S], t: TypeExpr[T]) \ + -> TypeExpr[S | T]: ... # return type + +:: + + STR_TYPE: TypeExpr = str # variable type + assert_type(STR_TYPE, TypeExpr[str]) + +Note however that an *unannotated* variable assigned a type expression literal +will not be inferred to be of ``TypeExpr`` type by type checkers because PEP +484 :pep:`reserves that syntax for defining type aliases <484#type-aliases>`: + +- No: + + :: + + STR_TYPE = str # OOPS; treated as a type alias! + +If you want a type checker to recognize a type expression literal in a bare +assignment you’ll need to explicitly declare the assignment-target as +having ``TypeExpr`` type: + +- Yes: + + :: + + STR_TYPE: TypeExpr = str + +- Yes: + + :: + + STR_TYPE: TypeExpr + STR_TYPE = str + +- Okay, but discouraged: + + :: + + STR_TYPE = str # type: TypeExpr # the type comment is significant + +``TypeExpr`` values can be passed around and assigned just like normal +values: + +:: + + def swap1[S, T](t1: TypeExpr[S], t2: TypeExpr[T]) -> tuple[TypeExpr[T], TypeExpr[S]]: + t1_new: TypeExpr[T] = t2 # assigns a TypeExpr value to a new annotated variable + t2_new: TypeExpr[S] = t1 + return (t1_new, t2_new) + + def swap2[S, T](t1: TypeExpr[S], t2: TypeExpr[T]) -> tuple[TypeExpr[T], TypeExpr[S]]: + t1_new = t2 # assigns a TypeExpr value to a new unannotated variable + t2_new = t1 + assert_type(t1_new, TypeExpr[T]) + assert_type(t2_new, TypeExpr[S]) + return (t1_new, t2_new) + + # NOTE: A more straightforward implementation would use isinstance() + def ensure_int(value: object) -> None: + value_type: TypeExpr = type(value) # assigns a type (a subtype of TypeExpr) + assert value_type == int + + +TypeExpr Values +--------------- + +A variable of type ``TypeExpr[T]`` where ``T`` is a type, can hold any +**type expression object** - the result of evaluating a +:ref:`type expression ` +at runtime - which is a subtype of ``T``. + +Incomplete expressions like a bare ``Optional`` or ``Union`` which do +not spell a type are not ``TypeExpr`` values. + +``TypeExpr[...]`` is itself a ``TypeExpr`` value: + +:: + + OPTIONAL_INT_TYPE: TypeExpr = TypeExpr[int | None] # OK + assert isassignable(Optional[int], OPTIONAL_INT_TYPE) + +.. _non_universal_typeexpr: + +``TypeExpr[]`` values include *all* type expressions including some +**non-universal type expressions** which are not valid in all annotation contexts. +In particular: + +- ``Self`` (valid only in some contexts) +- ``TypeGuard[...]`` (valid only in some contexts) +- ``TypeIs[...]`` (valid only in some contexts) + + +Explicit TypeExpr Values +'''''''''''''''''''''''' + +The syntax ``TypeExpr(T)`` (with parentheses) can be used to +spell a ``TypeExpr[T]`` value explicitly: + +:: + + NONE = TypeExpr(None) + INT1 = TypeExpr('int') # stringified type expression + INT2 = TypeExpr(int) + +At runtime the ``TypeExpr(...)`` callable returns its single argument unchanged. + + +.. _implicit_typeexpr_values: + +Implicit TypeExpr Values +'''''''''''''''''''''''' + +Historically static type checkers have only needed to recognize +*type expressions* in contexts where a type expression was expected. +Now *type expression objects* must also be recognized in contexts where a +value expression is expected. + +Static type checkers already recognize **class objects** (``type[C]``): + +- As a value expression, ``C`` has type ``type[C]``, + for each of the following values of C: + + - ``name`` (where ``name`` must refer to a valid in-scope class, type alias, or TypeVar) + - ``name '[' ... ']'`` + - `` '[' ... ']'`` + +The following **unparameterized type expressions** can be recognized unambiguously: + +- As a value expression, ``X`` has type ``TypeForm[X]``, + for each of the following values of X: + + - ```` + - ```` + - ```` + - ```` + - ```` + +**None**: The type expression ``None`` (``NoneType``) is ambiguous with the value ``None``, +so must use the explicit ``TypeForm(...)`` syntax: + +- As a value expression, ``TypeForm(None)`` has type ``TypeForm[None]``. +- As a value expression, ``None`` continues to have type ``None``. + +The following **parameterized type expressions** can be recognized unambiguously: + +- As a value expression, ``X`` has type ``TypeForm[X]``, + for each of the following values of X: + + - `` '[' ... ']'`` + - `` '[' ... ']'`` + - `` '[' ... ']'`` + - `` '[' ... ']'`` + - `` '[' ... ']'`` + - `` '[' ... ']'`` + - `` '[' ... ']'`` + +.. _recognizing_annotated: + +**Annotated**: The type expression ``Annotated[...]`` is ambiguous with +the annotation expression ``Annotated[...]``, +so must be disambiguated based on its argument type: + +- As a value expression, ``Annotated[x, ...]`` has type ``type[C]`` + if ``x`` has type ``type[C]``. +- As a value expression, ``Annotated[x, ...]`` has type ``TypeExpr[T]`` + if ``x`` has type ``TypeExpr[T]``. +- As a value expression, ``Annotated[x, ...]`` has type ``object`` + if ``x`` has a type that is not ``type[C]`` or ``TypeExpr[T]``. + +**Union**: The type expression ``T1 | T2`` is ambiguous with the value ``int1 | int2``, +so must be disambiguated based on its argument type: + +- As a value expression, ``x | y`` has type ``TypeForm[x | y]`` + if ``x`` has type ``TypeForm[t1]`` (or ``type[t1]``) + and ``y`` has type ``TypeForm[t2]`` (or ``type[t2]``). +- As a value expression, ``x | y`` has type ``int`` + if ``x`` has type ``int`` and ``y`` has type ``int`` + +The **stringified type expression** ``"T"`` is ambiguous with both +the stringified annotation expression ``"T"`` +and the string literal ``"T"``, +so must use the explicit ``TypeForm(...)`` syntax: + +- As a value expression, ``TypeForm("T")`` has type ``TypeForm[T]``, + where ``T`` is a valid type expression +- As a value expression, ``"T"`` continues to have type ``Literal["T"]``. + +No other kinds of type expressions currently exist. + +New kinds of type expressions that are introduced should define how they +will be recognized in a value expression context. + + +Implicit Annotation Expression Values +''''''''''''''''''''''''''''''''''''' + +Extending the rules for :ref:`recognizing type expressions ` +in a value expression context, the following rules for recognizing +annotation expressions are defined: + +The following **unparameterized annotation expressions** can be recognized unambiguously: + +- As a value expression, ``X`` has type ``object``, + for each of the following values of X: + + - ```` + +The following **parameterized annotation expressions** can be recognized unambiguously: + +- As a value expression, ``X`` has type ``object``, + for each of the following values of X: + + - `` '[' ... ']'`` + - `` '[' ... ']'`` + - `` '[' ... ']'`` + - `` '[' ... ']'`` + - `` '[' ... ']'`` + - `` '[' ... ']'`` + - `` '[' ... ']'`` + +**Annotated**: The annotation expression ``Annotated[...]`` is ambiguous with +the type expression ``Annotated[...]``, +so must be :ref:`disambiguated based on its argument type `. + +The following **syntactic annotation expressions** +cannot be recognized in a value expression context at all: + +- ``'*' unpackable`` +- ``name '.' 'args'`` (where ``name`` must be an in-scope ParamSpec) +- ``name '.' 'kwargs'`` (where ``name`` must be an in-scope ParamSpec) + +The **stringified annotation expression** ``"T"`` is ambiguous with both +the stringified type expression ``"T"`` +and the string literal ``"T"``, and +cannot be recognized in a value expression context at all: + +- As a value expression, ``"T"`` continues to have type ``Literal["T"]``. + +No other kinds of annotation expressions currently exist. + +New kinds of annotation expressions that are introduced should define how they +will (or will not) be recognized in a value expression context. + + +Literal[] TypeExprs +''''''''''''''''''' + +To simplify static type checking, a ``Literal[...]`` value is *not* +considered assignable to a ``TypeExpr`` variable even if all of its members +spell valid types: + +:: + + STRS_TYPE_NAME: Literal['str', 'list[str]'] = 'str' + STRS_TYPE: TypeExpr = STRS_TYPE_NAME # ERROR: Literal[] value is not a TypeExpr + + +Static vs. Runtime Representations of TypeExprs +''''''''''''''''''''''''''''''''''''''''''''''' + +A ``TypeExpr`` value appearing statically in a source file may be normalized +to a different representation at runtime. For example string-based +forward references are normalized at runtime to be ``ForwardRef`` instances +in some contexts: [#forward_ref_normalization]_ + +:: + + >>> IntTree = list[typing.Union[int, 'IntTree']] + >>> IntTree + list[typing.Union[int, ForwardRef('IntTree')]] + +The runtime representations of ``TypeExpr``\ s are considered implementation +details that may change over time and therefore static type checkers are +not required to recognize them: + +:: + + INT_TREE: TypeExpr = ForwardRef('IntTree') # ERROR: Runtime-only form + +Runtime type checkers that wish to assign a runtime-only representation +of a type expression to a ``TypeExpr[]`` variable must use ``cast()`` to +avoid errors from static type checkers: + +:: + + INT_TREE = cast(TypeExpr, ForwardRef('IntTree')) # OK + + +Subtyping +--------- + +Whether a ``TypeExpr`` value can be assigned from one variable to another is +determined by the following rules: + +``TypeExpr[]`` is covariant in its argument type, just like ``type[]``: + +- ``TypeExpr[T1]`` is a subtype of ``TypeExpr[T2]`` iff ``T1`` is a + subtype of ``T2``. +- ``type[C1]`` is a subtype of ``TypeExpr[C2]`` iff ``C1`` is a subtype + of ``C2``. + +A plain ``type`` can be assigned to a plain ``TypeExpr`` but not the +other way around: + +- ``type[Any]`` is assignable to ``TypeExpr[Any]``. (But not the + other way around.) + +``TypeExpr[]`` is a kind of ``object``, just like ``type[]``: + +- ``TypeExpr[T]`` for any ``T`` is a subtype of ``object``. + +``TypeExpr[T]``, where ``T`` is a type variable, is assumed to have all +the attributes and methods of ``object`` and is not callable. + + +Interactions with isinstance() and issubclass() +----------------------------------------------- + +The ``TypeExpr`` special form cannot be used as the ``type`` argument to +``isinstance``: + +:: + + >>> isinstance(str, TypeExpr) + TypeError: typing.TypeExpr cannot be used with isinstance() + + >>> isinstance(str, TypeExpr[str]) + TypeError: isinstance() argument 2 cannot be a parameterized generic + +The ``TypeExpr`` special form cannot be used as any argument to +``issubclass``: + +:: + + >>> issubclass(TypeExpr, object) + TypeError: issubclass() arg 1 must be a class + + >>> issubclass(object, TypeExpr) + TypeError: typing.TypeExpr cannot be used with issubclass() + + +Affected signatures in the standard library +------------------------------------------- + +Changed signatures +'''''''''''''''''' + +The following signatures related to type expressions introduce +``TypeExpr`` where previously ``object`` existed: + +- ``typing.cast`` +- ``typing.assert_type`` + + +Unchanged signatures +'''''''''''''''''''' + +The following signatures related to annotation expressions continue to +use ``object`` and remain unchanged: + +- ``typing.get_origin`` +- ``typing.get_args`` + +The following signatures related to class objects continue to use +``type`` and remain unchanged: + +- ``builtins.isinstance`` +- ``builtins.issubclass`` +- ``builtins.type`` + +``typing.get_type_hints(..., include_extras=False)`` nearly returns only type +expressions in Python 3.12, stripping out most type qualifiers +(``Required, NotRequired, ReadOnly, Annotated``) but currently preserves a +few type qualifiers which are only allowed in annotation expressions +(``ClassVar, Final, InitVar, Unpack``). It may be desirable to alter the +behavior of this function in the future to also strip out those +qualifiers and actually return type expressions, although this PEP does +not propose those changes now: + +- ``typing.get_type_hints(..., include_extras=False)`` + + - Almost returns only type expressions, but not quite + +- ``typing.get_type_hints(..., include_extras=True)`` + + - Returns annotation expressions + + +Backwards Compatibility +======================= + +Previously the rules for recognizing type expression objects +in a value expression context were not defined, so static type checkers +`varied in what types were assigned `_ +to such objects. Existing programs manipulating type expression objects +were already limited in manipulating them as plain ``object`` values, +and such programs should not break with +:ref:`the newly-defined rules `. + + +How to Teach This +================= + +Most users interacting with ``TypeExpr`` will do so only +in a limited way, by passing a literal type expression to a function +accepting a ``TypeExpr`` input, imported from a runtime type checker +library. + +For those implementing runtime type checkers, +``TypeExpr``\ s can be used in combination with other typing features to +write useful functions that accept type expression objects: + + +Combining with a type variable +------------------------------ + +``TypeExpr[]`` can be parameterized by a type variable that is used elsewhere within +the same function definition: + +:: + + def as_instance[T](typx: TypeExpr[T]) -> T | None: + return typx() if isinstance(typx, type) else None + + +Combining with type[] +--------------------- + +Both ``TypeExpr[]`` and ``type[]`` can be parameterized by the same type +variable within the same function definition: + +:: + + def as_type[T](typx: TypeExpr[T]) -> type[T] | None: + return typx if isinstance(typx, type) else None + + +Combining with TypeIs[] and TypeGuard[] +--------------------------------------- + +A type variable parameterizing a ``TypeExpr[]`` can also be used by a ``TypeIs[]`` +within the same function definition: + +:: + + def isassignable[T](value: object, typx: TypeExpr[T]) -> TypeIs[T]: ... + + count: int | str = ... + if isassignable(count, int): + assert_type(count, int) + else: + assert_type(count, str) + +or by a ``TypeGuard[]`` within the same function definition: + +:: + + def isdefault[T](value: object, typx: TypeExpr[T]) -> TypeGuard[T]: + return (value == typx()) if isinstance(typx, type) else False + + value: int | str = '' + if isdefault(value, int): + assert_type(value, int) + assert 0 == value + elif isdefault(value, str): + assert_type(value, str) + assert '' == value + else: + assert_type(value, int | str) + + +Introspecting TypeExpr Values +----------------------------- + +A ``TypeExpr`` is very similar to an ``object`` at runtime, with no additional +attributes or methods defined. + +You can use existing introspection functions like ``typing.get_origin`` and +``typing.get_args`` to extract the components of a type expression that looks +like ``Origin[Arg1, Arg2, ..., ArgN]``: + +:: + + import typing + + def strip_annotated_metadata(typx: TypeExpr[T]) -> TypeExpr[T]: + if typing.get_origin(typx) == typing.Annotated: + typx = cast(TypeExpr[T], typing.get_args(typx)[0]) + return typx + +You can also use ``isinstance`` and ``==`` to distinguish one kind of +type expression from another: + +:: + + import types + import typing + + def split_union(typx: TypeExpr) -> tuple[TypeExpr, ...]: + if isinstance(typx, types.UnionType): # X | Y + return cast(tuple[TypeExpr, ...], typing.get_args(typx)) + if typing.get_origin(typx) == typing.Union: # Union[X, Y] + return cast(tuple[TypeExpr, ...], typing.get_args(typx)) + if typx in (typing.Never, typing.NoReturn,): + return () + return (typx,) + + +Challenges When Accepting All TypeExprs +--------------------------------------- + +A function that takes an *arbitrary* ``TypeExpr`` as +input must support a large variety of possible type expressions and is +not easy to write. Some challenges faced by such a function include: + +- An ever-increasing number of typing special forms are introduced with + each new Python version which must be recognized, with special + handling required for each one. +- Stringified type annotations [#strann_less_common]_ (like ``'list[str]'``) + must be *parsed* (to something like ``typing.List[str]``) to be introspected. + + - In practice it is extremely difficult for stringified type + annotations to be handled reliably at runtime, so runtime type + checkers may opt to not support them at all. + +- Resolving string-based forward references inside type + expressions to actual values must typically be done using ``eval()``, + which is difficult/impossible to use in a safe way. +- Recursive types like ``IntTree = list[typing.Union[int, 'IntTree']]`` + are not possible to fully resolve. +- Supporting user-defined generic types (like Django’s + ``QuerySet[User]``) requires user-defined functions to + recognize/parse, which a runtime type checker must provide a + registration API for. + +Consider importing a function from an existing runtime type checker +library rather than writing your own. + + +Reference Implementation +======================== + +The following will be true when +`mypy#9773 `__ is implemented: + + The mypy type checker supports ``TypeExpr`` types. + A reference implementation of the runtime component is provided in the + ``typing_extensions`` module. + + +Rejected Ideas +============== + +.. _widen_type_C_to_support_all_type_expressions: + +Widen type[C] to support all type expressions +--------------------------------------------- + +``type`` was `designed`_ to only be used to describe class objects. A +class object can always be used as the second argument of ``isinstance()`` +and can usually be instantiated by calling it. + +``TypeExpr`` on the other hand is typically introspected by the user in +some way, is not necessarily directly instantiable, and is not +necessarily directly usable in a regular ``isinstance()`` check. + +It would be possible to widen ``type`` to include the additional values +allowed by ``TypeExpr`` but it would reduce clarity about the user’s +intentions when working with a ``type``. Different concepts and usage +patterns; different spellings. + +.. _designed: https://mail.python.org/archives/list/typing-sig@python.org/message/D5FHORQVPHX3BHUDGF3A3TBZURBXLPHD/ + + +Accept arbitrary annotation expressions +--------------------------------------- + +Certain typing special forms can be used in *some* but not *all* +annotation contexts: + +For example ``Final[]`` can be used as a variable type but not as a +parameter type or a return type: + +:: + + some_const: Final[str] = ... # OK + + def foo(not_reassignable: Final[object]): ... # ERROR: Final[] not allowed here + + def nonsense() -> Final[object]: ... # ERROR: Final[] not meaningful here + +``TypeExpr[T]`` does not allow matching such annotation expressions +because it is not clear what it would mean for such an expression +to parameterized by a type variable in position ``T``: + +:: + + def ismatch[T](value: object, typx: TypeExpr[T]) -> TypeGuard[T]: ... + + def foo(some_arg): + if ismatch(some_arg, Final[int]): # ERROR: Final[int] is not a TypeExpr + reveal_type(some_arg) # ? NOT Final[int], because invalid for a parameter + +Functions that wish to operate on *all* kinds of annotation expressions, +including those that are not ``TypeExpr``\ s, can continue to accept such +inputs as ``object`` parameters, as they must do so today. + + +Accept only universal type expressions +-------------------------------------- + +Earlier drafts of this PEP only allowed ``TypeExpr[]`` to match the subset +of type expressions which are valid in *all* contexts, excluding +:ref:`non-universal type expressions `. +However doing that would effectively +create a new subset of annotation expressions that Python typing users +would have to understand, on top of all the existing distinctions between +“class objects”, “type expressions”, and “annotation expressions”. + +To avoid introducing yet another concept that everyone has to learn, +this proposal just rounds ``TypeExpr[]`` to exactly match the existing +definition of a “type expression”. + + +Support pattern matching on type expressions +-------------------------------------------- + +It was asserted that some functions may wish to pattern match on the +interior of type expressions in their signatures. + +One use case is to allow a function to explicitly enumerate all the +*specific* kinds of type expressions it supports as input. +Consider the following possible pattern matching syntax: + +:: + + @overload + def checkcast(typx: TypeExpr[AT=Annotated[T, *Anns]], value: str) -> T: ... + @overload + def checkcast(typx: TypeExpr[UT=Union[*Ts]], value: str) -> Union[*Ts]: ... + @overload + def checkcast(typx: type[C], value: str) -> C: ... + # ... (more) + +All functions observed in the wild that conceptually take a ``TypeExpr[]`` +generally try to support *all* kinds of type expressions, so it doesn’t +seem valuable to enumerate a particular subset. + +Additionally the above syntax isn’t precise enough to fully describe the +actual input constraints for a typical function in the wild. For example +many functions recognize un-stringified type expressions like +``list[Movie]`` but may not recognize type expressions with stringified +subcomponents like ``list['Movie']``. + +A second use case for pattern matching on the interior of type +expressions is to explicitly match an ``Annotated[]`` form to pull out the +interior type argument and strip away the metadata: + +:: + + def checkcast( + typx: TypeExpr[T] | TypeExpr[AT=Annotated[T, *Anns]], + value: object + ) -> T: + +However ``Annotated[T, metadata]`` is already treated equivalent to ``T`` anyway. +There’s no additional value in being explicit about this behavior. +The example above could be more-straightforwardly written as the equivalent: + +:: + + def checkcast(typx: TypeExpr[T], value: object) -> T: + + +Footnotes +========= + +.. [#type_c] + :pep:`Type[C] spells a class object <484#the-type-of-class-objects>` + +.. [#TypeIsPep] + :pep:`TypeIs[T] is similar to bool <742>` + +.. [#DataclassInitVar] + ``dataclass.make_dataclass`` accepts ``InitVar[...]`` as a special case + in addition to type expressions. Therefore it may unfortunately be necessary + to continue annotating its ``type`` parameter as ``object`` rather + than ``TypeExpr``. + +.. [#forward_ref_normalization] + Special forms normalize string arguments to ``ForwardRef`` instances + at runtime using internal helper functions in the ``typing`` module. + Runtime type checkers may wish to implement similar functions when + working with string-based forward references. + +.. [#strann_less_common] + Stringified type annotations are expected to become less common + starting in Python 3.14 when :pep:`deferred annotations <649>` + become available. However there is a large amount of existing code from + earlier Python versions relying on stringified type annotations that will + still need to be supported for several years. + + +Copyright +========= + +This document is placed in the public domain or under the +CC0-1.0-Universal license, whichever is more permissive.