PEP: 747 Title: Annotating Type Forms Author: David Foster , Eric Traut Sponsor: Jelle Zijlstra Discussions-To: https://discuss.python.org/t/pep-747-typeexpr-type-hint-for-a-type-expression/55984 Status: Draft Type: Standards Track Topic: Typing Created: 27-May-2024 Python-Version: 3.14 Post-History: `19-Apr-2024 `__, `04-May-2024 `__, `17-Jun-2024 `__ Abstract ======== :ref:`Type expressions ` provide a standardized way to specify types in the Python type system. When a type expression is evaluated at runtime, the resulting *type form object* encodes the information supplied in the type expression. This enables a variety of use cases including runtime type checking, introspection, and metaprogramming. Such use cases have proliferated, but there is currently no way to accurately annotate functions that accept type form objects. Developers are forced to use an overly-wide type like ``object``, which makes some use cases impossible and generally reduces type safety. This PEP addresses this limitation by introducing a new special form ``typing.TypeForm``. This PEP makes no changes to the Python grammar. ``TypeForm`` is intended to be enforced only by type checkers, not by the Python runtime. Motivation ========== A function that operates on type form objects must understand how type expression details are encoded in these objects. For example, ``int | str``, ``"int | str"``, ``list[int]``, and ``MyTypeAlias`` are all valid type expressions, and they evaluate to instances of ``types.UnionType``, ``builtins.str``, ``types.GenericAlias``, and ``typing.TypeAliasType``, respectively. There is currently no way to indicate to a type checker that a function accepts type form objects and knows how to work with them. ``TypeForm`` addresses this limitation. For example, here is a function that checks whether a value is assignable to a specified type and returns None if it is not:: def trycast[T](typx: TypeForm[T], value: object) -> T | None: ... The use of ``TypeForm`` and the type variable ``T`` describes a relationship between the type form passed to parameter ``typx`` and the function's return type. ``TypeForm`` can also be used with :ref:`typing:typeis` to define custom type narrowing behaviors:: def isassignable[T](value: object, typx: TypeForm[T]) -> TypeIs[T]: ... request_json: object = ... if isassignable(request_json, MyTypedDict): assert_type(request_json, MyTypedDict) # Type of variable is narrowed The ``isassignable`` function implements something like an enhanced ``isinstance`` check. This is useful for validating whether a value decoded from JSON conforms to a particular structure of nested ``TypedDict``\ s, lists, unions, ``Literal``\ s, or any other type form that can be described with a type expression. This kind of check was alluded to in :pep:`PEP 589 <589#using-typeddict-types>` but could not be implemented without ``TypeForm``. Why not ``type[C]``? -------------------- One might think that ``type[C]`` would suffice for these use cases. However, only class objects (instances of the ``builtins.type`` class) are assignable to ``type[C]``. Many type form objects do not meet this requirement:: def trycast[T](typx: type[T], value: object) -> T | None: ... trycast(str, 'hi') # OK trycast(Literal['hi'], 'hi') # Type violation trycast(str | None, 'hi') # Type violation trycast(MyProtocolClass, obj) # Type violation TypeForm use cases ------------------ `A survey of Python libraries`_ reveals several categories of functions that would benefit from ``TypeForm``: .. _A survey of Python libraries: https://github.com/python/mypy/issues/9773#issuecomment-2017998886 - Assignability checkers: - Determines whether a value is assignable to a specified type - Pattern 1: ``def is_assignable[T](value: object, typx: TypeForm[T]) -> TypeIs[T]`` - Pattern 2: ``def is_match[T](value: object, typx: TypeForm[T]) -> TypeGuard[T]`` - Examples: beartype.\ `is_bearable`_, trycast.\ `isassignable`_, typeguard.\ `check_type`_, xdsl.\ `isa`_ .. _is_bearable: https://github.com/beartype/beartype/issues/255 .. _isassignable: https://github.com/davidfstr/trycast?tab=readme-ov-file#isassignable-api .. _check_type: https://typeguard.readthedocs.io/en/latest/api.html#typeguard.check_type .. _isa: https://github.com/xdslproject/xdsl/blob/ac12c9ab0d64618475efb98d1d197bdd79f593c3/xdsl/utils/hints.py#L23 - Converters: - If a value is assignable to (or coercible to) a specified type, a *converter* returns the value narrowed to (or coerced to) that type. Otherwise, an exception is raised. - Pattern 1: ``def convert[T](value: object, typx: TypeForm[T]) -> T`` - Examples: cattrs.BaseConverter.\ `structure`_, trycast.\ `checkcast`_, typedload.\ `load`_ - Pattern 2: :: class Converter[T]: def __init__(self, typx: TypeForm[T]) -> None: ... def convert(self, value: object) -> T: ... - Examples: pydantic.\ `TypeAdapter(T).validate_python`_, mashumaro.\ `JSONDecoder(T).decode`_ .. _structure: https://github.com/python-attrs/cattrs/blob/5f5c11627a7f67a23d6212bc7df9f96243c62dc5/src/cattrs/converters.py#L332-L334 .. _checkcast: https://github.com/davidfstr/trycast#checkcast-api .. _load: https://ltworf.github.io/typedload/ .. _TypeAdapter(T).validate_python: https://stackoverflow.com/a/61021183/604063 .. _JSONDecoder(T).decode: https://github.com/Fatal1ty/mashumaro?tab=readme-ov-file#usage-example - Typed field definitions: - Pattern: :: class Field[T]: value_type: TypeForm[T] - Examples: attrs.\ `make_class`_, dataclasses.\ `make_dataclass`_ [#DataclassInitVar]_, `openapify`_ .. _make_class: https://www.attrs.org/en/stable/api.html#attrs.make_class .. _make_dataclass: https://github.com/python/typeshed/issues/11653 .. _openapify: https://github.com/Fatal1ty/openapify/blob/c8d968c7c9c8fd7d4888bd2ddbe18ffd1469f3ca/openapify/core/models.py#L16 The survey also identified some introspection functions that accept runtime type forms as input. Today, these functions are annotated with ``object``: - General introspection operations: - Pattern: ``def get_annotation_info(typx: object) -> object`` - Examples: typing.{`get_origin`_, `get_args`_}, `typing_inspect`_.{is_*_type, get_origin, get_parameters} These functions accept values evaluated from arbitrary annotation expressions, not just type expressions, so they cannot be altered to use ``TypeForm``. .. _get_origin: https://docs.python.org/3/library/typing.html#typing.get_origin .. _get_args: https://docs.python.org/3/library/typing.html#typing.get_args .. _typing_inspect: https://github.com/ilevkivskyi/typing_inspect?tab=readme-ov-file#readme Specification ============= When a type expression is evaluated at runtime, the resulting value is a *type form* object. This value encodes the information supplied in the type expression, and it represents the type described by that type expression. ``TypeForm`` is a special form that, when used in a type expression, describes a set of type form objects. It accepts a single type argument, which must be a valid type expression. ``TypeForm[T]`` describes the set of all type form objects that represent the type ``T`` or types that are :term:`assignable to ` ``T``. For example, ``TypeForm[str | None]`` describes the set of all type form objects that represent a type assignable to ``str | None``:: ok1: TypeForm[str | None] = str | None # OK ok2: TypeForm[str | None] = str # OK ok3: TypeForm[str | None] = None # OK ok4: TypeForm[str | None] = Literal[None] # OK ok5: TypeForm[str | None] = Optional[str] # OK ok6: TypeForm[str | None] = "str | None" # OK ok7: TypeForm[str | None] = Any # OK err1: TypeForm[str | None] = str | int # Error err2: TypeForm[str | None] = list[str | None] # Error By this same definition, ``TypeForm[Any]`` describes a type form object that represents the type ``Any`` or any type that is assignable to ``Any``. Since all types in the Python type system are assignable to ``Any``, ``TypeForm[Any]`` describes the set of all type form objects evaluated from all valid type expressions. The type expression ``TypeForm``, with no type argument provided, is equivalent to ``TypeForm[Any]``. Implicit ``TypeForm`` Evaluation -------------------------------- When a static type checker encounters an expression that follows all of the syntactic, semantic and contextual rules for a type expression as detailed in the typing spec, the evaluated type of this expression should be assignable to ``TypeForm[T]`` if the type it describes is assignable to ``T``. For example, if a static type checker encounters the expression ``str | None``, it may normally evaluate its type as ``UnionType`` because it produces a runtime value that is an instance of ``types.UnionType``. However, because this expression is a valid type expression, it is also assignable to the type ``TypeForm[str | None]``: v1_actual: UnionType = str | None # OK v1_type_form: TypeForm[str | None] = str | None # OK v2_actual: type = list[int] # OK v2_type_form: TypeForm = list[int] # OK The ``Annotated`` special form is allowed in type expressions, so it can also appear in an expression that is assignable to ``TypeForm``. Consistent with the typing spec's rules for ``Annotated``, a static type checker may choose to ignore any ``Annotated`` metadata that it does not understand:: v3: TypeForm[int | str] = Annotated[int | str, "metadata"] # OK v4: TypeForm[Annotated[int | str, "metadata"]] = int | str # OK A string literal expression containing a valid type expression should likewise be assignable to ``TypeForm``:: v5: TypeForm[set[str]] = "set[str]" # OK Expressions that violate one or more of the syntactic, semantic, or contextual rules for type expressions should not evaluate to a ``TypeForm`` type. The rules for type expression validity are explained in detail within the typing spec, so they are not repeated here:: bad1: TypeForm = tuple() # Error: Call expression not allowed in type expression bad2: TypeForm = (1, 2) # Error: Tuple expression not allowed in type expression bad3: TypeForm = 1 # Non-class object not allowed in type expression bad4: TypeForm = Self # Error: Self not allowed outside of a class bad5: TypeForm = Literal[var] # Error: Variable not allowed in type expression bad6: TypeForm = Literal[f""] # Error: f-strings not allowed in type expression bad7: TypeForm = ClassVar[int] # Error: ClassVar not allowed in type expression bad8: TypeForm = Required[int] # Error: Required not allowed in type expression bad9: TypeForm = Final[int] # Error: Final not allowed in type expression bad10: TypeForm = Unpack[Ts] # Error: Unpack not allowed in this context bad11: TypeForm = Optional # Error: Invalid use of Optional special form bad12: TypeForm = T # Error if T is an out-of-scope TypeVar bad13: TypeForm = "int + str" # Error: invalid quoted type expression Explicit ``TypeForm`` Evaluation -------------------------------- ``TypeForm`` also acts as a function that can be called with a single argument. Type checkers should validate that this argument is a valid type expression:: x1 = TypeForm(str | None) reveal_type(v1) # Revealed type is "TypeForm[str | None]" x2 = TypeForm("list[int]") revealed_type(v2) # Revealed type is "TypeForm[list[int]]" x3 = TypeForm('type(1)') # Error: invalid type expression At runtime the ``TypeForm(...)`` callable simply returns the value passed to it. This explicit syntax serves two purposes. First, it documents the developer's intent to use the value as a type form object. Second, static type checkers validate that all rules for type expressions are followed:: x4 = type(int) # No error, evaluates to "type[int]" x5 = TypeForm(type(int)) # Error: call not allowed in type expression Assignability ------------- ``TypeForm`` has a single type parameter, which is covariant. That means ``TypeForm[B]`` is assignable to ``TypeForm[A]`` if ``B`` is assignable to ``A``:: def get_type_form() -> TypeForm[int]: ... t1: TypeForm[int | str] = get_type_form() # OK t2: TypeForm[str] = get_type_form() # Error ``type[T]`` is a subtype of ``TypeForm[T]``, which means that ``type[B]`` is assignable to ``TypeForm[A]`` if ``B`` is assignable to ``A``:: def get_type() -> type[int]: ... t3: TypeForm[int | str] = get_type() # OK t4: TypeForm[str] = get_type() # Error ``TypeForm`` is a subtype of ``object`` and is assumed to have all of the attributes and methods of ``object``. Backward Compatibility ====================== This PEP clarifies static type checker behaviors when evaluating type expressions in "value expression" contexts (that is, contexts where type expressions are not mandated by the typing spec). In the absence of a ``TypeForm`` type annotation, existing type evaluation behaviors persist, so no backward compatibility issues are anticipated. For example, if a static type checker previously evaluated the type of expression ``str | None`` as ``UnionType``, it will continue to do so unless this expression is assigned to a variable or parameter whose type is annotated as ``TypeForm``. How to Teach This ================= Type expressions are used in annotations to describe which values are accepted by a function parameter, returned by a function, or stored in a variable: .. code-block:: text parameter type return type | | v v def plus(n1: int, n2: int) -> int: sum: int = n1 + n2 ^ | variable type return sum Type expressions evaluate to valid *type form* objects at runtime and can be assigned to variables and manipulated like any other data in a program: .. code-block:: text a variable a type expression | | v v int_type_form: TypeForm = int | None ^ | the type of a type form object ``TypeForm[]`` is how you spell the type of a *type form* object, which is a runtime representation of a type. ``TypeForm`` is similar to ``type``, but ``type`` is compatible only with **class objects** like ``int``, ``str``, ``list``, or ``MyClass``. ``TypeForm`` accommodates any type form that can be expressed using a valid type expression, including those with brackets (``list[int]``), union operators (``int | None``), and special forms (``Any``, ``LiteralString``, ``Never``, etc.). Most programmers will not define their *own* functions that accept a ``TypeForm`` parameter or return a ``TypeForm`` value. It is more common to pass a type form object to a library function that knows how to decode and use such objects. For example, the ``isassignable`` function in the ``trycast`` library can be used like Python's built-in ``isinstance`` function to check whether a value matches the shape of a particular type. ``isassignable`` accepts *any* type form object as input. - Yes: :: from trycast import isassignable if isassignable(some_object, MyTypedDict): # OK: MyTypedDict is a TypeForm[] ... - No: :: if isinstance(some_object, MyTypedDict): # ERROR: MyTypedDict is not a type[] ... Advanced Examples ================= If you want to write your own runtime type checker or a function that manipulates type form objects as values at runtime, this section provides examples of how such a function can use ``TypeForm``. Introspecting type form objects ------------------------------- Functions like ``typing.get_origin`` and ``typing.get_args`` can be used to extract components of some type form objects. :: import typing def strip_annotated_metadata(typx: TypeForm[T]) -> TypeForm[T]: if typing.get_origin(typx) is typing.Annotated: typx = cast(TypeForm[T], typing.get_args(typx)[0]) return typx ``isinstance`` and ``is`` can also be used to distinguish between different kinds of type form objects: :: import types import typing def split_union(typx: TypeForm) -> tuple[TypeForm, ...]: if isinstance(typ, types.UnionType): # X | Y return cast(tuple[TypeForm, ...], typing.get_args(typ)) if typing.get_origin(typ) is typing.Union: # Union[X, Y] return cast(tuple[TypeForm, ...], typing.get_args(typ)) if typ in (typing.Never, typing.NoReturn,): return () return (typ,) Combining with a type variable ------------------------------ ``TypeForm`` can be parameterized by a type variable that is used elsewhere within the same function definition: :: def as_instance[T](typx: TypeForm[T]) -> T | None: return typ() if isinstance(typ, type) else None Combining with ``type`` ----------------------- Both ``TypeForm`` and ``type`` can be parameterized by the same type variable within the same function definition: :: def as_type[T](typx: TypeForm[T]) -> type[T] | None: return typ if isinstance(typ, type) else None Combining with ``TypeIs`` and ``TypeGuard`` ------------------------------------------- A type variable can also be used by a ``TypeIs`` or ``TypeGuard`` return type: :: def isassignable[T](value: object, typx: TypeForm[T]) -> TypeIs[T]: ... count: int | str = ... if isassignable(count, int): assert_type(count, int) else: assert_type(count, str) Challenges When Accepting All TypeForms --------------------------------------- A function that takes an *arbitrary* ``TypeForm`` as input must support a variety of possible type form objects. Such functions are not easy to write. - New special forms are introduced with each new Python version, and special handling may be required for each one. - Quoted annotations [#quoted_less_common]_ (like ``'list[str]'``) must be *parsed* (to something like ``list[str]``). - Resolving quoted forward references inside type expressions is typically done with ``eval()``, which is difficult to use in a safe way. - Recursive types like ``IntTree = list[int | 'IntTree']`` are difficult to resolve. - User-defined generic types (like Django’s ``QuerySet[User]``) can introduce non-standard behaviors that require runtime support. Reference Implementation ======================== Pyright (version 1.1.379) provides a reference implementation for ``TypeForm``. Mypy contributors also `plan to implement `__ support for ``TypeForm``. A reference implementation of the runtime component is provided in the ``typing_extensions`` module. Rejected Ideas ============== Alternative names ----------------- Alternate names were considered for ``TypeForm``. ``TypeObject`` and ``TypeType`` were deemed too generic. ``TypeExpression`` and ``TypeExpr`` were also considered, but these were considered confusing because these objects are not themselves "expressions" but rather the result of evaluating a type expression. Widen ``type[C]`` to support all type expressions ------------------------------------------------- ``type`` was `designed`_ to describe class objects, subclasses of the ``type`` class. A value with the type ``type`` is assumed to be instantiable through a constructor call. Widening the meaning of ``type`` to represent arbitrary type form objects would present backward compatibility problems and would eliminate a way to describe the set of values limited to subclasses of ``type``. .. _designed: https://mail.python.org/archives/list/typing-sig@python.org/message/D5FHORQVPHX3BHUDGF3A3TBZURBXLPHD/ Accept arbitrary annotation expressions --------------------------------------- Certain special forms act as type qualifiers and can be used in *some* but not *all* annotation contexts: For example. the type qualifier ``Final`` can be used as a variable type but not as a parameter type or a return type: :: some_const: Final[str] = ... # OK def foo(not_reassignable: Final[object]): ... # Error: Final not allowed here def nonsense() -> Final[object]: ... # Error: Final not alowed here With the exception of ``Annotated``, type qualifiers are not allowed in type expressions. ``TypeForm`` is limited to type expressions because its assignability rules are based on the assignability rules for types. It is nonsensical to ask whether ``Final[int]`` is assignable to ``int`` because the former is not a valid type expression. Functions that wish to operate on objects that are evaluated from annotation expressions can continue to accept such inputs as ``object`` parameters. Pattern matching on type forms ------------------------------ It was asserted that some functions may wish to pattern match on the interior of type expressions in their signatures. One use case is to allow a function to explicitly enumerate all the *specific* kinds of type expressions it supports as input. Consider the following possible pattern matching syntax: :: @overload def checkcast(typx: TypeForm[AT=Annotated[T, *A]], value: str) -> T: ... @overload def checkcast(typx: TypeForm[UT=Union[*Ts]], value: str) -> Union[*Ts]: ... @overload def checkcast(typx: type[C], value: str) -> C: ... # ... (more) All functions observed in the wild that conceptually accept type form objects generally try to support *all* kinds of type expressions, so it doesn’t seem valuable to enumerate a particular subset. Additionally, the above syntax isn’t precise enough to fully describe the input constraints for a typical function in the wild. For example, many functions do not support type expressions with quoted subexpressions like ``list['Movie']``. A second use case for pattern matching is to explicitly match an ``Annotated`` form to extract the interior type argument and strip away any metadata: :: def checkcast( typx: TypeForm[T] | TypeForm[AT=Annotated[T, *A]], value: object ) -> T: However, ``Annotated[T, metadata]`` is already treated equivalent to ``T`` by static type checkers. There’s no additional value in being explicit about this behavior. The example above could more simply be written as the equivalent: :: def checkcast(typx: TypeForm[T], value: object) -> T: Footnotes ========= .. [#type_t] :ref:`Type[T] ` spells a class object .. [#TypeIs] :ref:`TypeIs[T] ` is similar to bool .. [#DataclassInitVar] ``dataclass.make_dataclass`` allows the type qualifier ``InitVar[...]``, so ``TypeForm`` cannot be used in this case. .. [#forward_ref_normalization] Special forms normalize string arguments to ``ForwardRef`` instances at runtime using internal helper functions in the ``typing`` module. Runtime type checkers may wish to implement similar functions when working with string-based forward references. .. [#quoted_less_common] Quoted annotations are expected to become less common starting in Python 3.14 when :pep:`deferred annotations <649>` is implemented. However, code written for earlier Python versions relies on quoted annotations and will need to be supported for several years. Copyright ========= This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive.