From 02a70341a2d34b7ad4c86a5773629d587e47c295 Mon Sep 17 00:00:00 2001 From: Eric Traut Date: Sun, 15 Sep 2024 15:10:35 -0700 Subject: [PATCH] PEP 747: Revised TypeForm specification (#3929) Co-authored-by: David Foster Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com> --- peps/pep-0747.rst | 1077 ++++++++++++++------------------------------- 1 file changed, 326 insertions(+), 751 deletions(-) diff --git a/peps/pep-0747.rst b/peps/pep-0747.rst index 1f362c060..dd3a053ae 100644 --- a/peps/pep-0747.rst +++ b/peps/pep-0747.rst @@ -1,6 +1,6 @@ PEP: 747 -Title: TypeExpr: Type Hint for a Type Expression -Author: David Foster +Title: Annotating Type Forms +Author: David Foster , Eric Traut Sponsor: Jelle Zijlstra Discussions-To: https://discuss.python.org/t/pep-747-typeexpr-type-hint-for-a-type-expression/55984 Status: Draft @@ -14,121 +14,95 @@ Post-History: `19-Apr-2024 ` objects such -as the runtime object ``str | None``, even if ``C`` is an unbounded -``TypeVar``. [#type_c]_ In cases where that restriction is unwanted, this -PEP proposes a new notation ``TypeExpr[T]`` where ``T`` is a type, to -refer to a either a class object or some other type expression object -that is a subtype of ``T``, allowing any kind of type to be referenced. +:ref:`Type expressions ` provide a standardized way +to specify types in the Python type system. When a type expression is +evaluated at runtime, the resulting *type form object* encodes the information +supplied in the type expression. This enables a variety of use cases including +runtime type checking, introspection, and metaprogramming. -This PEP makes no Python grammar changes. Correct usage of -``TypeExpr[]`` is intended to be enforced only by static and runtime -type checkers and need not be enforced by Python itself at runtime. +Such use cases have proliferated, but there is currently no way to accurately +annotate functions that accept type form objects. Developers are forced to use +an overly-wide type like ``object``, which makes some use cases impossible and +generally reduces type safety. This PEP addresses this limitation by +introducing a new special form ``typing.TypeForm``. +This PEP makes no changes to the Python grammar. ``TypeForm`` is +intended to be enforced only by type checkers, not by the Python runtime. -.. _motivation: Motivation ========== -The introduction of ``TypeExpr`` allows new kinds of metaprogramming -functions that operate on type expressions to be type-annotated and -understood by type checkers. +A function that operates on type form objects must understand how type +expression details are encoded in these objects. For example, ``int | str``, +``"int | str"``, ``list[int]``, and ``MyTypeAlias`` are all valid type +expressions, and they evaluate to instances of ``types.UnionType``, +``builtins.str``, ``types.GenericAlias``, and ``typing.TypeAliasType``, +respectively. -For example, here is a function that checks whether a value is -assignable to a variable of a particular type, and if so returns the -original value: +There is currently no way to indicate to a type checker that a function accepts +type form objects and knows how to work with them. ``TypeForm`` addresses this +limitation. For example, here is a function that checks whether a value is +assignable to a specified type and returns None if it is not:: -:: + def trycast[T](typx: TypeForm[T], value: object) -> T | None: ... - def trycast[T](typx: TypeExpr[T], value: object) -> T | None: ... +The use of ``TypeForm`` and the type variable ``T`` describes a relationship +between the type form passed to parameter ``typx`` and the function's +return type. -The use of ``TypeExpr[]`` and the type variable ``T`` enables the return -type of this function to be influenced by a ``typx`` value passed at -runtime, which is quite powerful. +``TypeForm`` can also be used with :ref:`typing:typeis` to define custom type +narrowing behaviors:: -Here is another function that checks whether a value is assignable to a -variable of a particular type, and if so returns ``True`` (as a special -``TypeIs[]`` bool [#TypeIsPep]_): + def isassignable[T](value: object, typx: TypeForm[T]) -> TypeIs[T]: ... -:: + request_json: object = ... + if isassignable(request_json, MyTypedDict): + assert_type(request_json, MyTypedDict) # Type of variable is narrowed - def isassignable[T](value: object, typx: TypeExpr[T]) -> TypeIs[T]: ... - -The use of ``TypeExpr[]`` and ``TypeIs[]`` together enables type -checkers to narrow the return type appropriately depending on what type -expression is passed in: - -:: - - request_json: object = ... - if isassignable(request_json, MyTypedDict): - assert_type(request_json, MyTypedDict) # type is narrowed! - -That ``isassignable`` function enables a kind of enhanced ``isinstance`` -check which is useful for `checking whether a value decoded from JSON -conforms to a particular structure`_ of nested ``TypedDict``\ s, -lists, unions, ``Literal``\ s, and other types. This kind -of check was alluded to in :pep:`PEP 589 <589#using-typeddict-types>` but could -not be implemented at the time without a notation similar to -``TypeExpr[]``. - -.. _checking whether a value decoded from JSON conforms to a particular structure: https://mail.python.org/archives/list/typing-sig@python.org/thread/I5ZOQICTJCENTCDPHLZR7NT42QJ43GP4/ +The ``isassignable`` function implements something like an enhanced +``isinstance`` check. This is useful for validating whether a value decoded +from JSON conforms to a particular structure of nested ``TypedDict``\ s, +lists, unions, ``Literal``\ s, or any other type form that can be described +with a type expression. This kind of check was alluded to in +:pep:`PEP 589 <589#using-typeddict-types>` but could not be implemented without +``TypeForm``. -Why can’t ``type[]`` be used? ------------------------------ +Why not ``type[C]``? +-------------------- -One might think you could define the example functions above to take a -``type[C]`` - which is syntax that already exists - rather than a -``TypeExpr[T]``. However if you were to do that then certain type -expressions like ``str | None`` - which are not class objects and -therefore not ``type``\ s at runtime - would be rejected: +One might think that ``type[C]`` would suffice for these use cases. However, +only class objects (instances of the ``builtins.type`` class) are assignable +to ``type[C]``. Many type form objects do not meet this requirement:: -:: + def trycast[T](typx: type[T], value: object) -> T | None: ... - # NOTE: Uses a type[C] parameter rather than a TypeExpr[T] - def trycast_type[C](typ: type[C], value: object) -> T | None: ... - - trycast_type(str, 'hi') # ok; str is a type - trycast_type(Optional[str], 'hi') # ERROR; Optional[str] is not a type - trycast_type(str | int, 'hi') # ERROR; (str | int) is not a type - trycast_type(MyTypedDict, dict(value='hi')) # questionable; accepted by mypy 1.9.0 - -To solve that problem, ``type[]`` could be widened to include the -additional values allowed by ``TypeExpr``. However doing so would lose -``type[]``\ ’s current ability to spell a class object which always -supports instantiation and ``isinstance`` checks, unlike arbitrary type -expression objects. Therefore ``TypeExpr`` is proposed as new notation -instead. - -For a longer explanation of why we don’t just widen ``type[T]`` to -accept all type expressions, see -:ref:`widen_type_C_to_support_all_type_expressions`. + trycast(str, 'hi') # OK + trycast(Literal['hi'], 'hi') # Type violation + trycast(str | None, 'hi') # Type violation + trycast(MyProtocolClass, obj) # Type violation -.. _runtime_type_checkers_using_typeexpr: +TypeForm use cases +------------------ -Common kinds of functions that would benefit from TypeExpr ----------------------------------------------------------- +`A survey of Python libraries`_ reveals several categories of functions that +would benefit from ``TypeForm``: -`A survey of various Python libraries`_ revealed a few kinds of commonly -defined functions which would benefit from ``TypeExpr[]``: - -.. _A survey of various Python libraries: https://github.com/python/mypy/issues/9773#issuecomment-2017998886 +.. _A survey of Python libraries: https://github.com/python/mypy/issues/9773#issuecomment-2017998886 - Assignability checkers: - - Returns whether a value is assignable to a type expression. If so - then also narrows the type of the value to match the type - expression. + - Determines whether a value is assignable to a specified type - Pattern 1: - ``def isassignable[T](value: object, typx: TypeExpr[T]) -> TypeIs[T]`` + + ``def is_assignable[T](value: object, typx: TypeForm[T]) -> TypeIs[T]`` + - Pattern 2: - ``def ismatch[T](value: object, typx: TypeExpr[T]) -> TypeGuard[T]`` + + ``def is_match[T](value: object, typx: TypeForm[T]) -> TypeGuard[T]`` + - Examples: beartype.\ `is_bearable`_, trycast.\ `isassignable`_, typeguard.\ `check_type`_, xdsl.\ `isa`_ @@ -139,11 +113,13 @@ defined functions which would benefit from ``TypeExpr[]``: - Converters: - - If a value is assignable to (or coercible to) a type expression, - a *converter* returns the value narrowed to (or coerced to) that type - expression. Otherwise, it raises an exception. + - If a value is assignable to (or coercible to) a specified type, + a *converter* returns the value narrowed to (or coerced to) that type. + Otherwise, an exception is raised. + - Pattern 1: - ``def convert[T](value: object, typx: TypeExpr[T]) -> T`` + + ``def convert[T](value: object, typx: TypeForm[T]) -> T`` - Examples: cattrs.BaseConverter.\ `structure`_, trycast.\ `checkcast`_, typedload.\ `load`_ @@ -153,7 +129,7 @@ defined functions which would benefit from ``TypeExpr[]``: :: class Converter[T]: - def __init__(self, typx: TypeExpr[T]) -> None: ... + def __init__(self, typx: TypeForm[T]) -> None: ... def convert(self, value: object) -> T: ... - Examples: pydantic.\ `TypeAdapter(T).validate_python`_, @@ -172,7 +148,7 @@ defined functions which would benefit from ``TypeExpr[]``: :: class Field[T]: - value_type: TypeExpr[T] + value_type: TypeForm[T] - Examples: attrs.\ `make_class`_, dataclasses.\ `make_dataclass`_ [#DataclassInitVar]_, `openapify`_ @@ -181,473 +157,180 @@ defined functions which would benefit from ``TypeExpr[]``: .. _make_dataclass: https://github.com/python/typeshed/issues/11653 .. _openapify: https://github.com/Fatal1ty/openapify/blob/c8d968c7c9c8fd7d4888bd2ddbe18ffd1469f3ca/openapify/core/models.py#L16 -The survey also identified some introspection functions that take -annotation expressions as input using plain ``object``\ s which would -*not* gain functionality by marking those inputs as ``TypeExpr[]``: +The survey also identified some introspection functions that accept runtime +type forms as input. Today, these functions are annotated with ``object``: - General introspection operations: - - Pattern: ``def get_annotation_info(maybe_annx: object) -> object`` + - Pattern: ``def get_annotation_info(typx: object) -> object`` + - Examples: typing.{`get_origin`_, `get_args`_}, `typing_inspect`_.{is_*_type, get_origin, get_parameters} +These functions accept values evaluated from arbitrary annotation expressions, +not just type expressions, so they cannot be altered to use ``TypeForm``. + .. _get_origin: https://docs.python.org/3/library/typing.html#typing.get_origin .. _get_args: https://docs.python.org/3/library/typing.html#typing.get_args .. _typing_inspect: https://github.com/ilevkivskyi/typing_inspect?tab=readme-ov-file#readme -Rationale -========= - -Before this PEP existed there were already a few definitions in use to describe -different kinds of type annotations: - -.. code-block:: text - - +----------------------------------+ - | +------------------------------+ | - | | +-------------------------+ | | - | | | +---------------------+ | | | - | | | | Class object | | | | = type[C] - | | | +---------------------+ | | | - | | | Type expression object | | | = TypeExpr[T] <-- new! - | | +-------------------------+ | | - | | Annotation expression object | | - | +------------------------------+ | - | Object | = object - +----------------------------------+ - -- :ref:`Class objects `, - spelled as ``type[C]``, support ``isinstance`` checks and are callable. - - - Examples: ``int``, ``str``, ``MyClass`` - -- :ref:`Type expressions ` - include any type annotation which describes a type. - - - Examples: ``list[int]``, ``MyTypedDict``, ``int | str``, - ``Literal['square']``, any class object - -- :ref:`Annotation expressions ` - include any type annotation, including those only valid in specific contexts. - - - Examples: ``Final[int]``, ``Required[str]``, ``ClassVar[str]``, - any type expression - -``TypeExpr`` aligns with an existing definition from the above list - -*type expression* - to avoid introducing yet another subset of type annotations -that users of Python typing need to think about. - -``TypeExpr`` aligns with *type expression* specifically -because a type expression is already used to parameterize type variables, -which are used in combination with ``TypeIs`` and ``TypeGuard`` to enable -the compelling examples mentioned in :ref:`Motivation `. - -``TypeExpr`` does not align with *annotation expression* for reasons given in -:ref:`Rejected Ideas » Accept arbitrary annotation expressions `. - - Specification ============= -A ``TypeExpr`` value represents a :ref:`type expression ` -such as ``str | None``, ``dict[str, int]``, or ``MyTypedDict``. -A ``TypeExpr`` type is written as -``TypeExpr[T]`` where ``T`` is a type or a type variable. It can also be -written without brackets as just ``TypeExpr``, which is treated the same as -to ``TypeExpr[Any]``. +When a type expression is evaluated at runtime, the resulting value is a +*type form* object. This value encodes the information supplied in the type +expression, and it represents the type described by that type expression. + +``TypeForm`` is a special form that, when used in a type expression, describes +a set of type form objects. It accepts a single type argument, which must be a +valid type expression. ``TypeForm[T]`` describes the set of all type form +objects that represent the type ``T`` or types that are +:term:`assignable to ` ``T``. For example, +``TypeForm[str | None]`` describes the set of all type form objects +that represent a type assignable to ``str | None``:: + ok1: TypeForm[str | None] = str | None # OK + ok2: TypeForm[str | None] = str # OK + ok3: TypeForm[str | None] = None # OK + ok4: TypeForm[str | None] = Literal[None] # OK + ok5: TypeForm[str | None] = Optional[str] # OK + ok6: TypeForm[str | None] = "str | None" # OK + ok7: TypeForm[str | None] = Any # OK -Using TypeExprs ---------------- + err1: TypeForm[str | None] = str | int # Error + err2: TypeForm[str | None] = list[str | None] # Error -A ``TypeExpr`` is a new kind of type expression, usable in any context where a -type expression is valid, as a function parameter type, a return type, -or a variable type: +By this same definition, ``TypeForm[Any]`` describes a type form object +that represents the type ``Any`` or any type that is assignable to ``Any``. +Since all types in the Python type system are assignable to ``Any``, +``TypeForm[Any]`` describes the set of all type form objects +evaluated from all valid type expressions. -:: +The type expression ``TypeForm``, with no type argument provided, is +equivalent to ``TypeForm[Any]``. - def is_union_type(typx: TypeExpr) -> bool: ... # parameter type -:: +Implicit ``TypeForm`` Evaluation +-------------------------------- - def union_of[S, T](s: TypeExpr[S], t: TypeExpr[T]) \ - -> TypeExpr[S | T]: ... # return type +When a static type checker encounters an expression that follows all of the +syntactic, semantic and contextual rules for a type expression as detailed +in the typing spec, the evaluated type of this expression should be assignable +to ``TypeForm[T]`` if the type it describes is assignable to ``T``. -:: +For example, if a static type checker encounters the expression ``str | None``, +it may normally evaluate its type as ``UnionType`` because it produces a +runtime value that is an instance of ``types.UnionType``. However, because +this expression is a valid type expression, it is also assignable to the +type ``TypeForm[str | None]``: - STR_TYPE: TypeExpr = str # variable type + v1_actual: UnionType = str | None # OK + v1_type_form: TypeForm[str | None] = str | None # OK -Note however that an *unannotated* variable assigned a type expression literal -will not be inferred to be of ``TypeExpr`` type by type checkers because PEP -484 :pep:`reserves that syntax for defining type aliases <484#type-aliases>`: + v2_actual: type = list[int] # OK + v2_type_form: TypeForm = list[int] # OK -- No: +The ``Annotated`` special form is allowed in type expressions, so it can +also appear in an expression that is assignable to ``TypeForm``. Consistent +with the typing spec's rules for ``Annotated``, a static type checker may choose +to ignore any ``Annotated`` metadata that it does not understand:: - :: + v3: TypeForm[int | str] = Annotated[int | str, "metadata"] # OK + v4: TypeForm[Annotated[int | str, "metadata"]] = int | str # OK - STR_TYPE = str # OOPS; treated as a type alias! +A string literal expression containing a valid type expression should likewise +be assignable to ``TypeForm``:: -If you want a type checker to recognize a type expression literal in a bare -assignment you’ll need to explicitly declare the assignment-target as -having ``TypeExpr`` type: + v5: TypeForm[set[str]] = "set[str]" # OK -- Yes: +Expressions that violate one or more of the syntactic, semantic, or contextual +rules for type expressions should not evaluate to a ``TypeForm`` type. The rules +for type expression validity are explained in detail within the typing spec, so +they are not repeated here:: - :: + bad1: TypeForm = tuple() # Error: Call expression not allowed in type expression + bad2: TypeForm = (1, 2) # Error: Tuple expression not allowed in type expression + bad3: TypeForm = 1 # Non-class object not allowed in type expression + bad4: TypeForm = Self # Error: Self not allowed outside of a class + bad5: TypeForm = Literal[var] # Error: Variable not allowed in type expression + bad6: TypeForm = Literal[f""] # Error: f-strings not allowed in type expression + bad7: TypeForm = ClassVar[int] # Error: ClassVar not allowed in type expression + bad8: TypeForm = Required[int] # Error: Required not allowed in type expression + bad9: TypeForm = Final[int] # Error: Final not allowed in type expression + bad10: TypeForm = Unpack[Ts] # Error: Unpack not allowed in this context + bad11: TypeForm = Optional # Error: Invalid use of Optional special form + bad12: TypeForm = T # Error if T is an out-of-scope TypeVar + bad13: TypeForm = "int + str" # Error: invalid quoted type expression - STR_TYPE: TypeExpr = str -- Yes: +Explicit ``TypeForm`` Evaluation +-------------------------------- - :: +``TypeForm`` also acts as a function that can be called with a single argument. +Type checkers should validate that this argument is a valid type expression:: - STR_TYPE: TypeExpr - STR_TYPE = str + x1 = TypeForm(str | None) + reveal_type(v1) # Revealed type is "TypeForm[str | None]" -- Okay, but discouraged: + x2 = TypeForm("list[int]") + revealed_type(v2) # Revealed type is "TypeForm[list[int]]" - :: + x3 = TypeForm('type(1)') # Error: invalid type expression - STR_TYPE = str # type: TypeExpr # the type comment is significant +At runtime the ``TypeForm(...)`` callable simply returns the value passed to it. -``TypeExpr`` values can be passed around and assigned just like normal -values: +This explicit syntax serves two purposes. First, it documents the developer's +intent to use the value as a type form object. Second, static type checkers +validate that all rules for type expressions are followed:: -:: + x4 = type(int) # No error, evaluates to "type[int]" + + x5 = TypeForm(type(int)) # Error: call not allowed in type expression - def swap1[S, T](t1: TypeExpr[S], t2: TypeExpr[T]) -> tuple[TypeExpr[T], TypeExpr[S]]: - t1_new: TypeExpr[T] = t2 # assigns a TypeExpr value to a new annotated variable - t2_new: TypeExpr[S] = t1 - return (t1_new, t2_new) - def swap2[S, T](t1: TypeExpr[S], t2: TypeExpr[T]) -> tuple[TypeExpr[T], TypeExpr[S]]: - t1_new = t2 # assigns a TypeExpr value to a new unannotated variable - t2_new = t1 - assert_type(t1_new, TypeExpr[T]) - assert_type(t2_new, TypeExpr[S]) - return (t1_new, t2_new) +Assignability +------------- - # NOTE: A more straightforward implementation would use isinstance() - def ensure_int(value: object) -> None: - value_type: TypeExpr = type(value) # assigns a type (a subtype of TypeExpr) - assert value_type == int +``TypeForm`` has a single type parameter, which is covariant. That means +``TypeForm[B]`` is assignable to ``TypeForm[A]`` if ``B`` is assignable to +``A``:: + def get_type_form() -> TypeForm[int]: ... -TypeExpr Values ---------------- + t1: TypeForm[int | str] = get_type_form() # OK + t2: TypeForm[str] = get_type_form() # Error -A variable of type ``TypeExpr[T]`` where ``T`` is a type, can hold any -**type expression object** - the result of evaluating a -:ref:`type expression ` -at runtime - which is a subtype of ``T``. +``type[T]`` is a subtype of ``TypeForm[T]``, which means that ``type[B]`` is +assignable to ``TypeForm[A]`` if ``B`` is assignable to ``A``:: -Incomplete expressions like a bare ``Optional`` or ``Union`` which do -not spell a type are not ``TypeExpr`` values. + def get_type() -> type[int]: ... -``TypeExpr[...]`` is itself a ``TypeExpr`` value: + t3: TypeForm[int | str] = get_type() # OK + t4: TypeForm[str] = get_type() # Error -:: +``TypeForm`` is a subtype of ``object`` and is assumed to have all of the +attributes and methods of ``object``. - OPTIONAL_INT_TYPE: TypeExpr = TypeExpr[int | None] # OK - assert isassignable(int | None, OPTIONAL_INT_TYPE) -.. _non_universal_typeexpr: +Backward Compatibility +====================== -``TypeExpr[]`` values include *all* type expressions including some -**non-universal type expressions** which are not valid in all annotation contexts. -In particular: - -- ``Self`` (valid only in some contexts) -- ``TypeGuard[...]`` (valid only in some contexts) -- ``TypeIs[...]`` (valid only in some contexts) - - -Explicit TypeExpr Values -'''''''''''''''''''''''' - -The syntax ``TypeExpr(T)`` (with parentheses) can be used to -spell a ``TypeExpr[T]`` value explicitly: - -:: - - NONE = TypeExpr(None) - INT1 = TypeExpr('int') # stringified type expression - INT2 = TypeExpr(int) - -At runtime the ``TypeExpr(...)`` callable returns its single argument unchanged. - - -.. _implicit_typeexpr_values: - -Implicit TypeExpr Values -'''''''''''''''''''''''' - -Historically static type checkers have only needed to recognize -*type expressions* in contexts where a type expression was expected. -Now *type expression objects* must also be recognized in contexts where a -value expression is expected. - -Static type checkers already recognize **class objects** (``type[C]``): - -- As a value expression, ``C`` has type ``type[C]``, - for each of the following values of C: - - - ``name`` (where ``name`` must refer to a valid in-scope class, type alias, or TypeVar) - - ``name '[' ... ']'`` - - `` '[' ... ']'`` - -The following **unparameterized type expressions** can be recognized unambiguously: - -- As a value expression, ``X`` has type ``TypeExpr[X]``, - for each of the following values of X: - - - ```` - - ```` - - ```` - - ```` - - ```` - -**None**: The type expression ``None`` (``NoneType``) is ambiguous with the value ``None``, -so must use the explicit ``TypeExpr(...)`` syntax: - -- As a value expression, ``TypeExpr(None)`` has type ``TypeExpr[None]``. -- As a value expression, ``None`` continues to have type ``None``. - -The following **parameterized type expressions** can be recognized unambiguously: - -- As a value expression, ``X`` has type ``TypeExpr[X]``, - for each of the following values of X: - - - `` '[' ... ']'`` - - `` '[' ... ']'`` - - `` '[' ... ']'`` - - `` '[' ... ']'`` - - `` '[' ... ']'`` - - `` '[' ... ']'`` - - `` '[' ... ']'`` - -.. _recognizing_annotated: - -**Annotated**: The type expression ``Annotated[...]`` is ambiguous with -the annotation expression ``Annotated[...]``, -so must be disambiguated based on its argument type: - -- As a value expression, ``Annotated[x, ...]`` has type ``type[C]`` - if ``x`` has type ``type[C]``. -- As a value expression, ``Annotated[x, ...]`` has type ``TypeExpr[T]`` - if ``x`` has type ``TypeExpr[T]``. -- As a value expression, ``Annotated[x, ...]`` has type ``object`` - if ``x`` has a type that is not ``type[C]`` or ``TypeExpr[T]``. - -**Union**: The type expression ``T1 | T2`` is ambiguous with -the value ``int1 | int2``, ``set1 | set2``, ``dict1 | dict2``, and more, -so must use the explicit ``TypeExpr(...)`` syntax: - -- Yes: - - :: - - if isassignable(value, TypeExpr(int | str)): ... - -- No: - - :: - - if isassignable(value, int | str): ... - -Future PEPs may make it possible to recognize the value expression ``T1 | T2`` directly as an -implicit TypeExpr value and avoid the need to use the explicit ``TypeExpr(...)`` syntax, -but that work is :ref:`deferred for now `. - -The **stringified type expression** ``"T"`` is ambiguous with both -the stringified annotation expression ``"T"`` -and the string literal ``"T"``, -so must use the explicit ``TypeExpr(...)`` syntax: - -- As a value expression, ``TypeExpr("T")`` has type ``TypeExpr[T]``, - where ``T`` is a valid type expression -- As a value expression, ``"T"`` continues to have type ``Literal["T"]``. - -No other kinds of type expressions currently exist. - -New kinds of type expressions that are introduced should define how they -will be recognized in a value expression context. - - -Literal[] TypeExprs -''''''''''''''''''' - -A value of ``Literal[...]`` type is *not* considered assignable to -a ``TypeExpr`` variable even if all of its members spell valid types because -dynamic values are not allowed in type expressions: - -:: - - STRS_TYPE_NAME: Literal['str', 'list[str]'] = 'str' - STRS_TYPE: TypeExpr = STRS_TYPE_NAME # ERROR: Literal[] value is not a TypeExpr - -However ``Literal[...]`` itself is still a ``TypeExpr``: - -:: - - DIRECTION_TYPE: TypeExpr[Literal['left', 'right']] = Literal['left', 'right'] # OK - - -Static vs. Runtime Representations of TypeExprs -''''''''''''''''''''''''''''''''''''''''''''''' - -A ``TypeExpr`` value appearing statically in a source file may be normalized -to a different representation at runtime. For example string-based -forward references are normalized at runtime to be ``ForwardRef`` instances -in some contexts: [#forward_ref_normalization]_ - -:: - - >>> IntTree = list[typing.Union[int, 'IntTree']] - >>> IntTree - list[typing.Union[int, ForwardRef('IntTree')]] - -The runtime representations of ``TypeExpr``\ s are considered implementation -details that may change over time and therefore static type checkers are -not required to recognize them: - -:: - - INT_TREE: TypeExpr = ForwardRef('IntTree') # ERROR: Runtime-only form - -Runtime type checkers that wish to assign a runtime-only representation -of a type expression to a ``TypeExpr[]`` variable must use ``cast()`` to -avoid errors from static type checkers: - -:: - - INT_TREE = cast(TypeExpr, ForwardRef('IntTree')) # OK - - -Subtyping ---------- - -Whether a ``TypeExpr`` value can be assigned from one variable to another is -determined by the following rules: - -Relationship with type -'''''''''''''''''''''' - -``TypeExpr[]`` is covariant in its argument type, just like ``type[]``: - -- ``TypeExpr[T1]`` is a subtype of ``TypeExpr[T2]`` iff ``T1`` is a - subtype of ``T2``. -- ``type[C1]`` is a subtype of ``TypeExpr[C2]`` iff ``C1`` is a subtype - of ``C2``. - -An unparameterized ``type`` can be assigned to an unparameterized ``TypeExpr`` -but not the other way around: - -- ``type[Any]`` is assignable to ``TypeExpr[Any]``. (But not the - other way around.) - -Relationship with object -'''''''''''''''''''''''' - -``TypeExpr[]`` is a kind of ``object``, just like ``type[]``: - -- ``TypeExpr[T]`` for any ``T`` is a subtype of ``object``. - -``TypeExpr[T]``, where ``T`` is a type variable, is assumed to have all -the attributes and methods of ``object`` and is not callable. - - -Interactions with isinstance() and issubclass() ------------------------------------------------ - -The ``TypeExpr`` special form cannot be used as the ``type`` argument to -``isinstance``: - -:: - - >>> isinstance(str, TypeExpr) - TypeError: typing.TypeExpr cannot be used with isinstance() - - >>> isinstance(str, TypeExpr[str]) - TypeError: isinstance() argument 2 cannot be a parameterized generic - -The ``TypeExpr`` special form cannot be used as any argument to -``issubclass``: - -:: - - >>> issubclass(TypeExpr, object) - TypeError: issubclass() arg 1 must be a class - - >>> issubclass(object, TypeExpr) - TypeError: typing.TypeExpr cannot be used with issubclass() - - -Affected signatures in the standard library -------------------------------------------- - -Changed signatures -'''''''''''''''''' - -The following signatures related to type expressions introduce -``TypeExpr`` where previously ``object`` or ``Any`` existed: - -- ``typing.cast`` -- ``typing.assert_type`` - -Unchanged signatures -'''''''''''''''''''' - -The following signatures related to annotation expressions continue to -use ``object`` and remain unchanged: - -- ``typing.get_origin`` -- ``typing.get_args`` - -The following signatures related to class objects continue to use -``type`` and remain unchanged: - -- ``builtins.isinstance`` -- ``builtins.issubclass`` -- ``builtins.type`` - -``typing.get_type_hints(..., include_extras=False)`` nearly returns only type -expressions in Python 3.12, stripping out most type qualifiers -(``Required, NotRequired, ReadOnly, Annotated``) but currently preserves a -few type qualifiers which are only allowed in annotation expressions -(``ClassVar, Final, InitVar, Unpack``). It may be desirable to alter the -behavior of this function in the future to also strip out those -qualifiers and actually return type expressions, although this PEP does -not propose those changes now: - -- ``typing.get_type_hints(..., include_extras=False)`` - - - Almost returns only type expressions, but not quite - -- ``typing.get_type_hints(..., include_extras=True)`` - - - Returns annotation expressions - - -Backwards Compatibility -======================= - -The rules for recognizing type expression objects -in a value expression context were not previously defined, so static type checkers -`varied in what types were assigned `_ -to such objects. Existing programs manipulating type expression objects -were already limited in manipulating them as plain ``object`` values, -and such programs should not break with -:ref:`the newly-defined rules `. +This PEP clarifies static type checker behaviors when evaluating type +expressions in "value expression" contexts (that is, contexts where type +expressions are not mandated by the typing spec). In the absence of a +``TypeForm`` type annotation, existing type evaluation behaviors persist, +so no backward compatibility issues are anticipated. For example, if a static +type checker previously evaluated the type of expression ``str | None`` as +``UnionType``, it will continue to do so unless this expression is assigned +to a variable or parameter whose type is annotated as ``TypeForm``. How to Teach This ================= -Normally when using type annotations in Python you're concerned with defining -the shape of values allowed to be passed to a function parameter, returned -by a function, or stored in a variable: +Type expressions are used in annotations to describe which values are accepted +by a function parameter, returned by a function, or stored in a variable: .. code-block:: text @@ -662,65 +345,37 @@ by a function, or stored in a variable: return sum -However type annotations themselves are valid values in Python and can be +Type expressions evaluate to valid *type form* objects at runtime and can be assigned to variables and manipulated like any other data in a program: .. code-block:: text - a variable a type - | | - v v - MAYBE_INT_TYPE: TypeExpr = int | None + a variable a type expression + | | + v v + int_type_form: TypeForm = int | None ^ | - the type of a type + the type of a type form object -``TypeExpr[]`` is how you spell the type of a variable containing a -type annotation object describing a type. +``TypeForm[]`` is how you spell the type of a *type form* object, which is +a runtime representation of a type. -``TypeExpr[]`` is similar to ``type[]``, but ``type[]`` can only -spell simple **class objects** like ``int``, ``str``, ``list``, or ``MyClass``. -``TypeExpr[]`` by contrast can additionally spell more complex types, -including those with brackets (like ``list[int]``) or pipes (like ``int | None``), -and including special types like ``Any``, ``LiteralString``, or ``Never``. +``TypeForm`` is similar to ``type``, but ``type`` is compatible only with +**class objects** like ``int``, ``str``, ``list``, or ``MyClass``. +``TypeForm`` accommodates any type form that can be expressed using +a valid type expression, including those with brackets (``list[int]``), union +operators (``int | None``), and special forms (``Any``, ``LiteralString``, +``Never``, etc.). -A ``TypeExpr`` variable (``maybe_float: TypeExpr``) looks similar to -a ``TypeAlias`` definition (``MaybeFloat: TypeAlias``), but ``TypeExpr`` -can only be used where a dynamic value is expected: +Most programmers will not define their *own* functions that accept a ``TypeForm`` +parameter or return a ``TypeForm`` value. It is more common to pass a type +form object to a library function that knows how to decode and use such objects. -- No: - - :: - - maybe_float: TypeExpr = float | None - def sqrt(n: float) -> maybe_float: ... # ERROR: Can't use TypeExpr value in a type annotation - -- Okay, but discouraged in Python 3.12+: - - :: - - MaybeFloat: TypeAlias = float | None - def sqrt(n: float) -> MaybeFloat: ... - -- Yes: - - :: - - type MaybeFloat = float | None - def sqrt(n: float) -> MaybeFloat: ... - -It is uncommon for a programmer to define their *own* function which accepts -a ``TypeExpr`` parameter or returns a ``TypeExpr`` value. Instead it is more common -for a programmer to pass a literal type expression to an *existing* function -accepting a ``TypeExpr`` input which was imported from a runtime type checker -library. - -For example the ``isassignable`` function from the ``trycast`` library +For example, the ``isassignable`` function in the ``trycast`` library can be used like Python's built-in ``isinstance`` function to check whether -a value matches the shape of a particular type. -``isassignable`` will accept *any* kind of type as an input because its input -is a ``TypeExpr``. By contrast ``isinstance`` only accepts a simple class object -(a ``type[]``) as input: +a value matches the shape of a particular type. ``isassignable`` accepts *any* +type form object as input. - Yes: @@ -728,7 +383,7 @@ is a ``TypeExpr``. By contrast ``isinstance`` only accepts a simple class object from trycast import isassignable - if isassignable(some_object, MyTypedDict): # OK: MyTypedDict is a TypeExpr[] + if isassignable(some_object, MyTypedDict): # OK: MyTypedDict is a TypeForm[] ... - No: @@ -738,91 +393,80 @@ is a ``TypeExpr``. By contrast ``isinstance`` only accepts a simple class object if isinstance(some_object, MyTypedDict): # ERROR: MyTypedDict is not a type[] ... -There are :ref:`many other runtime type checkers ` -providing useful functions that accept a ``TypeExpr``. - - -.. _advanced_examples: Advanced Examples ================= -If you want to write your own runtime type checker or some other -kind of function that manipulates types as values at runtime, -this section gives examples of how you might implement such a function -using ``TypeExpr``. +If you want to write your own runtime type checker or a function that +manipulates type form objects as values at runtime, this section provides +examples of how such a function can use ``TypeForm``. -Introspecting TypeExpr Values ------------------------------ +Introspecting type form objects +------------------------------- -A ``TypeExpr`` is very similar to an ``object`` at runtime, with no additional -attributes or methods defined. - -You can use existing introspection functions like ``typing.get_origin`` and -``typing.get_args`` to extract the components of a type expression that looks -like ``Origin[Arg1, Arg2, ..., ArgN]``: +Functions like ``typing.get_origin`` and ``typing.get_args`` can be used to +extract components of some type form objects. :: import typing - def strip_annotated_metadata(typx: TypeExpr[T]) -> TypeExpr[T]: + def strip_annotated_metadata(typx: TypeForm[T]) -> TypeForm[T]: if typing.get_origin(typx) is typing.Annotated: - typx = cast(TypeExpr[T], typing.get_args(typx)[0]) + typx = cast(TypeForm[T], typing.get_args(typx)[0]) return typx -You can also use ``isinstance`` and ``is`` to distinguish one kind of -type expression from another: +``isinstance`` and ``is`` can also be used to distinguish between different +kinds of type form objects: :: import types import typing - def split_union(typx: TypeExpr) -> tuple[TypeExpr, ...]: - if isinstance(typx, types.UnionType): # X | Y - return cast(tuple[TypeExpr, ...], typing.get_args(typx)) - if typing.get_origin(typx) is typing.Union: # Union[X, Y] - return cast(tuple[TypeExpr, ...], typing.get_args(typx)) - if typx in (typing.Never, typing.NoReturn,): + def split_union(typx: TypeForm) -> tuple[TypeForm, ...]: + if isinstance(typ, types.UnionType): # X | Y + return cast(tuple[TypeForm, ...], typing.get_args(typ)) + if typing.get_origin(typ) is typing.Union: # Union[X, Y] + return cast(tuple[TypeForm, ...], typing.get_args(typ)) + if typ in (typing.Never, typing.NoReturn,): return () - return (typx,) + return (typ,) Combining with a type variable ------------------------------ -``TypeExpr[]`` can be parameterized by a type variable that is used elsewhere within -the same function definition: - -:: - - def as_instance[T](typx: TypeExpr[T]) -> T | None: - return typx() if isinstance(typx, type) else None - - -Combining with type[] ---------------------- - -Both ``TypeExpr[]`` and ``type[]`` can be parameterized by the same type -variable within the same function definition: - -:: - - def as_type[T](typx: TypeExpr[T]) -> type[T] | None: - return typx if isinstance(typx, type) else None - - -Combining with TypeIs[] and TypeGuard[] ---------------------------------------- - -A type variable parameterizing a ``TypeExpr[]`` can also be used by a ``TypeIs[]`` +``TypeForm`` can be parameterized by a type variable that is used elsewhere within the same function definition: :: - def isassignable[T](value: object, typx: TypeExpr[T]) -> TypeIs[T]: ... + def as_instance[T](typx: TypeForm[T]) -> T | None: + return typ() if isinstance(typ, type) else None + + +Combining with ``type`` +----------------------- + +Both ``TypeForm`` and ``type`` can be parameterized by the same type +variable within the same function definition: + +:: + + def as_type[T](typx: TypeForm[T]) -> type[T] | None: + return typ if isinstance(typ, type) else None + + +Combining with ``TypeIs`` and ``TypeGuard`` +------------------------------------------- + +A type variable can also be used by a ``TypeIs`` or ``TypeGuard`` return type: + +:: + + def isassignable[T](value: object, typx: TypeForm[T]) -> TypeIs[T]: ... count: int | str = ... if isassignable(count, int): @@ -830,59 +474,32 @@ within the same function definition: else: assert_type(count, str) -or by a ``TypeGuard[]`` within the same function definition: -:: - - def isdefault[T](value: object, typx: TypeExpr[T]) -> TypeGuard[T]: - return (value == typx()) if isinstance(typx, type) else False - - value: int | str = '' - if isdefault(value, int): - assert_type(value, int) - assert 0 == value - elif isdefault(value, str): - assert_type(value, str) - assert '' == value - else: - assert_type(value, int | str) - - -Challenges When Accepting All TypeExprs +Challenges When Accepting All TypeForms --------------------------------------- -A function that takes an *arbitrary* ``TypeExpr`` as -input must support a large variety of possible type expressions and is -not easy to write. Some challenges faced by such a function include: +A function that takes an *arbitrary* ``TypeForm`` as input must support a +variety of possible type form objects. Such functions are not easy to write. -- An ever-increasing number of typing special forms are introduced with - each new Python version which must be recognized, with special - handling required for each one. -- Stringified type annotations [#strann_less_common]_ (like ``'list[str]'``) - must be *parsed* (to something like ``typing.List[str]``) to be introspected. - - - In practice it is extremely difficult for stringified type - annotations to be handled reliably at runtime, so runtime type - checkers may opt to not support them at all. - -- Resolving string-based forward references inside type - expressions to actual values must typically be done using ``eval()``, - which is difficult/impossible to use in a safe way. -- Recursive types like ``IntTree = list[typing.Union[int, 'IntTree']]`` - are not possible to fully resolve. -- Supporting user-defined generic types (like Django’s - ``QuerySet[User]``) requires user-defined functions to - recognize/parse, which a runtime type checker should provide a - registration API for. +- New special forms are introduced with each new Python version, and + special handling may be required for each one. +- Quoted annotations [#quoted_less_common]_ (like ``'list[str]'``) + must be *parsed* (to something like ``list[str]``). +- Resolving quoted forward references inside type expressions is typically + done with ``eval()``, which is difficult to use in a safe way. +- Recursive types like ``IntTree = list[int | 'IntTree']`` are difficult + to resolve. +- User-defined generic types (like Django’s ``QuerySet[User]``) can introduce + non-standard behaviors that require runtime support. Reference Implementation ======================== -The following will be true when -`mypy#9773 `__ is implemented: +Pyright (version 1.1.379) provides a reference implementation for ``TypeForm``. - The mypy type checker supports ``TypeExpr`` types. +Mypy contributors also `plan to implement `__ +support for ``TypeForm``. A reference implementation of the runtime component is provided in the ``typing_extensions`` module. @@ -891,81 +508,58 @@ A reference implementation of the runtime component is provided in the Rejected Ideas ============== -.. _widen_type_C_to_support_all_type_expressions: +Alternative names +----------------- -Widen type[C] to support all type expressions ---------------------------------------------- +Alternate names were considered for ``TypeForm``. ``TypeObject`` +and ``TypeType`` were deemed too generic. ``TypeExpression`` and ``TypeExpr`` +were also considered, but these were considered confusing because these objects +are not themselves "expressions" but rather the result of evaluating a type +expression. -``type`` was `designed`_ to only be used to describe class objects. A -class object can always be used as the second argument of ``isinstance()`` -and can usually be instantiated by calling it. -``TypeExpr`` on the other hand is typically introspected by the user in -some way, is not necessarily directly instantiable, and is not -necessarily directly usable in a regular ``isinstance()`` check. +Widen ``type[C]`` to support all type expressions +------------------------------------------------- -It would be possible to widen ``type`` to include the additional values -allowed by ``TypeExpr`` but it would reduce clarity about the user’s -intentions when working with a ``type``. Different concepts and usage -patterns; different spellings. +``type`` was `designed`_ to describe class objects, subclasses of the +``type`` class. A value with the type ``type`` is assumed to be instantiable +through a constructor call. Widening the meaning of ``type`` to represent +arbitrary type form objects would present backward compatibility problems +and would eliminate a way to describe the set of values limited to subclasses +of ``type``. .. _designed: https://mail.python.org/archives/list/typing-sig@python.org/message/D5FHORQVPHX3BHUDGF3A3TBZURBXLPHD/ -.. _accept_arbitrary_annotation_expressions: - Accept arbitrary annotation expressions --------------------------------------- -Certain typing special forms can be used in *some* but not *all* -annotation contexts: +Certain special forms act as type qualifiers and can be used in +*some* but not *all* annotation contexts: -For example ``Final[]`` can be used as a variable type but not as a -parameter type or a return type: +For example. the type qualifier ``Final`` can be used as a variable type but +not as a parameter type or a return type: :: some_const: Final[str] = ... # OK - def foo(not_reassignable: Final[object]): ... # ERROR: Final[] not allowed here + def foo(not_reassignable: Final[object]): ... # Error: Final not allowed here - def nonsense() -> Final[object]: ... # ERROR: Final[] not meaningful here + def nonsense() -> Final[object]: ... # Error: Final not alowed here -``TypeExpr[T]`` does not allow matching such annotation expressions -because it is not clear what it would mean for such an expression -to parameterized by a type variable in position ``T``: +With the exception of ``Annotated``, type qualifiers are not allowed in type +expressions. ``TypeForm`` is limited to type expressions because its +assignability rules are based on the assignability rules for types. It is +nonsensical to ask whether ``Final[int]`` is assignable to ``int`` because the +former is not a valid type expression. -:: - - def ismatch[T](value: object, typx: TypeExpr[T]) -> TypeGuard[T]: ... - - def foo(some_arg): - if ismatch(some_arg, Final[int]): # ERROR: Final[int] is not a TypeExpr - reveal_type(some_arg) # ? NOT Final[int], because invalid for a parameter - -Functions that wish to operate on *all* kinds of annotation expressions, -including those that are not ``TypeExpr``\ s, can continue to accept such -inputs as ``object`` parameters, as they must do so today. +Functions that wish to operate on objects that are evaluated from annotation +expressions can continue to accept such inputs as ``object`` parameters. -Accept only universal type expressions --------------------------------------- - -Earlier drafts of this PEP only allowed ``TypeExpr[]`` to match the subset -of type expressions which are valid in *all* contexts, excluding -:ref:`non-universal type expressions `. -However doing that would effectively -create a new subset of annotation expressions that Python typing users -would have to understand, on top of all the existing distinctions between -“class objects”, “type expressions”, and “annotation expressions”. - -To avoid introducing yet another concept that everyone has to learn, -this proposal just rounds ``TypeExpr[]`` to exactly match the existing -definition of a “type expression”. - - -Support pattern matching on type expressions --------------------------------------------- +Pattern matching on type forms +------------------------------ It was asserted that some functions may wish to pattern match on the interior of type expressions in their signatures. @@ -977,71 +571,53 @@ Consider the following possible pattern matching syntax: :: @overload - def checkcast(typx: TypeExpr[AT=Annotated[T, *Anns]], value: str) -> T: ... + def checkcast(typx: TypeForm[AT=Annotated[T, *A]], value: str) -> T: ... @overload - def checkcast(typx: TypeExpr[UT=Union[*Ts]], value: str) -> Union[*Ts]: ... + def checkcast(typx: TypeForm[UT=Union[*Ts]], value: str) -> Union[*Ts]: ... @overload def checkcast(typx: type[C], value: str) -> C: ... # ... (more) -All functions observed in the wild that conceptually take a ``TypeExpr[]`` -generally try to support *all* kinds of type expressions, so it doesn’t -seem valuable to enumerate a particular subset. +All functions observed in the wild that conceptually accept type form +objects generally try to support *all* kinds of type expressions, so it +doesn’t seem valuable to enumerate a particular subset. -Additionally the above syntax isn’t precise enough to fully describe the -actual input constraints for a typical function in the wild. For example -many functions recognize un-stringified type expressions like -``list[Movie]`` but may not recognize type expressions with stringified -subcomponents like ``list['Movie']``. +Additionally, the above syntax isn’t precise enough to fully describe the +input constraints for a typical function in the wild. For example, many +functions do not support type expressions with quoted subexpressions +like ``list['Movie']``. -A second use case for pattern matching on the interior of type -expressions is to explicitly match an ``Annotated[]`` form to pull out the -interior type argument and strip away the metadata: +A second use case for pattern matching is to explicitly match an ``Annotated`` +form to extract the interior type argument and strip away any metadata: :: def checkcast( - typx: TypeExpr[T] | TypeExpr[AT=Annotated[T, *Anns]], + typx: TypeForm[T] | TypeForm[AT=Annotated[T, *A]], value: object ) -> T: -However ``Annotated[T, metadata]`` is already treated equivalent to ``T`` anyway. -There’s no additional value in being explicit about this behavior. -The example above could be more-straightforwardly written as the equivalent: +However, ``Annotated[T, metadata]`` is already treated equivalent to ``T`` +by static type checkers. There’s no additional value in being explicit about +this behavior. The example above could more simply be written as the equivalent: :: - def checkcast(typx: TypeExpr[T], value: object) -> T: - - -.. _recognize_uniontype_as_implicit_typeexpr_value: - -Recognize (T1 | T2) as an implicit TypeExpr value -------------------------------------------------- - -It would be nice if a value expression like ``int | str`` could be recognized -as an implicit ``TypeExpr`` value and be used directly in a context where a -``TypeExpr`` was expected. However making that possible would require making -changes to the rules that type checkers use for the ``|`` operator. These rules -are currently underspecified and would need to be make explicit first, -before making changes to them. The PEP author is not sufficiently motivated to -take on that specification work at the time of writing. + def checkcast(typx: TypeForm[T], value: object) -> T: Footnotes ========= -.. [#type_c] - :pep:`Type[C] spells a class object <484#the-type-of-class-objects>` +.. [#type_t] + :ref:`Type[T] ` spells a class object -.. [#TypeIsPep] - :pep:`TypeIs[T] is similar to bool <742>` +.. [#TypeIs] + :ref:`TypeIs[T] ` is similar to bool .. [#DataclassInitVar] - ``dataclass.make_dataclass`` accepts ``InitVar[...]`` as a special case - in addition to type expressions. Therefore it may unfortunately be necessary - to continue annotating its ``type`` parameter as ``object`` rather - than ``TypeExpr``. + ``dataclass.make_dataclass`` allows the type qualifier ``InitVar[...]``, + so ``TypeForm`` cannot be used in this case. .. [#forward_ref_normalization] Special forms normalize string arguments to ``ForwardRef`` instances @@ -1049,12 +625,11 @@ Footnotes Runtime type checkers may wish to implement similar functions when working with string-based forward references. -.. [#strann_less_common] - Stringified type annotations are expected to become less common - starting in Python 3.14 when :pep:`deferred annotations <649>` - become available. However there is a large amount of existing code from - earlier Python versions relying on stringified type annotations that will - still need to be supported for several years. +.. [#quoted_less_common] + Quoted annotations are expected to become less common starting in Python + 3.14 when :pep:`deferred annotations <649>` is implemented. However, + code written for earlier Python versions relies on quoted annotations and + will need to be supported for several years. Copyright