PEP 746: Type checking Annotated metadata (#3785)

Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>

.github/CODEOWNERS

@@ -624,6 +624,7 @@ peps/pep-0742.rst  @JelleZijlstra
 peps/pep-0743.rst  @vstinner
 peps/pep-0744.rst  @brandtbucher
 peps/pep-0745.rst  @hugovk
+peps/pep-0746.rst  @jellezijlstra
 # ...
 # peps/pep-0754.rst
 # ...

peps/pep-0746.rst

@@ -0,0 +1,149 @@
+PEP: 746
+Title: Type checking Annotated metadata
+Author: Adrian Garcia Badaracco <adrian@adriangb.com>
+Sponsor: Jelle Zijlstra <jelle.zijlstra@gmail.com>
+Discussions-To: https://discuss.python.org/t/pep-746-typedmetadata-for-type-checking-of-pep-593-annotated/53834
+Status: Draft
+Type: Standards Track
+Topic: Typing
+Created: 20-May-2024
+Python-Version: 3.14
+Post-History: 20-May-2024
+
+Abstract
+========
+
+This PEP proposes a mechanism for type checking metadata that uses
+the :py:data:`typing.Annotated` type. Metadata objects that implement
+the new ``__supports_type__`` protocol will be type checked by static
+type checkers to ensure that the metadata is valid for the given type.
+
+Motivation
+==========
+
+:pep:`593` introduced ``Annotated`` as a way to attach runtime metadata to types.
+In general, the metadata is not meant for static type checkers, but even so,
+it is often useful to be able to check that the metadata makes sense for the given
+type.
+
+Take the first example in :pep:`593`, which uses ``Annotated`` to attach
+serialization information to a field::
+
+   class Student(struct2.Packed):
+       name: Annotated[str, struct2.ctype("<10s")]
+
+Here, the ``struct2.ctype("<10s")`` metadata is meant to be used by a serialization
+library to serialize the field. Such libraries can only serialize a subset of types:
+it would not make sense to write, for example, ``Annotated[list[str], struct2.ctype("<10s")]``.
+Yet the type system provides no way to enforce this. The metadata are completely
+ignored by type checkers.
+
+This use case comes up in libraries like :pypi:`pydantic`, which use
+``Annotated`` to attach validation and conversion information to fields.
+
+Specification
+=============
+
+This PEP introduces a new ``__supports_type__`` protocol that both static and
+runtime type checkers can use to understand if a metadata object in
+``Annotated`` is valid for the given type. Objects that implement this protocol
+must have a method named ``__supports_type__`` that takes a single positional argument and
+returns ``bool``::
+
+    class Int64:
+        def __supports_type__(self, obj: int) -> bool:
+            return isinstance(obj, int)
+
+When a static type checker encounters a type expression of the form ``Annotated[T, M1, M2, ...]``,
+it should enforce that for each metadata element in ``M1, M2, ...``, one of the following is true:
+
+* The metadata element evaluates to a type that does not have a ``__supports_type__`` method; or
+* The metadata element evaluates to an object ``M`` that has a ``__supports_type__`` method, and
+  a call to ``M.__supports_type__(T)`` type checks without errors (i.e., ``T`` is assignable to the
+  parameter type of the ``__supports_type__`` method), and that call does not
+  evaluate to ``Literal[False]``.
+
+The body of the ``__supports_type__`` method is not used to check the validity of the metadata
+and static type checkers can ignore it. However, tools that use the annotation at
+runtime may call the method to check that a particular value is valid.
+
+For example, to support a generic ``Gt`` metadata, one might write::
+
+    from typing import Protocol
+
+    class SupportsGt[T](Protocol):
+        def __gt__(self, __other: T) -> bool:
+            ...
+
+    class Gt[T]:
+        def __init__(self, value: T) -> None:
+            self.value = value
+
+        def __supports_type__(self, obj: SupportsGt[T]) -> bool:
+            return obj > self.value
+
+    x1: Annotated[int, Gt(0)] = 1  # OK
+    x2: Annotated[str, Gt(0)] = 0  # type checker error: str is not assignable to SupportsGt[int]
+    x3: Annotated[int, Gt(1)] = 0  # OK for static type checkers; runtime type checkers may flag this
+
+Implementations may be generic and may use overloads that return ``Literal[True]`` or ``Literal[False]``
+to indicate if the metadata is valid for the given type.
+
+Backwards Compatibility
+=======================
+
+Metadata that does not implement the protocol will be considered valid for all types,
+so no breaking changes are introduced for existing code. The new checks only apply
+to metadata objects that explicitly implement the protocol specified by this PEP.
+
+Security Implications
+=====================
+
+None.
+
+How to Teach This
+=================
+
+This protocol is intended mostly for libraries that provide ``Annotated`` metadata;
+end users of those libraries are unlikely to need to implement the protocol themselves.
+The protocol should be mentioned in the documentation for :py:data:`typing.Annotated` and
+in the typing specification.
+
+Reference Implementation
+========================
+
+None yet.
+
+Rejected ideas
+==============
+
+Introducing a type variable instead of a generic class
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+We considered using a special type variable, ``AnnotatedT = TypeVar("AnnotatedT")``,
+to represent the type ``T`` of the inner type in ``Annotated``; metadata would be
+type checked against this type variable. However, this would require using the old
+type variable syntax (before :pep:`695`), which is now a discouraged feature.
+In addition, this would use type variables in an unusual way that does not fit well
+with the rest of the type system.
+
+Introducing a new type to ``typing.py`` that all metadata objects should subclass
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+A previous version of this PEP suggested adding a new generic base class, ``TypedMetadata[U]``,
+that metadata objects would subclass. If a metadata object is a subclass of ``TypedMetadata[U]``,
+then type checkers would check that the annotation's base type is assignable to ``U``.
+However, this mechanism does not integrate as well with the rest of the language; Python
+does not generally use marker base classes. In addition, it provides less flexibility than
+the current proposal: it would not allow overloads, and it would require metadata objects
+to add a new base class, which may make their runtime implementation more complex.
+
+Acknowledgments
+===============
+
+We thank Eric Traut for suggesting the idea of using a protocol.
+
+Copyright
+=========
+
+This document has been placed in the public domain.