python-peps/peps/pep-0746.rst

PEP: 746
Title: Type checking Annotated metadata
Author: Adrian Garcia Badaracco <adrian@adriangb.com>
Sponsor: Jelle Zijlstra <jelle.zijlstra@gmail.com>
Discussions-To: https://discuss.python.org/t/pep-746-typedmetadata-for-type-checking-of-pep-593-annotated/53834
Status: Draft
Type: Standards Track
Topic: Typing
Created: 20-May-2024
Python-Version: 3.14
Post-History: 20-May-2024

Abstract
========

This PEP proposes a mechanism for type checking metadata that uses
the :py:data:`typing.Annotated` type. Metadata objects that implement
the new ``__supports_type__`` protocol will be type checked by static
type checkers to ensure that the metadata is valid for the given type.

Motivation
==========

:pep:`593` introduced ``Annotated`` as a way to attach runtime metadata to types.
In general, the metadata is not meant for static type checkers, but even so,
it is often useful to be able to check that the metadata makes sense for the given
type.

Take the first example in :pep:`593`, which uses ``Annotated`` to attach
serialization information to a field::

   class Student(struct2.Packed):
       name: Annotated[str, struct2.ctype("<10s")]

Here, the ``struct2.ctype("<10s")`` metadata is meant to be used by a serialization
library to serialize the field. Such libraries can only serialize a subset of types:
it would not make sense to write, for example, ``Annotated[list[str], struct2.ctype("<10s")]``.
Yet the type system provides no way to enforce this. The metadata are completely
ignored by type checkers.

This use case comes up in libraries like :pypi:`pydantic` and :pypi:`msgspec`, which use
``Annotated`` to attach validation and conversion information to fields or :pypi:`fastapi`,
which uses ``Annotated`` to mark parameters as extracted from headers, query strings or
dependency injection.

Specification
=============
This PEP introduces a protocol that can be used by static and runtime type checkers to validate
the consistency between ``Annotated`` metadata and a given type.
Objects that implement this protocol have an attribute called ``__supports_type__``
that specifies whether the metadata is valid for a given type::

    class Int64:
        __supports_type__: int

The attribute may also be marked as a ``ClassVar`` to avoid interaction with dataclasses::
    
    from dataclasses import dataclass
    from typing import ClassVar

    @dataclass
    class Gt:
        value: int
        __supports_type__: ClassVar[int]

When a static type checker encounters a type expression of the form ``Annotated[T, M1, M2, ...]``,
it should enforce that for each metadata element in ``M1, M2, ...``, one of the following is true:

* The metadata element evaluates to an object that does not have a ``__supports_type__`` attribute; or
* The metadata element evaluates to an object ``M`` that has a ``__supports_type__`` attribute;
  and ``T`` is assignable to the type of ``M.__supports_type__``.

To support generic ``Gt`` metadata, one might write::

    from typing import Protocol

    class SupportsGt[T](Protocol):
        def __gt__(self, __other: T) -> bool:
            ...
    
    class Gt[T]:
        __supports_type__: ClassVar[SupportsGt[T]]

        def __init__(self, value: T) -> None:
            self.value = value

    x1: Annotated[int, Gt(0)] = 1  # OK
    x2: Annotated[str, Gt(0)] = 0  # type checker error: str is not assignable to SupportsGt[int]
    x3: Annotated[int, Gt(1)] = 0  # OK for static type checkers; runtime type checkers may flag this

Backwards Compatibility
=======================

Metadata that does not implement the protocol will be considered valid for all types,
so no breaking changes are introduced for existing code. The new checks only apply
to metadata objects that explicitly implement the protocol specified by this PEP.

Security Implications
=====================

None.

How to Teach This
=================

This protocol is intended mostly for libraries that provide ``Annotated`` metadata;
end users of those libraries are unlikely to need to implement the protocol themselves.
The protocol should be mentioned in the documentation for :py:data:`typing.Annotated` and
in the typing specification.

Reference Implementation
========================

None yet.

Rejected ideas
==============

Introducing a type variable instead of a generic class
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

We considered using a special type variable, ``AnnotatedT = TypeVar("AnnotatedT")``,
to represent the type ``T`` of the inner type in ``Annotated``; metadata would be
type checked against this type variable. However, this would require using the old
type variable syntax (before :pep:`695`), which is now a discouraged feature.
In addition, this would use type variables in an unusual way that does not fit well
with the rest of the type system.

Introducing a new type to ``typing.py`` that all metadata objects should subclass
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

A previous version of this PEP suggested adding a new generic base class, ``TypedMetadata[U]``,
that metadata objects would subclass. If a metadata object is a subclass of ``TypedMetadata[U]``,
then type checkers would check that the annotation's base type is assignable to ``U``.
However, this mechanism does not integrate as well with the rest of the language; Python
does not generally use marker base classes. In addition, it provides less flexibility than
the current proposal: it would not allow overloads, and it would require metadata objects
to add a new base class, which may make their runtime implementation more complex.

Using a method instead of an attribute for ``__supports_type__``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

We considered using a method instead of an attribute for the protocol, so that this method can be used
at runtime to check the validity of the metadata and to support overloads or returning boolean literals.
However, using a method adds boilerplate to the implementation and the value of the runtime use cases or
more complex scenarios involving overloads and returning boolean literals was not clear.

Acknowledgments
===============

We thank Eric Traut for suggesting the idea of using a protocol and implementing provisional support in Pyright.
Thank you to Jelle Zijlstra for sponsoring this PEP.

Copyright
=========

This document has been placed in the public domain.