python-peps/peps/pep-0749.rst

PEP: 749
Title: Implementing PEP 649
Author: Jelle Zijlstra <jelle.zijlstra@gmail.com>
Discussions-To: https://discuss.python.org/t/pep-749-implementing-pep-649/54974
Status: Draft
Type: Standards Track
Topic: Typing
Requires: 649
Created: 28-May-2024
Python-Version: 3.14
Post-History: `04-Jun-2024 <https://discuss.python.org/t/pep-749-implementing-pep-649/54974>`__


Abstract
========

This PEP supplements :pep:`649` by providing various tweaks and additions to its
specification:

* ``from __future__ import annotations`` (:pep:`563`) will continue to exist with
  its current behavior at least until Python 3.13 reaches its end-of-life. Subsequently,
  it will be deprecated and eventually removed.
* A new standard library module, ``annotationlib``, is added to provide tooling for
  annotations. It will include the ``get_annotations()`` function, an enum for annotation
  formats, a ``ForwardRef`` class, and a helper function for calling ``__annotate__`` functions.
* Annotations in the REPL are lazily evaluated, just like other module-level annotations.
* We specify the behavior of wrapper objects that provide annotations, such as :py:func:`classmethod`
  and code that uses :py:func:`functools.wraps`.
* There will not be a code flag for marking ``__annotate__`` functions
  that can be run in a "fake globals" environment.
* Setting the ``__annotations__`` attribute directly will not affect the ``__annotate__`` attribute.
* We add functionality to allow evaluating type alias values and type parameter bounds and defaults
  (which were added by :pep:`695` and :pep:`696`) using PEP 649-like semantics.

Motivation
==========

:pep:`649` provides an excellent framework for creating better semantics for
annotations in Python. It solves a common pain point for users of annotations,
including those using static type hints as well as those using runtime typing,
and it makes the language more elegant and powerful.
The PEP was originally proposed in 2021 for Python 3.10,
and it was accepted in 2023. However, the implementation took longer than anticipated,
and now the PEP is expected to be implemented in Python 3.14.

I have started working on the implementation of the PEP in CPython. I found that
the PEP leaves some areas underspecified, and some
of its decisions in corner cases are questionable. This new PEP proposes several
changes and additions to the specification to address these issues.

This PEP supplements rather than supersedes PEP 649. The changes proposed here
should make the overall user experience better, but they do not change the
general framework of the earlier PEP.


The future of ``from __future__ import annotations``
====================================================

:pep:`563` previously introduced the future import ``from __future__ import annotations``,
which changes all annotations to strings. :pep:`649` proposes an alternative approach
that does not require this future import, and states:

    If this PEP is accepted, PEP 563 will be deprecated and eventually removed. 

However, the PEP does not provide a detailed plan for this deprecation.

There is some previous discussion of this topic `on Discourse <https://discuss.python.org/t/pep-649-deferred-evaluation-of-annotations-tentatively-accepted/21331/44>`__
(note that in the linked post I proposed something different from what is proposed here).

Specification
-------------

We suggest the following deprecation plan:

- In Python 3.14, ``from __future__ import annotations`` will continue to work as it
  did before, converting annotations into strings.
- Sometime after the last release that did not support :pep:`649` semantics (expected to be 3.13)
  reaches its end-of-life, ``from __future__ import annotations`` is deprecated. Compiling
  any code that uses the future import will emit a :py:exc:`DeprecationWarning`. This will
  happen no sooner than the first release after Python 3.13 reaches its end-of-life, but
  the community may decide to wait longer.
- After at least two releases, the future import is removed, and annotations are always
  evaluated as per :pep:`649`. Code that continues to use the future import will raise
  a :py:exc:`SyntaxError`, similar to any other undefined future import.

Rejected alternatives
---------------------

*Immediately make the future import a no-op*: We considered applying :pep:`649` semantics
to all code in Python 3.14, making the future import a no-op. However, this would break
code that works in 3.13 under the following set of conditions:

* ``__future__ import annotations`` is active
* There are annotations that rely on forward references
* Annotations are eagerly evaluated at import time, for example by a metaclass or
  class or function decorator. For example, this currently applies to the
  released version of ``typing_extensions.TypedDict``.

This is expected to be a common pattern, so we cannot afford to break such code during
the upgrade from 3.13 to 3.14.

Such code would still break when the future import is eventually removed. However, this
is many years in the future, giving affected decorators plenty of time to update their code.

*Immediately deprecate the future import*: Instead of waiting until Python 3.13 reaches
its end-of-life, we could immediately start emitting warnings when the future import is
used. However, many libraries are already using ``from __future__ import annotations`` as
an elegant way to enable unrestricted forward references in their annotations. If we deprecate
the future import immediately, it would be impossible for these libraries to use unrestricted
forward references on all supported Python versions while avoiding deprecation warnings:
unlike other features deprecated from the standard library, a ``__future__`` import must
be the first statement in a given module, meaning it would be impossible to only
conditionally import ``__future__.annotations`` on Python 3.13 and lower. (The necessary
``sys.version_info`` check would count as a statement preceding the ``__future__`` import.)

*Keep the future import around forever*: We could also decide to keep the future import
indefinitely. However, this would permanently bifurcate the behavior of the Python
language. This is undesirable; the language should have only a single set of semantics,
not two permanently different modes.

*Make the future import a no-op in the future*: Instead of eventually making
``from __future__ import annotations`` a ``SyntaxError``, we could make it do nothing
instead at some point after Python 3.13 reaches its end-of-life. This still has some
of the same issues outlined above around making it a no-op now, although the ecosystem
would have had much longer to adapt. It is better to have users explicitly remove
the future import from their code in the future once they have confirmed they do not
rely on stringized annotations.

New ``annotationlib`` module
============================

:pep:`649` proposes to add tooling related to annotations to the :py:mod:`inspect`
module. However, that module is rather large, has direct or indirect dependencies
on at least 35 other standard library modules, and is so slow to import that other
standard library modules are often discouraged from importing it. Furthermore, we
anticipate adding more tools in addition to the :py:func:`inspect.get_annotations`
function and the ``VALUE``, ``FORWARDREF``, and ``SOURCE`` formats.

A new standard library module provides a logical home for this functionality and
also enables us to add more tooling that is useful for consumers of annotations.

Rationale
---------

:pep:`649` indicates that :py:class:`!typing.ForwardRef` should be used to implement the
``FORWARDREF`` format in :py:func:`inspect.get_annotations`. However, the existing implementation
of :py:class:`!typing.ForwardRef` is intertwined with the rest of the :py:mod:`!typing` module,
and it would not make sense to add :py:mod:`!typing`-specific behavior to the generic ``get_annotations()``
function. Furthermore, :py:class:`!typing.ForwardRef` is a problematic
class: it is public and documented, but the documentation lists no attributes or methods
for it. Nonetheless, third-party libraries make use of some of its undocumented
attributes. For instance, `Pydantic <https://github.com/pydantic/pydantic/blob/00ff77ed37589d924d3c10e0d5a48a7ef679a0d7/pydantic/v1/typing.py#L66>`__
and `Typeguard <https://github.com/agronholm/typeguard/blob/016f8139f5a0a63147d68df9558cc5584cd2c49a/src/typeguard/_utils.py#L44>`__
use the ``_evaluate`` method; `beartype <https://github.com/beartype/beartype/blob/0b4453f83c7ed4be054d8733aab8075e1478e166/beartype/_util/hint/pep/proposal/pep484585/utilpep484585ref.py#L210>`__
and `pyanalyze <https://github.com/quora/pyanalyze/blob/9e401724f9d035cf138b72612834b6d5a00eb8e8/pyanalyze/annotations.py#L509>`__
use the ``__forward_arg__`` attribute.

We replace the existing but poorly specified :py:class:`!typing.ForwardRef` with a new class,
``annotationlib.ForwardRef``. It is designed to be mostly compatible with existing uses
of the :py:class:`!typing.ForwardRef` class, but without the behaviors specific to the
:py:mod:`!typing` module. For compatibility with existing users, we keep the private
``_evaluate`` method, but mark it as deprecated. It delegates to a new public function in
the :py:mod:`!typing` module, ``typing.evaluate_forward_ref``, that is designed to
evaluate forward references in a way that is specific to type hints.

We add a function ``annotationlib.call_annotate_function`` as a helper for calling
``__annotate__`` functions. This is a useful building block when implementing functionality
that needs to partially evaluate annotations while a class is being constructed.
For example, the implementation of :py:class:`typing.NamedTuple` needs to retrieve
the annotations from a class namespace dictionary before the namedtuple class itself
can be constructed, because the annotations determine what fields exist on the namedtuple.

Specification
-------------

A new module, ``annotationlib``, is added to the standard library. Its aim is to
provide tooling for introspecting and wrapping annotations.

The exact contents of the module are not yet specified. We will add support for
:pep:`649` semantics to standard library functionality that uses annotations, such
as :py:mod:`dataclasses` and :py:class:`typing.TypedDict`, and use the experience
to inform the design of the new module.

The module will contain the following functionality:

* ``get_annotations()``: A function that returns the annotations of a function,
  module, or class. This will replace :py:func:`inspect.get_annotations`. The latter
  will delegate to the new function. It may eventually be deprecated, but to
  minimize disruption, we do not propose an immediate deprecation.
* ``get_annotate_function()``: A function that returns the ``__annotate__`` function
  of an object, if it has one, or ``None`` if it does not. This is usually equivalent
  to accessing the ``.__annotate__`` attribute, except in the presence of metaclasses
  (see :ref:`below <pep749-metaclasses>`).
* ``Format``: an enum that contains the possible formats of annotations. This will
  replace the ``VALUE``, ``FORWARDREF``, and ``SOURCE`` formats in :pep:`649`.
  PEP 649 proposed to make these values global members of the :py:mod:`inspect`
  module; we prefer to place them within an enum.
* ``ForwardRef``: a class representing a forward reference; it may be returned by
  ``get_annotations()`` when the format is ``FORWARDREF``. The existing class
  :py:class:`typing.ForwardRef` will become an alias of this class. Its members include:

  * ``__forward_arg__``: the string argument of the forward reference
  * ``evaluate(globals=None, locals=None, type_params=None, owner=None)``: a method that attempts to evaluate
    the forward reference. The ``ForwardRef`` object may hold a reference to the
    globals and other namespaces of the object that it originated from. If so, these
    namespaces may be used to evaluate the forward reference. The *owner* argument
    may be the object that holds the original annotation, such as the class or module
    object; it is used to extract the globals and locals namespaces if these are not
    provided.
  * ``_evaluate()``, with the same interface as the existing ``ForwardRef._evaluate``
    method. It will be undocumented and immediately deprecated. It is provided for
    compatibility with existing users of ``typing.ForwardRef``.

* ``call_annotate_function(func: Callable, format: Format)``: a helper for calling
  an ``__annotate__`` function with a given format. If the function does not support
  this format, ``call_annotate_function()`` will set up a "fake globals" environment,
  as described in :pep:`649`, and use that environment to return the desired annotations
  format.
* ``call_evaluate_function(func: Callable | None, format: Format)``: similar to
  ``call_annotate_function``, but does not rely on the function returning an annotations
  dictionary. This is intended to be used for evaluating deferred attributes introduced by
  :pep:`695` and :pep:`696`; see below for details. *func* may be ``None``
  for convenience; if ``None`` is passed, the function also returns ``None``.

A new function is also added to the :py:mod:`!typing` module, ``typing.evaluate_forward_ref``.
This function is a wrapper around the ``ForwardRef.evaluate`` method, but it performs
additional work that is specific to type hints. For example, it recurses into complex
types and evaluates additional forward references within these types.

Contrary to :pep:`649`, the annotation formats (``VALUE``, ``FORWARDREF``, and ``SOURCE``)
will not be added as global members of the :py:mod:`inspect` module. The only recommended
way to refer to these constants will be as ``annotationlib.Format.VALUE``.

Open issues
-----------

What should this module be called? Some ideas:

- ``annotations``: The most obvious name, but it may cause confusion with the existing
  ``from __future__ import annotations``, because users may have both ``import annotations``
  and ``from __future__ import annotations`` in the same module. The use of a common word
  as the name will make the module harder to search for. There is a PyPI package :pypi:`annotations`,
  but it had only a single release in 2015 and looks abandoned.
- ``annotools``: Analogous to :py:mod:`itertools` and :py:mod:`functools`, but "anno" is a less
  obvious abbreviation than "iter" or "func". As of this writing, there
  is no PyPI package with this name.
- ``annotationtools``: A more explicit version. There is a PyPI package
  :pypi:`annotationtools`, which had a release in 2023.
- ``annotation_tools``: A variation of the above but without a PyPI conflict. However,
  no other public standard library module has an underscore in its name.
- ``annotationslib``: Analogous to :py:mod:`tomllib`, :py:mod:`pathlib`, and :py:mod:`importlib`.
  There is no PyPI package with this name.
- ``annotationlib``: Similar to the above, but one character shorter and subjectively reads
  better. Also not taken on PyPI.

Rejected alternatives
---------------------

*Add the functionality to the inspect module*: As described above, the
:py:mod:`inspect` module is already quite large, and its import time is prohibitive
for some use cases.

*Add the functionality to the typing module*: While annotations are mostly
used for typing, they may also be used for other purposes. We prefer to keep a clean
separation between functionality for introspecting annotations and functionality that
is exclusively meant for type hints.

*Add the functionality to the types module*: The :py:mod:`types` module is
meant for functionality related to *types*, and annotations can exist on functions
and modules, not only on types.

*Develop this functionality in a third-party package*: The functionality in this new
module will be pure Python code, and it is possible to implement a third-party package
that provides the same functionality by interacting directly with ``__annotate__``
functions generated by the interpreter. However, the functionality of the proposed new
module will certainly be useful in the standard library itself (e.g., for implementing
:py:mod:`dataclasses` and :py:class:`typing.NamedTuple`), so it makes sense to include
it in the standard library.

*Add this functionality to a private module*: It would be possible to initially develop
the module in a private standard library module (e.g., ``_annotations``), and publicize
it only after we have gained more experience with the API. However, we already know
that we will need parts of this module for the standard library itself (e.g., for
implementing :py:mod:`!dataclasses` and :py:class:`!typing.NamedTuple`). Even if we make
it private, the module will inevitably get used by third-party users. It is preferable
to start with a clear, documented API from the beginning, to enable third-party users to
support :pep:`649` semantics as thoroughly as the standard library. The module will
immediately be used in other parts of the standard library, ensuring that it covers a
reasonable set of use cases.

Behavior of the REPL
====================

:pep:`649` specifies the following behavior of the interactive REPL:

    For the sake of simplicity, in this case we forego delayed evaluation.
    Module-level annotations in the REPL shell will continue to work exactly
    as they do with “stock semantics”, evaluating immediately and setting the
    result directly inside the ``__annotations__`` dict.

There are several problems with this proposed behavior. It makes the REPL the
only context where annotations are still evaluated immediately, which is
confusing for users and complicates the language.

It also makes the implementation of the REPL more complex, as it needs to
ensure that all statements are compiled in "interactive" mode, even if their
output does not need to be displayed. (This matters if there are multiple
statements in a single line evaluated by the REPL.)

Most importantly, this breaks some plausible use cases that inexperienced
users could run into. A user might write the following in a file::

    a: X | None = None
    class X: ...

Under :pep:`649` this would work fine: ``X`` is not yet defined when it is used
in the annotation for ``a``, but the annotation is lazily evaluated. However,
if a user were to paste this same code into the REPL and execute it line by
line, it would throw a ``NameError``, because the name ``X`` is not yet defined.

This topic was previously discussed `on Discourse <https://discuss.python.org/t/pep-649-behavior-of-the-repl/54109>`__.

Specification
-------------

We propose to treat the interactive console like any other module-level code, and
make annotations lazily evaluated. This makes the language more consistent and
avoids subtle behavior changes between modules and the REPL.

Because the REPL is evaluated line by line, we would generate a new ``__annotate__``
function for every evaluated statement in the global scope that contains annotations. Whenever a line
containing annotations is evaluated, the previous ``__annotate__`` function is
lost:

.. code:: pycon

    >>> x: int
    >>> __annotate__(1)
    {'x': <class 'int'>}
    >>> y: str
    >>> __annotate__(1)
    {'y': <class 'str'>}
    >>> z: doesntexist
    >>> __annotate__(1)
    Traceback (most recent call last):
    File "<python-input-5>", line 1, in <module>
        __annotate__(1)
        ~~~~~~~~~~~~^^^
    File "<python-input-4>", line 1, in __annotate__
        z: doesntexist
           ^^^^^^^^^^^
    NameError: name 'doesntexist' is not defined

There will be no ``__annotations__`` key in the global namespace of the REPL.
In module namespaces, this key is created lazily when the ``__annotations__``
descriptor of the module object is accessed, but in the REPL there is no such module
object.

Classes and functions defined within the REPL will also work like any other classes,
so evaluation of their annotations will be deferred. It is possible to access the
``__annotations__`` and ``__annotate__`` attributes or use the ``annotationlib`` module
to introspect the annotations.

Wrappers that provide ``__annotations__``
=========================================

Several objects in the standard library and elsewhere provide annotations for their
wrapped object. :pep:`649` does not specify how such wrappers should behave.

Specification
-------------

Wrappers that provide annotations should be designed with the following goals
in mind:

* Evaluation of ``__annotations__`` should be deferred for as long as possible,
  consistent with the behavior of built-in functions, classes, and modules.
* Backward compatibility with the behavior prior to the implementation of :pep:`649`
  should be preserved.
* The ``__annotate__`` and ``__annotations__`` attributes should both be supplied
  with semantics consistent to those of the wrapped object.

More specifically:

* :py:func:`functools.update_wrapper` (and therefore :py:func:`functools.wraps`)
  will copy only the ``__annotate__`` attribute
  from the wrapped object to the wrapper. The ``__annotations__`` descriptor on the
  wrapper function will use the copied ``__annotate__``.
* The constructors for :py:func:`classmethod` and :py:func:`staticmethod` currently
  copy the ``__annotations__`` attribute from the wrapped object to the wrapper.
  They will instead have writable attributes for
  ``__annotate__`` and ``__annotations__``. Reading these attributes will retrieve
  the corresponding attribute from the underlying callable and cache it in the wrapper's
  ``__dict__``. Writing to these attributes will directly update the ``__dict__``,
  without affecting the wrapped callable.

.. _pep749-metaclasses:

Annotations and metaclasses
===========================

Testing of the initial implementation of this PEP revealed serious problems with
the interaction between metaclasses and class annotations.

Pre-existing bugs
-----------------

We found several bugs in the existing behavior of ``__annotations__`` on classes
while investigating the behaviors to be specified in this PEP. Fixing these bugs
on Python 3.13 and earlier is outside the scope of this PEP, but they are noted here
to explain the corner cases that need to be dealt with.

For context, on Python 3.10 through 3.13 the ``__annotations__`` dictionary is
placed in the class namespace if the class has any annotations. If it does not,
there is no ``__annotations__`` class dictionary key when the class is created,
but accessing ``cls.__annotations__`` invokes a descriptor defined on ``type``
that returns an empty dictionary and stores it in the class dictionary.
:py:ref:`Static types <static-types>` are an exception: they never have
annotations, and accessing ``.__annotations__`` raises :py:exc:`AttributeError`.
On Python 3.9 and earlier, the behavior was different; see
`gh-88067 <https://github.com/python/cpython/issues/88067>`__.

The following code fails identically on Python 3.10 through 3.13::

    class Meta(type): pass

    class X(metaclass=Meta):
        a: str

    class Y(X): pass

    Meta.__annotations__  # important
    assert Y.__annotations__ == {}, Y.__annotations__  # fails: {'a': <class 'str'>}

If the annotations on the metaclass ``Meta`` are accessed before the annotations
on ``Y``, then the annotations for the base class ``X`` are leaked to ``Y``.
However, if the metaclass's annotations are *not* accessed (i.e., the line ``Meta.__annotations__``
above is removed), then the annotations for ``Y`` are correctly empty.

Similarly, annotations from annotated metaclasses leak to unannotated
classes that are instances of the metaclass::

    class Meta(type):
        a: str

    class X(metaclass=Meta):
        pass

    assert X.__annotations__ == {}, X.__annotations__  # fails: {'a': <class 'str'>}

The reason for these behaviors is that if the metaclass contains an
``__annotations__`` entry in its class dictionary, this prevents
instances of the metaclass from using the ``__annotations__`` data descriptor
on the base :py:class:`type` class. In the first case, accessing ``Meta.__annotations__``
sets ``Meta.__dict__["__annotations__"] = {}`` as a side effect. Then, looking
up the ``__annotations__`` attribute on ``Y`` first sees the metaclass attribute,
but skips it because it is a data descriptor. Next, it looks in the class dictionaries
of the classes in its method resolution order (MRO), finds ``X.__annotations__``,
and returns it. In the second example, there are no annotations
anywhere in the MRO, so ``type.__getattribute__`` falls back to
returning the metaclass attribute.

Metaclass behavior with PEP 649
-------------------------------

With :pep:`649`, the behavior of accessing the ``.__annotations__`` attribute
on classes when metaclasses are involved becomes even more erratic, because now
``__annotations__`` is only lazily added to the class dictionary even for classes
with annotations. The new ``__annotate__`` attribute is also lazily created
on classes without annotations, which causes further misbehaviors when
metaclasses are involved.

The cause of these problems is that we set the ``__annotate__`` and ``__annotations__``
class dictionary entries only under some circumstances, and rely on descriptors
defined on :py:class:`type` to fill them in if they are not set. When normal
attribute lookup is used, this approach breaks down in the presence of
metaclasses, because entries in the metaclass's own class dictionary can render
the descriptors invisible.

While we considered several approaches that would allow ``cls.__annotations__``
and ``cls.__annotate__`` to work reliably when ``cls`` is a type with a custom
metaclass, any such approach would expose significant complexity to advanced users.
Instead, we recommend a simpler approach that confines the complexity to the
``annotationlib`` module: in ``annotationlib.get_annotations``, we bypass normal
attribute lookup by using the ``type.__annotations__`` descriptor directly.

Specification
-------------

Users should always use ``annotationlib.get_annotations`` to access the
annotations of a class object, and ``annotationlib.get_annotate_function``
to access the ``__annotate__`` function. These functions will return only
the class's own annotations, even when metaclasses are involved.

The behavior of accessing the ``__annotations__`` and ``__annotate__``
attributes on classes with a metaclass other than ``builtins.type`` is
unspecified. The documentation should warn against direct use of these
attributes and recommend using the ``annotationlib`` module instead.

Similarly, the presence of ``__annotations__`` and ``__annotate__`` keys
in the class dictionary is an implementation detail and should not be relied
upon.

Rejected alternatives
---------------------

We considered two broad approaches for dealing with the behavior
of the ``__annotations__`` and ``__annotate__`` entries in classes:

* Ensure that the entry is *always* present in the class dictionary, even if it
  is empty or has not yet been evaluated. This means we do not have to rely on
  the descriptors defined on :py:class:`type` to fill in the field, and
  therefore the metaclass's attributes will not interfere. (Prototype
  in `gh-120719 <https://github.com/python/cpython/pull/120719>`__.)
* Ensure that the entry is *never* present in the class dictionary, or at least
  never added by logic in the language core. This means that the descriptors
  on :py:class:`type` will always be used, without interference from the metaclass.
  (Prototype in `gh-120816 <https://github.com/python/cpython/pull/120816>`__.)

Alex Waygood suggested an implementation using the first approach. When a
heap type (such as a class created through the ``class`` statement) is created,
``cls.__dict__["__annotations__"]`` is set to a special descriptor.
On ``__get__``, the descriptor evaluates the annotations by calling ``__annotate__``
and returning the result. The annotations dictionary is cached within the
descriptor instance. The descriptor also behaves like a mapping,
so that code that uses ``cls.__dict__["__annotations__"]`` will still usually
work: treating the object as a mapping will evaluate the annotations and behave
as if the descriptor itself was the annotations dictionary. (Code that assumes
that ``cls.__dict__["__annotations__"]`` is specifically an instance of ``dict``
may break, however.)

This approach is also straightforward to implement for ``__annotate__``: this
attribute is already always set for classes with annotations, and we can set
it explicitly to ``None`` for classes without annotations.

While this approach would fix the known edge cases with metaclasses, it
introduces significant complexity to all classes, including a new built-in type
(for the annotations descriptor) with unusual behavior.

The alternative approach would be to never set ``__dict__["__annotations__"]``
and use some other storage to store the cached annotations. This behavior
change would have to apply even to classes defined under
``from __future__ import annotations``, because otherwise there could be buggy
behavior if a class is defined without ``from __future__ import annotations``
but its metaclass does have the future enabled. As :pep:`649` previously noted,
removing ``__annotations__`` from class dictionaries also has backwards compatibility
implications: ``cls.__dict__.get("__annotations__")`` is a common idiom to
retrieve annotations.

This approach would also mean that accessing ``.__annotations__`` on an instance
of an annotated class no longer works. While this behavior is not documented,
it is a long-standing feature of Python and is relied upon by some users.

Remove code flag for marking ``__annotate__`` functions
=======================================================

:pep:`649` specifies:

    This PEP assumes that
    third-party libraries may implement their own ``__annotate__``
    methods, and those functions would almost certainly work
    incorrectly when run in this "fake globals" environment.
    For that reason, this PEP allocates a flag on code objects,
    one of the unused bits in ``co_flags``, to mean "This code
    object can be run in a 'fake globals' environment."  This
    makes the "fake globals" environment strictly opt-in, and
    it's expected that only ``__annotate__`` methods generated
    by the Python compiler will set it.

We have not found a need for this mechanism during our work to
add :pep:`649` support to the standard library. While it is true
that custom ``__annotate__`` functions may not work well with the
"fake globals" environment, this technique is used only when the
``__annotate__`` function raises :py:exc:`NotImplementedError` to
signal that it does not support the requested format. However,
manually implemented ``__annotate__`` functions are likely to support
all three annotation formats; often, they will consist of a call to
``annotationlib.call_annotate_function`` plus some transformation of the
result.

In addition, the proposed mechanism couples the implementation with
low-level details of the code object. The code object flags are
CPython-specific and the documentation :py:ref:`explicitly warns <inspect-module-co-flags>`
against relying on the values.

Specification
-------------

The standard library will use the "fake globals" technique on any
``__annotate__`` function that raises :py:exc:`NotImplementedError`
when the requested format is not supported.

Third-party code that implements ``__annotate__`` functions should either
support all three annotation formats, or be prepared to handle the
"fake globals" environment. This should be mentioned in the data model
documentation for ``__annotate__``.

Effect of setting ``__annotations__``
=====================================

:pep:`649` specifies:

  Setting ``o.__annotations__`` to a legal value
  automatically sets ``o.__annotate__`` to ``None``.

We would prefer to keep ``__annotate__`` unchanged when ``__annotations__``
is written to. Conceptually, ``__annotate__`` provides the ground truth
and ``__annotations__`` is merely a cache, and we shouldn't throw away the
ground truth if the cache is modified.

The motivation for :pep:`649`'s behavior is to keep the two attributes in sync.
However, this is impossible in general; if the ``__annotations__`` dictionary
is modified in place, this will not be reflected in the ``__annotate__`` attribute.
The overall mental model for this area will be simpler if setting ``__annotations__``
has no effect on ``__annotate__``.

Specification
-------------

The value of ``__annotate__`` is not changed when ``__annotations__`` is set.

Deferred evaluation of PEP 695 and 696 objects
==============================================

Since :pep:`649` was written, Python 3.12 and 3.13 gained support for
several new features that also use deferred evaluation, similar to the
behavior this PEP proposes for annotations:

* The value of type aliases created through the :py:keyword:`type`
  statement (:pep:`695`)
* The bound and constraints of :py:class:`typing.TypeVar` objects
  created through the syntax for generics (:pep:`695`)
* The default value of :py:class:`typing.TypeVar`, :py:class:`ParamSpec`,
  and :py:class:`typing.TypeVarTuple` objects (:pep:`696`)

Currently, these objects use deferred evaluation, but there is no direct
access to the function object used for deferred evaluation. To enable
the same kind of introspection that is now possible for annotations, we propose
to expose the internal function objects, allowing users to evaluate them
using the FORWARDREF and SOURCE formats.

Specification
-------------

We will add the following new attributes:

* ``evaluate_value`` on :py:class:`typing.TypeAliasType`
* ``evaluate_bound``, ``evaluate_constraints``, and ``evaluate_default`` on :py:class:`typing.TypeVar`
* ``evaluate_default`` on :py:class:`typing.ParamSpec`
* ``evaluate_default`` on :py:class:`typing.TypeVarTuple`

Except for ``evaluate_value``, these attributes may be ``None`` if the object
does not have a bound, constraints, or default. Otherwise, the attribute is a
callable, similar to an ``__annotate__`` function, that takes a single integer
argument and returns the evaluated value. Unlike ``__annotate__`` functions,
these callables return a single value, not a dictionary of annotations.
These attributes are read-only.

Usually, users would use these attributes in combinations with
``annotationlib.call_evaluate_function``. For example, to get a ``TypeVar``'s bound
in SOURCE format, one could write
``annotationlib.call_evaluate_function(T.evaluate_bound, annotationlib.Format.SOURCE)``.

Miscellaneous implementation details
====================================

:pep:`649` goes into considerable detail on some aspects of the implementation.
To avoid confusion, we describe a few aspects where the current implementation
differs from that described in the PEP. However, these details are not guaranteed
to hold in the future, and they may change without notice in the future, unless
they are documented in the language reference.

Supported operations on ``ForwardRef`` objects
----------------------------------------------

The ``SOURCE`` format is implemented by the "stringizer" technique,
where the globals dictionary of a function is augmented so that every
lookup results in a special object that can be used to reconstruct the
operations that are performed on the object.

:pep:`649` specifies:

    In practice, the "stringizer" functionality will be implemented
    in the ``ForwardRef`` object currently defined in the
    ``typing`` module.  ``ForwardRef`` will be extended to
    implement all stringizer functionality; it will also be
    extended to support evaluating the string it contains,
    to produce the real value (assuming all symbols referenced
    are defined).

However, this is likely to lead to confusion in practice. An object
that implements stringizer functionality must implement almost all
special methods, including ``__getattr__`` and ``__eq__``, to return
a new stringizer. Such an object is confusing to work with: all operations
succeed, but they are likely to return different objects than the user
expects.

The current implementation instead implements only a few useful methods
on the ``ForwardRef`` class. During the evaluation of annotations,
an instance of a private stringizer class is used instead of ``ForwardRef``.
After evaluation completes, the implementation of the FORWARDREF format
converts these internal objects into ``ForwardRef`` objects.

Signature of ``__annotate__`` functions
---------------------------------------

:pep:`649` specifies the signature of ``__annotate__`` functions as:

  ``__annotate__(format: int) -> dict``

However, using ``format`` as a parameter name could lead to collisions
if an annotation uses a class named ``format``. The parameter should be
positional-only and have a name that cannot be a legal identifier in order
to avoid this problem.

The current implementation uses the name ``.format`` with a leading
dot, but the exact name should be considered an implementation detail
and cannot be relied upon.

The documentation may still use the name ``format`` for simplicity.

Backwards Compatibility
=======================

:pep:`649` provides a thorough discussion of the backwards compatibility implications
on existing code that uses either stock or :pep:`563` semantics.

However, there is another set of compatibility problems: new code that is written
assuming :pep:`649` semantics, but uses existing tools that eagerly evaluate annotations.
For example, consider a ``dataclass``-like class decorator ``@annotator`` that retrieves the annotated
fields in the class it decorates, either by accessing ``__annotations__`` directly
or by calling :py:func:`inspect.get_annotations`.

Once :pep:`649` is implemented, code like this will work fine::

    class X:
        y: Y

    class Y: pass

But this will not, unless ``@annotator`` is changed to use the new ``FORWARDREF``
format::

    @annotator
    class X:
        y: Y

    class Y: pass

This is not strictly a backwards compatibility issue, since no previously working code
would break; before :pep:`649`, this code would have raised ``NameError`` at runtime.
In a sense, it is no different from any other new Python feature that needs
to be supported by third-party libraries. Nevertheless, it is a serious issue for libraries
that perform introspection, and it is important that we make it as easy as possible for
libraries to support the new semantics in a straightforward, user-friendly way.

We will update those parts of the standard library that are affected by this problem,
and we propose to add commonly useful functionality to the new ``annotationlib`` module,
so third-party tools can use the same set of tools.


Security Implications
=====================

None.


How to Teach This
=================

The semantics of :pep:`649`, as modified by this PEP, should largely be intuitive for
users who add annotations to their code. We eliminate the need for manually adding
quotes around annotations that require forward references, a major source of confusion
for users.

For advanced users who need to introspect annotations, the story becomes more complex.
The documentation of the new ``annotationlib`` module will serve as a reference for users
who need to interact programmatically with annotations.


Reference Implementation
========================

The in-progress PR `#119891 <https://github.com/python/cpython/pull/119891>`__
implements much of this PEP.

Open Issues
===========

We may discover additional areas where :pep:`649` needs clarification or amendment
as we make progress on implementing it. Readers are encouraged to follow the
`CPython issue <https://github.com/python/cpython/issues/119180>`__ tracking the
implementation of the PEP and try out the draft implementation. Any feedback may
be incorporated into future versions of this PEP.

Should dataclass field types use deferred evaluation?
-----------------------------------------------------

The current draft implementation already supports deferred evaluation in dataclasses,
so this works:

.. code:: pycon

  >>> from dataclasses import dataclass
  >>> @dataclass
  ... class D:
  ...     x: undefined
  ...     

However, the ``FORWARDREF`` format leaks into the field types of the dataclass:

.. code:: pycon

  >>> fields(D)[0].type
  ForwardRef('undefined')

We could instead add deferred evaluation for the field type, similar to that outlined
above for type alias values.

Accessing ``.type`` might throw an error:

.. code:: pycon

  >>> @dataclass
  ... class D:
  ...     x: undefined
  ...     
  >>> field = fields(D)[0]
  >>> field.type
  Traceback (most recent call last):
    File "<python-input-4>", line 1, in <module>
      field.type
    File ".../dataclasses.py", line 308, in type
      annos = self._annotate(annotationlib.Format.VALUE)
    File "<python-input-2>", line 3, in __annotate__
      x: undefined
        ^^^^^^^^^
  NameError: name 'undefined' is not defined

But users could use ``annotationlib.call_evaluate_function`` to get the type in other formats:

.. code:: pycon

  >>> annotationlib.call_evaluate_function(field.evaluate_type, annotationlib.Format.SOURCE)
  'undefined'
  >>> annotationlib.call_evaluate_function(field.evaluate_type, annotationlib.Format.FORWARDREF)
  ForwardRef('undefined')

Other variations are possible. For example, we could leave the ``type`` attribute unchanged,
and only add the ``evaluate_type`` method. This avoids unpleasant surprises where accessing
``.type`` may throw an exception.

Acknowledgments
===============

First of all, I thank Larry Hastings for writing :pep:`649`. This PEP modifies some of his
initial decisions, but the overall design is still his.

I thank Carl Meyer and Alex Waygood for feedback on early drafts of this PEP. Alex Waygood,
Alyssa Coghlan, and David Ellis provided insightful feedback and suggestions on the
interaction between metaclasses and ``__annotations__``.

Appendix
========

Which expressions can be stringified?
-------------------------------------

:pep:`649` acknowledges that the stringifier cannot handle all expressions. Now that we
have a draft implementation, we can be more precise about the expressions that can and
cannot be handled. Below is a list of all expressions in the Python AST that can and
cannot be recovered by the stringifier. The full list should probably not be added to
the documentation, but creating it is a useful exercise.

First, the stringifier of course cannot recover any information that is not present in
the compiled code, including comments, whitespace, parenthesization, and operations that
get simplified by the AST optimizer.

Second, the stringifier can intercept almost all operations that involve names looked
up in some scope, but it cannot intercept operations that operate fully on constants.
As a corollary, this also means it is not safe to request the ``SOURCE`` format on
untrusted code: Python is powerful enough that it is possible to achieve arbitrary
code execution even with no access to any globals or builtins. For example:

.. code-block:: pycon

  >>> def f(x: (1).__class__.__base__.__subclasses__()[-1].__init__.__builtins__["print"]("Hello world")): pass
  ... 
  >>> annotationlib.get_annotations(f, format=annotationlib.Format.SOURCE)
  Hello world
  {'x': 'None'}

(This particular example worked for me on the current implementation of a draft of this PEP;
the exact code may not keep working in the future.)

The following are supported (sometimes with caveats):

* ``BinOp``
* ``UnaryOp``

  * ``Invert`` (``~``), ``UAdd`` (``+``), and ``USub`` (``-``) are supported
  * ``Not`` (``not``) is not supported

* ``Dict`` (except when using ``**`` unpacking)
* ``Set``
* ``Compare``

  * ``Eq`` and ``NotEq`` are supported
  * ``Lt``, ``LtE``, ``Gt``, and ``GtE`` are supported, but the operand may be flipped
  * ``Is``, ``IsNot``, ``In``, and ``NotIn`` are not supported

* ``Call`` (except when using ``**`` unpacking)
* ``Constant`` (though not the exact representation of the constant; for example, escape
  sequences in strings are lost; hexadecimal numbers are converted to decimal)
* ``Attribute`` (assuming the value is not a constant)
* ``Subscript`` (assuming the value is not a constant)
* ``Starred`` (``*`` unpacking)
* ``Name``
* ``List``
* ``Tuple``
* ``Slice``

The following are unsupported, but throw an informative error when encountered by the
stringifier:

* ``FormattedValue`` (f-strings; error is not detected if conversion specifiers like ``!r``
  are used)
* ``JoinedStr`` (f-strings)

The following are unsupported and result in incorrect output:

* ``BoolOp`` (``and`` and ``or``)
* ``IfExp``
* ``Lambda``
* ``ListComp``
* ``SetComp``
* ``DictComp``
* ``GeneratorExp``

The following are disallowed in annotation scopes and therefore not relevant:

* ``NamedExpr`` (``:=``)
* ``Await``
* ``Yield``
* ``YieldFrom``


Copyright
=========

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.