953 lines
44 KiB
ReStructuredText
953 lines
44 KiB
ReStructuredText
PEP: 749
|
|
Title: Implementing PEP 649
|
|
Author: Jelle Zijlstra <jelle.zijlstra@gmail.com>
|
|
Discussions-To: https://discuss.python.org/t/pep-749-implementing-pep-649/54974
|
|
Status: Draft
|
|
Type: Standards Track
|
|
Topic: Typing
|
|
Requires: 649
|
|
Created: 28-May-2024
|
|
Python-Version: 3.14
|
|
Post-History: `04-Jun-2024 <https://discuss.python.org/t/pep-749-implementing-pep-649/54974>`__
|
|
|
|
|
|
Abstract
|
|
========
|
|
|
|
This PEP supplements :pep:`649` by providing various tweaks and additions to its
|
|
specification:
|
|
|
|
* ``from __future__ import annotations`` (:pep:`563`) will continue to exist with
|
|
its current behavior at least until Python 3.13 reaches its end-of-life. Subsequently,
|
|
it will be deprecated and eventually removed.
|
|
* A new standard library module, ``annotationlib``, is added to provide tooling for
|
|
annotations. It will include the ``get_annotations()`` function, an enum for annotation
|
|
formats, a ``ForwardRef`` class, and a helper function for calling ``__annotate__`` functions.
|
|
* Annotations in the REPL are lazily evaluated, just like other module-level annotations.
|
|
* We specify the behavior of wrapper objects that provide annotations, such as :py:func:`classmethod`
|
|
and code that uses :py:func:`functools.wraps`.
|
|
* There will not be a code flag for marking ``__annotate__`` functions
|
|
that can be run in a "fake globals" environment.
|
|
* Setting the ``__annotations__`` attribute directly will not affect the ``__annotate__`` attribute.
|
|
* We add functionality to allow evaluating type alias values and type parameter bounds and defaults
|
|
(which were added by :pep:`695` and :pep:`696`) using PEP 649-like semantics.
|
|
|
|
Motivation
|
|
==========
|
|
|
|
:pep:`649` provides an excellent framework for creating better semantics for
|
|
annotations in Python. It solves a common pain point for users of annotations,
|
|
including those using static type hints as well as those using runtime typing,
|
|
and it makes the language more elegant and powerful.
|
|
The PEP was originally proposed in 2021 for Python 3.10,
|
|
and it was accepted in 2023. However, the implementation took longer than anticipated,
|
|
and now the PEP is expected to be implemented in Python 3.14.
|
|
|
|
I have started working on the implementation of the PEP in CPython. I found that
|
|
the PEP leaves some areas underspecified, and some
|
|
of its decisions in corner cases are questionable. This new PEP proposes several
|
|
changes and additions to the specification to address these issues.
|
|
|
|
This PEP supplements rather than supersedes PEP 649. The changes proposed here
|
|
should make the overall user experience better, but they do not change the
|
|
general framework of the earlier PEP.
|
|
|
|
|
|
The future of ``from __future__ import annotations``
|
|
====================================================
|
|
|
|
:pep:`563` previously introduced the future import ``from __future__ import annotations``,
|
|
which changes all annotations to strings. :pep:`649` proposes an alternative approach
|
|
that does not require this future import, and states:
|
|
|
|
If this PEP is accepted, PEP 563 will be deprecated and eventually removed.
|
|
|
|
However, the PEP does not provide a detailed plan for this deprecation.
|
|
|
|
There is some previous discussion of this topic `on Discourse <https://discuss.python.org/t/pep-649-deferred-evaluation-of-annotations-tentatively-accepted/21331/44>`__
|
|
(note that in the linked post I proposed something different from what is proposed here).
|
|
|
|
Specification
|
|
-------------
|
|
|
|
We suggest the following deprecation plan:
|
|
|
|
- In Python 3.14, ``from __future__ import annotations`` will continue to work as it
|
|
did before, converting annotations into strings.
|
|
- Sometime after the last release that did not support :pep:`649` semantics (expected to be 3.13)
|
|
reaches its end-of-life, ``from __future__ import annotations`` is deprecated. Compiling
|
|
any code that uses the future import will emit a :py:exc:`DeprecationWarning`. This will
|
|
happen no sooner than the first release after Python 3.13 reaches its end-of-life, but
|
|
the community may decide to wait longer.
|
|
- After at least two releases, the future import is removed, and annotations are always
|
|
evaluated as per :pep:`649`. Code that continues to use the future import will raise
|
|
a :py:exc:`SyntaxError`, similar to any other undefined future import.
|
|
|
|
Rejected alternatives
|
|
---------------------
|
|
|
|
*Immediately make the future import a no-op*: We considered applying :pep:`649` semantics
|
|
to all code in Python 3.14, making the future import a no-op. However, this would break
|
|
code that works in 3.13 under the following set of conditions:
|
|
|
|
* ``__future__ import annotations`` is active
|
|
* There are annotations that rely on forward references
|
|
* Annotations are eagerly evaluated at import time, for example by a metaclass or
|
|
class or function decorator. For example, this currently applies to the
|
|
released version of ``typing_extensions.TypedDict``.
|
|
|
|
This is expected to be a common pattern, so we cannot afford to break such code during
|
|
the upgrade from 3.13 to 3.14.
|
|
|
|
Such code would still break when the future import is eventually removed. However, this
|
|
is many years in the future, giving affected decorators plenty of time to update their code.
|
|
|
|
*Immediately deprecate the future import*: Instead of waiting until Python 3.13 reaches
|
|
its end-of-life, we could immediately start emitting warnings when the future import is
|
|
used. However, many libraries are already using ``from __future__ import annotations`` as
|
|
an elegant way to enable unrestricted forward references in their annotations. If we deprecate
|
|
the future import immediately, it would be impossible for these libraries to use unrestricted
|
|
forward references on all supported Python versions while avoiding deprecation warnings:
|
|
unlike other features deprecated from the standard library, a ``__future__`` import must
|
|
be the first statement in a given module, meaning it would be impossible to only
|
|
conditionally import ``__future__.annotations`` on Python 3.13 and lower. (The necessary
|
|
``sys.version_info`` check would count as a statement preceding the ``__future__`` import.)
|
|
|
|
*Keep the future import around forever*: We could also decide to keep the future import
|
|
indefinitely. However, this would permanently bifurcate the behavior of the Python
|
|
language. This is undesirable; the language should have only a single set of semantics,
|
|
not two permanently different modes.
|
|
|
|
*Make the future import a no-op in the future*: Instead of eventually making
|
|
``from __future__ import annotations`` a ``SyntaxError``, we could make it do nothing
|
|
instead at some point after Python 3.13 reaches its end-of-life. This still has some
|
|
of the same issues outlined above around making it a no-op now, although the ecosystem
|
|
would have had much longer to adapt. It is better to have users explicitly remove
|
|
the future import from their code in the future once they have confirmed they do not
|
|
rely on stringized annotations.
|
|
|
|
New ``annotationlib`` module
|
|
============================
|
|
|
|
:pep:`649` proposes to add tooling related to annotations to the :py:mod:`inspect`
|
|
module. However, that module is rather large, has direct or indirect dependencies
|
|
on at least 35 other standard library modules, and is so slow to import that other
|
|
standard library modules are often discouraged from importing it. Furthermore, we
|
|
anticipate adding more tools in addition to the :py:func:`inspect.get_annotations`
|
|
function and the ``VALUE``, ``FORWARDREF``, and ``SOURCE`` formats.
|
|
|
|
A new standard library module provides a logical home for this functionality and
|
|
also enables us to add more tooling that is useful for consumers of annotations.
|
|
|
|
Rationale
|
|
---------
|
|
|
|
:pep:`649` indicates that :py:class:`!typing.ForwardRef` should be used to implement the
|
|
``FORWARDREF`` format in :py:func:`inspect.get_annotations`. However, the existing implementation
|
|
of :py:class:`!typing.ForwardRef` is intertwined with the rest of the :py:mod:`!typing` module,
|
|
and it would not make sense to add :py:mod:`!typing`-specific behavior to the generic ``get_annotations()``
|
|
function. Furthermore, :py:class:`!typing.ForwardRef` is a problematic
|
|
class: it is public and documented, but the documentation lists no attributes or methods
|
|
for it. Nonetheless, third-party libraries make use of some of its undocumented
|
|
attributes. For instance, `Pydantic <https://github.com/pydantic/pydantic/blob/00ff77ed37589d924d3c10e0d5a48a7ef679a0d7/pydantic/v1/typing.py#L66>`__
|
|
and `Typeguard <https://github.com/agronholm/typeguard/blob/016f8139f5a0a63147d68df9558cc5584cd2c49a/src/typeguard/_utils.py#L44>`__
|
|
use the ``_evaluate`` method; `beartype <https://github.com/beartype/beartype/blob/0b4453f83c7ed4be054d8733aab8075e1478e166/beartype/_util/hint/pep/proposal/pep484585/utilpep484585ref.py#L210>`__
|
|
and `pyanalyze <https://github.com/quora/pyanalyze/blob/9e401724f9d035cf138b72612834b6d5a00eb8e8/pyanalyze/annotations.py#L509>`__
|
|
use the ``__forward_arg__`` attribute.
|
|
|
|
We replace the existing but poorly specified :py:class:`!typing.ForwardRef` with a new class,
|
|
``annotationlib.ForwardRef``. It is designed to be mostly compatible with existing uses
|
|
of the :py:class:`!typing.ForwardRef` class, but without the behaviors specific to the
|
|
:py:mod:`!typing` module. For compatibility with existing users, we keep the private
|
|
``_evaluate`` method, but mark it as deprecated. It delegates to a new public function in
|
|
the :py:mod:`!typing` module, ``typing.evaluate_forward_ref``, that is designed to
|
|
evaluate forward references in a way that is specific to type hints.
|
|
|
|
We add a function ``annotationlib.call_annotate_function`` as a helper for calling
|
|
``__annotate__`` functions. This is a useful building block when implementing functionality
|
|
that needs to partially evaluate annotations while a class is being constructed.
|
|
For example, the implementation of :py:class:`typing.NamedTuple` needs to retrieve
|
|
the annotations from a class namespace dictionary before the namedtuple class itself
|
|
can be constructed, because the annotations determine what fields exist on the namedtuple.
|
|
|
|
Specification
|
|
-------------
|
|
|
|
A new module, ``annotationlib``, is added to the standard library. Its aim is to
|
|
provide tooling for introspecting and wrapping annotations.
|
|
|
|
The exact contents of the module are not yet specified. We will add support for
|
|
:pep:`649` semantics to standard library functionality that uses annotations, such
|
|
as :py:mod:`dataclasses` and :py:class:`typing.TypedDict`, and use the experience
|
|
to inform the design of the new module.
|
|
|
|
The module will contain the following functionality:
|
|
|
|
* ``get_annotations()``: A function that returns the annotations of a function,
|
|
module, or class. This will replace :py:func:`inspect.get_annotations`. The latter
|
|
will delegate to the new function. It may eventually be deprecated, but to
|
|
minimize disruption, we do not propose an immediate deprecation.
|
|
* ``get_annotate_function()``: A function that returns the ``__annotate__`` function
|
|
of an object, if it has one, or ``None`` if it does not. This is usually equivalent
|
|
to accessing the ``.__annotate__`` attribute, except in the presence of metaclasses
|
|
(see :ref:`below <pep749-metaclasses>`).
|
|
* ``Format``: an enum that contains the possible formats of annotations. This will
|
|
replace the ``VALUE``, ``FORWARDREF``, and ``SOURCE`` formats in :pep:`649`.
|
|
PEP 649 proposed to make these values global members of the :py:mod:`inspect`
|
|
module; we prefer to place them within an enum.
|
|
* ``ForwardRef``: a class representing a forward reference; it may be returned by
|
|
``get_annotations()`` when the format is ``FORWARDREF``. The existing class
|
|
:py:class:`typing.ForwardRef` will become an alias of this class. Its members include:
|
|
|
|
* ``__forward_arg__``: the string argument of the forward reference
|
|
* ``evaluate(globals=None, locals=None, type_params=None, owner=None)``: a method that attempts to evaluate
|
|
the forward reference. The ``ForwardRef`` object may hold a reference to the
|
|
globals and other namespaces of the object that it originated from. If so, these
|
|
namespaces may be used to evaluate the forward reference. The *owner* argument
|
|
may be the object that holds the original annotation, such as the class or module
|
|
object; it is used to extract the globals and locals namespaces if these are not
|
|
provided.
|
|
* ``_evaluate()``, with the same interface as the existing ``ForwardRef._evaluate``
|
|
method. It will be undocumented and immediately deprecated. It is provided for
|
|
compatibility with existing users of ``typing.ForwardRef``.
|
|
|
|
* ``call_annotate_function(func: Callable, format: Format)``: a helper for calling
|
|
an ``__annotate__`` function with a given format. If the function does not support
|
|
this format, ``call_annotate_function()`` will set up a "fake globals" environment,
|
|
as described in :pep:`649`, and use that environment to return the desired annotations
|
|
format.
|
|
* ``call_evaluate_function(func: Callable | None, format: Format)``: similar to
|
|
``call_annotate_function``, but does not rely on the function returning an annotations
|
|
dictionary. This is intended to be used for evaluating deferred attributes introduced by
|
|
:pep:`695` and :pep:`696`; see below for details. *func* may be ``None``
|
|
for convenience; if ``None`` is passed, the function also returns ``None``.
|
|
|
|
A new function is also added to the :py:mod:`!typing` module, ``typing.evaluate_forward_ref``.
|
|
This function is a wrapper around the ``ForwardRef.evaluate`` method, but it performs
|
|
additional work that is specific to type hints. For example, it recurses into complex
|
|
types and evaluates additional forward references within these types.
|
|
|
|
Contrary to :pep:`649`, the annotation formats (``VALUE``, ``FORWARDREF``, and ``SOURCE``)
|
|
will not be added as global members of the :py:mod:`inspect` module. The only recommended
|
|
way to refer to these constants will be as ``annotationlib.Format.VALUE``.
|
|
|
|
Open issues
|
|
-----------
|
|
|
|
What should this module be called? Some ideas:
|
|
|
|
- ``annotations``: The most obvious name, but it may cause confusion with the existing
|
|
``from __future__ import annotations``, because users may have both ``import annotations``
|
|
and ``from __future__ import annotations`` in the same module. The use of a common word
|
|
as the name will make the module harder to search for. There is a PyPI package :pypi:`annotations`,
|
|
but it had only a single release in 2015 and looks abandoned.
|
|
- ``annotools``: Analogous to :py:mod:`itertools` and :py:mod:`functools`, but "anno" is a less
|
|
obvious abbreviation than "iter" or "func". As of this writing, there
|
|
is no PyPI package with this name.
|
|
- ``annotationtools``: A more explicit version. There is a PyPI package
|
|
:pypi:`annotationtools`, which had a release in 2023.
|
|
- ``annotation_tools``: A variation of the above but without a PyPI conflict. However,
|
|
no other public standard library module has an underscore in its name.
|
|
- ``annotationslib``: Analogous to :py:mod:`tomllib`, :py:mod:`pathlib`, and :py:mod:`importlib`.
|
|
There is no PyPI package with this name.
|
|
- ``annotationlib``: Similar to the above, but one character shorter and subjectively reads
|
|
better. Also not taken on PyPI.
|
|
|
|
Rejected alternatives
|
|
---------------------
|
|
|
|
*Add the functionality to the inspect module*: As described above, the
|
|
:py:mod:`inspect` module is already quite large, and its import time is prohibitive
|
|
for some use cases.
|
|
|
|
*Add the functionality to the typing module*: While annotations are mostly
|
|
used for typing, they may also be used for other purposes. We prefer to keep a clean
|
|
separation between functionality for introspecting annotations and functionality that
|
|
is exclusively meant for type hints.
|
|
|
|
*Add the functionality to the types module*: The :py:mod:`types` module is
|
|
meant for functionality related to *types*, and annotations can exist on functions
|
|
and modules, not only on types.
|
|
|
|
*Develop this functionality in a third-party package*: The functionality in this new
|
|
module will be pure Python code, and it is possible to implement a third-party package
|
|
that provides the same functionality by interacting directly with ``__annotate__``
|
|
functions generated by the interpreter. However, the functionality of the proposed new
|
|
module will certainly be useful in the standard library itself (e.g., for implementing
|
|
:py:mod:`dataclasses` and :py:class:`typing.NamedTuple`), so it makes sense to include
|
|
it in the standard library.
|
|
|
|
*Add this functionality to a private module*: It would be possible to initially develop
|
|
the module in a private standard library module (e.g., ``_annotations``), and publicize
|
|
it only after we have gained more experience with the API. However, we already know
|
|
that we will need parts of this module for the standard library itself (e.g., for
|
|
implementing :py:mod:`!dataclasses` and :py:class:`!typing.NamedTuple`). Even if we make
|
|
it private, the module will inevitably get used by third-party users. It is preferable
|
|
to start with a clear, documented API from the beginning, to enable third-party users to
|
|
support :pep:`649` semantics as thoroughly as the standard library. The module will
|
|
immediately be used in other parts of the standard library, ensuring that it covers a
|
|
reasonable set of use cases.
|
|
|
|
Behavior of the REPL
|
|
====================
|
|
|
|
:pep:`649` specifies the following behavior of the interactive REPL:
|
|
|
|
For the sake of simplicity, in this case we forego delayed evaluation.
|
|
Module-level annotations in the REPL shell will continue to work exactly
|
|
as they do with “stock semantics”, evaluating immediately and setting the
|
|
result directly inside the ``__annotations__`` dict.
|
|
|
|
There are several problems with this proposed behavior. It makes the REPL the
|
|
only context where annotations are still evaluated immediately, which is
|
|
confusing for users and complicates the language.
|
|
|
|
It also makes the implementation of the REPL more complex, as it needs to
|
|
ensure that all statements are compiled in "interactive" mode, even if their
|
|
output does not need to be displayed. (This matters if there are multiple
|
|
statements in a single line evaluated by the REPL.)
|
|
|
|
Most importantly, this breaks some plausible use cases that inexperienced
|
|
users could run into. A user might write the following in a file::
|
|
|
|
a: X | None = None
|
|
class X: ...
|
|
|
|
Under :pep:`649` this would work fine: ``X`` is not yet defined when it is used
|
|
in the annotation for ``a``, but the annotation is lazily evaluated. However,
|
|
if a user were to paste this same code into the REPL and execute it line by
|
|
line, it would throw a ``NameError``, because the name ``X`` is not yet defined.
|
|
|
|
This topic was previously discussed `on Discourse <https://discuss.python.org/t/pep-649-behavior-of-the-repl/54109>`__.
|
|
|
|
Specification
|
|
-------------
|
|
|
|
We propose to treat the interactive console like any other module-level code, and
|
|
make annotations lazily evaluated. This makes the language more consistent and
|
|
avoids subtle behavior changes between modules and the REPL.
|
|
|
|
Because the REPL is evaluated line by line, we would generate a new ``__annotate__``
|
|
function for every evaluated statement in the global scope that contains annotations. Whenever a line
|
|
containing annotations is evaluated, the previous ``__annotate__`` function is
|
|
lost:
|
|
|
|
.. code:: pycon
|
|
|
|
>>> x: int
|
|
>>> __annotate__(1)
|
|
{'x': <class 'int'>}
|
|
>>> y: str
|
|
>>> __annotate__(1)
|
|
{'y': <class 'str'>}
|
|
>>> z: doesntexist
|
|
>>> __annotate__(1)
|
|
Traceback (most recent call last):
|
|
File "<python-input-5>", line 1, in <module>
|
|
__annotate__(1)
|
|
~~~~~~~~~~~~^^^
|
|
File "<python-input-4>", line 1, in __annotate__
|
|
z: doesntexist
|
|
^^^^^^^^^^^
|
|
NameError: name 'doesntexist' is not defined
|
|
|
|
There will be no ``__annotations__`` key in the global namespace of the REPL.
|
|
In module namespaces, this key is created lazily when the ``__annotations__``
|
|
descriptor of the module object is accessed, but in the REPL there is no such module
|
|
object.
|
|
|
|
Classes and functions defined within the REPL will also work like any other classes,
|
|
so evaluation of their annotations will be deferred. It is possible to access the
|
|
``__annotations__`` and ``__annotate__`` attributes or use the ``annotationlib`` module
|
|
to introspect the annotations.
|
|
|
|
Wrappers that provide ``__annotations__``
|
|
=========================================
|
|
|
|
Several objects in the standard library and elsewhere provide annotations for their
|
|
wrapped object. :pep:`649` does not specify how such wrappers should behave.
|
|
|
|
Specification
|
|
-------------
|
|
|
|
Wrappers that provide annotations should be designed with the following goals
|
|
in mind:
|
|
|
|
* Evaluation of ``__annotations__`` should be deferred for as long as possible,
|
|
consistent with the behavior of built-in functions, classes, and modules.
|
|
* Backward compatibility with the behavior prior to the implementation of :pep:`649`
|
|
should be preserved.
|
|
* The ``__annotate__`` and ``__annotations__`` attributes should both be supplied
|
|
with semantics consistent to those of the wrapped object.
|
|
|
|
More specifically:
|
|
|
|
* :py:func:`functools.update_wrapper` (and therefore :py:func:`functools.wraps`)
|
|
will copy only the ``__annotate__`` attribute
|
|
from the wrapped object to the wrapper. The ``__annotations__`` descriptor on the
|
|
wrapper function will use the copied ``__annotate__``.
|
|
* The constructors for :py:func:`classmethod` and :py:func:`staticmethod` currently
|
|
copy the ``__annotations__`` attribute from the wrapped object to the wrapper.
|
|
They will instead have writable attributes for
|
|
``__annotate__`` and ``__annotations__``. Reading these attributes will retrieve
|
|
the corresponding attribute from the underlying callable and cache it in the wrapper's
|
|
``__dict__``. Writing to these attributes will directly update the ``__dict__``,
|
|
without affecting the wrapped callable.
|
|
|
|
.. _pep749-metaclasses:
|
|
|
|
Annotations and metaclasses
|
|
===========================
|
|
|
|
Testing of the initial implementation of this PEP revealed serious problems with
|
|
the interaction between metaclasses and class annotations.
|
|
|
|
Pre-existing bugs
|
|
-----------------
|
|
|
|
We found several bugs in the existing behavior of ``__annotations__`` on classes
|
|
while investigating the behaviors to be specified in this PEP. Fixing these bugs
|
|
on Python 3.13 and earlier is outside the scope of this PEP, but they are noted here
|
|
to explain the corner cases that need to be dealt with.
|
|
|
|
For context, on Python 3.10 through 3.13 the ``__annotations__`` dictionary is
|
|
placed in the class namespace if the class has any annotations. If it does not,
|
|
there is no ``__annotations__`` class dictionary key when the class is created,
|
|
but accessing ``cls.__annotations__`` invokes a descriptor defined on ``type``
|
|
that returns an empty dictionary and stores it in the class dictionary.
|
|
:py:ref:`Static types <static-types>` are an exception: they never have
|
|
annotations, and accessing ``.__annotations__`` raises :py:exc:`AttributeError`.
|
|
On Python 3.9 and earlier, the behavior was different; see
|
|
`gh-88067 <https://github.com/python/cpython/issues/88067>`__.
|
|
|
|
The following code fails identically on Python 3.10 through 3.13::
|
|
|
|
class Meta(type): pass
|
|
|
|
class X(metaclass=Meta):
|
|
a: str
|
|
|
|
class Y(X): pass
|
|
|
|
Meta.__annotations__ # important
|
|
assert Y.__annotations__ == {}, Y.__annotations__ # fails: {'a': <class 'str'>}
|
|
|
|
If the annotations on the metaclass ``Meta`` are accessed before the annotations
|
|
on ``Y``, then the annotations for the base class ``X`` are leaked to ``Y``.
|
|
However, if the metaclass's annotations are *not* accessed (i.e., the line ``Meta.__annotations__``
|
|
above is removed), then the annotations for ``Y`` are correctly empty.
|
|
|
|
Similarly, annotations from annotated metaclasses leak to unannotated
|
|
classes that are instances of the metaclass::
|
|
|
|
class Meta(type):
|
|
a: str
|
|
|
|
class X(metaclass=Meta):
|
|
pass
|
|
|
|
assert X.__annotations__ == {}, X.__annotations__ # fails: {'a': <class 'str'>}
|
|
|
|
The reason for these behaviors is that if the metaclass contains an
|
|
``__annotations__`` entry in its class dictionary, this prevents
|
|
instances of the metaclass from using the ``__annotations__`` data descriptor
|
|
on the base :py:class:`type` class. In the first case, accessing ``Meta.__annotations__``
|
|
sets ``Meta.__dict__["__annotations__"] = {}`` as a side effect. Then, looking
|
|
up the ``__annotations__`` attribute on ``Y`` first sees the metaclass attribute,
|
|
but skips it because it is a data descriptor. Next, it looks in the class dictionaries
|
|
of the classes in its method resolution order (MRO), finds ``X.__annotations__``,
|
|
and returns it. In the second example, there are no annotations
|
|
anywhere in the MRO, so ``type.__getattribute__`` falls back to
|
|
returning the metaclass attribute.
|
|
|
|
Metaclass behavior with PEP 649
|
|
-------------------------------
|
|
|
|
With :pep:`649`, the behavior of accessing the ``.__annotations__`` attribute
|
|
on classes when metaclasses are involved becomes even more erratic, because now
|
|
``__annotations__`` is only lazily added to the class dictionary even for classes
|
|
with annotations. The new ``__annotate__`` attribute is also lazily created
|
|
on classes without annotations, which causes further misbehaviors when
|
|
metaclasses are involved.
|
|
|
|
The cause of these problems is that we set the ``__annotate__`` and ``__annotations__``
|
|
class dictionary entries only under some circumstances, and rely on descriptors
|
|
defined on :py:class:`type` to fill them in if they are not set. When normal
|
|
attribute lookup is used, this approach breaks down in the presence of
|
|
metaclasses, because entries in the metaclass's own class dictionary can render
|
|
the descriptors invisible.
|
|
|
|
While we considered several approaches that would allow ``cls.__annotations__``
|
|
and ``cls.__annotate__`` to work reliably when ``cls`` is a type with a custom
|
|
metaclass, any such approach would expose significant complexity to advanced users.
|
|
Instead, we recommend a simpler approach that confines the complexity to the
|
|
``annotationlib`` module: in ``annotationlib.get_annotations``, we bypass normal
|
|
attribute lookup by using the ``type.__annotations__`` descriptor directly.
|
|
|
|
Specification
|
|
-------------
|
|
|
|
Users should always use ``annotationlib.get_annotations`` to access the
|
|
annotations of a class object, and ``annotationlib.get_annotate_function``
|
|
to access the ``__annotate__`` function. These functions will return only
|
|
the class's own annotations, even when metaclasses are involved.
|
|
|
|
The behavior of accessing the ``__annotations__`` and ``__annotate__``
|
|
attributes on classes with a metaclass other than ``builtins.type`` is
|
|
unspecified. The documentation should warn against direct use of these
|
|
attributes and recommend using the ``annotationlib`` module instead.
|
|
|
|
Similarly, the presence of ``__annotations__`` and ``__annotate__`` keys
|
|
in the class dictionary is an implementation detail and should not be relied
|
|
upon.
|
|
|
|
Rejected alternatives
|
|
---------------------
|
|
|
|
We considered two broad approaches for dealing with the behavior
|
|
of the ``__annotations__`` and ``__annotate__`` entries in classes:
|
|
|
|
* Ensure that the entry is *always* present in the class dictionary, even if it
|
|
is empty or has not yet been evaluated. This means we do not have to rely on
|
|
the descriptors defined on :py:class:`type` to fill in the field, and
|
|
therefore the metaclass's attributes will not interfere. (Prototype
|
|
in `gh-120719 <https://github.com/python/cpython/pull/120719>`__.)
|
|
* Ensure that the entry is *never* present in the class dictionary, or at least
|
|
never added by logic in the language core. This means that the descriptors
|
|
on :py:class:`type` will always be used, without interference from the metaclass.
|
|
(Prototype in `gh-120816 <https://github.com/python/cpython/pull/120816>`__.)
|
|
|
|
Alex Waygood suggested an implementation using the first approach. When a
|
|
heap type (such as a class created through the ``class`` statement) is created,
|
|
``cls.__dict__["__annotations__"]`` is set to a special descriptor.
|
|
On ``__get__``, the descriptor evaluates the annotations by calling ``__annotate__``
|
|
and returning the result. The annotations dictionary is cached within the
|
|
descriptor instance. The descriptor also behaves like a mapping,
|
|
so that code that uses ``cls.__dict__["__annotations__"]`` will still usually
|
|
work: treating the object as a mapping will evaluate the annotations and behave
|
|
as if the descriptor itself was the annotations dictionary. (Code that assumes
|
|
that ``cls.__dict__["__annotations__"]`` is specifically an instance of ``dict``
|
|
may break, however.)
|
|
|
|
This approach is also straightforward to implement for ``__annotate__``: this
|
|
attribute is already always set for classes with annotations, and we can set
|
|
it explicitly to ``None`` for classes without annotations.
|
|
|
|
While this approach would fix the known edge cases with metaclasses, it
|
|
introduces significant complexity to all classes, including a new built-in type
|
|
(for the annotations descriptor) with unusual behavior.
|
|
|
|
The alternative approach would be to never set ``__dict__["__annotations__"]``
|
|
and use some other storage to store the cached annotations. This behavior
|
|
change would have to apply even to classes defined under
|
|
``from __future__ import annotations``, because otherwise there could be buggy
|
|
behavior if a class is defined without ``from __future__ import annotations``
|
|
but its metaclass does have the future enabled. As :pep:`649` previously noted,
|
|
removing ``__annotations__`` from class dictionaries also has backwards compatibility
|
|
implications: ``cls.__dict__.get("__annotations__")`` is a common idiom to
|
|
retrieve annotations.
|
|
|
|
This approach would also mean that accessing ``.__annotations__`` on an instance
|
|
of an annotated class no longer works. While this behavior is not documented,
|
|
it is a long-standing feature of Python and is relied upon by some users.
|
|
|
|
Remove code flag for marking ``__annotate__`` functions
|
|
=======================================================
|
|
|
|
:pep:`649` specifies:
|
|
|
|
This PEP assumes that
|
|
third-party libraries may implement their own ``__annotate__``
|
|
methods, and those functions would almost certainly work
|
|
incorrectly when run in this "fake globals" environment.
|
|
For that reason, this PEP allocates a flag on code objects,
|
|
one of the unused bits in ``co_flags``, to mean "This code
|
|
object can be run in a 'fake globals' environment." This
|
|
makes the "fake globals" environment strictly opt-in, and
|
|
it's expected that only ``__annotate__`` methods generated
|
|
by the Python compiler will set it.
|
|
|
|
We have not found a need for this mechanism during our work to
|
|
add :pep:`649` support to the standard library. While it is true
|
|
that custom ``__annotate__`` functions may not work well with the
|
|
"fake globals" environment, this technique is used only when the
|
|
``__annotate__`` function raises :py:exc:`NotImplementedError` to
|
|
signal that it does not support the requested format. However,
|
|
manually implemented ``__annotate__`` functions are likely to support
|
|
all three annotation formats; often, they will consist of a call to
|
|
``annotationlib.call_annotate_function`` plus some transformation of the
|
|
result.
|
|
|
|
In addition, the proposed mechanism couples the implementation with
|
|
low-level details of the code object. The code object flags are
|
|
CPython-specific and the documentation :py:ref:`explicitly warns <inspect-module-co-flags>`
|
|
against relying on the values.
|
|
|
|
Specification
|
|
-------------
|
|
|
|
The standard library will use the "fake globals" technique on any
|
|
``__annotate__`` function that raises :py:exc:`NotImplementedError`
|
|
when the requested format is not supported.
|
|
|
|
Third-party code that implements ``__annotate__`` functions should either
|
|
support all three annotation formats, or be prepared to handle the
|
|
"fake globals" environment. This should be mentioned in the data model
|
|
documentation for ``__annotate__``.
|
|
|
|
Effect of setting ``__annotations__``
|
|
=====================================
|
|
|
|
:pep:`649` specifies:
|
|
|
|
Setting ``o.__annotations__`` to a legal value
|
|
automatically sets ``o.__annotate__`` to ``None``.
|
|
|
|
We would prefer to keep ``__annotate__`` unchanged when ``__annotations__``
|
|
is written to. Conceptually, ``__annotate__`` provides the ground truth
|
|
and ``__annotations__`` is merely a cache, and we shouldn't throw away the
|
|
ground truth if the cache is modified.
|
|
|
|
The motivation for :pep:`649`'s behavior is to keep the two attributes in sync.
|
|
However, this is impossible in general; if the ``__annotations__`` dictionary
|
|
is modified in place, this will not be reflected in the ``__annotate__`` attribute.
|
|
The overall mental model for this area will be simpler if setting ``__annotations__``
|
|
has no effect on ``__annotate__``.
|
|
|
|
Specification
|
|
-------------
|
|
|
|
The value of ``__annotate__`` is not changed when ``__annotations__`` is set.
|
|
|
|
Deferred evaluation of PEP 695 and 696 objects
|
|
==============================================
|
|
|
|
Since :pep:`649` was written, Python 3.12 and 3.13 gained support for
|
|
several new features that also use deferred evaluation, similar to the
|
|
behavior this PEP proposes for annotations:
|
|
|
|
* The value of type aliases created through the :py:keyword:`type`
|
|
statement (:pep:`695`)
|
|
* The bound and constraints of :py:class:`typing.TypeVar` objects
|
|
created through the syntax for generics (:pep:`695`)
|
|
* The default value of :py:class:`typing.TypeVar`, :py:class:`ParamSpec`,
|
|
and :py:class:`typing.TypeVarTuple` objects (:pep:`696`)
|
|
|
|
Currently, these objects use deferred evaluation, but there is no direct
|
|
access to the function object used for deferred evaluation. To enable
|
|
the same kind of introspection that is now possible for annotations, we propose
|
|
to expose the internal function objects, allowing users to evaluate them
|
|
using the FORWARDREF and SOURCE formats.
|
|
|
|
Specification
|
|
-------------
|
|
|
|
We will add the following new attributes:
|
|
|
|
* ``evaluate_value`` on :py:class:`typing.TypeAliasType`
|
|
* ``evaluate_bound``, ``evaluate_constraints``, and ``evaluate_default`` on :py:class:`typing.TypeVar`
|
|
* ``evaluate_default`` on :py:class:`typing.ParamSpec`
|
|
* ``evaluate_default`` on :py:class:`typing.TypeVarTuple`
|
|
|
|
Except for ``evaluate_value``, these attributes may be ``None`` if the object
|
|
does not have a bound, constraints, or default. Otherwise, the attribute is a
|
|
callable, similar to an ``__annotate__`` function, that takes a single integer
|
|
argument and returns the evaluated value. Unlike ``__annotate__`` functions,
|
|
these callables return a single value, not a dictionary of annotations.
|
|
These attributes are read-only.
|
|
|
|
Usually, users would use these attributes in combinations with
|
|
``annotationlib.call_evaluate_function``. For example, to get a ``TypeVar``'s bound
|
|
in SOURCE format, one could write
|
|
``annotationlib.call_evaluate_function(T.evaluate_bound, annotationlib.Format.SOURCE)``.
|
|
|
|
Miscellaneous implementation details
|
|
====================================
|
|
|
|
:pep:`649` goes into considerable detail on some aspects of the implementation.
|
|
To avoid confusion, we describe a few aspects where the current implementation
|
|
differs from that described in the PEP. However, these details are not guaranteed
|
|
to hold in the future, and they may change without notice in the future, unless
|
|
they are documented in the language reference.
|
|
|
|
Supported operations on ``ForwardRef`` objects
|
|
----------------------------------------------
|
|
|
|
The ``SOURCE`` format is implemented by the "stringizer" technique,
|
|
where the globals dictionary of a function is augmented so that every
|
|
lookup results in a special object that can be used to reconstruct the
|
|
operations that are performed on the object.
|
|
|
|
:pep:`649` specifies:
|
|
|
|
In practice, the "stringizer" functionality will be implemented
|
|
in the ``ForwardRef`` object currently defined in the
|
|
``typing`` module. ``ForwardRef`` will be extended to
|
|
implement all stringizer functionality; it will also be
|
|
extended to support evaluating the string it contains,
|
|
to produce the real value (assuming all symbols referenced
|
|
are defined).
|
|
|
|
However, this is likely to lead to confusion in practice. An object
|
|
that implements stringizer functionality must implement almost all
|
|
special methods, including ``__getattr__`` and ``__eq__``, to return
|
|
a new stringizer. Such an object is confusing to work with: all operations
|
|
succeed, but they are likely to return different objects than the user
|
|
expects.
|
|
|
|
The current implementation instead implements only a few useful methods
|
|
on the ``ForwardRef`` class. During the evaluation of annotations,
|
|
an instance of a private stringizer class is used instead of ``ForwardRef``.
|
|
After evaluation completes, the implementation of the FORWARDREF format
|
|
converts these internal objects into ``ForwardRef`` objects.
|
|
|
|
Signature of ``__annotate__`` functions
|
|
---------------------------------------
|
|
|
|
:pep:`649` specifies the signature of ``__annotate__`` functions as:
|
|
|
|
``__annotate__(format: int) -> dict``
|
|
|
|
However, using ``format`` as a parameter name could lead to collisions
|
|
if an annotation uses a class named ``format``. The parameter should be
|
|
positional-only and have a name that cannot be a legal identifier in order
|
|
to avoid this problem.
|
|
|
|
The current implementation uses the name ``.format`` with a leading
|
|
dot, but the exact name should be considered an implementation detail
|
|
and cannot be relied upon.
|
|
|
|
The documentation may still use the name ``format`` for simplicity.
|
|
|
|
Backwards Compatibility
|
|
=======================
|
|
|
|
:pep:`649` provides a thorough discussion of the backwards compatibility implications
|
|
on existing code that uses either stock or :pep:`563` semantics.
|
|
|
|
However, there is another set of compatibility problems: new code that is written
|
|
assuming :pep:`649` semantics, but uses existing tools that eagerly evaluate annotations.
|
|
For example, consider a ``dataclass``-like class decorator ``@annotator`` that retrieves the annotated
|
|
fields in the class it decorates, either by accessing ``__annotations__`` directly
|
|
or by calling :py:func:`inspect.get_annotations`.
|
|
|
|
Once :pep:`649` is implemented, code like this will work fine::
|
|
|
|
class X:
|
|
y: Y
|
|
|
|
class Y: pass
|
|
|
|
But this will not, unless ``@annotator`` is changed to use the new ``FORWARDREF``
|
|
format::
|
|
|
|
@annotator
|
|
class X:
|
|
y: Y
|
|
|
|
class Y: pass
|
|
|
|
This is not strictly a backwards compatibility issue, since no previously working code
|
|
would break; before :pep:`649`, this code would have raised ``NameError`` at runtime.
|
|
In a sense, it is no different from any other new Python feature that needs
|
|
to be supported by third-party libraries. Nevertheless, it is a serious issue for libraries
|
|
that perform introspection, and it is important that we make it as easy as possible for
|
|
libraries to support the new semantics in a straightforward, user-friendly way.
|
|
|
|
We will update those parts of the standard library that are affected by this problem,
|
|
and we propose to add commonly useful functionality to the new ``annotationlib`` module,
|
|
so third-party tools can use the same set of tools.
|
|
|
|
|
|
Security Implications
|
|
=====================
|
|
|
|
None.
|
|
|
|
|
|
How to Teach This
|
|
=================
|
|
|
|
The semantics of :pep:`649`, as modified by this PEP, should largely be intuitive for
|
|
users who add annotations to their code. We eliminate the need for manually adding
|
|
quotes around annotations that require forward references, a major source of confusion
|
|
for users.
|
|
|
|
For advanced users who need to introspect annotations, the story becomes more complex.
|
|
The documentation of the new ``annotationlib`` module will serve as a reference for users
|
|
who need to interact programmatically with annotations.
|
|
|
|
|
|
Reference Implementation
|
|
========================
|
|
|
|
The in-progress PR `#119891 <https://github.com/python/cpython/pull/119891>`__
|
|
implements much of this PEP.
|
|
|
|
Open Issues
|
|
===========
|
|
|
|
We may discover additional areas where :pep:`649` needs clarification or amendment
|
|
as we make progress on implementing it. Readers are encouraged to follow the
|
|
`CPython issue <https://github.com/python/cpython/issues/119180>`__ tracking the
|
|
implementation of the PEP and try out the draft implementation. Any feedback may
|
|
be incorporated into future versions of this PEP.
|
|
|
|
Should dataclass field types use deferred evaluation?
|
|
-----------------------------------------------------
|
|
|
|
The current draft implementation already supports deferred evaluation in dataclasses,
|
|
so this works:
|
|
|
|
.. code:: pycon
|
|
|
|
>>> from dataclasses import dataclass
|
|
>>> @dataclass
|
|
... class D:
|
|
... x: undefined
|
|
...
|
|
|
|
However, the ``FORWARDREF`` format leaks into the field types of the dataclass:
|
|
|
|
.. code:: pycon
|
|
|
|
>>> fields(D)[0].type
|
|
ForwardRef('undefined')
|
|
|
|
We could instead add deferred evaluation for the field type, similar to that outlined
|
|
above for type alias values.
|
|
|
|
Accessing ``.type`` might throw an error:
|
|
|
|
.. code:: pycon
|
|
|
|
>>> @dataclass
|
|
... class D:
|
|
... x: undefined
|
|
...
|
|
>>> field = fields(D)[0]
|
|
>>> field.type
|
|
Traceback (most recent call last):
|
|
File "<python-input-4>", line 1, in <module>
|
|
field.type
|
|
File ".../dataclasses.py", line 308, in type
|
|
annos = self._annotate(annotationlib.Format.VALUE)
|
|
File "<python-input-2>", line 3, in __annotate__
|
|
x: undefined
|
|
^^^^^^^^^
|
|
NameError: name 'undefined' is not defined
|
|
|
|
But users could use ``annotationlib.call_evaluate_function`` to get the type in other formats:
|
|
|
|
.. code:: pycon
|
|
|
|
>>> annotationlib.call_evaluate_function(field.evaluate_type, annotationlib.Format.SOURCE)
|
|
'undefined'
|
|
>>> annotationlib.call_evaluate_function(field.evaluate_type, annotationlib.Format.FORWARDREF)
|
|
ForwardRef('undefined')
|
|
|
|
Other variations are possible. For example, we could leave the ``type`` attribute unchanged,
|
|
and only add the ``evaluate_type`` method. This avoids unpleasant surprises where accessing
|
|
``.type`` may throw an exception.
|
|
|
|
Acknowledgments
|
|
===============
|
|
|
|
First of all, I thank Larry Hastings for writing :pep:`649`. This PEP modifies some of his
|
|
initial decisions, but the overall design is still his.
|
|
|
|
I thank Carl Meyer and Alex Waygood for feedback on early drafts of this PEP. Alex Waygood,
|
|
Alyssa Coghlan, and David Ellis provided insightful feedback and suggestions on the
|
|
interaction between metaclasses and ``__annotations__``.
|
|
|
|
Appendix
|
|
========
|
|
|
|
Which expressions can be stringified?
|
|
-------------------------------------
|
|
|
|
:pep:`649` acknowledges that the stringifier cannot handle all expressions. Now that we
|
|
have a draft implementation, we can be more precise about the expressions that can and
|
|
cannot be handled. Below is a list of all expressions in the Python AST that can and
|
|
cannot be recovered by the stringifier. The full list should probably not be added to
|
|
the documentation, but creating it is a useful exercise.
|
|
|
|
First, the stringifier of course cannot recover any information that is not present in
|
|
the compiled code, including comments, whitespace, parenthesization, and operations that
|
|
get simplified by the AST optimizer.
|
|
|
|
Second, the stringifier can intercept almost all operations that involve names looked
|
|
up in some scope, but it cannot intercept operations that operate fully on constants.
|
|
As a corollary, this also means it is not safe to request the ``SOURCE`` format on
|
|
untrusted code: Python is powerful enough that it is possible to achieve arbitrary
|
|
code execution even with no access to any globals or builtins. For example:
|
|
|
|
.. code-block:: pycon
|
|
|
|
>>> def f(x: (1).__class__.__base__.__subclasses__()[-1].__init__.__builtins__["print"]("Hello world")): pass
|
|
...
|
|
>>> annotationlib.get_annotations(f, format=annotationlib.Format.SOURCE)
|
|
Hello world
|
|
{'x': 'None'}
|
|
|
|
(This particular example worked for me on the current implementation of a draft of this PEP;
|
|
the exact code may not keep working in the future.)
|
|
|
|
The following are supported (sometimes with caveats):
|
|
|
|
* ``BinOp``
|
|
* ``UnaryOp``
|
|
|
|
* ``Invert`` (``~``), ``UAdd`` (``+``), and ``USub`` (``-``) are supported
|
|
* ``Not`` (``not``) is not supported
|
|
|
|
* ``Dict`` (except when using ``**`` unpacking)
|
|
* ``Set``
|
|
* ``Compare``
|
|
|
|
* ``Eq`` and ``NotEq`` are supported
|
|
* ``Lt``, ``LtE``, ``Gt``, and ``GtE`` are supported, but the operand may be flipped
|
|
* ``Is``, ``IsNot``, ``In``, and ``NotIn`` are not supported
|
|
|
|
* ``Call`` (except when using ``**`` unpacking)
|
|
* ``Constant`` (though not the exact representation of the constant; for example, escape
|
|
sequences in strings are lost; hexadecimal numbers are converted to decimal)
|
|
* ``Attribute`` (assuming the value is not a constant)
|
|
* ``Subscript`` (assuming the value is not a constant)
|
|
* ``Starred`` (``*`` unpacking)
|
|
* ``Name``
|
|
* ``List``
|
|
* ``Tuple``
|
|
* ``Slice``
|
|
|
|
The following are unsupported, but throw an informative error when encountered by the
|
|
stringifier:
|
|
|
|
* ``FormattedValue`` (f-strings; error is not detected if conversion specifiers like ``!r``
|
|
are used)
|
|
* ``JoinedStr`` (f-strings)
|
|
|
|
The following are unsupported and result in incorrect output:
|
|
|
|
* ``BoolOp`` (``and`` and ``or``)
|
|
* ``IfExp``
|
|
* ``Lambda``
|
|
* ``ListComp``
|
|
* ``SetComp``
|
|
* ``DictComp``
|
|
* ``GeneratorExp``
|
|
|
|
The following are disallowed in annotation scopes and therefore not relevant:
|
|
|
|
* ``NamedExpr`` (``:=``)
|
|
* ``Await``
|
|
* ``Yield``
|
|
* ``YieldFrom``
|
|
|
|
|
|
Copyright
|
|
=========
|
|
|
|
This document is placed in the public domain or under the
|
|
CC0-1.0-Universal license, whichever is more permissive.
|