PEP 749: Add section on metaclasses (#3847)

Co-authored-by: Carl Meyer <carl@oddbird.net>
This commit is contained in:
Jelle Zijlstra 2024-07-23 13:49:18 -07:00 committed by GitHub
parent 6906bb2e07
commit c1c52a5832
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 164 additions and 1 deletions

View File

@ -187,6 +187,10 @@ The module will contain the following functionality:
module, or class. This will replace :py:func:`inspect.get_annotations`. The latter module, or class. This will replace :py:func:`inspect.get_annotations`. The latter
will delegate to the new function. It may eventually be deprecated, but to will delegate to the new function. It may eventually be deprecated, but to
minimize disruption, we do not propose an immediate deprecation. minimize disruption, we do not propose an immediate deprecation.
* ``get_annotate_function()``: A function that returns the ``__annotate__`` function
of an object, if it has one, or ``None`` if it does not. This is usually equivalent
to accessing the ``.__annotate__`` attribute, except in the presence of metaclasses
(see :ref:`below <pep749-metaclasses>`).
* ``Format``: an enum that contains the possible formats of annotations. This will * ``Format``: an enum that contains the possible formats of annotations. This will
replace the ``VALUE``, ``FORWARDREF``, and ``SOURCE`` formats in :pep:`649`. replace the ``VALUE``, ``FORWARDREF``, and ``SOURCE`` formats in :pep:`649`.
PEP 649 proposed to make these values global members of the :py:mod:`inspect` PEP 649 proposed to make these values global members of the :py:mod:`inspect`
@ -390,6 +394,163 @@ More specifically:
``__dict__``. Writing to these attributes will directly update the ``__dict__``, ``__dict__``. Writing to these attributes will directly update the ``__dict__``,
without affecting the wrapped callable. without affecting the wrapped callable.
.. _pep749-metaclasses:
Annotations and metaclasses
===========================
Testing of the initial implementation of this PEP revealed serious problems with
the interaction between metaclasses and class annotations.
Pre-existing bugs
-----------------
We found several bugs in the existing behavior of ``__annotations__`` on classes
while investigating the behaviors to be specified in this PEP. Fixing these bugs
on Python 3.13 and earlier is outside the scope of this PEP, but they are noted here
to explain the corner cases that need to be dealt with.
For context, on Python 3.10 through 3.13 the ``__annotations__`` dictionary is
placed in the class namespace if the class has any annotations. If it does not,
there is no ``__annotations__`` class dictionary key when the class is created,
but accessing ``cls.__annotations__`` invokes a descriptor defined on ``type``
that returns an empty dictionary and stores it in the class dictionary.
:py:ref:`Static types <static-types>` are an exception: they never have
annotations, and accessing ``.__annotations__`` raises :py:exc:`AttributeError`.
On Python 3.9 and earlier, the behavior was different; see
`gh-88067 <https://github.com/python/cpython/issues/88067>`__.
The following code fails identically on Python 3.10 through 3.13::
class Meta(type): pass
class X(metaclass=Meta):
a: str
class Y(X): pass
Meta.__annotations__ # important
assert Y.__annotations__ == {}, Y.__annotations__ # fails: {'a': <class 'str'>}
If the annotations on the metaclass ``Meta`` are accessed before the annotations
on ``Y``, then the annotations for the base class ``X`` are leaked to ``Y``.
However, if the metaclass's annotations are *not* accessed (i.e., the line ``Meta.__annotations__``
above is removed), then the annotations for ``Y`` are correctly empty.
Similarly, annotations from annotated metaclasses leak to unannotated
classes that are instances of the metaclass::
class Meta(type):
a: str
class X(metaclass=Meta):
pass
assert X.__annotations__ == {}, X.__annotations__ # fails: {'a': <class 'str'>}
The reason for these behaviors is that if the metaclass contains an
``__annotations__`` entry in its class dictionary, this prevents
instances of the metaclass from using the ``__annotations__`` data descriptor
on the base :py:class:`type` class. In the first case, accessing ``Meta.__annotations__``
sets ``Meta.__dict__["__annotations__"] = {}`` as a side effect. Then, looking
up the ``__annotations__`` attribute on ``Y`` first sees the metaclass attribute,
but skips it because it is a data descriptor. Next, it looks in the class dictionaries
of the classes in its method resolution order (MRO), finds ``X.__annotations__``,
and returns it. In the second example, there are no annotations
anywhere in the MRO, so ``type.__getattribute__`` falls back to
returning the metaclass attribute.
Metaclass behavior with PEP 649
-------------------------------
With :pep:`649`, the behavior of accessing the ``.__annotations__`` attribute
on classes when metaclasses are involved becomes even more erratic, because now
``__annotations__`` is only lazily added to the class dictionary even for classes
with annotations. The new ``__annotate__`` attribute is also lazily created
on classes without annotations, which causes further misbehaviors when
metaclasses are involved.
The cause of these problems is that we set the ``__annotate__`` and ``__annotations__``
class dictionary entries only under some circumstances, and rely on descriptors
defined on :py:class:`type` to fill them in if they are not set. When normal
attribute lookup is used, this approach breaks down in the presence of
metaclasses, because entries in the metaclass's own class dictionary can render
the descriptors invisible.
While we considered several approaches that would allow ``cls.__annotations__``
and ``cls.__annotate__`` to work reliably when ``cls`` is a type with a custom
metaclass, any such approach would expose significant complexity to advanced users.
Instead, we recommend a simpler approach that confines the complexity to the
``annotationlib`` module: in ``annotationlib.get_annotations``, we bypass normal
attribute lookup by using the ``type.__annotations__`` descriptor directly.
Specification
-------------
Users should always use ``annotationlib.get_annotations`` to access the
annotations of a class object, and ``annotationlib.get_annotate_function``
to access the ``__annotate__`` function. These functions will return only
the class's own annotations, even when metaclasses are involved.
The behavior of accessing the ``__annotations__`` and ``__annotate__``
attributes on classes with a metaclass other than ``builtins.type`` is
unspecified. The documentation should warn against direct use of these
attributes and recommend using the ``annotationlib`` module instead.
Similarly, the presence of ``__annotations__`` and ``__annotate__`` keys
in the class dictionary is an implementation detail and should not be relied
upon.
Rejected alternatives
---------------------
We considered two broad approaches for dealing with the behavior
of the ``__annotations__`` and ``__annotate__`` entries in classes:
* Ensure that the entry is *always* present in the class dictionary, even if it
is empty or has not yet been evaluated. This means we do not have to rely on
the descriptors defined on :py:class:`type` to fill in the field, and
therefore the metaclass's attributes will not interfere. (Prototype
in `gh-120719 <https://github.com/python/cpython/pull/120719>`__.)
* Ensure that the entry is *never* present in the class dictionary, or at least
never added by logic in the language core. This means that the descriptors
on :py:class:`type` will always be used, without interference from the metaclass.
(Prototype in `gh-120816 <https://github.com/python/cpython/pull/120816>`__.)
Alex Waygood suggested an implementation using the first approach. When a
heap type (such as a class created through the ``class`` statement) is created,
``cls.__dict__["__annotations__"]`` is set to a special descriptor.
On ``__get__``, the descriptor evaluates the annotations by calling ``__annotate__``
and returning the result. The annotations dictionary is cached within the
descriptor instance. The descriptor also behaves like a mapping,
so that code that uses ``cls.__dict__["__annotations__"]`` will still usually
work: treating the object as a mapping will evaluate the annotations and behave
as if the descriptor itself was the annotations dictionary. (Code that assumes
that ``cls.__dict__["__annotations__"]`` is specifically an instance of ``dict``
may break, however.)
This approach is also straightforward to implement for ``__annotate__``: this
attribute is already always set for classes with annotations, and we can set
it explicitly to ``None`` for classes without annotations.
While this approach would fix the known edge cases with metaclasses, it
introduces significant complexity to all classes, including a new built-in type
(for the annotations descriptor) with unusual behavior.
The alternative approach would be to never set ``__dict__["__annotations__"]``
and use some other storage to store the cached annotations. This behavior
change would have to apply even to classes defined under
``from __future__ import annotations``, because otherwise there could be buggy
behavior if a class is defined without ``from __future__ import annotations``
but its metaclass does have the future enabled. As :pep:`649` previously noted,
removing ``__annotations__`` from class dictionaries also has backwards compatibility
implications: ``cls.__dict__.get("__annotations__")`` is a common idiom to
retrieve annotations.
This approach would also mean that accessing ``.__annotations__`` on an instance
of an annotated class no longer works. While this behavior is not documented,
it is a long-standing feature of Python and is relied upon by some users.
Remove code flag for marking ``__annotate__`` functions Remove code flag for marking ``__annotate__`` functions
======================================================= =======================================================
@ -695,7 +856,9 @@ Acknowledgments
First of all, I thank Larry Hastings for writing :pep:`649`. This PEP modifies some of his First of all, I thank Larry Hastings for writing :pep:`649`. This PEP modifies some of his
initial decisions, but the overall design is still his. initial decisions, but the overall design is still his.
I thank Carl Meyer and Alex Waygood for feedback on early drafts of this PEP. I thank Carl Meyer and Alex Waygood for feedback on early drafts of this PEP. Alex Waygood,
Alyssa Coghlan, and David Ellis provided insightful feedback and suggestions on the
interaction between metaclasses and ``__annotations__``.
Appendix Appendix
======== ========