From c1c52a5832c9ea67865f934e91bb6b4c85ab7d99 Mon Sep 17 00:00:00 2001 From: Jelle Zijlstra Date: Tue, 23 Jul 2024 13:49:18 -0700 Subject: [PATCH] PEP 749: Add section on metaclasses (#3847) Co-authored-by: Carl Meyer --- peps/pep-0749.rst | 165 +++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 164 insertions(+), 1 deletion(-) diff --git a/peps/pep-0749.rst b/peps/pep-0749.rst index 7b76fc6d8..6503ef832 100644 --- a/peps/pep-0749.rst +++ b/peps/pep-0749.rst @@ -187,6 +187,10 @@ The module will contain the following functionality: module, or class. This will replace :py:func:`inspect.get_annotations`. The latter will delegate to the new function. It may eventually be deprecated, but to minimize disruption, we do not propose an immediate deprecation. +* ``get_annotate_function()``: A function that returns the ``__annotate__`` function + of an object, if it has one, or ``None`` if it does not. This is usually equivalent + to accessing the ``.__annotate__`` attribute, except in the presence of metaclasses + (see :ref:`below `). * ``Format``: an enum that contains the possible formats of annotations. This will replace the ``VALUE``, ``FORWARDREF``, and ``SOURCE`` formats in :pep:`649`. PEP 649 proposed to make these values global members of the :py:mod:`inspect` @@ -390,6 +394,163 @@ More specifically: ``__dict__``. Writing to these attributes will directly update the ``__dict__``, without affecting the wrapped callable. +.. _pep749-metaclasses: + +Annotations and metaclasses +=========================== + +Testing of the initial implementation of this PEP revealed serious problems with +the interaction between metaclasses and class annotations. + +Pre-existing bugs +----------------- + +We found several bugs in the existing behavior of ``__annotations__`` on classes +while investigating the behaviors to be specified in this PEP. Fixing these bugs +on Python 3.13 and earlier is outside the scope of this PEP, but they are noted here +to explain the corner cases that need to be dealt with. + +For context, on Python 3.10 through 3.13 the ``__annotations__`` dictionary is +placed in the class namespace if the class has any annotations. If it does not, +there is no ``__annotations__`` class dictionary key when the class is created, +but accessing ``cls.__annotations__`` invokes a descriptor defined on ``type`` +that returns an empty dictionary and stores it in the class dictionary. +:py:ref:`Static types ` are an exception: they never have +annotations, and accessing ``.__annotations__`` raises :py:exc:`AttributeError`. +On Python 3.9 and earlier, the behavior was different; see +`gh-88067 `__. + +The following code fails identically on Python 3.10 through 3.13:: + + class Meta(type): pass + + class X(metaclass=Meta): + a: str + + class Y(X): pass + + Meta.__annotations__ # important + assert Y.__annotations__ == {}, Y.__annotations__ # fails: {'a': } + +If the annotations on the metaclass ``Meta`` are accessed before the annotations +on ``Y``, then the annotations for the base class ``X`` are leaked to ``Y``. +However, if the metaclass's annotations are *not* accessed (i.e., the line ``Meta.__annotations__`` +above is removed), then the annotations for ``Y`` are correctly empty. + +Similarly, annotations from annotated metaclasses leak to unannotated +classes that are instances of the metaclass:: + + class Meta(type): + a: str + + class X(metaclass=Meta): + pass + + assert X.__annotations__ == {}, X.__annotations__ # fails: {'a': } + +The reason for these behaviors is that if the metaclass contains an +``__annotations__`` entry in its class dictionary, this prevents +instances of the metaclass from using the ``__annotations__`` data descriptor +on the base :py:class:`type` class. In the first case, accessing ``Meta.__annotations__`` +sets ``Meta.__dict__["__annotations__"] = {}`` as a side effect. Then, looking +up the ``__annotations__`` attribute on ``Y`` first sees the metaclass attribute, +but skips it because it is a data descriptor. Next, it looks in the class dictionaries +of the classes in its method resolution order (MRO), finds ``X.__annotations__``, +and returns it. In the second example, there are no annotations +anywhere in the MRO, so ``type.__getattribute__`` falls back to +returning the metaclass attribute. + +Metaclass behavior with PEP 649 +------------------------------- + +With :pep:`649`, the behavior of accessing the ``.__annotations__`` attribute +on classes when metaclasses are involved becomes even more erratic, because now +``__annotations__`` is only lazily added to the class dictionary even for classes +with annotations. The new ``__annotate__`` attribute is also lazily created +on classes without annotations, which causes further misbehaviors when +metaclasses are involved. + +The cause of these problems is that we set the ``__annotate__`` and ``__annotations__`` +class dictionary entries only under some circumstances, and rely on descriptors +defined on :py:class:`type` to fill them in if they are not set. When normal +attribute lookup is used, this approach breaks down in the presence of +metaclasses, because entries in the metaclass's own class dictionary can render +the descriptors invisible. + +While we considered several approaches that would allow ``cls.__annotations__`` +and ``cls.__annotate__`` to work reliably when ``cls`` is a type with a custom +metaclass, any such approach would expose significant complexity to advanced users. +Instead, we recommend a simpler approach that confines the complexity to the +``annotationlib`` module: in ``annotationlib.get_annotations``, we bypass normal +attribute lookup by using the ``type.__annotations__`` descriptor directly. + +Specification +------------- + +Users should always use ``annotationlib.get_annotations`` to access the +annotations of a class object, and ``annotationlib.get_annotate_function`` +to access the ``__annotate__`` function. These functions will return only +the class's own annotations, even when metaclasses are involved. + +The behavior of accessing the ``__annotations__`` and ``__annotate__`` +attributes on classes with a metaclass other than ``builtins.type`` is +unspecified. The documentation should warn against direct use of these +attributes and recommend using the ``annotationlib`` module instead. + +Similarly, the presence of ``__annotations__`` and ``__annotate__`` keys +in the class dictionary is an implementation detail and should not be relied +upon. + +Rejected alternatives +--------------------- + +We considered two broad approaches for dealing with the behavior +of the ``__annotations__`` and ``__annotate__`` entries in classes: + +* Ensure that the entry is *always* present in the class dictionary, even if it + is empty or has not yet been evaluated. This means we do not have to rely on + the descriptors defined on :py:class:`type` to fill in the field, and + therefore the metaclass's attributes will not interfere. (Prototype + in `gh-120719 `__.) +* Ensure that the entry is *never* present in the class dictionary, or at least + never added by logic in the language core. This means that the descriptors + on :py:class:`type` will always be used, without interference from the metaclass. + (Prototype in `gh-120816 `__.) + +Alex Waygood suggested an implementation using the first approach. When a +heap type (such as a class created through the ``class`` statement) is created, +``cls.__dict__["__annotations__"]`` is set to a special descriptor. +On ``__get__``, the descriptor evaluates the annotations by calling ``__annotate__`` +and returning the result. The annotations dictionary is cached within the +descriptor instance. The descriptor also behaves like a mapping, +so that code that uses ``cls.__dict__["__annotations__"]`` will still usually +work: treating the object as a mapping will evaluate the annotations and behave +as if the descriptor itself was the annotations dictionary. (Code that assumes +that ``cls.__dict__["__annotations__"]`` is specifically an instance of ``dict`` +may break, however.) + +This approach is also straightforward to implement for ``__annotate__``: this +attribute is already always set for classes with annotations, and we can set +it explicitly to ``None`` for classes without annotations. + +While this approach would fix the known edge cases with metaclasses, it +introduces significant complexity to all classes, including a new built-in type +(for the annotations descriptor) with unusual behavior. + +The alternative approach would be to never set ``__dict__["__annotations__"]`` +and use some other storage to store the cached annotations. This behavior +change would have to apply even to classes defined under +``from __future__ import annotations``, because otherwise there could be buggy +behavior if a class is defined without ``from __future__ import annotations`` +but its metaclass does have the future enabled. As :pep:`649` previously noted, +removing ``__annotations__`` from class dictionaries also has backwards compatibility +implications: ``cls.__dict__.get("__annotations__")`` is a common idiom to +retrieve annotations. + +This approach would also mean that accessing ``.__annotations__`` on an instance +of an annotated class no longer works. While this behavior is not documented, +it is a long-standing feature of Python and is relied upon by some users. + Remove code flag for marking ``__annotate__`` functions ======================================================= @@ -695,7 +856,9 @@ Acknowledgments First of all, I thank Larry Hastings for writing :pep:`649`. This PEP modifies some of his initial decisions, but the overall design is still his. -I thank Carl Meyer and Alex Waygood for feedback on early drafts of this PEP. +I thank Carl Meyer and Alex Waygood for feedback on early drafts of this PEP. Alex Waygood, +Alyssa Coghlan, and David Ellis provided insightful feedback and suggestions on the +interaction between metaclasses and ``__annotations__``. Appendix ========