PEP 749: Add section on metaclasses (#3847)

Co-authored-by: Carl Meyer <carl@oddbird.net>

peps/pep-0749.rst

@@ -187,6 +187,10 @@ The module will contain the following functionality:
   module, or class. This will replace :py:func:`inspect.get_annotations`. The latter
   will delegate to the new function. It may eventually be deprecated, but to
   minimize disruption, we do not propose an immediate deprecation.
+* ``get_annotate_function()``: A function that returns the ``__annotate__`` function
+  of an object, if it has one, or ``None`` if it does not. This is usually equivalent
+  to accessing the ``.__annotate__`` attribute, except in the presence of metaclasses
+  (see :ref:`below <pep749-metaclasses>`).
 * ``Format``: an enum that contains the possible formats of annotations. This will
   replace the ``VALUE``, ``FORWARDREF``, and ``SOURCE`` formats in :pep:`649`.
   PEP 649 proposed to make these values global members of the :py:mod:`inspect`
@@ -390,6 +394,163 @@ More specifically:
   ``__dict__``. Writing to these attributes will directly update the ``__dict__``,
   without affecting the wrapped callable.
 
+.. _pep749-metaclasses:
+
+Annotations and metaclasses
+===========================
+
+Testing of the initial implementation of this PEP revealed serious problems with
+the interaction between metaclasses and class annotations.
+
+Pre-existing bugs
+-----------------
+
+We found several bugs in the existing behavior of ``__annotations__`` on classes
+while investigating the behaviors to be specified in this PEP. Fixing these bugs
+on Python 3.13 and earlier is outside the scope of this PEP, but they are noted here
+to explain the corner cases that need to be dealt with.
+
+For context, on Python 3.10 through 3.13 the ``__annotations__`` dictionary is
+placed in the class namespace if the class has any annotations. If it does not,
+there is no ``__annotations__`` class dictionary key when the class is created,
+but accessing ``cls.__annotations__`` invokes a descriptor defined on ``type``
+that returns an empty dictionary and stores it in the class dictionary.
+:py:ref:`Static types <static-types>` are an exception: they never have
+annotations, and accessing ``.__annotations__`` raises :py:exc:`AttributeError`.
+On Python 3.9 and earlier, the behavior was different; see
+`gh-88067 <https://github.com/python/cpython/issues/88067>`__.
+
+The following code fails identically on Python 3.10 through 3.13::
+
+    class Meta(type): pass
+
+    class X(metaclass=Meta):
+        a: str
+
+    class Y(X): pass
+
+    Meta.__annotations__  # important
+    assert Y.__annotations__ == {}, Y.__annotations__  # fails: {'a': <class 'str'>}
+
+If the annotations on the metaclass ``Meta`` are accessed before the annotations
+on ``Y``, then the annotations for the base class ``X`` are leaked to ``Y``.
+However, if the metaclass's annotations are *not* accessed (i.e., the line ``Meta.__annotations__``
+above is removed), then the annotations for ``Y`` are correctly empty.
+
+Similarly, annotations from annotated metaclasses leak to unannotated
+classes that are instances of the metaclass::
+
+    class Meta(type):
+        a: str
+
+    class X(metaclass=Meta):
+        pass
+
+    assert X.__annotations__ == {}, X.__annotations__  # fails: {'a': <class 'str'>}
+
+The reason for these behaviors is that if the metaclass contains an
+``__annotations__`` entry in its class dictionary, this prevents
+instances of the metaclass from using the ``__annotations__`` data descriptor
+on the base :py:class:`type` class. In the first case, accessing ``Meta.__annotations__``
+sets ``Meta.__dict__["__annotations__"] = {}`` as a side effect. Then, looking
+up the ``__annotations__`` attribute on ``Y`` first sees the metaclass attribute,
+but skips it because it is a data descriptor. Next, it looks in the class dictionaries
+of the classes in its method resolution order (MRO), finds ``X.__annotations__``,
+and returns it. In the second example, there are no annotations
+anywhere in the MRO, so ``type.__getattribute__`` falls back to
+returning the metaclass attribute.
+
+Metaclass behavior with PEP 649
+-------------------------------
+
+With :pep:`649`, the behavior of accessing the ``.__annotations__`` attribute
+on classes when metaclasses are involved becomes even more erratic, because now
+``__annotations__`` is only lazily added to the class dictionary even for classes
+with annotations. The new ``__annotate__`` attribute is also lazily created
+on classes without annotations, which causes further misbehaviors when
+metaclasses are involved.
+
+The cause of these problems is that we set the ``__annotate__`` and ``__annotations__``
+class dictionary entries only under some circumstances, and rely on descriptors
+defined on :py:class:`type` to fill them in if they are not set. When normal
+attribute lookup is used, this approach breaks down in the presence of
+metaclasses, because entries in the metaclass's own class dictionary can render
+the descriptors invisible.
+
+While we considered several approaches that would allow ``cls.__annotations__``
+and ``cls.__annotate__`` to work reliably when ``cls`` is a type with a custom
+metaclass, any such approach would expose significant complexity to advanced users.
+Instead, we recommend a simpler approach that confines the complexity to the
+``annotationlib`` module: in ``annotationlib.get_annotations``, we bypass normal
+attribute lookup by using the ``type.__annotations__`` descriptor directly.
+
+Specification
+-------------
+
+Users should always use ``annotationlib.get_annotations`` to access the
+annotations of a class object, and ``annotationlib.get_annotate_function``
+to access the ``__annotate__`` function. These functions will return only
+the class's own annotations, even when metaclasses are involved.
+
+The behavior of accessing the ``__annotations__`` and ``__annotate__``
+attributes on classes with a metaclass other than ``builtins.type`` is
+unspecified. The documentation should warn against direct use of these
+attributes and recommend using the ``annotationlib`` module instead.
+
+Similarly, the presence of ``__annotations__`` and ``__annotate__`` keys
+in the class dictionary is an implementation detail and should not be relied
+upon.
+
+Rejected alternatives
+---------------------
+
+We considered two broad approaches for dealing with the behavior
+of the ``__annotations__`` and ``__annotate__`` entries in classes:
+
+* Ensure that the entry is *always* present in the class dictionary, even if it
+  is empty or has not yet been evaluated. This means we do not have to rely on
+  the descriptors defined on :py:class:`type` to fill in the field, and
+  therefore the metaclass's attributes will not interfere. (Prototype
+  in `gh-120719 <https://github.com/python/cpython/pull/120719>`__.)
+* Ensure that the entry is *never* present in the class dictionary, or at least
+  never added by logic in the language core. This means that the descriptors
+  on :py:class:`type` will always be used, without interference from the metaclass.
+  (Prototype in `gh-120816 <https://github.com/python/cpython/pull/120816>`__.)
+
+Alex Waygood suggested an implementation using the first approach. When a
+heap type (such as a class created through the ``class`` statement) is created,
+``cls.__dict__["__annotations__"]`` is set to a special descriptor.
+On ``__get__``, the descriptor evaluates the annotations by calling ``__annotate__``
+and returning the result. The annotations dictionary is cached within the
+descriptor instance. The descriptor also behaves like a mapping,
+so that code that uses ``cls.__dict__["__annotations__"]`` will still usually
+work: treating the object as a mapping will evaluate the annotations and behave
+as if the descriptor itself was the annotations dictionary. (Code that assumes
+that ``cls.__dict__["__annotations__"]`` is specifically an instance of ``dict``
+may break, however.)
+
+This approach is also straightforward to implement for ``__annotate__``: this
+attribute is already always set for classes with annotations, and we can set
+it explicitly to ``None`` for classes without annotations.
+
+While this approach would fix the known edge cases with metaclasses, it
+introduces significant complexity to all classes, including a new built-in type
+(for the annotations descriptor) with unusual behavior.
+
+The alternative approach would be to never set ``__dict__["__annotations__"]``
+and use some other storage to store the cached annotations. This behavior
+change would have to apply even to classes defined under
+``from __future__ import annotations``, because otherwise there could be buggy
+behavior if a class is defined without ``from __future__ import annotations``
+but its metaclass does have the future enabled. As :pep:`649` previously noted,
+removing ``__annotations__`` from class dictionaries also has backwards compatibility
+implications: ``cls.__dict__.get("__annotations__")`` is a common idiom to
+retrieve annotations.
+
+This approach would also mean that accessing ``.__annotations__`` on an instance
+of an annotated class no longer works. While this behavior is not documented,
+it is a long-standing feature of Python and is relied upon by some users.
+
 Remove code flag for marking ``__annotate__`` functions
 =======================================================
 
@@ -695,7 +856,9 @@ Acknowledgments
 First of all, I thank Larry Hastings for writing :pep:`649`. This PEP modifies some of his
 initial decisions, but the overall design is still his.
 
-I thank Carl Meyer and Alex Waygood for feedback on early drafts of this PEP.
+I thank Carl Meyer and Alex Waygood for feedback on early drafts of this PEP. Alex Waygood,
+Alyssa Coghlan, and David Ellis provided insightful feedback and suggestions on the
+interaction between metaclasses and ``__annotations__``.
 
 Appendix
 ========