PEP 487: New version from Martin

pep-0487.txt

@@ -7,8 +7,8 @@ Status: Draft
 Type: Standards Track
 Content-Type: text/x-rst
 Created: 27-Feb-2015
-Python-Version: 3.5
+Python-Version: 3.6
-Post-History: 27-Feb-2015
+Post-History: 27-Feb-2015, 5-Feb-2016
 Replaces: 422
 
 
@@ -20,127 +20,80 @@ This custom metaclass then persists for the entire lifecycle of the class,
 creating the potential for spurious metaclass conflicts.
 
 This PEP proposes to instead support a wide range of customisation
-scenarios through a new ``namespace`` parameter in the class header, and
+scenarios through a new ``__init_subclass__`` hook in the class body,
-a new ``__init_subclass__`` hook in the class body.
+a hook to initialize descriptors, and a way to keep the order in which
+attributes are defined.
+
+Those hooks should at first be defined in a metaclass in the standard
+library, with the option that this metaclass eventually becomes the
+default ``type`` metaclass.
 
 The new mechanism should be easier to understand and use than
 implementing a custom metaclass, and thus should provide a gentler
 introduction to the full power Python's metaclass machinery.
 
 
-Connection to other PEP
-=======================
-
-This is a competing proposal to PEP 422 by Nick Coughlan and Daniel Urban.
-It shares both most of the PEP text and proposed code, but has major
-differences in how to achieve its goals.
-
 Background
 ==========
 
-For an already created class ``cls``, the term "metaclass" has a clear
+Metaclasses are a powerful tool to customize class creation. They have,
-meaning: it is the value of ``type(cls)``.
+however, the problem that there is no automatic way to combine metaclasses.
-
+If one wants to use two metaclasses for a class, a new metaclass combining
-*During* class creation, it has another meaning: it is also used to refer to
+those two needs to be created, typically manually.
-the metaclass hint that may be provided as part of the class definition.
-While in many cases these two meanings end up referring to one and the same
-object, there are two situations where that is not the case:
-
-* If the metaclass hint refers to an instance of ``type``, then it is
-  considered as a candidate metaclass along with the metaclasses of all of
-  the parents of the class being defined. If a more appropriate metaclass is
-  found amongst the candidates, then it will be used instead of the one
-  given in the metaclass hint.
-* Otherwise, an explicit metaclass hint is assumed to be a factory function
-  and is called directly to create the class object. In this case, the final
-  metaclass will be determined by the factory function definition. In the
-  typical case (where the factory functions just calls ``type``, or, in
-  Python 3.3 or later, ``types.new_class``) the actual metaclass is then
-  determined based on the parent classes.
-
-It is notable that only the actual metaclass is inherited - a factory
-function used as a metaclass hook sees only the class currently being
-defined, and is not invoked for any subclasses.
-
-In Python 3, the metaclass hint is provided using the ``metaclass=Meta``
-keyword syntax in the class header. This allows the ``__prepare__`` method
-on the metaclass to be used to create the ``locals()`` namespace used during
-execution of the class body (for example, specifying the use of
-``collections.OrderedDict`` instead of a regular ``dict``).
-
-In Python 2, there was no ``__prepare__`` method (that API was added for
-Python 3 by PEP 3115). Instead, a class body could set the ``__metaclass__``
-attribute, and the class creation process would extract that value from the
-class namespace to use as the metaclass hint. There is `published code`_ that
-makes use of this feature.
-
-Another new feature in Python 3 is the zero-argument form of the ``super()``
-builtin, introduced by PEP 3135. This feature uses an implicit ``__class__``
-reference to the class being defined to replace the "by name" references
-required in Python 2. Just as code invoked during execution of a Python 2
-metaclass could not call methods that referenced the class by name (as the
-name had not yet been bound in the containing scope), similarly, Python 3
-metaclasses cannot call methods that rely on the implicit ``__class__``
-reference (as it is not populated until after the metaclass has returned
-control to the class creation machinery).
-
-Finally, when a class uses a custom metaclass, it can pose additional
-challenges to the use of multiple inheritance, as a new class cannot
-inherit from parent classes with unrelated metaclasses. This means that
-it is impossible to add a metaclass to an already published class: such
-an addition is a backwards incompatible change due to the risk of metaclass
-conflicts.
 
+This need often occurs as a surprise to a user: inheriting from two base
+classes coming from two different libraries suddenly raises the necessity
+to manually create a combined metaclass, where typically one is not
+interested in those details about the libraries at all. This becomes
+even worse if one library starts to make use of a metaclass which it
+has not done before. While the library itself continues to work perfectly,
+suddenly every code combining those classes with classes from another library
+fails.
 
 Proposal
 ========
 
-This PEP proposes that a new mechanism to customise class creation be
+While there are many possible ways to use a metaclass, the vast majority
-added to Python 3.5 that meets the following criteria:
+of use cases falls into just three categories: some initialization code
+running after class creation, the initalization of descriptors and
+keeping the order in which class attributes were defined.
 
-1. Integrates nicely with class inheritance structures (including mixins and
+Those three use cases can easily be performed by just one metaclass. If
-   multiple inheritance),
+this metaclass is put into the standard library, and all libraries that
-2. Integrates nicely with the implicit ``__class__`` reference and
+wish to customize class creation use this very metaclass, no combination
-   zero-argument ``super()`` syntax introduced by PEP 3135,
+of metaclasses is necessary anymore.
-3. Can be added to an existing base class without a significant risk of
-   introducing backwards compatibility problems, and
-4. Restores the ability for class namespaces to have some influence on the
-   class creation process (above and beyond populating the namespace itself),
-   but potentially without the full flexibility of the Python 2 style
-   ``__metaclass__`` hook.
 
-Those goals can be achieved by adding two functionalities:
+The three use cases are achieved as follows:
 
-1. A ``__init_subclass__`` hook that initializes all subclasses of a
+1. The metaclass contains an ``__init_subclass__`` hook that initializes
-   given class, and
+   all subclasses of a given class,
-2. A new keyword parameter ``namespace`` to the class creation statement,
+2. the metaclass calls an ``__init_descriptor__`` hook for all descriptors
-   that gives an initialization of the namespace.
+   defined in the class, and
+3. an ``__attribute_order__`` tuple is left in the class in order to inspect
+   the order in which attributes were defined.
 
-As an example, the first proposal looks as follows::
+For ease of use, a base class ``SubclassInit`` is defined, which uses said
+metaclass and contains an empty stub for the hook described for use case 1.
 
-   class SpamBase:
+As an example, the first use case looks as follows::
+
+   class SpamBase(SubclassInit):
        # this is implicitly a @classmethod
-       def __init_subclass__(cls, ns, **kwargs):
+       def __init_subclass__(cls, **kwargs):
            # This is invoked after a subclass is created, but before
            # explicit decorators are called.
            # The usual super() mechanisms are used to correctly support
            # multiple inheritance.
-  # ns is the classes namespace
            # **kwargs are the keyword arguments to the subclasses'
            # class creation statement
-           super().__init_subclass__(cls, ns, **kwargs)
+           super().__init_subclass__(cls, **kwargs)
 
    class Spam(SpamBase):
        pass
    # the new hook is called on Spam
 
-To simplify the cooperative multiple inheritance case, ``object`` will gain
+The base class ``SubclassInit`` contains an empty ``__init_subclass__``
-a default implementation of the hook that does nothing::
+method which serves as an endpoint for cooperative multiple inheritance.
-
-   class object:
-       def __init_subclass__(cls, ns):
-           pass
-
 Note that this method has no keyword arguments, meaning that all
 methods which are more specialized have to process all keyword
 arguments.
@@ -151,58 +104,46 @@ similar mechanism has long been supported by `Zope's ExtensionClass`_),
 but the situation has changed sufficiently in recent years that
 the idea is worth reconsidering for inclusion.
 
-The second part of the proposal is to have a ``namespace`` keyword
+The second part of the proposal adds an ``__init_descriptor__``
-argument to the class declaration statement. If present, its value
+initializer for descriptors.  Descriptors are defined in the body of a
-will be called without arguments to initialize a subclasses
+class, but they do not know anything about that class, they do not
-namespace, very much like a metaclass ``__prepare__`` method would
+even know the name they are accessed with. They do get to know their
-do.
+owner once ``__get__`` is called, but still they do not know their
+name. This is unfortunate, for example they cannot put their
+associated value into their object's ``__dict__`` under their name,
+since they do not know that name.  This problem has been solved many
+times, and is one of the most important reasons to have a metaclass in
+a library. While it would be easy to implement such a mechanism using
+the first part of the proposal, it makes sense to have one solution
+for this problem for everyone.
 
-In addition, the introduction of the metaclass ``__prepare__`` method
+To give an example of its usage, imagine a descriptor representing weak
-in PEP 3115 allows a further enhancement that was not possible in
+referenced values (this is an insanely simplified, yet working example)::
-Python 2: this PEP also proposes that ``type.__prepare__`` be updated
-to accept a factory function as a ``namespace`` keyword-only argument.
-If present, the value provided as the ``namespace`` argument will be
-called without arguments to create the result of ``type.__prepare__``
-instead of using a freshly created dictionary instance. For example,
-the following will use an ordered dictionary as the class namespace::
 
-   class OrderedBase(namespace=collections.OrderedDict):
+    import weakref
-        pass
 
-   class Ordered(OrderedBase):
+    class WeakAttribute:
-        # cls.__dict__ is still a read-only proxy to the class namespace,
+        def __get__(self, instance, owner):
-        # but the underlying storage is an OrderedDict instance
+            return instance.__dict__[self.name]
 
+        def __set__(self, instance, value):
+            instance.__dict__[self.name] = weakref.ref(value)
 
-.. note::
+        # this is the new initializer:
+        def __init_descriptor__(self, owner, name):
+            self.name = name
 
-    This PEP, along with the existing ability to use  __prepare__ to share a
+The third part of the proposal is to leave a tuple called
-    single namespace amongst multiple class objects, highlights a possible
+``__attribute_order__`` in the class that contains the order in which
-    issue with the attribute lookup caching: when the underlying mapping is
+the attributes were defined. This is a very common usecase, many
-    updated by other means, the attribute lookup cache is not invalidated
+libraries use an ``OrderedDict`` to store this order. This is a very
-    correctly (this is a key part of the reason class ``__dict__`` attributes
+simple way to achieve the same goal.
-    produce a read-only view of the underlying storage).
-
-    Since the optimisation provided by that cache is highly desirable,
-    the use of a preexisting namespace as the class namespace may need to
-    be declared as officially unsupported (since the observed behaviour is
-    rather strange when the caches get out of sync).
 
 
 Key Benefits
 ============
 
 
-Easier use of custom namespaces for a class
--------------------------------------------
-
-Currently, to use a different type (such as ``collections.OrderedDict``) for
-a class namespace, or to use a pre-populated namespace, it is necessary to
-write and use a custom metaclass. With this PEP, using a custom namespace
-becomes as simple as specifying an appropriate factory function in the
-class header.
-
-
 Easier inheritance of definition time behaviour
 -----------------------------------------------
 
@@ -235,40 +176,20 @@ it is recognised as a bug in the subclass rather than the library author
 breaching backwards compatibility guarantees.
 
 
-Integrates cleanly with \PEP 3135
----------------------------------
-
-Given that the method is called on already existing classes, the new
-hook will be able to freely invoke class methods that rely on the
-implicit ``__class__`` reference introduced by PEP 3135, including
-methods that use the zero argument form of ``super()``.
-
-
-Replaces many use cases for dynamic setting of ``__metaclass__``
-----------------------------------------------------------------
-
-For use cases that don't involve completely replacing the defined
-class, Python 2 code that dynamically set ``__metaclass__`` can now
-dynamically set ``__init_subclass__`` instead. For more advanced use
-cases, introduction of an explicit metaclass (possibly made available
-as a required base class) will still be necessary in order to support
-Python 3.
-
-
 A path of introduction into Python
 ==================================
 
 Most of the benefits of this PEP can already be implemented using
 a simple metaclass. For the ``__init_subclass__`` hook this works
-all the way down to python 2.7, while the namespace needs python 3.0
+all the way down to Python 2.7, while the attribute order needs Python 3.0
 to work. Such a class has been `uploaded to PyPI`_.
 
 The only drawback of such a metaclass are the mentioned problems with
 metaclasses and multiple inheritance. Two classes using such a
 metaclass can only be combined, if they use exactly the same such
 metaclass. This fact calls for the inclusion of such a class into the
-standard library, let's call it ``SubclassMeta``, with a base class
+standard library, let's call it ``SubclassMeta``, with the base class
-using it called ``SublassInit``. Once all users use this standard
+using it called ``SubclassInit``. Once all users use this standard
 library metaclass, classes from different packages can easily be
 combined.
 
@@ -277,21 +198,32 @@ using other metaclasses. Authors of metaclasses should bear that in
 mind and inherit from the standard metaclass if it seems useful
 for users of the metaclass to add more functionality. Ultimately,
 if the need for combining with other metaclasses is strong enough,
-the proposed functionality may be introduced into python's ``type``.
+the proposed functionality may be introduced into Python's ``type``.
 
 Those arguments strongly hint to the following procedure to include
-the proposed functionality into python:
+the proposed functionality into Python:
 
 1. The metaclass implementing this proposal is put onto PyPI, so that
    it can be used and scrutinized.
-2. Once the code is properly mature, it can be added to the python
+2. Once the code is properly mature, it can be added to the Python
    standard library. There should be a new module called
    ``metaclass`` which collects tools for metaclass authors, as well
    as a documentation of the best practices of how to write
    metaclasses.
 3. If the need of combining this metaclass with other metaclasses is
-   strong enough, it may be included into python itself.
+   strong enough, it may be included into Python itself.
 
+While the metaclass is still in the standard library and not in the
+language, it may still clash with other metaclasses.  The most
+prominent metaclass in use is probably ABCMeta.  It is also a
+particularly good example for the need of combining metaclasses. For
+users who want to define a ABC with subclass initialization, we should
+support a ``ABCSubclassInit`` class, or let ABCMeta inherit from this
+PEP's metaclass.
+
+Extensions written in C or C++ also often define their own metaclass.
+It would be very useful if those could also inherit from the metaclass
+defined here, but this is probably not possible.
 
 New Ways of Using Classes
 =========================
@@ -309,8 +241,8 @@ subclasses of a plugin baseclass. This can be done as follows::
    class PluginBase(SubclassInit):
        subclasses = []
 
-       def __init_subclass__(cls, ns, **kwargs):
+       def __init_subclass__(cls, **kwargs):
-           super().__init_subclass__(ns, **kwargs)
+           super().__init_subclass__(**kwargs)
            cls.subclasses.append(cls)
 
 One should note that this also works nicely as a mixin class.
@@ -318,7 +250,7 @@ One should note that this also works nicely as a mixin class.
 Trait descriptors
 -----------------
 
-There are many designs of python descriptors in the wild which, for
+There are many designs of Python descriptors in the wild which, for
 example, check boundaries of values. Often those "traits" need some support
 of a metaclass to work. This is how this would look like with this
 PEP::
@@ -330,55 +262,14 @@ PEP::
        def __set__(self, instance, value):
            instance.__dict__[self.key] = value
 
+       def __init_descriptor__(self, owner, name):
+           self.key = name
+
    class Int(Trait):
        def __set__(self, instance, value):
            # some boundary check code here
            super().__set__(instance, value)
 
-   class HasTraits(SubclassInit):
-       def __init_subclass__(cls, ns, **kwargs):
-  super().__init_subclass__(ns, **kwargs)
-           for k, v in ns.items():
-               if isinstance(v, Trait):
-                   v.key = k
-
-The new ``namespace`` keyword in the class header enables a number of
-interesting options for controlling the way a class is initialised,
-including some aspects of the object models of both Javascript and Ruby.
-
-
-Order preserving classes
-------------------------
-
-::
-
-    class OrderedClassBase(namespace=collections.OrderedDict):
-        pass
-
-    class OrderedClass(OrderedClassBase):
-        a = 1
-        b = 2
-        c = 3
-
-
-Prepopulated namespaces
------------------------
-
-::
-
-    seed_data = dict(a=1, b=2, c=3)
-    class PrepopulatedClass(namespace=seed_data.copy):
-        pass
-
-
-Cloning a prototype class
--------------------------
-
-::
-
-    class NewClass(namespace=Prototype.__dict__.copy):
-        pass
-
 
 Rejected Design Options
 =======================
@@ -402,7 +293,7 @@ complete class on its own.
 
 The original proposal also made major changes in the class
 initialization process, rendering it impossible to back-port the
-proposal to older python versions.
+proposal to older Python versions.
 
 
 Other variants of calling the hook
@@ -430,28 +321,47 @@ documents it as an ordinary method, and the current documentation doesn't
 explicitly say anything one way or the other).
 
 
-Passing in the namespace directly rather than a factory function
+Defining arbitrary namespaces
-----------------------------------------------------------------
+-----------------------------
 
-At one point, PEP 422 proposed that the class namespace be passed
+PEP 422 defined a generic way to add arbitrary namespaces for class
-directly as a keyword argument, rather than passing a factory function.
+definitions. This approach is much more flexible than just leaving
-However, this encourages an unsupported behaviour (that is, passing the
+the definition order in a tuple. The ``__prepare__`` method in a metaclass
-same namespace to multiple classes, or retaining direct write access
+supports exactly this behavior. But given that effectively
-to a mapping used as a class namespace), so the API was switched to
+the only use cases that could be found out in the wild were the
-the factory function version.
+``OrderedDict`` way of determining the attribute order, it seemed
+reasonable to only support this special case.
+
+The metaclass described in this PEP has been designed to be very simple
+such that it could be reasonably made the default metaclass. This was
+especially important when designing the attribute order functionality:
+This was a highly demanded feature and has been enabled through the
+``__prepare__`` method of metaclasses. This method can be abused in
+very weird ways, making it hard to correctly maintain this feature in
+CPython. This is why it has been proposed to deprecated this feature,
+and instead use ``OrderedDict`` as the standard namespace, supporting
+the most important feature while dropping most of the complexity. But
+this would have meant that ``OrderedDict`` becomes a language builtin
+like dict and set, and not just a standard library class. The choice
+of the ``__attribute_order__`` tuple is a much simpler solution to the
+problem.
+
+A more ``__new__``-like hook
+----------------------------
+
+In PEP 422 the hook worked more like the ``__new__`` method than the
+``__init__`` method, meaning that it returned a class instead of
+modifying one. This allows a bit more flexibility, but at the cost
+of much harder implementation and undesired side effects.
 
 
-Possible Extensions
+History
-===================
+=======
 
-Some extensions to this PEP are imaginable, which are postponed to a
+This used to be a competing proposal to PEP 422 by Nick Coughlan and
-later pep:
+Daniel Urban. It shares both most of the PEP text and proposed code, but
-
+has major differences in how to achieve its goals. In the meantime, PEP 422
-* A ``__new_subclass__`` method could be defined which acts like a
+has been withdrawn favouring this approach.
-  ``__new__`` for classes. This would be very close to
-  ``__autodecorate__`` in PEP 422.
-* ``__subclasshook__`` could be made a classmethod in a class instead
-  of a method in the metaclass.
 
 References
 ==========