568 lines
22 KiB
Plaintext
568 lines
22 KiB
Plaintext
PEP: 422
|
|
Title: Simpler customisation of class creation
|
|
Version: $Revision$
|
|
Last-Modified: $Date$
|
|
Author: Nick Coghlan <ncoghlan@gmail.com>,
|
|
Daniel Urban <urban.dani+py@gmail.com>
|
|
Status: Deferred
|
|
Type: Standards Track
|
|
Content-Type: text/x-rst
|
|
Created: 5-Jun-2012
|
|
Python-Version: 3.5
|
|
Post-History: 5-Jun-2012, 10-Feb-2013
|
|
|
|
|
|
Abstract
|
|
========
|
|
|
|
Currently, customising class creation requires the use of a custom metaclass.
|
|
This custom metaclass then persists for the entire lifecycle of the class,
|
|
creating the potential for spurious metaclass conflicts.
|
|
|
|
This PEP proposes to instead support a wide range of customisation
|
|
scenarios through a new ``namespace`` parameter in the class header, and
|
|
a new ``__autodecorate__`` hook in the class body.
|
|
|
|
The new mechanism should be easier to understand and use than
|
|
implementing a custom metaclass, and thus should provide a gentler
|
|
introduction to the full power Python's metaclass machinery.
|
|
|
|
|
|
PEP Deferral
|
|
============
|
|
|
|
Currently deferred pending updates to address feedback received prior to the
|
|
Python 3.4 release (see TODO note below).
|
|
|
|
Note that I'd be open to relinquishing the PEP to a new champion if anyone
|
|
would like to propose it for Python 3.5 (while I've made a few updates to
|
|
the design recently it doesn't currently look like I will be in a position
|
|
to push it forward myself in the 3.5 time frame).
|
|
|
|
|
|
Background
|
|
==========
|
|
|
|
For an already created class ``cls``, the term "metaclass" has a clear
|
|
meaning: it is the value of ``type(cls)``.
|
|
|
|
*During* class creation, it has another meaning: it is also used to refer to
|
|
the metaclass hint that may be provided as part of the class definition.
|
|
While in many cases these two meanings end up referring to one and the same
|
|
object, there are two situations where that is not the case:
|
|
|
|
* If the metaclass hint refers to an instance of ``type``, then it is
|
|
considered as a candidate metaclass along with the metaclasses of all of
|
|
the parents of the class being defined. If a more appropriate metaclass is
|
|
found amongst the candidates, then it will be used instead of the one
|
|
given in the metaclass hint.
|
|
* Otherwise, an explicit metaclass hint is assumed to be a factory function
|
|
and is called directly to create the class object. In this case, the final
|
|
metaclass will be determined by the factory function definition. In the
|
|
typical case (where the factory functions just calls ``type``, or, in
|
|
Python 3.3 or later, ``types.new_class``) the actual metaclass is then
|
|
determined based on the parent classes.
|
|
|
|
It is notable that only the actual metaclass is inherited - a factory
|
|
function used as a metaclass hook sees only the class currently being
|
|
defined, and is not invoked for any subclasses.
|
|
|
|
In Python 3, the metaclass hint is provided using the ``metaclass=Meta``
|
|
keyword syntax in the class header. This allows the ``__prepare__`` method
|
|
on the metaclass to be used to create the ``locals()`` namespace used during
|
|
execution of the class body (for example, specifying the use of
|
|
``collections.OrderedDict`` instead of a regular ``dict``).
|
|
|
|
In Python 2, there was no ``__prepare__`` method (that API was added for
|
|
Python 3 by PEP 3115). Instead, a class body could set the ``__metaclass__``
|
|
attribute, and the class creation process would extract that value from the
|
|
class namespace to use as the metaclass hint. There is `published code`_ that
|
|
makes use of this feature.
|
|
|
|
Another new feature in Python 3 is the zero-argument form of the ``super()``
|
|
builtin, introduced by PEP 3135. This feature uses an implicit ``__class__``
|
|
reference to the class being defined to replace the "by name" references
|
|
required in Python 2. Just as code invoked during execution of a Python 2
|
|
metaclass could not call methods that referenced the class by name (as the
|
|
name had not yet been bound in the containing scope), similarly, Python 3
|
|
metaclasses cannot call methods that rely on the implicit ``__class__``
|
|
reference (as it is not populated until after the metaclass has returned
|
|
control to the class creation machinery).
|
|
|
|
Finally, when a class uses a custom metaclass, it can pose additional
|
|
challenges to the use of multiple inheritance, as a new class cannot
|
|
inherit from parent classes with unrelated metaclasses. This means that
|
|
it is impossible to add a metaclass to an already published class: such
|
|
an addition is a backwards incompatible change due to the risk of metaclass
|
|
conflicts.
|
|
|
|
|
|
Proposal
|
|
========
|
|
|
|
This PEP proposes that a new mechanism to customise class creation be
|
|
added to Python 3.4 that meets the following criteria:
|
|
|
|
1. Integrates nicely with class inheritance structures (including mixins and
|
|
multiple inheritance)
|
|
2. Integrates nicely with the implicit ``__class__`` reference and
|
|
zero-argument ``super()`` syntax introduced by PEP 3135
|
|
3. Can be added to an existing base class without a significant risk of
|
|
introducing backwards compatibility problems
|
|
4. Restores the ability for class namespaces to have some influence on the
|
|
class creation process (above and beyond populating the namespace itself),
|
|
but potentially without the full flexibility of the Python 2 style
|
|
``__metaclass__`` hook
|
|
|
|
One mechanism that can achieve this goal is to add a new implicit class
|
|
decoration hook, modelled directly on the existing explicit class
|
|
decorators, but defined in the class body or in a parent class, rather than
|
|
being part of the class definition header.
|
|
|
|
Specifically, it is proposed that class definitions be able to provide a
|
|
class initialisation hook as follows::
|
|
|
|
class Example:
|
|
def __autodecorate__(cls):
|
|
# This is invoked after the class is created, but before any
|
|
# explicit decorators are called
|
|
# The usual super() mechanisms are used to correctly support
|
|
# multiple inheritance. The class decorator style signature helps
|
|
# ensure that invoking the parent class is as simple as possible.
|
|
cls = super().__autodecorate__()
|
|
return cls
|
|
|
|
To simplify the cooperative multiple inheritance case, ``object`` will gain
|
|
a default implementation of the hook that returns the class unmodified:
|
|
|
|
class object:
|
|
def __autodecorate__(cls):
|
|
return cls
|
|
|
|
If a metaclass wishes to block implicit class decoration for some reason, it
|
|
must arrange for ``cls.__autodecorate__`` to trigger ``AttributeError``.
|
|
|
|
If present on the created object, this new hook will be called by the class
|
|
creation machinery *after* the ``__class__`` reference has been initialised.
|
|
For ``types.new_class()``, it will be called as the last step before
|
|
returning the created class object. ``__autodecorate__`` is implicitly
|
|
converted to a class method when the class is created (prior to the hook
|
|
being invoked).
|
|
|
|
Note, that when ``__autodecorate__`` is called, the name of the class is not
|
|
yet bound to the new class object. As a consequence, the two argument form
|
|
of ``super()`` cannot be used to call methods (e.g., ``super(Example, cls)``
|
|
wouldn't work in the example above). However, the zero argument form of
|
|
``super()`` works as expected, since the ``__class__`` reference is already
|
|
initialised.
|
|
|
|
This general proposal is not a new idea (it was first suggested for
|
|
inclusion in the language definition `more than 10 years ago`_, and a
|
|
similar mechanism has long been supported by `Zope's ExtensionClass`_),
|
|
but the situation has changed sufficiently in recent years that
|
|
the idea is worth reconsidering for inclusion as a native language feature.
|
|
|
|
In addition, the introduction of the metaclass ``__prepare__`` method in PEP
|
|
3115 allows a further enhancement that was not possible in Python 2: this
|
|
PEP also proposes that ``type.__prepare__`` be updated to accept a factory
|
|
function as a ``namespace`` keyword-only argument. If present, the value
|
|
provided as the ``namespace`` argument will be called without arguments
|
|
to create the result of ``type.__prepare__`` instead of using a freshly
|
|
created dictionary instance. For example, the following will use
|
|
an ordered dictionary as the class namespace::
|
|
|
|
class OrderedExample(namespace=collections.OrderedDict):
|
|
def __autodecorate__(cls):
|
|
# cls.__dict__ is still a read-only proxy to the class namespace,
|
|
# but the underlying storage is an OrderedDict instance
|
|
|
|
.. note::
|
|
|
|
This PEP, along with the existing ability to use __prepare__ to share a
|
|
single namespace amongst multiple class objects, highlights a possible
|
|
issue with the attribute lookup caching: when the underlying mapping is
|
|
updated by other means, the attribute lookup cache is not invalidated
|
|
correctly (this is a key part of the reason class ``__dict__`` attributes
|
|
produce a read-only view of the underlying storage).
|
|
|
|
Since the optimisation provided by that cache is highly desirable,
|
|
the use of a preexisting namespace as the class namespace may need to
|
|
be declared as officially unsupported (since the observed behaviour is
|
|
rather strange when the caches get out of sync).
|
|
|
|
|
|
Key Benefits
|
|
============
|
|
|
|
|
|
Easier use of custom namespaces for a class
|
|
-------------------------------------------
|
|
|
|
Currently, to use a different type (such as ``collections.OrderedDict``) for
|
|
a class namespace, or to use a pre-populated namespace, it is necessary to
|
|
write and use a custom metaclass. With this PEP, using a custom namespace
|
|
becomes as simple as specifying an appropriate factory function in the
|
|
class header.
|
|
|
|
|
|
Easier inheritance of definition time behaviour
|
|
-----------------------------------------------
|
|
|
|
Understanding Python's metaclasses requires a deep understanding of
|
|
the type system and the class construction process. This is legitimately
|
|
seen as challenging, due to the need to keep multiple moving parts (the code,
|
|
the metaclass hint, the actual metaclass, the class object, instances of the
|
|
class object) clearly distinct in your mind. Even when you know the rules,
|
|
it's still easy to make a mistake if you're not being extremely careful.
|
|
An earlier version of this PEP actually included such a mistake: it
|
|
stated "subclass of type" for a constraint that is actually "instance of
|
|
type".
|
|
|
|
Understanding the proposed implicit class decoration hook only requires
|
|
understanding decorators and ordinary method inheritance, which isn't
|
|
quite as daunting a task. The new hook provides a more gradual path
|
|
towards understanding all of the phases involved in the class definition
|
|
process.
|
|
|
|
|
|
Reduced chance of metaclass conflicts
|
|
-------------------------------------
|
|
|
|
One of the big issues that makes library authors reluctant to use metaclasses
|
|
(even when they would be appropriate) is the risk of metaclass conflicts.
|
|
These occur whenever two unrelated metaclasses are used by the desired
|
|
parents of a class definition. This risk also makes it very difficult to
|
|
*add* a metaclass to a class that has previously been published without one.
|
|
|
|
By contrast, adding an ``__autodecorate__`` method to an existing type poses
|
|
a similar level of risk to adding an ``__init__`` method: technically, there
|
|
is a risk of breaking poorly implemented subclasses, but when that occurs,
|
|
it is recognised as a bug in the subclass rather than the library author
|
|
breaching backwards compatibility guarantees. In fact, due to the constrained
|
|
signature of ``__autodecorate__``, the risk in this case is actually even
|
|
lower than in the case of ``__init__``.
|
|
|
|
|
|
Integrates cleanly with \PEP 3135
|
|
---------------------------------
|
|
|
|
Unlike code that runs as part of the metaclass, code that runs as part of
|
|
the new hook will be able to freely invoke class methods that rely on the
|
|
implicit ``__class__`` reference introduced by PEP 3135, including methods
|
|
that use the zero argument form of ``super()``.
|
|
|
|
|
|
Replaces many use cases for dynamic setting of ``__metaclass__``
|
|
-----------------------------------------------------------------
|
|
|
|
For use cases that don't involve completely replacing the defined class,
|
|
Python 2 code that dynamically set ``__metaclass__`` can now dynamically
|
|
set ``__autodecorate__`` instead. For more advanced use cases, introduction of
|
|
an explicit metaclass (possibly made available as a required base class) will
|
|
still be necessary in order to support Python 3.
|
|
|
|
|
|
Design Notes
|
|
============
|
|
|
|
|
|
Determining if the class being decorated is the base class
|
|
----------------------------------------------------------
|
|
|
|
In the body of an ``__autodecorate__`` method, as in any other class method,
|
|
``__class__`` will be bound to the class declaring the method, while the
|
|
value passed in may be a subclass.
|
|
|
|
This makes it relatively straightforward to skip processing the base class
|
|
if necessary::
|
|
|
|
class Example:
|
|
def __autodecorate__(cls):
|
|
cls = super().__autodecorate__()
|
|
# Don't process the base class
|
|
if cls is __class__:
|
|
return
|
|
# Process subclasses here
|
|
...
|
|
|
|
|
|
Replacing a class with a different kind of object
|
|
-------------------------------------------------
|
|
|
|
As an implicit decorator, ``__autodecorate__`` is able to relatively easily
|
|
replace the defined class with a different kind of object. Technically
|
|
custom metaclasses and even ``__new__`` methods can already do this
|
|
implicitly, but the decorator model makes such code much easier to understand
|
|
and implement.
|
|
|
|
class BuildDict:
|
|
def __autodecorate__(cls):
|
|
cls = super().__autodecorate__()
|
|
# Don't process the base class
|
|
if cls is __class__:
|
|
return
|
|
# Convert subclasses to ordinary dictionaries
|
|
return cls.__dict__.copy()
|
|
|
|
It's not clear why anyone would ever do this implicitly based on inheritance
|
|
rather than just using an explicit decorator, but the possibility seems worth
|
|
noting.
|
|
|
|
|
|
Open Questions
|
|
==============
|
|
|
|
Is the ``namespace`` concept worth the extra complexity?
|
|
--------------------------------------------------------
|
|
|
|
Unlike the new ``__autodecorate__`` hook the proposed ``namespace`` keyword
|
|
argument is not automatically inherited by subclasses. Given the way this
|
|
proposal is currently written , the only way to get a special namespace used
|
|
consistently in subclasses is still to write a custom metaclass with a
|
|
suitable ``__prepare__`` implementation.
|
|
|
|
Changing the custom namespace factory to also be inherited would
|
|
significantly increase the complexity of this proposal, and introduce a
|
|
number of the same potential base class conflict issues as arise with the
|
|
use of custom metaclasses.
|
|
|
|
Eric Snow has put forward a
|
|
`separate proposal <https://mail.python.org/pipermail/python-dev/2013-June/127103.html>`__
|
|
to instead make the execution namespace for class bodies an ordered dictionary
|
|
by default, and capture the class attribute definition order for future
|
|
reference as an attribute (e.g. ``__definition_order__``) on the class object.
|
|
|
|
Eric's suggested approach may be a better choice for a new default behaviour
|
|
for type that combines well with the proposed ``__autodecorate__`` hook,
|
|
leaving the more complex configurable namespace factory idea to a custom
|
|
metaclass like the one shown below.
|
|
|
|
|
|
New Ways of Using Classes
|
|
=========================
|
|
|
|
The new ``namespace`` keyword in the class header enables a number of
|
|
interesting options for controlling the way a class is initialised,
|
|
including some aspects of the object models of both Javascript and Ruby.
|
|
|
|
All of the examples below are actually possible today through the use of a
|
|
custom metaclass::
|
|
|
|
class CustomNamespace(type):
|
|
@classmethod
|
|
def __prepare__(meta, name, bases, *, namespace=None, **kwds):
|
|
parent_namespace = super().__prepare__(name, bases, **kwds)
|
|
return namespace() if namespace is not None else parent_namespace
|
|
def __new__(meta, name, bases, ns, *, namespace=None, **kwds):
|
|
return super().__new__(meta, name, bases, ns, **kwds)
|
|
def __init__(cls, name, bases, ns, *, namespace=None, **kwds):
|
|
return super().__init__(name, bases, ns, **kwds)
|
|
|
|
The advantage of implementing the new keyword directly in
|
|
``type.__prepare__`` is that the *only* persistent effect is then
|
|
the change in the underlying storage of the class attributes. The metaclass
|
|
of the class remains unchanged, eliminating many of the drawbacks
|
|
typically associated with these kinds of customisations.
|
|
|
|
|
|
Order preserving classes
|
|
------------------------
|
|
|
|
::
|
|
|
|
class OrderedClass(namespace=collections.OrderedDict):
|
|
a = 1
|
|
b = 2
|
|
c = 3
|
|
|
|
|
|
Prepopulated namespaces
|
|
-----------------------
|
|
|
|
::
|
|
|
|
seed_data = dict(a=1, b=2, c=3)
|
|
class PrepopulatedClass(namespace=seed_data.copy):
|
|
pass
|
|
|
|
|
|
Cloning a prototype class
|
|
-------------------------
|
|
|
|
::
|
|
|
|
class NewClass(namespace=Prototype.__dict__.copy):
|
|
pass
|
|
|
|
|
|
Extending a class
|
|
-----------------
|
|
|
|
.. note:: Just because the PEP makes it *possible* to do this relatively
|
|
cleanly doesn't mean anyone *should* do this!
|
|
|
|
::
|
|
|
|
from collections import MutableMapping
|
|
|
|
# The MutableMapping + dict combination should give something that
|
|
# generally behaves correctly as a mapping, while still being accepted
|
|
# as a class namespace
|
|
class ClassNamespace(MutableMapping, dict):
|
|
def __init__(self, cls):
|
|
self._cls = cls
|
|
def __len__(self):
|
|
return len(dir(self._cls))
|
|
def __iter__(self):
|
|
for attr in dir(self._cls):
|
|
yield attr
|
|
def __contains__(self, attr):
|
|
return hasattr(self._cls, attr)
|
|
def __getitem__(self, attr):
|
|
return getattr(self._cls, attr)
|
|
def __setitem__(self, attr, value):
|
|
setattr(self._cls, attr, value)
|
|
def __delitem__(self, attr):
|
|
delattr(self._cls, attr)
|
|
|
|
def extend(cls):
|
|
return lambda: ClassNamespace(cls)
|
|
|
|
class Example:
|
|
pass
|
|
|
|
class ExtendedExample(namespace=extend(Example)):
|
|
a = 1
|
|
b = 2
|
|
c = 3
|
|
|
|
>>> Example.a, Example.b, Example.c
|
|
(1, 2, 3)
|
|
|
|
|
|
Rejected Design Options
|
|
=======================
|
|
|
|
|
|
Calling ``__autodecorate__`` from ``type.__init__``
|
|
---------------------------------------------------
|
|
|
|
Calling the new hook automatically from ``type.__init__``, would achieve most
|
|
of the goals of this PEP. However, using that approach would mean that
|
|
``__autodecorate__`` implementations would be unable to call any methods that
|
|
relied on the ``__class__`` reference (or used the zero-argument form of
|
|
``super()``), and could not make use of those features themselves.
|
|
|
|
The current design instead ensures that the implicit decorator hook is able
|
|
to do anything an explicit decorator can do by running it after the initial
|
|
class creation is already complete.
|
|
|
|
Calling the automatic decoration hook ``__init_class__``
|
|
--------------------------------------------------------
|
|
|
|
Earlier versions of the PEP used the name ``__init_class__`` for the name
|
|
of the new hook. There were three significant problems with this name:
|
|
|
|
* it was hard to remember if the correct spelling was ``__init_class__`` or
|
|
``__class_init__``
|
|
* the use of "init" in the name suggested the signature should match that
|
|
of ``type.__init__``, which is not the case
|
|
* the use of "init" in the name suggested the method would be run as part
|
|
of initial class object creation, which is not the case
|
|
|
|
The new name ``__autodecorate__`` was chosen to make it clear that the new
|
|
initialisation hook is most usefully thought of as an implicitly invoked
|
|
class decorator, rather than as being like an ``__init__`` method.
|
|
|
|
|
|
Requiring an explicit decorator on ``__autodecorate__``
|
|
-------------------------------------------------------
|
|
|
|
Originally, this PEP required the explicit use of ``@classmethod`` on the
|
|
``__autodecorate__`` decorator. It was made implicit since there's no
|
|
sensible interpretation for leaving it out, and that case would need to be
|
|
detected anyway in order to give a useful error message.
|
|
|
|
This decision was reinforced after noticing that the user experience of
|
|
defining ``__prepare__`` and forgetting the ``@classmethod`` method
|
|
decorator is singularly incomprehensible (particularly since PEP 3115
|
|
documents it as an ordinary method, and the current documentation doesn't
|
|
explicitly say anything one way or the other).
|
|
|
|
|
|
Making ``__autodecorate__`` implicitly static, like ``__new__``
|
|
---------------------------------------------------------------
|
|
|
|
While it accepts the class to be instantiated as the first argument,
|
|
``__new__`` is actually implicitly treated as a static method rather than
|
|
as a class method. This allows it to be readily extracted from its
|
|
defining class and called directly on a subclass, rather than being
|
|
coupled to the class object it is retrieved from.
|
|
|
|
Such behaviour initially appears to be potentially useful for the
|
|
new ``__autodecorate__`` hook, as it would allow ``__autodecorate__``
|
|
methods to readily be used as explicit decorators on other classes.
|
|
|
|
However, that apparent support would be an illusion as it would only work
|
|
correctly if invoked on a subclass, in which case the method can just as
|
|
readily be retrieved from the subclass and called that way. Unlike
|
|
``__new__``, there's no issue with potentially changing method signatures at
|
|
different points in the inheritance chain.
|
|
|
|
|
|
Passing in the namespace directly rather than a factory function
|
|
----------------------------------------------------------------
|
|
|
|
At one point, this PEP proposed that the class namespace be passed
|
|
directly as a keyword argument, rather than passing a factory function.
|
|
However, this encourages an unsupported behaviour (that is, passing the
|
|
same namespace to multiple classes, or retaining direct write access
|
|
to a mapping used as a class namespace), so the API was switched to
|
|
the factory function version.
|
|
|
|
|
|
Reference Implementation
|
|
========================
|
|
|
|
A reference implementation for ``__autodecorate__`` has been posted to the
|
|
`issue tracker`_. It uses the original ``__init_class__`` naming. does not yet
|
|
allow the implicit decorator to replace the class with a different object and
|
|
does not implement the suggested ``namespace`` parameter for
|
|
``type.__prepare__``.
|
|
|
|
TODO
|
|
====
|
|
|
|
* address the 5 points in http://mail.python.org/pipermail/python-dev/2013-February/123970.html
|
|
|
|
References
|
|
==========
|
|
|
|
.. _published code:
|
|
http://mail.python.org/pipermail/python-dev/2012-June/119878.html
|
|
|
|
.. _more than 10 years ago:
|
|
http://mail.python.org/pipermail/python-dev/2001-November/018651.html
|
|
|
|
.. _Zope's ExtensionClass:
|
|
http://docs.zope.org/zope_secrets/extensionclass.html
|
|
|
|
.. _issue tracker:
|
|
http://bugs.python.org/issue17044
|
|
|
|
Copyright
|
|
=========
|
|
|
|
This document has been placed in the public domain.
|
|
|
|
|
|
..
|
|
Local Variables:
|
|
mode: indented-text
|
|
indent-tabs-mode: nil
|
|
sentence-end-double-space: t
|
|
fill-column: 70
|
|
coding: utf-8
|
|
End:
|
|
|