python-peps/pep-3115.txt

PEP: 3115
Title: Metaclasses in Python 3000
Version: $Revision$
Last-Modified: $Date$
Author: Talin <talin at acm.org>
Status: Accepted
Type: Standards
Content-Type: text/plain
Created: 07-Mar-2007
Python-Version: 3.0
Post-History: 11-March-2007, 14-March-2007

Abstract

     This PEP proposes changing the syntax for declaring metaclasses,
     and alters the semantics for how classes with metaclasses are
     constructed.


Rationale

     There are two rationales for this PEP, both of which are somewhat
     subtle.

     The primary reason for changing the way metaclasses work, is that
     there are a number of interesting use cases that require the
     metaclass to get involved earlier in the class construction process
     than is currently possible. Currently, the metaclass mechanism is
     essentially a post-processing step. With the advent of class
     decorators, much of these post-processing chores can be taken over
     by the decorator mechanism.

     In particular, there is an important body of use cases where it
     would be useful to preserve the order in which a class members are
     declared. Ordinary Python objects store their members in a
     dictionary, in which ordering is unimportant, and members are
     accessed strictly by name. However, Python is often used to
     interface with external systems in which the members are organized
     according to an implicit ordering. Examples include declaration of C
     structs; COM objects; Automatic translation of Python classes into
     IDL or database schemas, such as used in an ORM; and so on.

     In such cases, it would be useful for a Python programmer to specify
     such ordering directly using the declaration order of class members.
     Currently, such orderings must be specified explicitly, using some
     other mechanism (see the ctypes module for an example.)

     Unfortunately, the current method for declaring a metaclass does
     not allow for this, since the ordering information has already been
     lost by the time the metaclass comes into play. By allowing the
     metaclass to get involved in the class construction process earlier,
     the new system allows the ordering or other early artifacts of
     construction to be preserved and examined.

     There proposed metaclass mechanism also supports a number of other
     interesting use cases beyond preserving the ordering of declarations.
     One use case is to insert symbols into the namespace of the class
     body which are only valid during class construction. An example of
     this might be "field constructors", small functions that are used in
     the creation of class members. Another interesting possibility is
     supporting forward references, i.e. references to Python
     symbols that are declared further down in the class body.

     The other, weaker, rationale is purely cosmetic: The current method
     for specifying a metaclass is by assignment to the special variable
     __metaclass__, which is considered by some to be aesthetically less
     than ideal. Others disagree strongly with that opinion. This PEP
     will not address this issue, other than to note it, since aesthetic
     debates cannot be resolved via logical proofs.


Specification

     In the new model, the syntax for specifying a metaclass is via a
     keyword argument in the list of base classes:

       class Foo(base1, base2, metaclass=mymeta):
         ...

     Additional keywords will also be allowed here, and will be passed to
     the metaclass, as in the following example:

       class Foo(base1, base2, metaclass=mymeta, private=True):
         ...

     Note that this PEP makes no attempt to define what these other
     keywords might be - that is up to metaclass implementors to
     determine.

     More generally, the parameter list passed to a class definition will
     now support all of the features of a function call, meaning that you
     can now use *args and **kwargs-style arguments in the class base
     list:

        class Foo(*bases, **kwds):
           ...

Invoking the Metaclass

     In the current metaclass system, the metaclass object can be any
     callable type. This does not change, however in order to fully
     exploit all of the new features the metaclass will need to have an
     extra attribute which is used during class pre-construction.

     This attribute is named __prepare__, which is invoked as a function
     before the evaluation of the class body. The __prepare__ function
     takes two positional arguments, and an arbitrary number of keyword
     arguments. The two positional arguments are:

       'name' - the name of the class being created.
       'bases' - the list of base classes.

     The interpreter always tests for the existence of __prepare__ before
     calling it; If it is not present, then a regular dictionary is used,
     as illustrated in the following Python snippet.

       def prepare_class(name, *bases, metaclass=None, **kwargs):
          if metaclass is None:
             metaclass = compute_default_metaclass(bases)
          prepare = getattr(metaclass, '__prepare__', None)
          if prepare is not None:
             return prepare(name, bases, **kwargs)
          else:
             return dict()

     The example above illustrates how the arguments to 'class' are
     interpreted. The class name is the first argument, followed by
     an arbitrary length list of base classes. After the base classes,
     there may be one or more keyword arguments, one of which can be
     'metaclass'. Note that the 'metaclass' argument is not included
     in kwargs, since it is filtered out by the normal parameter
     assignment algorithm. (Note also that 'metaclass' is a keyword-
     only argument as per PEP 3102 [6].)

     __prepare__ returns a dictionary-like object which is used to store
     the class member definitions during evaluation of the class body.
     In other words, the class body is evaluated as a function block
     (just like it is now), except that the local variables dictionary
     is replaced by the dictionary returned from __prepare__. This
     dictionary object can be a regular dictionary or a custom mapping
     type.

     This dictionary-like object is not required to support the full
     dictionary interface. A dictionary which supports a limited set of
     dictionary operations will restrict what kinds of actions can occur
     during evaluation of the class body. A minimal implementation might
     only support adding and retrieving values from the dictionary - most
     class bodies will do no more than that during evaluation. For some
     classes, it may be desirable to support deletion as well. Many
     metaclasses will need to make a copy of this dictionary afterwards,
     so iteration or other means for reading out the dictionary contents
     may also be useful.

     The __prepare__ method will most often be implemented as a class
     method rather than an instance method because it is called before
     the metaclass instance (i.e. the class itself) is created.

     Once the class body has finished evaluating, the metaclass will be
     called (as a callable) with the class dictionary, which is no
     different from the current metaclass mechanism.

     Typically, a metaclass will create a custom dictionary - either a
     subclass of dict, or a wrapper around it - that will contain
     additional properties that are set either before or during the
     evaluation of the class body. Then in the second phase, the
     metaclass can use these additional properties to further customize
     the class.

     An example would be a metaclass that uses information about the
     ordering of member declarations to create a C struct. The metaclass
     would provide a custom dictionary that simply keeps a record of the
     order of insertions. This does not need to be a full 'ordered dict'
     implementation, but rather just a Python list of (key,value) pairs
     that is appended to for each insertion.

     Note that in such a case, the metaclass would be required to deal
     with the possibility of duplicate keys, but in most cases that is
     trivial. The metaclass can use the first declaration, the last,
     combine them in some fashion, or simply throw an exception. It's up
     to the metaclass to decide how it wants to handle that case.

Example:

     Here's a simple example of a metaclass which creates a list of
     the names of all class members, in the order that they were
     declared:

     # The custom dictionary
     class member_table(dict):
        def __init__(self):
           self.member_names = []

        def __setitem__(self, key, value):
           # if the key is not already defined, add to the
           # list of keys.
           if key not in self:
              self.member_names.append(key)

           # Call superclass
           dict.setitem(self, key, value)

     # The metaclass
     class OrderedClass(type):

         # The prepare function
         @classmethod
         def __prepare__(metacls, name, bases): # No keywords in this case
            return member_table()

         # The metaclass invocation
         def __init__(self, name, bases, classdict):
            # Note that we replace the classdict with a regular
            # dict before passing it to the superclass, so that we
            # don't continue to record member names after the class
            # has been created.
            result = type(name, bases, dict(classdict))
            result.member_names = classdict.member_names
            return result

     class MyClass(metaclass=OrderedClass):
        # method1 goes in array element 0
        def method1(self):
           pass

        # method2 goes in array element 1
        def method2(self):
           pass

Sample Implementation:

     Guido van Rossum has created a patch which implements the new
     functionality:
     
        http://python.org/sf/1681101      

Alternate Proposals

     Josiah Carlson proposed using the name 'type' instead of
     'metaclass', on the theory that what is really being specified is
     the type of the type. While this is technically correct, it is also
     confusing from the point of view of a programmer creating a new
     class. From the application programmer's point of view, the 'type'
     that they are interested in is the class that they are writing; the
     type of that type is the metaclass.

     There were some objections in the discussion to the 'two-phase'
     creation process, where the metaclass is invoked twice, once to
     create the class dictionary and once to 'finish' the class. Some
     people felt that these two phases should be completely separate, in
     that there ought to be separate syntax for specifying the custom
     dict as for specifying the metaclass. However, in most cases, the
     two will be intimately tied together, and the metaclass will most
     likely have an intimate knowledge of the internal details of the
     class dict. Requiring the programmer to insure that the correct dict
     type and the correct metaclass type are used together creates an
     additional and unneeded burden on the programmer.

     Another good suggestion was to simply use an ordered dict for all
     classes, and skip the whole 'custom dict' mechanism. This was based
     on the observation that most use cases for a custom dict were for
     the purposes of preserving order information. However, this idea has
     several drawbacks, first because it means that an ordered dict
     implementation would have to be added to the set of built-in types
     in Python, and second because it would impose a slight speed (and
     complexity) penalty on all class declarations. Later, several people
     came up with ideas for use cases for custom dictionaries other
     than preserving field orderings, so this idea was dropped.


Backwards Compatibility

     It would be possible to leave the existing __metaclass__ syntax in
     place. Alternatively, it would not be too difficult to modify the
     syntax rules of the Py3K translation tool to convert from the old to
     the new syntax.


References

     [1] [Python-3000] Metaclasses in Py3K (original proposal)
         http://mail.python.org/pipermail/python-3000/2006-December/005030.html

     [2] [Python-3000] Metaclasses in Py3K (Guido's suggested syntax)
         http://mail.python.org/pipermail/python-3000/2006-December/005033.html

     [3] [Python-3000] Metaclasses in Py3K (Objections to two-phase init)
         http://mail.python.org/pipermail/python-3000/2006-December/005108.html

     [4] [Python-3000] Metaclasses in Py3K (Always use an ordered dict)
         http://mail.python.org/pipermail/python-3000/2006-December/005118.html

     [5] PEP 359: The 'make' statement -
         http://www.python.org/dev/peps/pep-0359/

     [6] PEP 3102: Keyword-only arguments -
         http://www.python.org/dev/peps/pep-3102/

Copyright

     This document has been placed in the public domain.


Local Variables:
mode: indented-text
indent-tabs-mode: nil
sentence-end-double-space: t
fill-column: 70
coding: utf-8
End: