python-peps/pep-0253.txt

PEP: 253
Title: Subtyping Built-in Types
Version: $Revision$
Author: guido@python.org (Guido van Rossum)
Status: Draft
Type: Standards Track
Python-Version: 2.2
Created: 14-May-2001
Post-History:

Abstract

    This PEP proposes ways for creating subtypes of existing built-in
    types, either in C or in Python.  The text is currently long and
    rambling; I'll go over it again later to make it shorter.

    Traditionally, types in Python have been created statically, by
    declaring a global variable of type PyTypeObject and initializing
    it with a static initializer.  The fields in the type object
    describe all aspects of a Python object that are relevant to the
    Python interpreter.  A few fields contain dimensional information
    (e.g. the basic allocation size of instances), others contain
    various flags, but most fields are pointers to functions to
    implement various kinds of behaviors.  A NULL pointer means that
    the type does not implement the specific behavior; in that case
    the system may provide a default behavior in that case or raise an
    exception when the behavior is invoked.  Some collections of
    functions pointers that are usually defined together are obtained
    indirectly via a pointer to an additional structure containing.

    While the details of initializing a PyTypeObject structure haven't
    been documented as such, they are easily glanced from the examples
    in the source code, and I am assuming that the reader is
    sufficiently familiar with the traditional way of creating new
    Python types in C.

    This PEP will introduce the following features:

    - a type, like a class, can be a factory for its instances

    - types can be subtyped in C by specifying a base type pointer

    - types can be subtyped in Python using the class statement

    - multiple inheritance from types (insofar as practical)

    - the standard coercions (int, tuple, str etc.) will be the
      corresponding type objects

    - a standard type hierarchy

    This PEP builds on pep-0252, which adds standard introspection to
    types; in particular, types are assumed to have e.g. a __hash__
    method when the type object defines the tp_hash slot.  pep-0252 also
    adds a dictionary to type objects which contains all methods.  At
    the Python level, this dictionary is read-only; at the C level, it
    is accessible directly (but modifying it is not recommended except
    as part of initialization).


Metatypes

    Inevitably the following discussion will come to mention metatypes
    (or metaclasses).  Metatypes are nothing new in Python: Python has
    always been able to talk about the type of a type:

    >>> a = 0
    >>> type(a)
    <type 'int'>
    >>> type(type(a))
    <type 'type'>
    >>> type(type(type(a)))
    <type 'type'>
    >>> 

    In this example, type(a) is a "regular" type, and type(type(a)) is
    a metatype.  While as distributed all types have the same metatype
    (which is also its own metatype), this is not a requirement, and
    in fact a useful 3rd party extension (ExtensionClasses by Jim
    Fulton) creates an additional metatype.  A related feature is the
    "Don Beaudry hook", which says that if a metatype is callable, its
    instances (which are regular types) can be subclassed (really
    subtyped) using a Python class statement.  We will use this rule
    to support subtyping of built-in types, and in fact it greatly
    simplifies the logic of class creation to always simply call the
    metatype.  When no base class is specified, a default metatype is
    called -- the default metatype is the "ClassType" object, so the
    class statement will behave as before in the normal case.

    Python uses the concept of metatypes or metaclasses in a
    different way than Smalltalk.  In Smalltalk-80, there is a
    hierarchy of metaclasses that mirrors the hierarchy of regular
    classes, metaclasses map 1-1 to classes (except for some funny
    business at the root of the hierarchy), and each class statement
    creates both a regular class and its metaclass, putting class
    methods in the metaclass and instance methods in the regular
    class.

    Nice though this may be in the context of Smalltalk, it's not
    compatible with the traditional use of metatypes in Python, and I
    prefer to continue in the Python way.  This means that Python
    metatypes are typically written in C, and may be shared between
    many regular types. (It will be possible to subtype metatypes in
    Python, so it won't be absolutely necessary to write C in order to
    use metatypes; but the power of Python metatypes will be limited,
    e.g. Python code will never be allowed to allocate raw memory and
    initialize it at will.)


Instantiation by calling the type object

    Traditionally, for each type there is at least one C function that
    creates instances of the type (e.g. PyInt_FromLong(),
    PyTuple_New() and so on).  This function has to take care of
    both allocating memory for the object and initializing that
    memory.  As of Python 2.0, it also has to interface with the
    garbage collection subsystem, if the type chooses to participate
    in garbage collection (which is optional, but strongly recommended
    for so-called "container" types: types that may contain arbitrary
    references to other objects, and hence may participate in
    reference cycles).

    If we're going to implement subtyping, we must separate allocation
    and initialization: typically, the most derived subtype is in
    charge of allocation (and hence deallocation!), but in most cases
    each base type's initializer (constructor) must still be called,
    from the "most base" type to the most derived type.

    But let's first get the interface for instantiation right.  If we
    call an object, the tp_call slot if its type gets invoked.  Thus,
    if we call a type, this invokes the tp_call slot of the type's
    type: in other words, the tp_call slot of the metatype.
    Traditionally this has been a NULL pointer, meaning that types
    can't be called.  Now we're adding a tp_call slot to the metatype,
    which makes all types "callable" in a trivial sense.  But
    obviously the metatype's tp_call implementation doesn't know how
    to initialize the instances of individual types.  So the type
    defines a new slot, tp_new, which is invoked by the metatype's
    tp_call slot.  If the tp_new slot is NULL, the metatype's tp_call
    issues a nice error message: the type isn't callable.

    This mechanism gives the maximum freedom to the type: a type's
    tp_new doesn't necessarily have to return a new object, or even an
    object that is an instance of the type (although the latter should
    be rare).

    HIRO

    The deallocation mechanism chosen should match the allocation
    mechanism: an allocation policy should prescribe both the
    allocation and deallocation mechanism.  And again, planning ahead
    for subtyping would be nice.  But the available mechanisms are
    different.  The deallocation function has always been part of the
    type structure, as tp_dealloc, which combines the
    "uninitialization" with deallocation.  This was good enough for
    the traditional situation, where it matched the combined
    allocation and initialization of the creation function.  But now
    imagine a type whose creation function uses a special free list
    for allocation.  It's deallocation function puts the object's
    memory back on the same free list.  But when allocation and
    creation are separate, the object may have been allocated from the
    regular heap, and it would be wrong (in some cases disastrous) if
    it were placed on the free list by the deallocation function.

    A solution would be for the tp_construct function to somehow mark
    whether the object was allocated from the special free list, so
    that the tp_dealloc function can choose the right deallocation
    method (assuming that the only two alternatives are a special free
    list or the regular heap).  A variant that doesn't require space
    for an allocation flag bit would be to have two type objects,
    identical in the contents of all their slots except for their
    deallocation slot.  But this requires that all type-checking code
    (e.g. the PyDict_Check()) recognizes both types.  We'll come back
    to this solution in the context of subtyping.  Another alternative
    is to require the metatype's tp_call to leave the allocation to
    the tp_construct method, by passing in a NULL pointer.  But this
    doesn't work once we allow subtyping.

    Eventually, when we add any form of subtyping, we'll have to
    separate deallocation from uninitialization.  The way to do this
    is to add a separate slot to the type object that does the
    uninitialization without the deallocation.  Fortunately, there is
    already such a slot: tp_clear, currently used by the garbage
    collection subsystem.  A simple rule makes this slot reusable as
    an uninitialization: for types that support separate allocation
    and initialization, tp_clear must be defined (even if the object
    doesn't support garbage collection) and it must DECREF all
    contained objects and FREE all other memory areas the object owns.
    It must also be reentrant: it must be possible to clear an already
    cleared object.  The easiest way to do this is to replace all
    pointers DECREFed or FREEd with NULL pointers.


Subtyping in C

    The simplest form of subtyping is subtyping in C.  It is the
    simplest form because we can require the C code to be aware of the
    various problems, and it's acceptable for C code that doesn't
    follow the rules to dump core; while for Python subtyping we would
    need to catch all errors before they become core dumps.

    The idea behind subtyping is very similar to that of single
    inheritance in C++.  A base type is described by a structure
    declaration plus a type object.  A derived type can extend the
    structure (but must leave the names, order and type of the fields
    of the base structure unchanged) and can override certain slots in
    the type object, leaving others the same.

    Not every type can serve as a base type.  The base type must
    support separation of allocation and initialization by having a
    tp_construct slot that can be called with a preallocated object,
    and it must support uninitialization without deallocation by
    having a tp_clear slot as described above.  The derived type must
    also export the structure declaration for its instances through a
    header file, as it is needed in order to derive a subtype.  The
    type object for the base type must also be exported.

    If the base type has a type-checking macro (e.g. PyDict_Check()),
    this macro may be changed to recognize subtypes.  This can be done
    by using the new PyObject_TypeCheck(object, type) macro, which
    calls a function that follows the base class links.  There are
    arguments for and against changing the type-checking macro in this
    way.  The argument for the change should be clear: it allows
    subtypes to be used in places where the base type is required,
    which is often the prime attraction of subtyping (as opposed to
    sharing implementation).  An argument against changing the
    type-checking macro could be that the type check is used
    frequently and a function call would slow things down too much
    (hard to believe); or one could fear that a subtype might break an
    invariant assumed by the support functions of the base type.
    Sometimes it would be wise to change the base type to remove this
    reliance; other times, it would be better to require that derived
    types (implemented in C) maintain the invariants.

    The derived type begins by declaring a type structure which
    contains the base type's structure.  For example, here's the type
    structure for a subtype of the built-in list type:

    typedef struct {
        PyListObject list;
        int state;
    } spamlistobject;

    Note that the base type structure field (here PyListObject) must
    be the first field in the structure; any following fields are
    extension fields.  Also note that the base type is not referenced
    via a pointer; the actual contents of its structure must be
    included! (The goal is for the memory lay out of the beginning of
    the subtype instance to be the same as that of the base type
    instance.)

    Next, the derived type must declare a type object and initialize
    it.  Most of the slots in the type object may be initialized to
    zero, which is a signal that the base type slot must be copied
    into it.  Some fields that must be initialized properly:

    - the object header must be filled in as usual; the type should be
      PyType_Type

    - the tp_basicsize field must be set to the size of the subtype
      instances

    - the tp_base field must be set to the address of the base type's
      type object

    - the tp_dealloc slot function must be a deallocation function for
      the subtype

    - the tp_flags field must be set to the usual Py_TPFLAGS_DEFAULT
      value

    - the tp_name field must be set (otherwise it will be inherited,
      which is wrong)

    Exception: if the subtype defines no additional fields in its
    structure (i.e., it only defines new behavior, no new data), the
    tp_basicsize and the tp_dealloc fields may be set to zero.  In
    order to complete the initialization of the type,
    PyType_InitDict() must be called.  This replaces zero slots in the
    subtype with the value of the corresponding base type slots.  It
    also fills in tp_dict, the type's dictionary; this is more a
    matter of pep-0252.

    The subtype's tp_dealloc slot deserves special attention.  It must
    uninitialize and deallocate the object in an orderly manner: first
    it must uninitialize the fields added by the extension type; then
    it must call the base type's tp_clear function; finally it must
    deallocate the memory of the object.  Usually, the base type's
    tp_clear function has no global name; it is permissible to call it
    via the base type's tp_clear slot, e.g. PyListType.tp_clear(obj).
    Only if it is known that the base type uses the same allocation
    method as the subtype and the subtype requires no uninitialization
    (e.g. it adds no data fields or all its data fields are numbers)
    is it permissible to leave tp_dealloc set to zero in the subtype's
    type object; it will be copied from the base type.

    A subtype is not usable until PyType_InitDict() is called for it;
    this is best done during module initialization, assuming the
    subtype belongs to a module.  An alternative for subtypes added to
    the Python core (which don't live in a particular module) would be
    to initialize the subtype in their constructor function.  It is
    allowed to call PyType_InitDict() more than once, the second and
    further calls have no effect.  In order to avoid unnecessary
    calls, a test for tp_dict==NULL can be made.

    If the subtype itself should be subtypable (usually desirable), it
    should follow the same rules are given above for base types: have
    a tp_construct that accepts a preallocated object and calls the
    base type's tp_construct, and have a tp_clear that calls the base
    type's tp_clear.


Subtyping in Python

    The next step is to allow subtyping of selected built-in types
    through a class statement in Python.  Limiting ourselves to single
    inheritance for now, here is what happens for a simple class
    statement:

    class C(B):
        var1 = 1
        def method1(self): pass
        # etc.

    The body of the class statement is executes in a fresh environment
    (basically, a new dictionary used as local namespace), and then C
    is created.  The following explains how C is created.

    Assume B is a type object.  Since type objects are objects, and
    every object has a type, B has a type.  B's type is accessible via
    type(B) or B.__class__ (the latter notation is new for types; it
    is introduced in pep-0252).  Let's say B's type is M (for
    Metatype).  The class statement will create a new type, C.  Since
    C will be a type object just like B, we view the creation of C as
    an instantiation of the metatype, M.  The information that needs
    to be provided for the creation of C is: its name (in this example
    the string "C"); the list of base classes (a singleton tuple
    containing B); and the results of executing the class body, in the
    form of a dictionary (e.g. {"var1": 1, "method1": <function...>,
    ...}).

    According to the Don Beaudry hook, the following call is made:

    C = M("C", (B,), dict)

    (where dict is the dictionary resulting from execution of the
    class body).  In other words, the metatype (M) is called.  Note
    that even though we currently require there to be exactly one base
    class, we still pass in a (singleton) sequence of base classes;
    this makes it possible to support multiple inheritance later (or
    for types with a different metaclass!) without changing this
    interface.

    Note that calling M requires that M itself has a type: the
    meta-metatype.  In the current implementation, I have introduced a
    new type object for this purpose, named turtle because of my
    fondness of the phrase "turtles all the way down".  However I now
    believe that it would be better if M were its own metatype, just
    like before.  This can be accomplished by making M's tp_call slot
    slightly more flexible.

    In any case, the work for creating C is done by M's tp_construct
    slot.  It allocates space for an "extended" type structure, which
    contains space for: the type object; the auxiliary structures
    (as_sequence etc.); the string object containing the type name (to
    ensure that this object isn't deallocated while the type object is
    still referencing it); and some more auxiliary storage (to be
    described later).  It initializes this storage to zeros except for
    a few crucial slots (e.g. tp_name is set to point to the type
    name) and then sets the tp_base slot to point to B.  Then
    PyType_InitDict() is called to inherit B's slots.  Finally, C's
    tp_dict slot is updated with the contents of the namespace
    dictionary (the third argument to the call to M).


Implementation

    A prototype implementation of this PEP is available from CVS as a
    branch named "descr-branch".  To experiment with this
    implementation, proceed to check out Python from CVS according to
    the instructions at http://sourceforge.net/cvs/?group_id=5470 but
    add the arguments "-r descr-branch" to the cvs checkout command.
    (You can also start with an existing checkout and do "cvs update
    -r descr-branch".)  For some examples of the features described
    here, see the file Lib/test/test_descr.py and the extension module
    Modules/spam.c.

    Note: the code in this branch is for pep-0252, pep-0253, and
    pep-254.


References

    [1] "Putting Metaclasses to Work", by Ira R. Forman and Scott
        H. Danforth, Addison-Wesley 1999.
        (http://www.aw.com/product/0,2627,0201433052,00.html)


Copyright

    This document has been placed in the public domain.


Local Variables:
mode: indented-text
indent-tabs-mode: nil
End: