Checkpoint from Tim

This commit is contained in:
Neal Norwitz 2007-05-25 07:10:03 +00:00
parent be80897241
commit 345e7eafcb
1 changed files with 449 additions and 224 deletions

View File

@ -2,264 +2,457 @@ PEP: 367
Title: New Super
Version: $Revision$
Last-Modified: $Date$
Author: Calvin Spealman <ironfroggy@gmail.com>
Author: Calvin Spealman <ironfroggy@gmail.com>,
Tim Delaney <timothy.c.delaney@gmail.com>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 28-Apr-2007
Python-Version: 2.6
Post-History: 28-Apr-2007, 29-Apr-2007 (1), 29-Apr-2007 (2)
Post-History: 28-Apr-2007, 29-Apr-2007 (1), 29-Apr-2007 (2), 14-May-2007
Abstract
========
The PEP defines the proposal to enhance the ``super`` builtin to work
implicitly upon the class within which it is used and upon the
instance the current function was called on. The premise of the new
super usage suggested is as follows::
This PEP proposes syntactic sugar for use of the ``super`` type to automatically
construct instances of the super type binding to the class that a method was
defined in, and the instance (or class object for classmethods) that the method
is currently acting upon.
The premise of the new super usage suggested is as follows::
super.foo(1, 2)
to replace the old ::
to replace the old::
super(Foo, self).foo(1, 2)
and the current ``__builtin__.super`` be aliased to ``__builtin__.__super__``
(with ``__builtin__.super`` to be removed in Python 3.0).
It is further proposed that assignment to ``super`` become a ``SyntaxError``,
similar to the behaviour of ``None``.
Rationale
=========
The current usage of ``super`` requires an explicit passing of both
the class and instance it must operate from, requiring a breaking of
the *DRY* (Don't Repeat Yourself) rule. This hinders any change in
class name, and is often considered a wart by many.
The current usage of super requires an explicit passing of both the class and
instance it must operate from, requiring a breaking of the DRY (Don't Repeat
Yourself) rule. This hinders any change in class name, and is often considered
a wart by many.
Specification
=============
Within the specification section, some special terminology will be
used to distinguish similar and closely related concepts. "Super
type" will refer to the actual builtin type named ``super``. "Next
Class/Type in the MRO" will refer to the class where attribute lookups
will be performed by ``super``, for example, in the following, ``A``
is the "Next class in the MRO" for the use of ``super``. ::
Within the specification section, some special terminology will be used to
distinguish similar and closely related concepts. "super type" will refer to
the actual builtin type named "super". A "super instance" is simply an instance
of the super type, which is associated with a class and possibly with an
instance of that class.
class A(object):
def f(self):
return 'A'
Because the new ``super`` semantics are not backwards compatible with Python
2.5, the new semantics will require a ``__future__`` import::
class B(A):
def f(self):
super(B, self).f() # Here, A would be our "Next class
# in the MRO", of course.
from __future__ import new_super
A "super object" is simply an instance of the super type, which is
associated with a class and possibly with an instance of that class.
Finally, "new super" refers to the new super type, which will replace
the original.
The current ``__builtin__.super`` will be aliased to ``__builtin__.__super__``.
This will occur regardless of whether the new ``super`` semantics are active.
It is not possible to simply rename ``__builtin__.super``, as that would affect
modules that do not use the new ``super`` semantics. In Python 3.0 it is
proposed that the name ``__builtin__.super`` will be removed.
Replacing the old usage of ``super``, calls to the next class in the
MRO (method resolution order) will be made without an explicit super
object creation, by simply accessing an attribute on the super type
directly, which will automatically apply the class and instance to
perform the proper lookup. The following example demonstrates the use
of this. ::
Replacing the old usage of super, calls to the next class in the MRO (method
resolution order) can be made without explicitly creating a ``super``
instance (although doing so will still be supported via ``__super__``). Every
function will have an implicit local named ``super``. This name behaves
identically to a normal local, including use by inner functions via a cell,
with the following exceptions:
class A(object):
def f(self):
return 'A'
1. Assigning to the name ``super`` will raise a ``SyntaxError`` at compile time;
class B(A):
def f(self):
return 'B' + super.f()
2. Calling a static method or normal function that accesses the name ``super``
will raise a ``TypeError`` at runtime.
class C(A):
def f(self):
return 'C' + super.f()
Every function that uses the name ``super``, or has an inner function that
uses the name ``super``, will include a preamble that performs the equivalent
of::
class D(B, C):
def f(self):
return 'D' + super.f()
super = __builtin__.__super__(<class>, <instance>)
assert D().f() == 'DBCA'
where ``<class>`` is the class that the method was defined in, and
``<instance>`` is the first parameter of the method (normally ``self`` for
instance methods, and ``cls`` for class methods). For static methods and normal
functions, ``<class>`` will be ``None``, resulting in a ``TypeError`` being
raised during the preamble.
The proposal adds a dynamic attribute lookup to the super type, which
will automatically determine the proper class and instance parameters.
Each super attribute lookup identifies these parameters and performs
the super lookup on the instance, as the current super implementation
does with the explicit invocation of a super object upon a class and
instance.
Note: The relationship between ``super`` and ``__super__`` is similar to that
between ``import`` and ``__import__``.
The enhancements to the super type will define a new ``__getattr__``
classmethod of the super type, which must look backwards to the
previous frame and locate the instance object. This can be naively
determined by located the local named by the first argument to the
function. Using super outside of a function where this is a valid
lookup for the instance can be considered undocumented in its
behavior. This special method will actually be invoked on attribute
lookups to the super type itself, as opposed to super objects, as the
current implementation works. This may pose open issues, which are
detailed below.
Much of this was discussed in the thread of the python-dev list, "Fixing super
anyone?" [1]_.
"Every class will gain a new special attribute, ``__super__``, which
refers to an instance of the associated super object for that class."
In this capacity, the new super also acts as its own descriptor,
create an instance-specific super upon lookup.
Much of this was discussed in the thread of the python-dev list,
"Fixing super anyone?" [1]_.
Open Issues
-----------
__call__ methods
''''''''''''''''
Backward compatibility of the super type API raises some issues.
Names, the lookup of the ``__call__`` method of the super type itself,
which means a conflict with doing an actual super lookup of the
``__call__`` attribute. Namely, the following is ambiguous in the
current proposal::
Determining the class object to use
'''''''''''''''''''''''''''''''''''
super.__call__(arg)
The exact mechanism for associating the method with the defining class is not
specified in this PEP, and should be chosen for maximum performance. For
CPython, it is suggested that the class instance be held in a C-level variable
on the function object which is bound to one of ``NULL`` (not part of a class),
``Py_None`` (static method) or a class object (instance or class method).
Which means the backward compatible API, which involves instantiating
the super type, will either not be possible, because it will actually
do a super lookup on the ``__call__`` attribute, or there will be no
way to perform a super lookup on the ``__call__`` attribute. Both
seem unacceptable, so any suggestions are welcome.
Actually keeping the old super around in 2.x and creating a completely
new super type separately may be the best option. A future import or
even a simple import in 2.x of the new super type from some built-in
module may offer a way to choose which each module uses, even mixing
uses by binding to different names. Such a built-in module might be
called 'newsuper'. This module is also the reference implementation,
which I will present below.
super type's new getattr
''''''''''''''''''''''''
To give the behavior needed, the super type either needs a way to do
dynamic lookup of attributes on the super type object itself or define
a metaclass for the built-in type. This author is unsure which, if
either, is possible with C-defined types.
When should we create __super__ attributes?
Should ``super`` actually become a keyword?
'''''''''''''''''''''''''''''''''''''''''''
They either need to be created on class creation or on ``__super__``
attribute lookup. For the second, they could be cached, of course,
which seems like it may be the best idea, if implicit creation of a
super object for every class is considered too much overhead.
With this proposal, ``super`` would become a keyword to the same extent that
``None`` is a keyword. It is possible that further restricting the ``super``
name may simplify implementation, however some are against the actual keyword-
ization of super. The simplest solution is often the correct solution and the
simplest solution may well not be adding additional keywords to the language
when they are not needed. Still, it may solve other open issues.
How does it work in inner functions?
''''''''''''''''''''''''''''''''''''
If a method defines a function and super is used inside of it, how
does this work? The frame looking and instance detection breaks here.
However, if there can be some unambiguous way to use both the new
super form and still be able to explicitly name the type and instance,
I think its an acceptable tradeoff to simply be explicit in these
cases, rather than add weird super-specific lookup rules in these
cases.
Closed Issues
-------------
An example of such a problematic bit of code is::
super used with __call__ attributes
'''''''''''''''''''''''''''''''''''
class B(A):
def f(self):
def g():
return super.f()
return g()
It was considered that it might be a problem that instantiating super instances
the classic way, because calling it would lookup the __call__ attribute and
thus try to perform an automatic super lookup to the next class in the MRO.
However, this was found to be false, because calling an object only looks up
the __call__ method directly on the object's type. The following example shows
this in action.
Should super actually become a keyword?
'''''''''''''''''''''''''''''''''''''''
::
This would solve many of the problems and allow more direct
implementation of super into the language proper. However, some are
against the actual keywordization of super. The simplest solution is
often the correct solution and the simplest solution may well not be
adding additional keywords to the language when they are not needed.
Still, it may solve many of the other open issues.
class A(object):
def __call__(self):
return '__call__'
def __getattribute__(self, attr):
if attr == '__call__':
return lambda: '__getattribute__'
a = A()
assert a() == '__call__'
assert a.__call__() == '__getattribute__'
Can we also allow super()?
''''''''''''''''''''''''''
There is strong sentiment for and against this, but implementation and
style concerns are obvious. Particularly, that it's "magical" and
that ``super()`` would differ from ``super.__call__()``, being very
unpythonic.
In any case, with the renaming of ``__builtin__.super`` to
``__builtin__.__super__`` this issue goes away entirely.
Reference Implementation
========================
This implementation was a cooperative contribution in the original thread [1]_. ::
It is impossible to implement the above specification entirely in Python. This
reference implementation has the following differences to the specification:
1. New ``super`` semantics are implemented using bytecode hacking.
2. Assignment to ``super`` is not a ``SyntaxError``. Also see point #4.
3. Classes must either use the metaclass ``autosuper_meta`` or inherit from
the base class ``autosuper`` to acquire the new ``super`` semantics.
4. ``super`` is not an implicit local variable. In particular, for inner
functions to be able to use the super instance, there must be an assignment
of the form ``super = super`` in the method.
The reference implementation assumes that it is being run on Python 2.5+.
::
#!/usr/bin/env python
#
# newsuper.py
# autosuper.py
import sys
from array import array
import dis
import new
import types
import __builtin__
__builtin__.__super__ = __builtin__.super
del __builtin__.super
class SuperMetaclass(type):
def __getattr__(cls, attr):
calling_frame = sys._getframe().f_back
instance_name = calling_frame.f_code.co_varnames[0]
instance = calling_frame.f_locals[instance_name]
return getattr(instance.__super__, attr)
# We need these for modifying bytecode
from opcode import opmap, HAVE_ARGUMENT, EXTENDED_ARG
class Super(object):
__metaclass__ = SuperMetaclass
def __init__(self, type, obj=None):
if isinstance(obj, Super):
obj = obj.__obj__
self.__type__ = type
self.__obj__ = obj
def __get__(self, obj, cls=None):
if obj is None:
raise Exception('only supports instances')
else:
return Super(self.__type__, obj)
def __getattr__(self, attr):
mro = iter(self.__obj__.__class__.__mro__)
for cls in mro:
if cls is self.__type__:
break
for cls in mro:
if attr in cls.__dict__:
x = cls.__dict__[attr]
if hasattr(x, '__get__'):
x = x.__get__(self, cls)
return x
raise AttributeError, attr
LOAD_GLOBAL = opmap['LOAD_GLOBAL']
LOAD_NAME = opmap['LOAD_NAME']
LOAD_CONST = opmap['LOAD_CONST']
LOAD_FAST = opmap['LOAD_FAST']
LOAD_ATTR = opmap['LOAD_ATTR']
STORE_FAST = opmap['STORE_FAST']
LOAD_DEREF = opmap['LOAD_DEREF']
STORE_DEREF = opmap['STORE_DEREF']
CALL_FUNCTION = opmap['CALL_FUNCTION']
STORE_GLOBAL = opmap['STORE_GLOBAL']
DUP_TOP = opmap['DUP_TOP']
POP_TOP = opmap['POP_TOP']
NOP = opmap['NOP']
JUMP_FORWARD = opmap['JUMP_FORWARD']
ABSOLUTE_TARGET = dis.hasjabs
class autosuper(type):
def _oparg(code, opcode_pos):
return code[opcode_pos+1] + (code[opcode_pos+2] << 8)
def _bind_autosuper(func, cls):
co = func.func_code
name = func.func_name
newcode = array('B', co.co_code)
codelen = len(newcode)
newconsts = list(co.co_consts)
newvarnames = list(co.co_varnames)
# Check if the global 'super' keyword is already present
try:
sn_pos = list(co.co_names).index('super')
except ValueError:
sn_pos = None
# Check if the varname 'super' keyword is already present
try:
sv_pos = newvarnames.index('super')
except ValueError:
sv_pos = None
# Check if the callvar 'super' keyword is already present
try:
sc_pos = list(co.co_cellvars).index('super')
except ValueError:
sc_pos = None
# If 'super' isn't used anywhere in the function, we don't have anything to do
if sn_pos is None and sv_pos is None and sc_pos is None:
return func
c_pos = None
s_pos = None
n_pos = None
# Check if the 'cls_name' and 'super' objects are already in the constants
for pos, o in enumerate(newconsts):
if o is cls:
c_pos = pos
if o is __super__:
s_pos = pos
if o == name:
n_pos = pos
# Add in any missing objects to constants and varnames
if c_pos is None:
c_pos = len(newconsts)
newconsts.append(cls)
if n_pos is None:
n_pos = len(newconsts)
newconsts.append(name)
if s_pos is None:
s_pos = len(newconsts)
newconsts.append(__super__)
if sv_pos is None:
sv_pos = len(newvarnames)
newvarnames.append('super')
# This goes at the start of the function. It is:
#
# super = __super__(cls, self)
#
# If 'super' is a cell variable, we store to both the
# local and cell variables (i.e. STORE_FAST and STORE_DEREF).
#
preamble = [
LOAD_CONST, s_pos & 0xFF, s_pos >> 8,
LOAD_CONST, c_pos & 0xFF, c_pos >> 8,
LOAD_FAST, 0, 0,
CALL_FUNCTION, 2, 0,
]
if sc_pos is None:
# 'super' is not a cell variable - we can just use the local variable
preamble += [
STORE_FAST, sv_pos & 0xFF, sv_pos >> 8,
]
else:
# If 'super' is a cell variable, we need to handle LOAD_DEREF.
preamble += [
DUP_TOP,
STORE_FAST, sv_pos & 0xFF, sv_pos >> 8,
STORE_DEREF, sc_pos & 0xFF, sc_pos >> 8,
]
preamble = array('B', preamble)
# Bytecode for loading the local 'super' variable.
load_super = array('B', [
LOAD_FAST, sv_pos & 0xFF, sv_pos >> 8,
])
preamble_len = len(preamble)
need_preamble = False
i = 0
while i < codelen:
opcode = newcode[i]
need_load = False
remove_store = False
if opcode == EXTENDED_ARG:
raise TypeError("Cannot use 'super' in function with EXTENDED_ARG opcode")
# If the opcode is an absolute target it needs to be adjusted
# to take into account the preamble.
elif opcode in ABSOLUTE_TARGET:
oparg = _oparg(newcode, i) + preamble_len
newcode[i+1] = oparg & 0xFF
newcode[i+2] = oparg >> 8
# If LOAD_GLOBAL(super) or LOAD_NAME(super) then we want to change it into
# LOAD_FAST(super)
elif (opcode == LOAD_GLOBAL or opcode == LOAD_NAME) and _oparg(newcode, i) == sn_pos:
need_preamble = need_load = True
# If LOAD_FAST(super) then we just need to add the preamble
elif opcode == LOAD_FAST and _oparg(newcode, i) == sv_pos:
need_preamble = need_load = True
# If LOAD_DEREF(super) then we change it into LOAD_FAST(super) because
# it's slightly faster.
elif opcode == LOAD_DEREF and _oparg(newcode, i) == sc_pos:
need_preamble = need_load = True
if need_load:
newcode[i:i+3] = load_super
i += 1
if opcode >= HAVE_ARGUMENT:
i += 2
# No changes needed - get out.
if not need_preamble:
return func
# Our preamble will have 3 things on the stack
co_stacksize = max(3, co.co_stacksize)
# Conceptually, our preamble is on the `def` line.
co_lnotab = array('B', co.co_lnotab)
if co_lnotab:
co_lnotab[0] += preamble_len
co_lnotab = co_lnotab.tostring()
# Our code consists of the preamble and the modified code.
codestr = (preamble + newcode).tostring()
codeobj = new.code(co.co_argcount, len(newvarnames), co_stacksize,
co.co_flags, codestr, tuple(newconsts), co.co_names,
tuple(newvarnames), co.co_filename, co.co_name,
co.co_firstlineno, co_lnotab, co.co_freevars,
co.co_cellvars)
func.func_code = codeobj
func.func_class = cls
return func
class autosuper_meta(type):
def __init__(cls, name, bases, clsdict):
cls.__super__ = Super(cls)
UnboundMethodType = types.UnboundMethodType
for v in vars(cls):
o = getattr(cls, v)
if isinstance(o, UnboundMethodType):
_bind_autosuper(o.im_func, cls)
class autosuper(object):
__metaclass__ = autosuper_meta
if __name__ == '__main__':
class A(object):
__metaclass__ = autosuper
class A(autosuper):
def f(self):
return 'A'
class B(A):
def f(self):
return 'B' + Super.f()
return 'B' + super.f()
class C(A):
def f(self):
return 'C' + Super.f()
def inner():
return 'C' + super.f()
# Needed to put 'super' into a cell
super = super
return inner()
class D(B, C):
def f(self, arg=None):
var = None
return 'D' + Super.f()
return 'D' + super.f()
assert D().f() == 'DBCA'
Disassembly of B.f and C.f reveals the different preambles used when ``super``
is simply a local variable compared to when it is used by an inner function.
::
>>> dis.dis(B.f)
214 0 LOAD_CONST 4 (<type 'super'>)
3 LOAD_CONST 2 (<class '__main__.B'>)
6 LOAD_FAST 0 (self)
9 CALL_FUNCTION 2
12 STORE_FAST 1 (super)
215 15 LOAD_CONST 1 ('B')
18 LOAD_FAST 1 (super)
21 LOAD_ATTR 1 (f)
24 CALL_FUNCTION 0
27 BINARY_ADD
28 RETURN_VALUE
::
>>> dis.dis(C.f)
218 0 LOAD_CONST 4 (<type 'super'>)
3 LOAD_CONST 2 (<class '__main__.C'>)
6 LOAD_FAST 0 (self)
9 CALL_FUNCTION 2
12 DUP_TOP
13 STORE_FAST 1 (super)
16 STORE_DEREF 0 (super)
219 19 LOAD_CLOSURE 0 (super)
22 LOAD_CONST 1 (<code object inner at 00C160A0, file "autosuper.py", line 219>)
25 MAKE_CLOSURE 0
28 STORE_FAST 2 (inner)
223 31 LOAD_FAST 1 (super)
34 STORE_DEREF 0 (super)
224 37 LOAD_FAST 2 (inner)
40 CALL_FUNCTION 0
43 RETURN_VALUE
Note that in the final implementation, the preamble would not be part of the
bytecode of the method, but would occur immediately following unpacking of
parameters.
Alternative Proposals
=====================
@ -267,71 +460,103 @@ Alternative Proposals
No Changes
----------
Although its always attractive to just keep things how they are,
people have sought a change in the usage of super calling for some
time, and for good reason, all mentioned previously.
Although its always attractive to just keep things how they are, people have
sought a change in the usage of super calling for some time, and for good
reason, all mentioned previously.
* Decoupling from the class name (which might not even be bound to the
right class anymore!).
- Decoupling from the class name (which might not even be bound to the
right class anymore!)
- Simpler looking, cleaner super calls would be better
* Simpler looking, cleaner super calls would be better.
``super(__this_class__, self)``
Dynamic attribute on super type
-------------------------------
This is nearly an anti-proposal, as it basically relies on the
acceptance of the ``__this_class__`` PEP [#pep3130]_, which proposes a
special name that would always be bound to the class within which it
is used. If that is accepted, ``__this_class__`` could simply be used
instead of the class' name explicitly, solving the name binding issues.
The proposal adds a dynamic attribute lookup to the super type, which will
automatically determine the proper class and instance parameters. Each super
attribute lookup identifies these parameters and performs the super lookup on
the instance, as the current super implementation does with the explicit
invokation of a super instance upon a class and instance.
``self.__super__.foo(*args)``
-----------------------------
This proposal relies on sys._getframe(), which is not appropriate for anything
except a prototype implementation.
The ``__super__`` attribute is mentioned in this PEP in several
places, and could be a candidate for the complete solution, actually
using it explicitly instead of any super usage directly. However,
double-underscore names are usually an internal detail, and attempted
to be kept out of everyday code.
``super(self, *args) or __super__(self, *args)``
------------------------------------------------
super(__this_class__, self)
---------------------------
This solution only solves the problem of the type indication, does not
handle differently named super methods, and is explicit about the name
of the instance. It is less flexible without being able to enacted on
other method names, in cases where that is needed. One use case where
this fails is when a base class has a factory classmethod and a
subclass has two factory classmethods, both of which need to properly
make super calls to the one in the base class.
This is nearly an anti-proposal, as it basically relies on the acceptance of
the __this_class__ PEP, which proposes a special name that would always be
bound to the class within which it is used. If that is accepted, __this_class__
could simply be used instead of the class' name explicitly, solving the name
binding issues [2]_.
``super.foo(self, *args)``
self.__super__.foo(\*args)
--------------------------
This variation actually eliminates the problems with locating the
proper instance, and if any of the alternatives were pushed into the
spotlight, I would want it to be this one.
The __super__ attribute is mentioned in this PEP in several places, and could
be a candidate for the complete solution, actually using it explicitly instead
of any super usage directly. However, double-underscore names are usually an
internal detail, and attempted to be kept out of everyday code.
``super`` or ``super()``
------------------------
super(self, \*args) or __super__(self, \*args)
----------------------------------------------
This solution only solves the problem of the type indication, does not handle
differently named super methods, and is explicit about the name of the
instance. It is less flexable without being able to enacted on other method
names, in cases where that is needed. One use case this fails is where a base-
class has a factory classmethod and a subclass has two factory classmethods,
both of which needing to properly make super calls to the one in the base-
class.
super.foo(self, \*args)
-----------------------
This variation actually eliminates the problems with locating the proper
instance, and if any of the alternatives were pushed into the spotlight, I
would want it to be this one.
super or super()
----------------
This proposal leaves no room for different names, signatures, or application
to other classes, or instances. A way to allow some similar use alongside the
normal proposal would be favorable, encouraging good design of multiple
inheritence trees and compatible methods.
super(\*p, \*\*kw)
------------------
There has been the proposal that directly calling ``super(*p, **kw)`` would
be equivalent to calling the method on the ``super`` object with the same name
as the method currently being executed i.e. the following two methods would be
equivalent:
::
def f(self, *p, **kw):
super.f(*p, **kw)
::
def f(self, *p, **kw):
super(*p, **kw)
There is strong sentiment for and against this, but implementation and style
concerns are obvious. Guido has suggested that this should be excluded from
this PEP on the principle of KISS (Keep It Simple Stupid).
This proposal leaves no room for different names, signatures, or
application to other classes, or instances. A way to allow some
similar use alongside the normal proposal would be favorable,
encouraging good design of multiple inheritance trees and compatible
methods.
History
=======
29-Apr-2007 - Changed title from "Super As A Keyword" to "New Super"
- Updated much of the language and added a terminology section
for clarification in confusing places.
- Added reference implementation and history sections.
29-Apr-2007:
- Changed title from "Super As A Keyword" to "New Super"
- Updated much of the language and added a terminology section
for clarification in confusing places.
- Added reference implementation and history sections.
06-May-2007 - Updated by Tim Delaney to reflect discussions on the python-3000
and python-dev mailing lists.
References
==========
@ -339,8 +564,8 @@ References
.. [1] Fixing super anyone?
(http://mail.python.org/pipermail/python-3000/2007-April/006667.html)
.. [#pep3130] PEP 3130 (Access to Current Module/Class/Function)
http://www.python.org/dev/peps/pep-3130
.. [2] PEP 3130: Access to Module/Class/Function Currently Being Defined (this)
(http://mail.python.org/pipermail/python-ideas/2007-April/000542.html)
Copyright