python-peps/pep-0318.txt

PEP: 318
Title: Decorators for Functions and Methods
Version: $Revision$
Last-Modified: $Date$
Author: Kevin D. Smith <Kevin.Smith@theMorgue.org>,
        Jim Jewett <jimjjewett@users.sourceforge.net>,
        Skip Montanaro <skip@pobox.com>,
        Anthony Baxter <anthony@python.org>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 05-Jun-2003
Python-Version: 2.4
Post-History: 09-Jun-2003, 10-Jun-2003, 27-Feb-2004, 23-Mar-2004


WarningWarningWarning
=====================

This is not yet complete. This is still a work-in-progress. Feedback
to anthony. Please note that the point of this PEP is _not_ to provide
an open-slather of plusses and minuses for each syntax, but is instead
to justify the choice made. If, in 2.4a3, the syntax is changed, the
PEP will be updated to match this, complete with the arguments for the
change.

Abstract
========

The current method for transforming functions and methods (for instance,
declaring them as a class or static method) is awkward and can lead to
code that is difficult to understand. Ideally, these transformations
should be made at the same point in the code where the declaration
itself is made. This PEP introduces new syntax for transformations of a
function or method declaration.


Motivation
==========

The current method of applying a transformation to a function or
method places the actual translation after the function body.  For
large functions this separates a key component of the function's
behavior from the definition of the rest of the function's external
interface.  For example::

    def foo(self):
        perform method operation
    foo = classmethod(foo)

This becomes less readable with longer methods.  It also seems less
than pythonic to name the function three times for what is
conceptually a single declaration.  A solution to this problem is to
move the transformation of the method closer to the method's own
declaration.  While the new syntax is not yet final, the intent is to
replace::

    def foo(cls):
        pass
    foo = synchronized(lock)(foo)
    foo = classmethod(foo)

with an alternative that places the decoration in the function's
declaration::

    @classmethod
    @synchronized(lock)
    def foo(cls):
        pass

Modifying classes in this fashion is also possible, though the
benefits are not as immediately apparent.  Almost certainly, anything
which could be done with class decorators could be done using
metaclasses, but using metaclasses is sufficiently obscure that there
is some attraction to having an easier way to make simple
modifications to classes. For Python 2.4, only function/method 
decorators are being added.


Background
==========

There is general agreement that syntactic support is desirable to the
current state of affairs.  Guido mentioned `syntactic support for
decorators`_ in his DevDay keynote presentation at the `10th Python
Conference`_, though `he later said`_ it was only one of several
extensions he proposed there "semi-jokingly".  `Michael Hudson raised
the topic`_ on ``python-dev`` shortly after the conference,
attributing the initial bracketed syntax to an earlier proposal on
``comp.lang.python`` by `Gareth McCaughan`_.

.. _syntactic support for decorators:
   http://www.python.org/doc/essays/ppt/python10/py10keynote.pdf
.. _10th python conference:
   http://www.python.org/workshops/2002-02/
.. _michael hudson raised the topic:
   http://mail.python.org/pipermail/python-dev/2002-February/020005.html
.. _he later said:
   http://mail.python.org/pipermail/python-dev/2002-February/020017.html
.. _gareth mccaughan:
   http://groups.google.com/groups?hl=en&lr=&ie=UTF-8&oe=UTF-8&selm=slrna40k88.2h9o.Gareth.McCaughan%40g.local

Class decorations seem like an obvious next step because class
definition and function definition are syntactically similar.

The discussion continued on and off on python-dev from February 2002
through July 2004. Hundreds and hundreds of posts were made, with people
proposing many possible syntax variations.  Guido took a list of
proposals to `EuroPython 2004`_, where a discussion took place.
Subsequent to this, he decided that for 2.4a2 we'd have the `Java-style`_
@decorator syntax.  Barry Warsaw named this the 'pie-decorator'
syntax, in honor of the Pie-thon Parrot shootout which was announced
about the same time as the decorator syntax, and because the @ looks a
little like a pie.  Guido `outlined his case`_ on Python-dev,
including `this piece`_ on the various rejected forms.

.. _EuroPython 2004:
    http://www.python.org/doc/essays/ppt/euro2004/euro2004.pdf

.. _outlined his case:
    http://mail.python.org/pipermail/python-dev/2004-August/author.html

.. _this piece:
    http://mail.python.org/pipermail/python-dev/2004-August/046672.html

..  Java-style:
    http://java.sun.com/j2se/1.5.0/docs/guide/language/annotations.html

On the name 'Decorator'
=======================

There's been a number of complaints about the choice of the name 
'decorator' for this feature. The major one is that the name is
not consistent with it's use in the `GoF book`_. The name 'decorator'
probably owes more to it's use in the compiler area - a syntax tree
is walked and annotated. It's quite possible that a better name may
turn up.

.. GoF book:
    http://patterndigest.com/patterns/Decorator.html

Design Goals
============

The new syntax should

*  work for arbitrary wrappers, including user-defined callables and
   the existing builtins ``classmethod()`` and ``staticmethod()``

*  work with multiple wrappers per definition

*  make it obvious what is happening; at the very least it should be
   obvious that new users can safely ignore it when writing their own
   code

*  not make future extensions more difficult

*  be easy to type;  programs that use it are expected to use it very
   frequently

*  not make it more difficult to scan through code quickly.  It should
   still be easy to search for all definitions, a particular
   definition, or the arguments that a function accepts

*  not needlessly complicate secondary support tools such as
   language-sensitive editors and other "`toy parser tools out
   there`_"

*  move from the end of the function, where it's currently hidden, to
   the front where it is more `in your face`_

.. _toy parser tools out there:
   http://groups.google.com/groups?hl=en&lr=&ie=UTF-8&oe=UTF-8&selm=mailman.1010809396.32158.python-list%40python.org

.. in your face:
    http://mail.python.org/pipermail/python-dev/2004-August/047112.html

Andrew Kuchling has links to a bunch of the discussions about motivations
and use cases `in his blog`_. Particularly notable is `Jim Huginin's list
of use cases`_.

.. _in his blog:
    http://www.amk.ca/diary/archives/cat_python.html#003255

.. _Jim Huginin's list of use cases:
    http://mail.python.org/pipermail/python-dev/2004-April/044132.html

Current Syntax
==============

The current syntax for function decorators as implemented in Python
2.4a2 is::

    @dec2
    @dec1
    def func(arg1, arg2, ...):
        pass

This is equivalent to::

    def func(arg1, arg2, ...):
        pass
    func = dec2(dec1(func))

without the intermediate assignment to the variable ``func``.  The
decorators are near the function declaration.  The @ sign makes it
clear that something new is going on here.

The decorator statement is limited in what it can accept - arbitrary
expressions will not work. Guido preferred this because of a `gut feeling`_

.. _gut feeling:
    http://mail.python.org/pipermail/python-dev/2004-August/046711.html

Syntax Alternatives
===================

There have been `a large number`_ of different syntaxes proposed - rather
than attempting to work through these individual syntaxes, it's worthwhile
to break the syntax discussion down into a number of areas. Attempting to
discuss `each possible syntax`_ individually would be an act of madness,
and produce a completely unwieldly PEP.

.. a large number:
    http://www.python.org/moin/PythonDecorators
.. each possible syntax:
    http://ucsu.colorado.edu/~bethard/py/decorators-output.py

Decorator Location
------------------

The first syntax point is the location of the decorators. For the
following examples, we use the @syntax used in 2.4a2.

Decorators before the def statement are the first alternative,
and the syntax used in 2.4a2::

    @classmethod
    def foo(arg1,arg2):
        pass

    @accepts(int,int)
    @returns(float)
    def bar(low,high):
        pass

There have been a number of objections raised to this location -
the primary one is that it's the first real Python case where a 
line of code has a result on a following line. The syntax that
will be in 2.4a3 will also require one decorator per line (in a2,
multiple decorators can be specified on the same line).

Some of the advantages of this form are that the decorators live
outside the method body - they are obviously executed at the time
the function is defined

The second form is the decorators between the def and the function
name, or the function name and the argument list::

    def @classmethod foo(arg1,arg2):
        pass

    def @accepts(int,int),@returns(float) bar(low,high):
        pass

    def foo @classmethod (arg1,arg2):
        pass

    def bar @accepts(int,int),@returns(float) (low,high):
        pass

There are a couple of objections to this form.
The first is that it breaks easily 'greppability' of the source - you 
can no longer search for 'def foo(' and find the definition of the 
function. The second, more serious, objection is that in the case
of multiple decorators, the syntax would be extremely unwieldy.

The next form, which has had a number of strong proponents, is to
have the decorators between the argument list and the trailing ``:``
in the 'def' line::

    def foo(arg1,arg2) @classmethod:
        pass

    def bar(low,high) @accepts(int,int),@returns(float):
        pass

Guido `summarised the arguments`_ against this form (many of which
also apply to the previous form) as:

- it hides crucial information (e.g. that it is a static method)
  after the signature, where it is easily missed

- it's easy to miss the transition between a long argument list and a
  long decorator list

- it's cumbersome to cut and paste a decorator list for reuse, because
  it starts and ends in the middle of a line

.. summarised the arguments:
    http://mail.python.org/pipermail/python-dev/2004-August/047112.html

The next form is that the decorator syntax go inside the method
body at the start, in the same place that docstrings currently
live:

    def foo(arg1,arg2):
        @classmethod
        pass

    def bar(low,high):
        @accepts(int,int)
        @returns(float)
        pass

The primary objection to this form is that it requires "peeking inside"
the method body to determine the decorators. In addition, even though
the code is inside the method body, it is not executed when the method
is run. Guido felt that docstrings were not a good counter-example, and
that it was quite possible that a 'docstring' decorator could help move
the docstring to outside the function body.

The final form is a new block that encloses the method's code. For this
example, we'll use a 'decorate' keyword, as it makes no sense with the
@syntax.

    decorate:
        classmethod
        def foo(arg1,arg2):
            pass

    decorate:
        accepts(int,int)
        returns(float)
        def bar(low,high):
            pass

This form would result in inconsistent indentation for decorated and
undecorated methods. In addition, a decorated method's body would start
three indent levels in. 

Syntax forms
------------

@decorator

The major objections against this syntax are that the @ symbol
is not currently used in Python (and is used in both IPython and
Leo), that the @ symbol is not meaningful, 

|decorator

This is a variant on the @decorator syntax - it has the advantage
that it does not break IPython and Leo. It's major disadvantage 
compared to the @syntax is that the | symbol looks like both a 
capital I and a lowercase l.

* list syntax

The major objection to the list syntax is that it's currently
meaningful (when used in the form before the method). It's also
lacking any indication that the expression is a decorator.

* list syntax using other brackets ( <...>, [[...]], ... )


* decorate()

The decorate() proposal was that no new syntax be implemented - 
instead a magic function that used introspection to manipulate 
the following function. Both Jp Calderone and Philip Eby produced
implementations of functions that did this. Guido was pretty firmly
against this - with no new syntax, the magicness of a function like
this is extremely high.

* new keyword (and block)



Alternate Proposals
===================

The text in this section will be integrated into the previous section.
They're here for hysterical raisins for now.


Several other syntaxes have been proposed::

    def func(arg1, arg2, ...) as dec1, dec2, ...:
        pass

The absence of brackets makes it cumbersome to break long lists of
decorators across multiple lines, and the keyword "as" doesn't have
the same meaning as its use in the ``import`` statement.  Plenty of
`alternatives to "as"`_ have been proposed. :-)

.. _alternatives to "as":
   http://groups.google.com/groups?hl=en&lr=&ie=UTF-8&oe=UTF-8&threadm=mailman.236.1079968472.742.python-list%40python.org&rnum=2&prev=/groups%3Fq%3Dpython%2Bpep%2B318%26hl%3Den%26lr%3D%26ie%3DUTF-8%26oe%3DUTF-8%26selm%3Dmailman.236.1079968472.742.python-list%2540python.org%26rnum%3D2

::

    def [dec1, dec2, ...] func(arg1, arg2, ...):
        pass

This form has the disadvantage that the decorators visually assume
higher priority than the function name and argument list.

::

    def func [dec1, dec2, ...] (arg1, arg2, ...):
        pass

Quixote's `Python Template Language`_ uses this form, but only supports a
single decorator chosen from a restricted set.  For short lists it
works okay, but for long list it separates the argument list from the
function name.

.. _Python Template Language:
   http://www.mems-exchange.org/software/quixote/doc/PTL.html

::

    using:
        dec1
        dec2
        ...
    def foo(arg1, arg2, ...):
        pass

The function definition is not nested within the using: block making
it impossible to tell which objects following the block will be
decorated.  Nesting the function definition within the using: block
suggests nesting of namespaces that doesn't exist.  The name ``foo``
would actually exist at the same scope as the using: block.  Finally,
it would require the introduction of a new keyword.

The obvious alternative that nests the function within the block

::

    using:
        dec1
        dec2
        ...
        def foo(arg1, arg2, ...):
            pass

has its own set of drawbacks.  Having the minimal indent level be
three deep for methods is painful for those using limited-width
windows.  The inconsistent indentation between methods of the same
class with and without decorators would be a readability problem.
Finally, adding or removing decorators would require reindenting the
entire function/method body.

Guido proposed and implemented a patch to support interpretation of
a `list of decorators`_ as a prefix to function definitions ::

    [dec1, dec2, ...]
    def foo(arg1, arg2, ...):
        pass

For a while this was Guido's preferred solution, but negative sentiment ran
high, mostly because that syntax, though useless except for side
effects of the list, is already legal and thus creates a special case.

.. _list of decorators:
   http://python.org/sf/926860

Another variant on the list syntax that was initially favored was::

    def func(arg1, arg2, ...) [dec1, dec2]:
        pass

Guido decided `he preferred`_ having the decorators on the line before
the 'def', because it was felt that a long argument list would mean
that the decorators would be 'hidden'

.. _he preferred:
    http://mail.python.org/pipermail/python-dev/2004-March/043756.html

Phillip Eby and Jp Calderone both proposed variants that required
no new syntax, but instead used some fairly advanced introspection
to provide decorator-like behavoiur, but Guido was unimpressed by
these, stating::

    Using functions with "action-at-a-distance" through
    sys.settraceback may be okay for an obscure feature that can't be
    had any other way yet doesn't merit changes to the language, but
    that's not the situation for decorators.  The widely held view
    here is that decorators need to be added as a syntactic feature to
    avoid the problems with the postfix notation used in 2.2 and 2.3.
    Decorators are slated to be an important new language feature and
    their design needs to be forward-looking, not constrained by what
    can be implemented in 2.3.

A `page on the Python Wiki`_ was created to summarize a number of the
proposals.  Once it stabilizes perhaps someone would care to
incorporate its content into this PEP (hint, hint).

.. _page on the Python Wiki:
    http://www.python.org/moin/PythonDecorators


Why @? 
------

There is some history in Java using @ initially as a marker in
`Javadoc comments`_ and later in Java 1.5 for `annotations`_, 
which are similar to Python decorators. The fact that
@ was previously unused as a token in Python also means it's clear
there is no possibility of such code being parsed by an earlier
version of Python, leading to possibly subtle semantic bugs.  That
said, @ is still a fairly arbitrary choice.  Some have suggested using
| instead.

For syntax options which use a list-like syntax (no matter where it
appears) to specify the decorators a few alternatives were proposed:
``[|...|]``, ``*[...]*``, and ``<...>``.  None gained much traction.
The alternatives which involve square brackets only serve to make it
obvious that the decorator construct is not a list.  They do nothing
to make parsing any easier.  The '<...>' alternative presents parsing
problems because '<' and '>' already parse as un-paired.  They present
a further parsing ambiguity because a right angle bracket might be a
greater than symbol instead of a closer for the decorators.

.. _Javadoc comments:
    http://java.sun.com/j2se/javadoc/writingdoccomments/

.. annotations:
    http://java.sun.com/j2se/1.5.0/docs/guide/language/annotations.html

Current Implementation
======================

Guido asked for a voluteer to implement his preferred syntax, and Mark
Russell stepped up and posted a `patch`_ to SF. The syntax accepted
for 2.4a2 is::


    @dec2
    @dec1
    def func(arg1, arg2, ...):
        pass

is equivalent to::

    def func(arg1, arg2, ...):
        pass
    func = dec2(dec1(func))

though without the intermediate creation of a variable named ``func``.

.. _patch: http://www.python.org/sf/979728

A `previous patch`_ from Michael Hudson which implements the
list-after-def syntax is also still kicking around.

.. _previous patch: http://starship.python.net/crew/mwh/hacks/meth-syntax-sugar-3.diff


Examples
========

Much of the discussion on ``comp.lang.python`` and the ``python-dev``
mailing list focuses on the use of decorators as a cleaner way to use
the ``staticmethod()`` and ``classmethod()`` builtins.  This
capability is much more powerful than that.  This section presents
some examples of use.

1. Define a function to be executed at exit.  Note that the function
   isn't actually "wrapped" in the usual sense.

   ::

       def onexit(f):
           import atexit
           atexit.register(f)
           return f

       @onexit
       def func():
           ...

2. Define a class with a singleton instance.  Note that once the class
   disappears enterprising programmers would have to be more creative
   to create more instances.  (From Shane Hathaway on ``python-dev``.)

   ::

       def singleton(cls):
           instances = {}
           def getinstance():
               if cls not in instances:
                   instances[cls] = cls()
               return instances[cls]
           return getinstance

       @singleton
       class MyClass:
           ...

3. Add attributes to a function.  (Based on an example posted by
   Anders Munch on ``python-dev``.)

   ::

       def attrs(**kwds):
           def decorate(f):
               for k in kwds:
                   setattr(f, k, kwds[k])
               return f
           return decorate

       @attrs(versionadded="2.2",
              author="Guido van Rossum")
       def mymethod(f):
           ...

4. Enforce function argument and return types.  (Note that this is not
   exactly correct, as the returned new_f doesn't have "func" as its
   func_name attribute.)

   ::

       def accepts(*types):
           def check_accepts(f):
               assert len(types) == f.func_code.co_argcount
               def new_f(*args, **kwds):
                   for (a, t) in zip(args, types):
                       assert isinstance(a, t), \
                              "arg %r does not match %s" % (a,t)
                   return f(*args, **kwds)
               return new_f
           return check_accepts

       def returns(rtype):
           def check_returns(f):
               def new_f(*args, **kwds):
                   result = f(*args, **kwds)
                   assert isinstance(result, rtype), \
                          "return value %r does not match %s" % (result,rtype)
                   return result
               return new_f
           return check_returns

       @accepts(int, (int,float))
       @returns((int,float))
       def func(arg1, arg2):
           return arg1 * arg2

5. Declare that a class implements a particular (set of) interface(s).
   This is from a posting by Bob Ippolito on ``python-dev`` based on
   experience with `PyProtocols`_.

   .. _PyProtocols: http://peak.telecommunity.com/PyProtocols.html

   ::

       def provides(*interfaces):
            """
            An actual, working, implementation of provides for
            the current implementation of PyProtocols.  Not
            particularly important for the PEP text.
            """
            def provides(typ):
                declareImplementation(typ, instancesProvide=interfaces)
                return typ
            return provides

       class IBar(Interface):
            """Declare something about IBar here"""

       @provides(IBar)
       class Foo(object):
               """Implement something here..."""

Of course, all these examples are possible today, though without
syntactic support.


Open Issues
===========

1. It's not yet certain that class decorators will be incorporated
   into the language at this point.  Guido expressed skepticism about
   the concept, but various people have made some `strong arguments`_
   (search for ``PEP 318 - posting draft``) on their behalf in
   ``python-dev``.

   .. _strong arguments:
      http://mail.python.org/pipermail/python-dev/2004-March/thread.html


Copyright
=========

This document has been placed in the public domain.


..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   End: