python-peps/pep-0403.txt

PEP: 403
Title: Prefix syntax for post function definition operations
Version: $Revision$
Last-Modified: $Date$
Author: Nick Coghlan <ncoghlan@gmail.com>
Status: Withdrawn
Type: Standards Track
Content-Type: text/x-rst
Created: 2011-10-13
Python-Version: 3.x
Post-History: 2011-10-13
Resolution: TBD


Abstract
========

This PEP proposes the addition of ``postdef`` as a new function prefix
syntax (analogous to decorators) that permits the execution of a single simple
statement (potentially including substatements separated by semi-colons) after

In addition, the new syntax would allow the 'def' keyword to be used to refer
to the function being defined without needing to repeat the name.

When the 'postdef' prefix syntax is used, the associated statement would be
executed *in addition to* the normal local name binding implicit in function
definitions. Any name collision are expected to be minor, analagous to those
encountered with ``for`` loop iteration variables.

This PEP is based heavily on many of the ideas in PEP 3150 (Statement Local
Namespaces) so some elements of the rationale will be familiar to readers of
that PEP. That PEP has now been withdrawn in favour of this one.


PEP Withdrawal
==============

The python-ideas thread discussing this PEP [1]_ persuaded me that it was
essentially am unnecessarily cryptic, wholly inferior version of PEP 3150's
statement local namespaces. The discussion also resolved some of my concerns
with PEP 3150, so I am withdrawing this more limited version of the idea in
favour of resurrecting the original concept.


Basic Examples
==============

Before diving into the long history of this problem and the detailed
rationale for this specific proposed solution, here are a few simple
examples of the kind of code it is designed to simplify.

As a trivial example, weakref callbacks could be defined as follows::

    postdef x = weakref.ref(target, def)
    def report_destruction(obj):
        print("{} is being destroyed".format(obj))

This contrasts with the current repetitive "out of order" syntax for this
operation::

    def report_destruction(obj):
        print("{} is being destroyed".format(obj))

    x = weakref.ref(target, report_destruction)

That structure is OK when you're using the callable multiple times, but
it's irritating to be forced into it for one-off operations.

Similarly, a sorted operation on a particularly poorly defined type could
now be defined as::

    postdef sorted_list = sorted(original, key=def)
    def force_sort(item):
        try:
            return item.calc_sort_order()
        except NotSortableError:
            return float('inf')

Rather than::

    def force_sort(item):
        try:
            return item.calc_sort_order()
        except NotSortableError:
            return float('inf')

    sorted_list = sorted(original, key=force_sort)

And early binding semantics in a list comprehension could be attained via::

    postdef funcs = [def(i) for i in range(10)]
    def make_incrementor(i):
        postdef return def
        def incrementor(x):
            return x + i


Proposal
========

This PEP proposes the addition of an optional block prefix clause to the
syntax for function and class definitions.

This block prefix would be introduced by a leading ``postdef`` and would be
allowed to contain any simple statement (including those that don't
make any sense in that context - while such code would be legal,
there wouldn't be any point in writing it). This permissive structure is
easier to define and easier to explain, but a more restrictive approach that
only permits operations that "make sense" would also be possible (see PEP
3150 for a list of possible candidates)

The function definition keyword ``def`` would be repurposed inside the block prefix
to refer to the function being defined.

When a block prefix is provided, the standard local name binding implicit
in the function definition still takes place.


Background
==========

The question of "multi-line lambdas" has been a vexing one for many
Python users for a very long time, and it took an exploration of Ruby's
block functionality for me to finally understand why this bugs people
so much: Python's demand that the function be named and introduced
before the operation that needs it breaks the developer's flow of thought.
They get to a point where they go "I need a one-shot operation that does
<X>", and instead of being able to just *say* that, they instead have to back
up, name a function to do <X>, then call that function from the operation
they actually wanted to do in the first place. Lambda expressions can help
sometimes, but they're no substitute for being able to use a full suite.

Ruby's block syntax also heavily inspired the style of the solution in this
PEP, by making it clear that even when limited to *one* anonymous function per
statement, anonymous functions could still be incredibly useful. Consider how
many constructs Python has where one expression is responsible for the bulk of
the heavy lifting:

  * comprehensions, generator expressions, map(), filter()
  * key arguments to sorted(), min(), max()
  * partial function application
  * provision of callbacks (e.g. for weak references)
  * array broadcast operations in NumPy

However, adopting Ruby's block syntax directly won't work for Python, since
the effectiveness of Ruby's blocks relies heavily on various conventions in
the way functions are *defined* (specifically, Ruby's ``yield`` syntax to
call blocks directly and the ``&arg`` mechanism to accept a block as a
function's final argument).

Since Python has relied on named functions for so long, the signatures of
APIs that accept callbacks are far more diverse, thus requiring a solution
that allows anonymous functions to be slotted in at the appropriate location.


Relation to PEP 3150
====================

PEP 3150 (Statement Local Namespaces) described its primary motivation
as being to elevate ordinary assignment statements to be on par with ``class``
and ``def`` statements where the name of the item to be defined is presented
to the reader in advance of the details of how the value of that item is
calculated. This PEP achieves the same goal in a different way, by allowing
the simple name binding of a standard function definition to be replaced
with something else (like assigning the result of the function to a value).

This PEP also achieves most of the other effects described in PEP 3150
without introducing a new brainbending kind of scope. All of the complex
scoping rules in PEP 3150 are replaced in this PEP with the simple ``def``
reference to the associated function definition.


Keyword Choice
==============

The proposal definitely requires *some* kind of prefix to avoid parsing
ambiguity and backwards compatibility problems with existing constructs.
It also needs to be clearly highlighted to readers, since it declares that
the following piece of code is going to be executed out of order.

The 'postdef' keyword was chosen as a literal explanation of exactly what
the new clause does: execute the specified statement *after* the associated
function definition, even though it is physically written *before* the
definition in the source code.


Requirement to Name Functions
=============================

One of the objections to widespread use of lambda expressions is that they
have an atrocious effect on traceback intelligibility and other aspects of
introspection. Accordingly, this PEP requires that even throwaway functions
be given some kind of name.

To help encourage the use of meaningful names without users having to repeat
themselves, the PEP suggests the provision of the ``def`` shorthand reference
to the current function from the ``postdef`` clause.


Syntax Change
=============

Current::

    atom: ('(' [yield_expr|testlist_comp] ')' |
           '[' [testlist_comp] ']' |
           '{' [dictorsetmaker] '}' |
           NAME | NUMBER | STRING+ | '...' | 'None' | 'True' | 'False')

Changed::

    atom: ('(' [yield_expr|testlist_comp] ')' |
           '[' [testlist_comp] ']' |
           '{' [dictorsetmaker] '}' |
           NAME | NUMBER | STRING+ | '...' | 'None' | 'True' | 'False' | 'def')

New::

    blockprefix: 'postdef' simple_stmt
    block: blockprefix funcdef

The above is the general idea, but I suspect that the change to the 'atom'
definition may cause an ambiguity problem in the parser when it comes to
detecting function definitions. So the actual implementation may need to be
more complex than that.

Grammar: http://hg.python.org/cpython/file/default/Grammar/Grammar


Possible Implementation Strategy
================================

This proposal has one titanic advantage over PEP 3150: implementation
should be relatively straightforward.

The post definition statement can be incorporated into the AST for the
function node and simply visited out of sequence.

The one potentially tricky part is working out how to allow the dual
use of 'def' without rewriting half the grammar definition.


More Examples
=============

Calculating attributes without polluting the local namespace (from os.py)::

  # Current Python (manual namespace cleanup)
  def _createenviron():
      ... # 27 line function

  environ = _createenviron()
  del _createenviron

  # Becomes:
  postdef environ = def()
  def _createenviron():
      ... # 27 line function

Loop early binding::

  # Current Python (default argument hack)
  funcs = [(lambda x, i=i: x + i) for i in range(10)]

  # Becomes:
  postdef funcs = [def(i) for i in range(10)]
  def make_incrementor(i):
    return lambda x: x + i

  # Or even:
  postdef funcs = [def(i) for i in range(10)]
  def make_incrementor(i):
    postdef return def
    def incrementor(x):
        return x + i


Reference Implementation
========================

None as yet.


Acknowledgements
================

Huge thanks to Gary Bernhardt for being blunt in pointing out that I had no
idea what I was talking about in criticising Ruby's blocks, kicking off a
rather enlightening process of investigation.

Even though this PEP has been withdrawn, the process of writing and arguing
in its favour has been quite influential on the future direction of PEP 3150.


References
==========

.. [1] Start of python-ideas thread:
   http://mail.python.org/pipermail/python-ideas/2011-October/012276.html


Copyright
=========

This document has been placed in the public domain.


..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   coding: utf-8
   End: