python-peps/pep-0403.txt

PEP: 403
Title: Statement local functions and classes
Version: $Revision$
Last-Modified: $Date$
Author: Nick Coghlan <ncoghlan@gmail.com>
Status: Deferred
Type: Standards Track
Content-Type: text/x-rst
Created: 2011-10-13
Python-Version: 3.4
Post-History: 2011-10-13
Resolution: TBD


Abstract
========

This PEP proposes the addition of a new ``in`` statement that accepts a
statement local function or class definition.

The statement accepts a single simple statement that can make a forward
reference to a trailing function or class definition.

This new statement is designed to be used whenever a "one-shot" function or
class is needed, and placing the function or class definition before the
statement that uses it actually makes the code harder to read. It also
avoids any name shadowing concerns by making sure the new name is visible
only to the statement in the ``in`` clause.

This PEP is based heavily on many of the ideas in PEP 3150 (Statement Local
Namespaces) so some elements of the rationale will be familiar to readers of
that PEP. That PEP has now been withdrawn in favour of this one.


Basic Examples
==============

Before diving into the long history of this problem and the detailed
rationale for this specific proposed solution, here are a few simple
examples of the kind of code it is designed to simplify.

As a trivial example, a weakref callback could be defined as follows::

    in x = weakref.ref(target, report_destruction)
    def report_destruction(obj):
        print("{} is being destroyed".format(obj))

This contrasts with the current (conceptually) "out of order" syntax for
this operation::

    def report_destruction(obj):
        print("{} is being destroyed".format(obj))

    x = weakref.ref(target, report_destruction)

That structure is OK when you're using the callable multiple times, but
it's irritating to be forced into it for one-off operations.

If the repetition of the name seems especially annoying, then a throwaway
name like ``f`` can be used instead::

    in x = weakref.ref(target, f)
    def f(obj):
        print("{} is being destroyed".format(obj))


Similarly, a sorted operation on a particularly poorly defined type could
now be defined as::

    in sorted_list = sorted(original, key=f)
    def f(item):
        try:
            return item.calc_sort_order()
        except NotSortableError:
            return float('inf')

Rather than::

    def force_sort(item):
        try:
            return item.calc_sort_order()
        except NotSortableError:
            return float('inf')

    sorted_list = sorted(original, key=force_sort)

And early binding semantics in a list comprehension could be attained via::

    in funcs = [adder(i) for i in range(10)]
    def adder(i):
        return lambda x: x + i


Proposal
========

This PEP proposes the addition of a new ``in`` statement that is a variant
of the existing class and function definition syntax.

The new ``in`` clause replaces the decorator lines, and allows forward
references to the trailing function or class definition.

The trailing function or class definition is always named - the name of
the trailing definition is then used to make the forward reference from the
preceding statement.

The ``in`` clause is allowed to contain any simple statement (including those
that don't make any sense in that context, such as ``pass`` - while such code
would be legal, there wouldn't be any point in writing it). This permissive
structure is easier to define and easier to explain, but a more restrictive
approach that only permits operations that "make sense" would also be
possible (see PEP 3150 for a list of possible candidates).

The ``in`` statement will not create a new scope - all name binding
operations aside from the trailing function or class definition will affect
the containing scope.

The name used in the trailing function or class definition is only visible
from the associated ``in`` clause, and behaves as if it was an ordinary
variable defined in that scope. If any nested scopes are created in either
the ``in`` clause or the trailing function or class definition, those scopes
will see the trailing function or class definition rather than any other
bindings for that name in the containing scope.


Background
==========

The question of "multi-line lambdas" has been a vexing one for many
Python users for a very long time, and it took an exploration of Ruby's
block functionality for me to finally understand why this bugs people
so much: Python's demand that the function be named and introduced
before the operation that needs it breaks the developer's flow of thought.
They get to a point where they go "I need a one-shot operation that does
<X>", and instead of being able to just *say* that directly, they instead
have to back up, name a function to do <X>, then call that function from
the operation they actually wanted to do in the first place. Lambda
expressions can help sometimes, but they're no substitute for being able to
use a full suite.

Ruby's block syntax also heavily inspired the style of the solution in this
PEP, by making it clear that even when limited to *one* anonymous function per
statement, anonymous functions could still be incredibly useful. Consider how
many constructs Python has where one expression is responsible for the bulk of
the heavy lifting:

  * comprehensions, generator expressions, map(), filter()
  * key arguments to sorted(), min(), max()
  * partial function application
  * provision of callbacks (e.g. for weak references)
  * array broadcast operations in NumPy

However, adopting Ruby's block syntax directly won't work for Python, since
the effectiveness of Ruby's blocks relies heavily on various conventions in
the way functions are *defined* (specifically, using Ruby's ``yield`` syntax
to call blocks directly and the ``&arg`` mechanism to accept a block as a
function's final argument).

Since Python has relied on named functions for so long, the signatures of
APIs that accept callbacks are far more diverse, thus requiring a solution
that allows one-shot functions to be slotted in at the appropriate location.

The approach taken in this PEP is to retain the requirement to name the
function explicitly, but allow the relative order of the definition and the
statement that references it to be changed to match the developer's flow of
thought. The rationale is essentially the same as that used when introducing
decorators, but covering a broader set of applications.


Relation to PEP 3150
====================

PEP 3150 (Statement Local Namespaces) described its primary motivation
as being to elevate ordinary assignment statements to be on par with ``class``
and ``def`` statements where the name of the item to be defined is presented
to the reader in advance of the details of how the value of that item is
calculated. This PEP achieves the same goal in a different way, by allowing
the simple name binding of a standard function definition to be replaced
with something else (like assigning the result of the function to a value).

This PEP also achieves most of the other effects described in PEP 3150
without introducing a new brainbending kind of scope. All of the complex
scoping rules in PEP 3150 are replaced in this PEP with allowing a forward
reference to the associated function or class definition without creating an
actual name binding in the current scope.


Keyword Choice
==============

The proposal definitely requires *some* kind of prefix to avoid parsing
ambiguity and backwards compatibility problems with existing constructs.
It also needs to be clearly highlighted to readers, since it declares that
the following piece of code is going to be executed only after the trailing
function or class definition has been executed.

The ``in`` keyword was chosen as an existing keyword that can be used to
denote the concept of a forward reference.

For functions, the construct is intended to be read as "in <this statement
that references NAME> define NAME as a function that does <operation>".

The mapping to English prose isn't as obvious for the class definition case,
but the concept remains the same.


Better Debugging Support for Functions and Classes with Short Names
===================================================================

One of the objections to widespread use of lambda expressions is that they
have a negative effect on traceback intelligibility and other aspects of
introspection. Similarly objections are raised regarding constructs that
promote short, cryptic function names (including this one, which requires
that the name of the trailing definition be supplied at least twice)

However, the introduction of qualified names in PEP 3155 means that even
anonymous classes and functions will now have different representations if
they occur in different scopes. For example::

    >>> def f():
    ...     return lambda: y
    ...
    >>> f()
    <function f.<locals>.<lambda> at 0x7f6f46faeae0>

Anonymous functions (or functions that share a name) within the *same* scope
will still share representations (aside from the object ID), but this is
still a major improvement over the historical situation where everything
*except* the object ID was identical.

Syntax Change
=============

New::

    in_stmt: 'in' simple_stmt (classdef|funcdef)

Grammar: http://hg.python.org/cpython/file/default/Grammar/Grammar


Possible Implementation Strategy
================================

This proposal has at least one titanic advantage over PEP 3150:
implementation should be relatively straightforward.

The AST for the ``in`` statement will include both the function or class
definition and the statement that references it, so it should just be a
matter of emitting the two operations out of order and using a hidden
variable to link up any references.

The one potentially tricky part is changing the meaning of the references to
the statement local function or namespace while within the scope of the
``in`` statement, but that shouldn't be too hard to address by maintaining
some additional state within the compiler.


More Examples
=============

Calculating attributes without polluting the local namespace (from os.py)::

  # Current Python (manual namespace cleanup)
  def _createenviron():
      ... # 27 line function

  environ = _createenviron()
  del _createenviron

  # Becomes:
  in environ = _createenviron()
  def _createenviron():
      ... # 27 line function

Loop early binding::

  # Current Python (default argument hack)
  funcs = [(lambda x, i=i: x + i) for i in range(10)]

  # Becomes:
  in funcs = [adder(i) for i in range(10)]
  def adder(i):
    return lambda x: x + i

  # Or even:
  in funcs = [adder(i) for i in range(10)]
  def adder(i):
    in return incr
    def incr(x):
        return x + i

A trailing class can be used as a statement local namespace::

  # Evaluate subexpressions only once
  in c = math.sqrt(x.a*x.a + x.b*x.b)
  class x:
    a = calculate_a()
    b = calculate_b()


Reference Implementation
========================

None as yet.


Acknowledgements
================

Huge thanks to Gary Bernhardt for being blunt in pointing out that I had no
idea what I was talking about in criticising Ruby's blocks, kicking off a
rather enlightening process of investigation.


Rejected Concepts
=================

A previous incarnation of this PEP (see [1]) proposed a much uglier syntax
that (quite rightly) was not well received. The current proposal is
significantly easier both to read and write.

A more recent variant always used ``...`` for forward references, along
with genuinely anonymous function and class definitions. However, this
degenerated quickly into a mass of unintelligible dots in more complex
cases::

  in funcs = [...(i) for i in range(10)]
  def ...(i):
    in return ...
    def ...(x):
        return x + i

  in c = math.sqrt(....a*....a + ....b*....b)
  class ...:
    a = calculate_a()
    b = calculate_b()


References
==========

.. [1] Start of python-ideas thread:
   http://mail.python.org/pipermail/python-ideas/2011-October/012276.html


Copyright
=========

This document has been placed in the public domain.


..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   coding: utf-8
   End: