python-peps/pep-3150.txt

PEP: 3150
Title: Statement local namespaces (aka "where" clause)
Version: $Revision$
Last-Modified: $Date$
Author: Nick Coghlan <ncoghlan@gmail.com>
Status: Deferred
Type: Standards Track
Content-Type: text/x-rst
Created: 2010-07-09
Python-Version: 3.3
Post-History: 2010-07-14
Resolution: TBD


Abstract
========

A recurring proposal on python-ideas is the addition of some form of
statement local namespace.

This PEP is intended to serve as a focal point for those ideas, so we
can hopefully avoid retreading the same ground a couple of times a
year. Even if the proposal is never accepted having a PEP to point
people to can be valuable (e.g. having PEP 315 helps greatly in avoiding
endless rehashing of loop-and-a-half arguments).

The ideas in this PEP are just a sketch of a way this concept might work.
They avoid some pitfalls that have been encountered in the past, but
have not themselves been subject to the test of implementation.


PEP Deferral
============

This PEP is currently deferred at least until the language moratorium
(PEP 3003) is officially lifted by Guido. Even after that, it will
require input from at least the four major Python implementations
(CPython, PyPy, Jython, IronPython) on the feasibility of implementing
the proposed semantics to get it moving again.

Proposal
========

This PEP proposes the addition of an optional "where" clause to the
syntax for simple statements which may contain an expression. The
current list of simple statements that would be affected by this
addition is as follows:

* expression statement
* assignment statement
* augmented assignment statement
* del statement
* return statement
* yield statement
* raise statement
* assert statement

The ``where`` clause would allow subexpressions to be referenced by
name in the header line, with the actual definitions following in
the indented clause. As a simple example::

 c = sqrt(a*a + b*b) where:
    a = retrieve_a()
    b = retrieve_b()

Torture Test
============

An implementation of this PEP must support execution of the following
code at module, class and function scope::

  b = {}
  a = b[f(a)] = x where:
    x = 42
    def f(x):
      return x
  assert "x" not in locals()
  assert "f" not in locals()
  assert a == 42
  assert d[42] == 42 where:
    d = b
  assert "d" not in locals()

Most naive implementations will choke on the first complex assignment,
while less naive but still broken implementations will fail when
the torture test is executed at class scope.

And yes, that's a perfectly well-defined assignment statement. Insane,
I agree, but legal::

  >>> def f(x): return x
  ... 
  >>> x = 42
  >>> b = {}
  >>> a = b[f(a)] = x
  >>> a
  42
  >>> b
  {42: 42}


Syntax Change
=============

Current::

  expr_stmt: testlist_star_expr (augassign (yield_expr|testlist) |
               ('=' (yield_expr|testlist_star_expr))*)
  del_stmt: 'del' exprlist
  return_stmt: 'return' [testlist]
  yield_stmt: yield_expr
  raise_stmt: 'raise' [test ['from' test]]
  assert_stmt: 'assert' test [',' test]


New::

  expr_stmt: testlist_star_expr (augassign (yield_expr|testlist) |
               ('=' (yield_expr|testlist_star_expr))*) [where_clause]
  del_stmt: 'del' exprlist [where_clause]
  return_stmt: 'return' [testlist] [where_clause]
  yield_stmt: yield_expr [where_clause]
  raise_stmt: 'raise' [test ['from' test]] [where_clause]
  assert_stmt: 'assert' test [',' test] [where_clause]
  where_clause: "where" ":" suite

(Note that expr_stmt in the grammar covers assignment and augmented
assignment in addition to simple expression statements)

The new clause is added as an optional element of the existing statements
rather than as a new kind of compound statement in order to avoid creating
an ambiguity in the grammar. It is applied only to the specific elements
listed so that nonsense like the following is disallowed::

   pass where:
       a = b = 1

However, even this is inadequate, as it creates problems for the definition
of simple_stmt (which allows chaining of multiple single line statements
with ";" rather than "\\n").

So the above syntax change should instead be taken as a statement of intent.
Any actual proposal would need to resolve the simple_stmt parsing problem
before it could be seriously considered. This would likely require a
non-trivial restructuring of the grammar, breaking up small_stmt and
flow_stmt to separate the statements that potentially contain arbitrary
subexpressions and then allowing a single one of those statements with
a ``where`` clause at the simple_stmt level. Something along the lines of::

  simple_stmt: small_stmt (';' small_stmt)* [';'] NEWLINE
  where_stmt: subexpr_stmt (where_clause | (';' small_stmt)* [';']) NEWLINE
  subexpr_stmt: expr_stmt | del_stmt | flow_subexpr_stmt | assert_stmt
  small_stmt: (pass_stmt | flow_stmt | import_stmt |
               global_stmt | nonlocal_stmt)
  flow_stmt: break_stmt | continue_stmt
  flow_subexpr_stmt: return_stmt | raise_stmt | yield_stmt
  where_clause: "where" ":" suite


For reference, here are the current definitions at that level::

  simple_stmt: small_stmt (';' small_stmt)* [';'] NEWLINE
  small_stmt: (expr_stmt | del_stmt | pass_stmt | flow_stmt |
               import_stmt | global_stmt | nonlocal_stmt | assert_stmt)
  flow_stmt: break_stmt | continue_stmt | return_stmt | raise_stmt | yield_stmt


Common Objections
=================

* Two Ways To Do It: a lot of code may now be written with values
  defined either before the expression where they are used or
  afterwards in a ``where`` clause, creating two ways to do it,
  without an obvious way of choosing between them.

* Out of Order Execution: the ``where`` clause makes execution
  jump around a little strangely, as the body of the ``where``
  clause is executed before the simple statement in the clause
  header. The closest any other part of Python comes to this
  before is the out of order evaluation in conditional
  expressions.
  

Possible Additions
==================

* The current proposal allows the addition of a ``where`` clause only
  for simple statements. Extending the idea to allow the use of
  compound statements would be quite possible, but doing so raises
  serious readability concerns (as values defined in the ``where``
  clause may be used well before they are defined, exactly the kind
  of readability trap that decorators were designed to eliminate)

* Currently only the outermost clause of comprehensions and generator
  expressions can reference the surrounding namespace when executed
  at class level. If this proposal is implemented successfully, the
  associated namespace semantics could allow that restriction to be
  lifted. There would be backwards compatibility implications in doing
  so as existing code may be relying on the behaviour of ignoring
  class level variables.

Possible Implementation Strategy
================================

AKA How Class Scopes Screw You When Attempting To Implement This

The natural idea when setting out to implement this concept is to
use an ordinary nested function scope. This doesn't work for the
two reasons mentioned in the Torture Test section above:

* Non-local variables are not your friend because they ignore class scopes
  and (when writing back to the outer scope) aren't really on speaking
  terms with module scopes either.

* Return-based semantics struggle with complex assignment statements
  like the one in the torture test

The most promising approach is one based on symtable analysis and
copy-in-copy-out referencing semantics to move any required name
bindings between the inner and outer scopes. The torture test above
would then translate to something like the following::

  b = {}
  def _anon1(b): # 'b' reference copied in
    x = 42
    def f(x):
      return x
    a = b[f(a)] = x
    return a # 'a' reference copied out
  a = _anon1(b)
  assert "x" not in locals()
  assert "f" not in locals()
  assert a == 42
  def _anon2(b) # 'b' reference copied in
    d = b
    assert d[42] == 42
    # Nothing to copy out (not an assignment)
  _anon2()
  assert "d" not in locals()

However, as noted in the abstract, an actual implementation of
this idea has never been tried.


Reference implementation
========================

None as yet. If you want a crash course in Python namespace
semantics and code compilation, feel free to try ;)


References
==========

.. [1] http://mail.python.org/pipermail/python-ideas/2010-June/007476.html

.. [2] http://mail.python.org/pipermail/python-ideas/2010-July/007584.html


Copyright
=========

This document has been placed in the public domain.



..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   coding: utf-8
   End: