python-peps/pep-3104.txt

PEP: 3104
Title: Access to Names in Outer Scopes
Version: $Revision$
Last-Modified: $Date$
Author: Ka-Ping Yee <ping@zesty.ca>
Status: Draft
Type: Standards Track
Python-Version: 3.0
Content-Type: text/x-rst
Created: 12-Oct-2006


Abstract
========

In most languages that support nested scopes, code can refer to or
rebind (assign to) any name in the nearest enclosing scope.
Currently, Python code can refer to a name in any enclosing scope,
but it can only rebind names in two scopes: the local scope (by
simple assignment) or the module-global scope (using a ``global``
declaration).

This limitation has been raised many times on the Python-Dev mailing
list and elsewhere, and has led to extended discussion and many
proposals for ways to remove this limitation.  This PEP summarizes
the various alternatives that have been suggested, together with
advantages and disadvantages that have been mentioned for each.


Rationale
=========

Before version 2.1, Python's treatment of scopes resembled that of
standard C: within a file there were only two levels of scope, global
and local.  In C, this is a natural consequence of the fact that
function definitions cannot be nested.  But in Python, though
functions are usually defined at the top level, a function definition
can be executed anywhere.  This gave Python the syntactic appearance
of nested scoping without the semantics, and yielded inconsistencies
that were surprising to some programmers -- for example, a recursive
function that worked at the top level would cease to work when moved
inside another function, because the recursive function's own name
would no longer be visible in its body's scope.  This violates the
intuition that a function should behave consistently when placed in
different contexts.  Here's an example::

    def enclosing_function():
        def factorial(n):
            if n < 2:
                return 1
            return n * factorial(n - 1)  # fails with NameError
        print factorial(5)

Python 2.1 moved closer to static nested scoping by making visible
the names bound in all enclosing scopes (see PEP 227).  This change
makes the above code example work as expected.  However, because any
assignment to a name implicitly declares that name to be local, it is
impossible to rebind a name in an outer scope (except when a
``global`` declaration forces the name to be global).  Thus, the
following code, intended to display a number that can be incremented
and decremented by clicking buttons, doesn't work as someone familiar
with lexical scoping might expect::

    def make_scoreboard(frame, score=0):
        label = Label(frame)
        label.pack()
        for i in [-10, -1, 1, 10]:
            def increment(step=i):
                score = score + step  # fails with UnboundLocalError
                label['text'] = score
            button = Button(frame, text='%+d' % i, command=increment)
            button.pack()
        return label

Python syntax doesn't provide a way to indicate that the name
``score`` mentioned in ``increment`` refers to the variable ``score``
bound in ``make_scoreboard``, not a local variable in ``increment``.
Users and developers of Python have expressed an interest in removing
this limitation so that Python can have the full flexibility of the
Algol-style scoping model that is now standard in many programming
languages, including JavaScript, Perl, Ruby, Scheme, Smalltalk,
C with GNU extensions, and C# 2.0.

It has been argued that that such a feature isn't necessary, because
a rebindable outer variable can be simulated by wrapping it in a
mutable object::

    class Namespace:
        pass

    def make_scoreboard(frame, score=0):
        ns = Namespace()
        ns.score = 0
        label = Label(frame)
        label.pack()
        for i in [-10, -1, 1, 10]:
            def increment(step=i):
                ns.score = ns.score + step
                label['text'] = ns.score
            button = Button(frame, text='%+d' % i, command=increment)
            button.pack()
        return label

However, this workaround only highlights the shortcomings of existing
scopes: the purpose of a function is to encapsulate code in its own
namespace, so it seems unfortunate that the programmer should have to
create additional namespaces to make up for missing functionality in
the existing local scopes, and then have to decide whether each name
should reside in the real scope or the simulated scope.

Another common objection is that the desired functionality can be
written as a class instead, albeit somewhat more verbosely.  One
rebuttal to this objection is that the existence of a different
implementation style is not a reason to leave a supported programming
construct (nested scopes) functionally incomplete.  Python is
sometimes called a "multi-paradigm language" because it derives so
much strength, practical flexibility, and pedagogical power from its
support and graceful integration of multiple programming paradigms.

A proposal for scoping syntax appeared on Python-Dev as far back as
1994 [1]_, long before PEP 227's support for nested scopes was
adopted.  At the time, Guido's response was:

    This is dangerously close to introducing CSNS [classic static
    nested scopes]. *If* you were to do so, your proposed semantics
    of scoped seem allright. I still think there is not enough need
    for CSNS to warrant this kind of construct ...

After PEP 227, the "outer name rebinding discussion" has reappeared
on Python-Dev enough times that it has become a familiar event,
having recurred in its present form since at least 2003 [2]_.  
Although none of the language changes proposed in these discussions
have yet been adopted, Guido has acknowledged that a language change
is worth considering [12]_.


Other Languages
===============

To provide some background, this section describes how some other
languages handle nested scopes and rebinding.

JavaScript, Perl, Scheme, Smalltalk, GNU C, C# 2.0
--------------------------------------------------

These languages use variable declarations to indicate scope.  In
JavaScript, a lexically scoped variable is declared with the ``var``
keyword; undeclared variable names are assumed to be global.  In
Perl, a lexically scoped variable is declared with the ``my``
keyword; undeclared variable names are assumed to be global.  In
Scheme, all variables must be declared (with ``define`` or ``let``,
or as formal parameters).  In Smalltalk, any block can begin by
declaring a list of local variable names between vertical bars.
C and C# require type declarations for all variables.  For all these
cases, the variable belongs to the scope containing the declaration. 

Ruby (as of 1.8)
----------------

Ruby is an instructive example because it appears to be the only
other currently popular language that, like Python, tries to support
statically nested scopes without requiring variable declarations, and
thus has to come up with an unusual solution.  Functions in Ruby can
contain other function definitions, and they can also contain code
blocks enclosed in curly braces.  Blocks have access to outer
variables, but nested functions do not.  Within a block, an
assignment to a name implies a declaration of a local variable only
if it would not shadow a name already bound in an outer scope;
otherwise assignment is interpreted as rebinding of the outer name.
Ruby's scoping syntax and rules have also been debated at great
length, and changes seem likely in Ruby 2.0 [25]_.


Overview of Proposals
=====================

There have been many different proposals on Python-Dev for ways to
rebind names in outer scopes.  They all fall into two categories:
new syntax in the scope where the name is bound, or new syntax in
the scope where the name is used.

New Syntax in the Binding (Outer) Scope
---------------------------------------

Scope Override Declaration
''''''''''''''''''''''''''

The proposals in this category all suggest a new kind of declaration
statement similar to JavaScript's ``var``.  A few possible keywords
have been proposed for this purpose:

    - ``scope x`` [4]_
    - ``var x`` [4]_ [9]_
    - ``my x`` [13]_

In all these proposals, a declaration such as ``var x`` in a
particular scope S would cause all references to ``x`` in scopes
nested within S to refer to the ``x`` bound in S.

The primary objection to this category of proposals is that the
meaning of a function definition would become context-sensitive.
Moving a function definition inside some other block could cause any
of the local name references in the function to become nonlocal, due
to declarations in the enclosing block.  For blocks in Ruby 1.8,
this is actually the case; in the following example, the two setters
have different effects even though they look identical::

    setter1 = proc { | x | y = x }      # y is local here
    y = 13
    setter2 = proc { | x | y = x }      # y is nonlocal here
    setter1.call(99)
    puts y                              # prints 13
    setter2.call(77)
    puts y                              # prints 77

Note that although this proposal resembles declarations in JavaScript
and Perl, the effect on the language is different because in those
languages undeclared variables are global by default, whereas in
Python undeclared variables are local by default.  Thus, moving
a function inside some other block in JavaScript or Perl can only
reduce the scope of a previously global name reference, whereas in
Python with this proposal, it could expand the scope of a previously
local name reference.

Required Variable Declaration
'''''''''''''''''''''''''''''

A more radical proposal [21]_ suggests removing Python's scope-guessing
convention altogether and requiring that all names be declared in the
scope where they are to be bound, much like Scheme.  With this
proposal, ``var x = 3`` would both declare ``x`` to belong to the
local scope and bind it, where as ``x = 3`` would rebind the existing
visible ``x``.  In a context without an enclosing scope containing a
``var x`` declaration, the statement ``x = 3`` would be statically
determined to be illegal.

This proposal yields a simple and consistent model, but it would be
incompatible with all existing Python code.

New Syntax in the Referring (Inner) Scope
-----------------------------------------

There are three kinds of proposals in this category.

Outer Reference Expression
''''''''''''''''''''''''''

This type of proposal suggests a new way of referring to a variable
in an outer scope when using the variable in an expression.  One
syntax that has been suggested for this is ``.x`` [7]_, which would
refer to ``x`` without creating a local binding for it.  A concern
with this proposal is that in many contexts ``x`` and ``.x`` could
be used interchangeably, which would confuse the reader.  A closely
related idea is to use multiple dots to specify the number of scope
levels to ascend [8]_, but most consider this too error-prone [17]_.

Rebinding Operator
''''''''''''''''''

This proposal suggests a new assignment-like operator that rebinds
a name without declaring the name to be local [2]_.  Whereas the
statement ``x = 3`` both declares ``x`` a local variable and binds
it to 3, the statement ``x := 3`` would change the existing binding
of ``x`` without declaring it local.

This is a simple solution, but according to PEP 3099 it has been
rejected (perhaps because it would be too easy to miss or to confuse
with ``=``).

Scope Override Declaration
''''''''''''''''''''''''''

The proposals in this category suggest a new kind of declaration
statement in the inner scope that prevents a name from becoming
local.  This statement would be similar in nature to the ``global``
statement, but instead of making the name refer to a binding in the
top module-level scope, it would make the name refer to the binding
in the nearest enclosing scope.

This approach is attractive due to its parallel with a familiar
Python construct, and because it retains context-independence for
function definitions.

This approach also has advantages from a security and debugging
perspective.  The resulting Python would not only match the
functionality of other nested-scope languages but would do so with a
syntax that is arguably even better for defensive programming.  In
most other languages, a declaration contracts the scope of an
existing name, so inadvertently omitting the declaration could yield
farther-reaching (i.e. more dangerous) effects than expected.  In
Python with this proposal, the extra effort of adding the declaration
is aligned with the increased risk of non-local effects (i.e. the
path of least resistance is the safer path).

Many spellings have been suggested for such a declaration:

    - ``scoped x`` [1]_
    - ``global x in f`` [3]_ (explicitly specify which scope)
    - ``free x`` [5]_
    - ``outer x`` [6]_
    - ``use x`` [9]_
    - ``global x`` [10]_ (change the meaning of ``global``)
    - ``nonlocal x`` [11]_
    - ``global x outer`` [18]_
    - ``global in x`` [18]_
    - ``not global x`` [18]_
    - ``extern x`` [20]_
    - ``ref x`` [22]_
    - ``refer x`` [22]_
    - ``share x`` [22]_
    - ``sharing x`` [22]_
    - ``common x`` [22]_
    - ``using x`` [22]_
    - ``borrow x`` [22]_
    - ``reuse x`` [23]_

The most commonly discussed choices appear to be ``outer``,
``global``, and ``nonlocal``.  ``outer`` is already used as both a
variable name and an attribute name in the standard library.  The
word ``global`` has a conflicting meaning, because "global variable"
is generally understood to mean a variable with top-level scope [24]_.
In C, the keyword ``extern`` means that a name refers to a variable
in a different compilation unit.  While ``nonlocal`` is a bit long
and less pleasant-sounding than some of the other options, it does
have precisely the correct meaning: it declares a name not local.


Proposed Solution
=================

The solution proposed by this PEP is to add a scope override
declaration in the referring (inner) scope.  Guido has expressed a
preference for this category of solution on Python-Dev [14]_ and has
shown approval for ``nonlocal`` as the keyword [19]_.

The proposed declaration::

    nonlocal x

prevents ``x`` from becoming a local name in the current scope.  All
occurrences of ``x`` in the current scope will refer to the ``x``
bound in an outer enclosing scope.  As with ``global``, multiple
names are permitted::

    nonlocal x, y, z

If there is no pre-existing binding in an enclosing scope, the
compiler raises a SyntaxError.  (It may be a bit of a stretch to
call this a syntax error, but so far SyntaxError is used for all
compile-time errors, including, for example, __future__ import
with an unknown feature name.)  Guido has said that this kind of
declaration in the absence of an outer binding should be considered
an error [16]_.

If a ``nonlocal`` declaration collides with the name of a formal
parameter in the local scope, the compiler raises a SyntaxError.

A shorthand form is also permitted, in which ``nonlocal`` is
prepended to an assignment or augmented assignment::

    nonlocal x = 3

The above has exactly the same meaning as ``nonlocal x; x = 3``.
(Guido supports a similar form of the ``global`` statement [26]_.)

On the left side of the shorthand form, only identifiers are allowed,
not target expressions like ``x[0]``.  Otherwise, all forms of
assignment are allowed.  The proposed grammar of the ``nonlocal``
statement is::

    nonlocal_stmt ::=
        "nonlocal" identifier ("," identifier)*
                   ["=" (target_list "=")+ expression_list]
      | "nonlocal" identifier augop expression_list

The rationale for allowing all these forms of assignment is that it
simplifies understanding of the ``nonlocal`` statement.  Separating
the shorthand form into a declaration and an assignment is sufficient
to understand what it means and whether it is valid.


Backward Compatibility
======================

This PEP targets Python 3000, as suggested by Guido [19]_.  However,
others have noted that some options considered in this PEP may be
small enough changes to be feasible in Python 2.x [27]_, in which
case this PEP could possibly be moved to be a 2.x series PEP.

As a (very rough) measure of the impact of introducing a new keyword,
here is the number of times that some of the proposed keywords appear
as identifiers in the standard library, according to a scan of the
Python SVN repository on November 5, 2006::

    nonlocal    0
    use         2
    using       3
    reuse       4
    free        8
    outer     147

``global`` appears 214 times as an existing keyword.  As a measure
of the impact of using ``global`` as the outer-scope keyword, there
are 18 files in the standard library that would break as a result
of such a change (because a function declares a variable ``global``
before that variable has been introduced in the global scope)::

    cgi.py
    dummy_thread.py
    mhlib.py
    mimetypes.py
    idlelib/PyShell.py
    idlelib/run.py
    msilib/__init__.py
    test/inspect_fodder.py
    test/test_compiler.py
    test/test_decimal.py
    test/test_descr.py
    test/test_dummy_threading.py
    test/test_fileinput.py
    test/test_global.py (not counted: this tests the keyword itself)
    test/test_grammar.py (not counted: this tests the keyword itself)
    test/test_itertools.py
    test/test_multifile.py
    test/test_scope.py (not counted: this tests the keyword itself)
    test/test_threaded_import.py
    test/test_threadsignals.py
    test/test_warnings.py


References
==========

.. [1] Scoping (was Re: Lambda binding solved?) (Rafael Bracho)
   http://www.python.org/search/hypermail/python-1994q1/0301.html

.. [2] Extended Function syntax (Just van Rossum)
   http://mail.python.org/pipermail/python-dev/2003-February/032764.html

.. [3] Closure semantics (Guido van Rossum)
   http://mail.python.org/pipermail/python-dev/2003-October/039214.html

.. [4] Better Control of Nested Lexical Scopes (Almann T. Goo)
   http://mail.python.org/pipermail/python-dev/2006-February/061568.html

.. [5] PEP for Better Control of Nested Lexical Scopes (Jeremy Hylton)
   http://mail.python.org/pipermail/python-dev/2006-February/061602.html

.. [6] PEP for Better Control of Nested Lexical Scopes (Almann T. Goo)
   http://mail.python.org/pipermail/python-dev/2006-February/061603.html

.. [7] Using and binding relative names (Phillip J. Eby)
   http://mail.python.org/pipermail/python-dev/2006-February/061636.html

.. [8] Using and binding relative names (Steven Bethard)
   http://mail.python.org/pipermail/python-dev/2006-February/061749.html

.. [9] Lexical scoping in Python 3k (Ka-Ping Yee)
   http://mail.python.org/pipermail/python-dev/2006-July/066862.html

.. [10] Lexical scoping in Python 3k (Greg Ewing)
   http://mail.python.org/pipermail/python-dev/2006-July/066889.html

.. [11] Lexical scoping in Python 3k (Ka-Ping Yee)
   http://mail.python.org/pipermail/python-dev/2006-July/066942.html

.. [12] Lexical scoping in Python 3k (Guido van Rossum)
   http://mail.python.org/pipermail/python-dev/2006-July/066950.html

.. [13] Explicit Lexical Scoping (pre-PEP?) (Talin)
   http://mail.python.org/pipermail/python-dev/2006-July/066978.html

.. [14] Explicit Lexical Scoping (pre-PEP?) (Guido van Rossum)
   http://mail.python.org/pipermail/python-dev/2006-July/066991.html

.. [15] Explicit Lexical Scoping (pre-PEP?) (Guido van Rossum)
   http://mail.python.org/pipermail/python-dev/2006-July/066995.html

.. [16] Lexical scoping in Python 3k (Guido van Rossum)
   http://mail.python.org/pipermail/python-dev/2006-July/066968.html

.. [17] Explicit Lexical Scoping (pre-PEP?) (Guido van Rossum)
   http://mail.python.org/pipermail/python-dev/2006-July/067004.html

.. [18] Explicit Lexical Scoping (pre-PEP?) (Andrew Clover)
   http://mail.python.org/pipermail/python-dev/2006-July/067007.html

.. [19] Explicit Lexical Scoping (pre-PEP?) (Guido van Rossum)
   http://mail.python.org/pipermail/python-dev/2006-July/067067.html

.. [20] Explicit Lexical Scoping (pre-PEP?) (Matthew Barnes)
   http://mail.python.org/pipermail/python-dev/2006-July/067221.html

.. [21] Sky pie: a "var" keyword (a thread started by Neil Toronto)
   http://mail.python.org/pipermail/python-3000/2006-October/003968.html

.. [22] Alternatives to 'outer' (Talin)
   http://mail.python.org/pipermail/python-3000/2006-October/004021.html

.. [23] Alternatives to 'outer' (Jim Jewett)
   http://mail.python.org/pipermail/python-3000/2006-November/004153.html

.. [24] Global variable (version 2006-11-01T01:23:16)
   http://en.wikipedia.org/wiki/Global_variable

.. [25] Ruby 2.0 block local variable
   http://redhanded.hobix.com/inspect/ruby20BlockLocalVariable.html

.. [26] Draft PEP for outer scopes (Guido van Rossum)
   http://mail.python.org/pipermail/python-3000/2006-November/004166.html

.. [27] Draft PEP for outer scopes (Nick Coghlan)
   http://mail.python.org/pipermail/python-3000/2006-November/004237.html

Acknowledgements
================

The ideas and proposals mentioned in this PEP are gleaned from
countless Python-Dev postings.  Thanks to Jim Jewett, Mike Orr,
Jason Orendorff, and Christian Tanzer for suggesting specific
edits to this PEP.


Copyright
=========

This document has been placed in the public domain.


..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   coding: utf-8
   End: