PEP: 577 Title: Augmented Assignment Expressions Author: Nick Coghlan Status: Withdrawn Type: Standards Track Content-Type: text/x-rst Created: 14-May-2018 Python-Version: 3.8 Post-History: 22-May-2018 PEP Withdrawal ============== While working on this PEP, I realised that it didn't really address what was actually bothering me about PEP 572's proposed scoping rules for previously unreferenced assignment targets, and also had some significant undesirable consequences (most notably, allowing ``>>=` and ``<<=`` as inline augmented assignment operators that meant something entirely different from the ``>=`` and ``<=`` comparison operators). I also realised that even without dedicated syntax of their own, PEP 572 allows inline augmented assignments to be written using the ``operator`` module:: from operator import iadd if (target := iadd(target, value)) < limit: ... (The restriction to simple names as inline assignment targets means that the target expession can always be repeated without side effects) Accordingly, I'm withdrawing this PEP without submitting it for pronouncement, and will instead be writing a replacement PEP that focuses specifically on the handling of assignment targets which haven't already been declared as local variables in the current scope (for both regular block scopes, and for scoped expressions). Abstract ======== This is a proposal to allow augmented assignments such as ``x += 1`` to be used as expressions, not just statements. As part of this, ``NAME := EXPR`` is proposed as an inline assignment expression that uses the new augmented assignment scoping rules, rather than implicitly defining a new local variable name the way that existing name binding statements do. The question of allowing expression level local variable declarations at function scope is deliberately separated from the question of allowing expression level name bindings, and deferred to a later PEP. This PEP is a direct competitor to PEP 572 (although it borrows heavily from that PEP's motivation, and even shares the proposed syntax for inline assignments). See `Relationship with PEP 572`_ for more details on the connections between the two PEPs. To improve the usability of the new expressions, a semantic split is proposed between the handling of augmented assignments in regular block scopes (modules, classes, and functions), and the handling of augmented assignments in scoped expressions (lambda expressions, generator expressions, and comprehensions), such that all inline assignments default to targeting the nearest containing block scope. A new compile time ``TargetNameError`` is added as a subclass of ``SyntaxError`` to handle cases where it is deemed to be currently unclear which target is expected to be rebound by an inline assignment, or else the target scope for the inline assignment is considered invalid for another reason. Syntax and semantics ==================== Augmented assignment expressions -------------------------------- The language grammar would be adjusted to allow augmented assignments to appear as expressions, where the result of the augmented assignment expression is the same post-calculation reference as is being bound to the given target. For example:: >>> n = 0 >>> n += 5 5 >>> n -= 2 3 >>> n *= 3 9 >>> n 9 For mutable targets, this means the result is always just the original object:: >>> seq = [] >>> seq_id = id(seq) >>> seq += range(3) [0, 1, 2] >>> seq_id == id(seq) True Augmented assignments to attributes and container subscripts will be permitted, with the result being the post-calculation reference being bound to the target, just as it is for simple name targets:: def increment(self, step=1): return self._value += step In these cases, ``__getitem__`` and ``__getattribute__`` will *not* be called after the assignment has already taken place (they will only be called as needed to evaluate the in-place operation). Adding an inline assignment operator ------------------------------------ Given only the addition of augmented assignment expressions, it would be possible to abuse a symbol like ``|=`` as a general purpose assignment operator by defining a ``Target`` wrapper type that worked as follows:: >>> class Target: ... def __init__(self, value): ... self.value = value ... def __or__(self, other): ... return Target(other) ... >>> x = Target(10) >>> x.value 10 >>> x |= 42 <__main__.Target object at 0x7f608caa8048> >>> x.value 42 This is similar to the way that storing a single reference in a list was long used as a workaround for the lack of a ``nonlocal`` keyword, and can still be used today (in combination with ``operator.itemsetter``) to work around the lack of expression level assignments. Rather than requiring such workarounds, this PEP instead proposes that PEP 572's "NAME := EXPR" syntax be adopted as a new inline assignment expression that uses the augmented assignment scoping rules described below. This cleanly handles cases where only the new value is of interest, and the previously bound value (if any) can just be discarded completely. Note that for both simple names and complex assignment targets, the inline assignment operator does *not* read the previous reference before assigning the new one. However, when used at function scope (either directly or inside a scoped expression), it does *not* implicitly define a new local variable, and will instead raise ``TargetNameError`` (as described for augmented assignments below). Assignment operator precedence ------------------------------ To preserve the existing semantics of augmented assignment statements, inline assignment operators will be defined as being of lower precedence than all other operators, include the comma pseudo-operator. This ensures that when used as a top level expression the entire right hand side of the expression is still interpreted as the value to be processed (even when that value is a tuple without parentheses). The difference this introduces relative to PEP 572 is that where ``(n := first, second)`` sets ``n = first`` in PEP 572, in this PEP it would set ``n = (first, second)`, and getting the first meaning would require an extra set of parentheses (``((n := first), second)``). PEP 572 quite reasonably notes that this results in ambiguity when assignment expressions are used as function call arguments. This PEP resolves that concern a different way by requiring that assignment expressions be parenthesised when used as arguments to a function call (unless they're the sole argument). This is a more relaxed version of the restriction placed on generator expressions (which always require parentheses, except when they're the sole argument to a function call). Augmented assignment to names in block scopes --------------------------------------------- No target name binding changes are proposed for augmented assignments at module or class scope (this also includes code executed using "exec" or "eval"). These will continue to implicitly declare a new local variable as the binding target as they do today, and (if necessary) will be able to resolve the name from an outer scope before binding it locally. At function scope, augmented assignments will be changed to require that there be either a preceding name binding or variable declaration to explicitly establish the target name as being local to the function, or else an explicit ``global`` or ``nonlocal`` declaration. ``TargetNameError``, a new ``SyntaxError`` subclass, will be raised at compile time if no such binding or declaration is present. For example, the following code would compile and run as it does today:: x = 0 x += 1 # Sets global "x" to 1 class C: x += 1 # Sets local "x" to 2, leaves global "x" alone def local_target(): x = 0 x += 1 # Sets local "x" to 1, leaves global "x" alone def global_target(): global x x += 1 # Increments global "x" each time this runs def nonlocal_target(): x = 0 def g(): nonlocal x x += 1 # Increments "x" in outer scope each time this runs return x return g The follow examples would all still compile and then raise an error at runtime as they do today:: n += 1 # Raises NameError at runtime class C: n += 1 # Raises NameError at runtime def missing_global(): global n n += 1 # Raises NameError at runtime def delayed_nonlocal_initialisation(): def f(): nonlocal n n += 1 f() # Raises NameError at runtime n = 0 def skipped_conditional_initialisation(): if False: n = 0 n += 1 # Raises UnboundLocalError at runtime def local_declaration_without_initial_assignment(): n: typing.Any n += 1 # Raises UnboundLocalError at runtime Whereas the following would raise a compile time ``DeprecationWarning`` initially, and eventually change to report a compile time ``TargetNameError``:: def missing_target(): x += 1 # Compile time TargetNameError due to ambiguous target scope # Is there a missing initialisation of "x" here? Or a missing # global or nonlocal declaration? As a conservative implementation approach, the compile time function name resolution change would be introduced as a ``DeprecationWarning`` in Python 3.8, and then converted to ``TargetNameError`` in Python 3.9. This avoids potential problems in cases where an unused function would currently raise ``UnboundLocalError`` if it was ever actually called, but the code is actually unused - converting that latent runtime defect to a compile time error qualifies as a backwards incompatible change that requires a deprecation period. When augmented assignments are used as expressions in function scope (rather than as standalone statements), there aren't any backwards compatibility concerns, so the compile time name binding checks would be enforced immediately in Python 3.8. Similarly, the new inline assignment expressions would always require explicit predeclaration of their target scope when used as part of a function, at least for Python 3.8. (See the design discussion section for notes on potentially revisiting that restriction in the future). Augmented assignment to names in scoped expressions --------------------------------------------------- Scoped expressions is a new collective term being proposed for expressions that introduce a new nested scope of execution, either as an intrinsic part of their operation (lambda expressions, generator expressions), or else as a way of hiding name binding operations from the containing scope (container comprehensions). Unlike regular functions, these scoped expressions can't include explicit ``global`` or ``nonlocal`` declarations to rebind names directly in an outer scope. Instead, their name binding semantics for augmented assignment expressions would be defined as follows: * augmented assignment targets used in scoped expressions are expected to either be already bound in the containing block scope, or else have their scope explicitly declared in the containing block scope. If no suitable name binding or declaration can be found in that scope, then ``TargetNameError`` will be raised at compile time (rather than creating a new binding within the scoped expression). * if the containing block scope is a function scope, and the target name is explicitly declared as ``global`` or ``nonlocal``, then it will be use the same scope declaration in the body of the scoped expression * if the containing block scope is a function scope, and the target name is a local variable in that function, then it will be implicitly declared as ``nonlocal`` in the body of the scoped expression * if the containing block scope is a class scope, than ``TargetNameError`` will always be raised, with a dedicated message indicating that combining class scopes with augmented assignments in scoped expressions is not currently permitted. * if a name is declared as a formal parameter (lambda expressions), or as an iteration variable (generator expressions, comprehensions), then that name is considered local to that scoped expression, and attempting to use it as the target of an augmented assignment operation in that scope, or any nested scoped expression, will raise ``TargetNameError`` (this is a restriction that could potentially be lifted later, but is being proposed for now to simplify the initial set of compile time and runtime semantics that needs to be covered in the language reference and handled by the compiler and interpreter) For example, the following code would work as shown:: >>> global_target = 0 >>> incr_global_target = lambda: global_target += 1 >>> incr_global_target() 1 >>> incr_global_target() 2 >>> global_target 2 >>> def cumulative_sums(data, start=0) ... total = start ... yield from (total += value for value in data) ... return total ... >>> print(list(cumulative_sums(range(5)))) [0, 1, 3, 6, 10] While the following examples would all raise ``TargetNameError``:: class C: cls_target = 0 incr_cls_target = lambda: cls_target += 1 # Error due to class scope def missing_target(): incr_x = lambda: x += 1 # Error due to missing target "x" def late_target(): incr_x = lambda: x += 1 # Error due to "x" being declared after use x = 1 lambda arg: arg += 1 # Error due to attempt to target formal parameter [x += 1 for x in data] # Error due to attempt to target iteration variable As augmented assignments currently can't appear inside scoped expressions, the above compile time name resolution exceptions would be included as part of the initial implementation rather than needing to be phased in as a potentially backwards incompatible change. Design discussion ================= Allowing complex assignment targets ----------------------------------- The initial drafts of this PEP kept PEP 572's restriction to single name targets when augmented assignments were used as expressions, allowing attribute and subscript targets solely for the statement form. However, enforcing that required varying the permitted targets based on whether or not the augmented assignment was a top level expression or not, as well as explaining why ``n += 1``, ``(n += 1)``, and ``self.n += 1`` were all legal, but ``(self.n += 1)`` was prohibited, so the proposal was simplified to allow all existing augmented assignment targets for the expression form as well. Since this PEP defines ``TARGET := EXPR`` as a variant on augmented assignment, that also gained support for assignment and subscript targets. Augmented assignment or name binding only? ------------------------------------------ PEP 572 makes a reasonable case that the potential use cases for inline augmented assignment are notably weaker than those for inline assignment in general, so it's acceptable to require that they be spelled as ``x := x + 1``, bypassing any in-place augmented assignment methods. While this is at least arguably true for the builtin types (where potential counterexamples would probably need to focus on set manipulation use cases that the PEP author doesn't personally have), it would also rule out more memory intensive use cases like manipulation of NumPy arrays, where the data copying involved in out-of-place operations can make them impractical as alternatives to their in-place counterparts. That said, this PEP mainly exists because the PEP author found the inline assignment proposal much easier to grasp as "It's like ``+=``, only skipping the addition step", and also liked the way that that framing provides an actual semantic difference between ``NAME = EXPR`` and ``NAME := EXPR`` at function scope. That difference in target scoping behaviour means that the ``NAME := EXPR`` syntax would be expected to have two primary use cases: * as a way of allowing assignments to be embedded as an expression in an ``if`` or ``while`` statement, or as part of a scoped expression * as a way of requesting a compile time check that the target name be previously declared or bound in the current function scope At module or class scope, ``NAME = EXPR`` and ``NAME := EXPR`` would be semantically equivalent due to the compiler's lack of visibility into the set of names that will be resolvable at runtime, but code linters and static type checkers would be encouraged to enforce the same "declaration or assignment required before use" behaviour for ``NAME := EXPR`` as the compiler would enforce at function scope. Postponing a decision on expression level target declarations ------------------------------------------------------------- At least for Python 3.8, usage of inline assignments (whether augmented or not) at function scope would always require a preceding name binding or scope declaration to avoid getting ``TargetNameError``, even when used outside a scoped expression. The intent behind this requirement is to clearly separate the following two language design questions: 1. Can an expression rebind a name in the current scope? 2. Can an expression declare a new name in the current scope? For module global scopes, the answer to both of those questions is unequivocally "Yes", because it's a language level guarantee that mutating the ``globals()`` dict will immediately impact the runtime module scope, and ``global NAME`` declarations inside a function can have the same effect (as can importing the currently executing module and modifying its attributes). For class scopes, the answer to both questions is also "Yes" in practice, although less unequivocally so, since the semantics of ``locals()`` are currently formally unspecified. However, if the current behaviour of ``locals()`` at class scope is taken as normative (as PEP 558 proposes), then this is essentially the same scenario as manipulating the module globals, just using ``locals()`` instead. For function scopes, however, the current answers to these two questions are respectively "Yes" and "No". Expression level rebinding of function locals is already possible thanks to lexically nested scopes and explicit ``nonlocal NAME`` expressions. While this PEP will likely make expression level rebinding more common than it is today, it isn't a fundamentally new concept for the language. By contrast, declaring a *new* function local variable is currently a statement level action, involving one of: * an assignment statement (``NAME = EXPR``, ``OTHER_TARGET = NAME = EXPR``, etc) * a variable declaration (``NAME : EXPR``) * a nested function definition * a nested class definition * a ``for`` loop * a ``with`` statement * an ``except`` clause (with limited scope of access) The historical trend for the language has actually been to *remove* support for expression level declarations of function local names, first with the introduction of "fast locals" semantics (which made the introduction of names via ``locals()`` unsupported for function scopes), and again with the hiding of comprehension iteration variables in Python 3.0. Now, it may be that in Python 3.9, we decide to revisit this question based on our experience with expression level name binding in Python 3.8, and decide that we really do want expression level function local variable declarations as well, and that we want ``NAME := EXPR`` to be the way we spell that (rather than, for example, spelling inline declarations more explicitly as ``NAME := EXPR given NAME``, which would permit them to carry type annotations, and also permit them to declare new local variables in scoped expressions, rather than having to pollute the namespace in their containing scope). But the proposal in this PEP is that we explicitly give ourselves a full release to decide how much we want that feature, and exactly where we find its absence irritating. Python has survived happily without expression level name bindings *or* declarations for decades, so we can afford to give ourselves a couple of years to decide if we really want *both* of those, or if expression level bindings are sufficient. Ignoring scoped expressions when determining augmented assignment targets ------------------------------------------------------------------------- When discussing possible binding semantics for PEP 572's assignment expressions, Tim Peters made a plausible case [1_,2_,3_] for assignment expressions targeting the containing block scope, essentially ignoring any intervening scoped expressions. This approach allows use cases like cumulative sums, or extracting the final value from a generator expression to be written in a relatively straightforward way:: total = 0 partial_sums = [total := total + value for value in data] factor = 1 while any(n % (factor := p) == 0 for p in small_primes): n //= factor Guido also expressed his approval for this general approach [4_]. The proposal in this PEP differs from Tim's original proposal in three main areas: - it applies the proposal to all augmented assignment operators, not just a single new name binding operator - as far as is practical, it extends the augmented assignment requirement that the name already be defined to the new name binding operator (raising ``TargetNameError`` rather than implicitly declaring new local variables at function scope) - it includes lambda expressions in the set of scopes that get ignored for target name binding purposes, making this transparency to assignments common to all of the scoped expressions rather than being specific to comprehensions and generator expressions With scoped expressions being ignored when calculating binding targets, it's once again difficult to detect the scoping difference between the outermost iterable expressions in generator expressions and comprehensions (you have to mess about with either class scopes or attempting to rebind iteration Variables to detect it), so there's also no need to tinker with that. Treating inline assignment as an augmented assignment variant ------------------------------------------------------------- One of the challenges with PEP 572 is the fact that ``NAME = EXPR`` and ``NAME := EXPR`` are entirely semantically equivalent at every scope. This makes the two forms hard to teach, since there's no inherent nudge towards choosing one over the other at the statement level, so you end up having to resort to "``NAME = EXPR`` is preferred because it's been around longer" (and PEP 572 proposes to enfore that historical idiosyncrasy at the compiler level). That semantic equivalence is difficult to avoid at module and class scope while still having ``if NAME := EXPR:`` and ``while NAME := EXPR:`` work sensibly, but at function scope the compiler's comprehensive view of all local names makes it possible to require that the name be assigned or declared before use, providing a reasonable incentive to continue to default to using the ``NAME = EXPR`` form when possible, while also enabling the use of the ``NAME := EXPR`` as a kind of simple compile time assertion (i.e. explicitly indicating that the targeted name has already been bound or declared and hence should already be known to the compiler). If Guido were to declare that support for inline declarations was a hard design requirement, then this PEP would be updated to propose that ``EXPR given NAME`` also be introduced as a way to support inline name declarations after arbitrary expressions (this would allow the inline name declarations to be deferred until the end of a complex expression rather than needing to be embedded in the middle of it, and PEP 8 would gain a recommendation encouraging that style). Disallowing augmented assignments in class level scoped expressions ------------------------------------------------------------------- While modern classes do define an implicit closure that's visible to method implementations (in order to make ``__class__`` available for use in zero-arg ``super()`` calls), there's no way for user level code to explicitly add additional names to that scope. Meanwhile, attributes defined in a class body are ignored for the purpose of defining a method's lexical closure, which means adding them there wouldn't work at an implementation level. Rather than trying to resolve that inherent ambiguity, this PEP simply prohibits such usage, and requires that any affected logic be written somewhere other than directly inline in the class body (e.g. in a separate helper function). Comparison operators vs assignment operators -------------------------------------------- The ``OP=`` construct as an expression currently indicates a comparison operation:: x == y # Equals x >= y # Greater-than-or-equal-to x <= y # Less-than-or-equal-to Both this PEP and PEP 572 propose adding at least one operator that's somewhat similar in appearance, but defines an assignment instead:: x := y # Becomes This PEP then goes much further and allows all *13* augmented assignment symbols to be uses as binary operators:: x += y # In-place add x -= y # In-place minus x *= y # In-place multiply x @= y # In-place matrix multiply x /= y # In-place division x //= y # In-place int division x %= y # In-place mod x &= y # In-place bitwise and x |= y # In-place bitwise or x ^= y # In-place bitwise xor x <<= y # In-place left shift x >>= y # In-place right shift x **= y # In-place power Of those additional binary operators, the most questionable would be the bitshift assignment operators, since they're each only one doubled character away from one of the inclusive ordered comparison operators. Examples ======== Simplifying retry loops ----------------------- There are currently a few different options for writing retry loops, including:: # Post-decrementing a counter remaining_attempts = MAX_ATTEMPTS while remaining_attempts: remaining_attempts -= 1 try: result = attempt_operation() except Exception as exc: continue # Failed, so try again log.debug(f"Succeeded after {attempts} attempts") break # Success! else: raise OperationFailed(f"Failed after {MAX_ATTEMPTS} attempts") from exc # Loop-and-a-half with a pre-incremented counter attempt = 0 while True: attempts += 1 if attempts > MAX_ATTEMPTS: raise OperationFailed(f"Failed after {MAX_ATTEMPTS} attempts") from exc try: result = attempt_operation() except Exception as exc: continue # Failed, so try again log.debug(f"Succeeded after {attempts} attempts") break # Success! Each of the available options hides some aspect of the intended loop structure inside the loop body, whether that's the state modification, the exit condition, or both. The proposal in this PEP allows both the state modification and the exit condition to be included directly in the loop header:: attempt = 0 while (attempt += 1) <= MAX_ATTEMPTS: try: result = attempt_operation() except Exception as exc: continue # Failed, so try again log.debug(f"Succeeded after {attempts} attempts") break # Success! else: raise OperationFailed(f"Failed after {MAX_ATTEMPTS} attempts") from exc Simplifying if-elif chains -------------------------- if-elif chains that need to rebind the checked condition currently need to be written using nested if-else statements:: m = pattern.match(data) if m: ... else: m = other_pattern.match(data) if m: ... else: m = yet_another_pattern.match(data) if m: ... else: ... As with PEP 572, this PEP allows the else/if portions of that chain to be condensed, making their consistent and mutually exclusive structure more readily apparent:: m = pattern.match(data) if m: ... elif m := other_pattern.match(data): ... elif m := yet_another_pattern.match(data): ... else: ... Unlike PEP 572, this PEP requires that the assignment target be explicitly indicated as local before the first use as a ``:=`` target, either by binding it to a value (as shown above), or else by including an appropriate explicit type declaration:: m: typing.re.Match if m := pattern.match(data): ... elif m := other_pattern.match(data): ... elif m := yet_another_pattern.match(data): ... else: ... Capturing intermediate values from comprehensions ------------------------------------------------- The proposal in this PEP makes it straightforward to capture and reuse intermediate values in comprehensions and generator expressions by exporting them to the containing block scope:: factor: int while any(n % (factor := p) == 0 for p in small_primes): n //= factor total = 0 partial_sums = [total += value for value in data] Allowing lambda expressions to act more like re-usable code thunks ------------------------------------------------------------------ This PEP allows the classic closure usage example:: def make_counter(start=0): x = start def counter(step=1): nonlocal x x += step return x return counter To be abbreviated as:: def make_counter(start=0): x = start return lambda step=1: x += step While the latter form is still a conceptually dense piece of code, it can be reasonably argued that the lack of boilerplate (where the "def", "nonlocal", and "return" keywords and two additional repetitions of the "x" variable name have been replaced with the "lambda" keyword) may make it easier to read in practice. Relationship with PEP 572 ========================= The case for allowing inline assignments at all is made in PEP 572. This competing PEP was initially going to propose an alternate surface syntax (``EXPR given NAME = EXPR``), while retaining the expression semantics from PEP 572, but that changed when discussing one of the initial motivating use cases for allowing embedded assignments at all: making it possible to easily calculate cumulative sums in comprehensions and generator expressions. As a result of that, and unlike PEP 572, this PEP focuses primarily on use cases for inline augmented assignment. It also has the effect of converting cases that currently inevitably raise ``UnboundLocalError`` at function call time to report a new compile time ``TargetNameError``. New syntax for a name rebinding expression (``NAME := TARGET``) is then added not only to handle the same use cases as are identified in PEP 572, but also as a lower level primitive to help illustrate, implement and explain the new augmented assignment semantics, rather than being the sole change being proposed. The author of this PEP believes that this approach makes the value of the new flexibility in name rebinding clearer, while also mitigating many of the potential concerns raised with PEP 572 around explaining when to use ``NAME = EXPR`` over ``NAME := EXPR`` (and vice-versa), without resorting to prohibiting the bare statement form of ``NAME := EXPR`` outright (such that ``NAME := EXPR`` is a compile error, but ``(NAME := EXPR)`` is permitted). Acknowledgements ================ The PEP author wishes to thank Chris Angelico for his work on PEP 572, and his efforts to create a coherent summary of the great many sprawling discussions that spawned on both python-ideas and python-dev, as well as Tim Peters for the in-depth discussion of parent local scoping that prompted the above scoping proposal for augmented assignments inside scoped expressions. Eric Snow's feedback on a pre-release version of this PEP helped make it significantly more readable. References ========== .. [1] The beginning of Tim's genexp & comprehension scoping thread (https://mail.python.org/pipermail/python-ideas/2018-May/050367.html) .. [2] Reintroducing the original cumulative sums use case (https://mail.python.org/pipermail/python-ideas/2018-May/050544.html) .. [3] Tim's language reference level explanation of his proposed scoping semantics (https://mail.python.org/pipermail/python-ideas/2018-May/050729.html) .. [4] Guido's endorsement of Tim's proposed genexp & comprehension scoping (https://mail.python.org/pipermail/python-ideas/2018-May/050411.html) Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: