PEP: 403 Title: Prefix syntax for post function definition operations Version: $Revision$ Last-Modified: $Date$ Author: Nick Coghlan Status: Withdrawn Type: Standards Track Content-Type: text/x-rst Created: 2011-10-13 Python-Version: 3.x Post-History: 2011-10-13 Resolution: TBD Abstract ======== This PEP proposes the addition of ``postdef`` as a new function prefix syntax (analogous to decorators) that permits the execution of a single simple statement (potentially including substatements separated by semi-colons) after In addition, the new syntax would allow the 'def' keyword to be used to refer to the function being defined without needing to repeat the name. When the 'postdef' prefix syntax is used, the associated statement would be executed *in addition to* the normal local name binding implicit in function definitions. Any name collision are expected to be minor, analagous to those encountered with ``for`` loop iteration variables. This PEP is based heavily on many of the ideas in PEP 3150 (Statement Local Namespaces) so some elements of the rationale will be familiar to readers of that PEP. That PEP has now been withdrawn in favour of this one. PEP Withdrawal ============== The python-ideas thread discussing this PEP [1]_ persuaded me that it was essentially am unnecessarily cryptic, wholly inferior version of PEP 3150's statement local namespaces. The discussion also resolved some of my concerns with PEP 3150, so I am withdrawing this more limited version of the idea in favour of resurrecting the original concept. Basic Examples ============== Before diving into the long history of this problem and the detailed rationale for this specific proposed solution, here are a few simple examples of the kind of code it is designed to simplify. As a trivial example, weakref callbacks could be defined as follows:: postdef x = weakref.ref(target, def) def report_destruction(obj): print("{} is being destroyed".format(obj)) This contrasts with the current repetitive "out of order" syntax for this operation:: def report_destruction(obj): print("{} is being destroyed".format(obj)) x = weakref.ref(target, report_destruction) That structure is OK when you're using the callable multiple times, but it's irritating to be forced into it for one-off operations. Similarly, a sorted operation on a particularly poorly defined type could now be defined as:: postdef sorted_list = sorted(original, key=def) def force_sort(item): try: return item.calc_sort_order() except NotSortableError: return float('inf') Rather than:: def force_sort(item): try: return item.calc_sort_order() except NotSortableError: return float('inf') sorted_list = sorted(original, key=force_sort) And early binding semantics in a list comprehension could be attained via:: postdef funcs = [def(i) for i in range(10)] def make_incrementor(i): postdef return def def incrementor(x): return x + i Proposal ======== This PEP proposes the addition of an optional block prefix clause to the syntax for function and class definitions. This block prefix would be introduced by a leading ``postdef`` and would be allowed to contain any simple statement (including those that don't make any sense in that context - while such code would be legal, there wouldn't be any point in writing it). This permissive structure is easier to define and easier to explain, but a more restrictive approach that only permits operations that "make sense" would also be possible (see PEP 3150 for a list of possible candidates) The function definition keyword ``def`` would be repurposed inside the block prefix to refer to the function being defined. When a block prefix is provided, the standard local name binding implicit in the function definition still takes place. Background ========== The question of "multi-line lambdas" has been a vexing one for many Python users for a very long time, and it took an exploration of Ruby's block functionality for me to finally understand why this bugs people so much: Python's demand that the function be named and introduced before the operation that needs it breaks the developer's flow of thought. They get to a point where they go "I need a one-shot operation that does ", and instead of being able to just *say* that, they instead have to back up, name a function to do , then call that function from the operation they actually wanted to do in the first place. Lambda expressions can help sometimes, but they're no substitute for being able to use a full suite. Ruby's block syntax also heavily inspired the style of the solution in this PEP, by making it clear that even when limited to *one* anonymous function per statement, anonymous functions could still be incredibly useful. Consider how many constructs Python has where one expression is responsible for the bulk of the heavy lifting: * comprehensions, generator expressions, map(), filter() * key arguments to sorted(), min(), max() * partial function application * provision of callbacks (e.g. for weak references) * array broadcast operations in NumPy However, adopting Ruby's block syntax directly won't work for Python, since the effectiveness of Ruby's blocks relies heavily on various conventions in the way functions are *defined* (specifically, Ruby's ``yield`` syntax to call blocks directly and the ``&arg`` mechanism to accept a block as a function's final argument). Since Python has relied on named functions for so long, the signatures of APIs that accept callbacks are far more diverse, thus requiring a solution that allows anonymous functions to be slotted in at the appropriate location. Relation to PEP 3150 ==================== PEP 3150 (Statement Local Namespaces) described its primary motivation as being to elevate ordinary assignment statements to be on par with ``class`` and ``def`` statements where the name of the item to be defined is presented to the reader in advance of the details of how the value of that item is calculated. This PEP achieves the same goal in a different way, by allowing the simple name binding of a standard function definition to be replaced with something else (like assigning the result of the function to a value). This PEP also achieves most of the other effects described in PEP 3150 without introducing a new brainbending kind of scope. All of the complex scoping rules in PEP 3150 are replaced in this PEP with the simple ``def`` reference to the associated function definition. Keyword Choice ============== The proposal definitely requires *some* kind of prefix to avoid parsing ambiguity and backwards compatibility problems with existing constructs. It also needs to be clearly highlighted to readers, since it declares that the following piece of code is going to be executed out of order. The 'postdef' keyword was chosen as a literal explanation of exactly what the new clause does: execute the specified statement *after* the associated function definition, even though it is physically written *before* the definition in the source code. Requirement to Name Functions ============================= One of the objections to widespread use of lambda expressions is that they have an atrocious effect on traceback intelligibility and other aspects of introspection. Accordingly, this PEP requires that even throwaway functions be given some kind of name. To help encourage the use of meaningful names without users having to repeat themselves, the PEP suggests the provision of the ``def`` shorthand reference to the current function from the ``postdef`` clause. Syntax Change ============= Current:: atom: ('(' [yield_expr|testlist_comp] ')' | '[' [testlist_comp] ']' | '{' [dictorsetmaker] '}' | NAME | NUMBER | STRING+ | '...' | 'None' | 'True' | 'False') Changed:: atom: ('(' [yield_expr|testlist_comp] ')' | '[' [testlist_comp] ']' | '{' [dictorsetmaker] '}' | NAME | NUMBER | STRING+ | '...' | 'None' | 'True' | 'False' | 'def') New:: blockprefix: 'postdef' simple_stmt block: blockprefix funcdef The above is the general idea, but I suspect that the change to the 'atom' definition may cause an ambiguity problem in the parser when it comes to detecting function definitions. So the actual implementation may need to be more complex than that. Grammar: http://hg.python.org/cpython/file/default/Grammar/Grammar Possible Implementation Strategy ================================ This proposal has one titanic advantage over PEP 3150: implementation should be relatively straightforward. The post definition statement can be incorporated into the AST for the function node and simply visited out of sequence. The one potentially tricky part is working out how to allow the dual use of 'def' without rewriting half the grammar definition. More Examples ============= Calculating attributes without polluting the local namespace (from os.py):: # Current Python (manual namespace cleanup) def _createenviron(): ... # 27 line function environ = _createenviron() del _createenviron # Becomes: postdef environ = def() def _createenviron(): ... # 27 line function Loop early binding:: # Current Python (default argument hack) funcs = [(lambda x, i=i: x + i) for i in range(10)] # Becomes: postdef funcs = [def(i) for i in range(10)] def make_incrementor(i): return lambda x: x + i # Or even: postdef funcs = [def(i) for i in range(10)] def make_incrementor(i): postdef return def def incrementor(x): return x + i Reference Implementation ======================== None as yet. Acknowledgements ================ Huge thanks to Gary Bernhardt for being blunt in pointing out that I had no idea what I was talking about in criticising Ruby's blocks, kicking off a rather enlightening process of investigation. Even though this PEP has been withdrawn, the process of writing and arguing in its favour has been quite influential on the future direction of PEP 3150. References ========== .. [1] Start of python-ideas thread: http://mail.python.org/pipermail/python-ideas/2011-October/012276.html Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: