PEP: 342 Title: Enhanced Iterators Version: $Revision$ Last-Modified: $Date$ Author: Guido van Rossum Status: Draft Type: Standards Track Content-Type: text/plain Created: 10-May-2005 Post-History: Introduction This PEP proposes a new iterator API that allows values to be passed into an iterator using "continue EXPR". These values are received in the iterator as an argument to the new __next__ method, and can be accessed in a generator with a yield-expression. The content of this PEP is derived from the original content of PEP 340, broken off into its own PEP as the new iterator API is pretty much orthogonal from the anonymous block statement discussion. Thanks to Steven Bethard for doing the editing. Motivation and Summary TBD. Use Cases See the Examples section near the end. Specification: the __next__() Method A new method for iterators is proposed, called __next__(). It takes one optional argument, which defaults to None. Calling the __next__() method without argument or with None is equivalent to using the old iterator API, next(). For backwards compatibility, it is recommended that iterators also implement a next() method as an alias for calling the __next__() method without an argument. The argument to the __next__() method may be used by the iterator as a hint on what to do next. Specification: the next() Built-in Function This is a built-in function defined as follows: def next(itr, arg=None): nxt = getattr(itr, "__next__", None) if nxt is not None: return nxt(arg) if arg is None: return itr.next() raise TypeError("next() with arg for old-style iterator") This function is proposed because there is often a need to call the next() method outside a for-loop; the new API, and the backwards compatibility code, is too ugly to have to repeat in user code. Specification: a Change to the 'for' Loop A small change in the translation of the for-loop is proposed. The statement for VAR1 in EXPR1: BLOCK1 else: BLOCK2 will be translated as follows: itr = iter(EXPR1) arg = None # Set by "continue EXPR2", see below brk = False while True: try: VAR1 = next(itr, arg) except StopIteration: brk = True break arg = None BLOCK1 if brk: BLOCK2 (However, the variables 'itr' etc. are not user-visible and the built-in names used cannot be overridden by the user.) Specification: the Extended 'continue' Statement In the translation of the for-loop, inside BLOCK1, the new syntax continue EXPR2 is legal and is translated into arg = EXPR2 continue (Where 'arg' references the corresponding hidden variable from the previous section.) This is also the case in the body of the block-statement proposed below. EXPR2 may contain commas; "continue 1, 2, 3" is equivalent to "continue (1, 2, 3)". Specification: Generators and Yield-Expressions Generators will implement the new __next__() method API, as well as the old argument-less next() method which becomes an alias for calling __next__() without an argument. The yield-statement will be allowed to be used on the right-hand side of an assignment; in that case it is referred to as yield-expression. The value of this yield-expression is None unless __next__() was called with an argument; see below. A yield-expression must always be parenthesized except when it occurs at the top-level expression on the right-hand side of an assignment. So x = yield 42 x = yield x = 12 + (yield 42) x = 12 + (yield) foo(yield 42) foo(yield) are all legal, but x = 12 + yield 42 x = 12 + yield foo(yield 42, 12) foo(yield, 12) are all illegal. (Some of the edge cases are motivated by the current legality of "yield 12, 42".) Note that a yield-statement or yield-expression without an expression is now legal. This makes sense: when the information flow in the next() call is reversed, it should be possible to yield without passing an explicit value ("yield" is of course equivalent to "yield None"). When __next__() is called with an argument that is not None, the yield-expression that it resumes will return the argument. If it resumes a yield-statement, the value is ignored (this is similar to ignoring the value returned by a function call). When the *initial* call to __next__() receives an argument that is not None, TypeError is raised; this is likely caused by some logic error. When __next__() is called without an argument or with None as argument, and a yield-expression is resumed, the yield-expression returns None. Note: the syntactic extensions to yield make its use very similar to that in Ruby. This is intentional. Do note that in Python the block passes a value to the generator using "continue EXPR" rather than "return EXPR", and the underlying mechanism whereby control is passed between the generator and the block is completely different. Blocks in Python are not compiled into thunks; rather, yield suspends execution of the generator's frame. Some edge cases work differently; in Python, you cannot save the block for later use, and you cannot test whether there is a block or not. Alternative An alternative proposal is still under consideration, where instead of adding a __next__() method, the existing next() method is given an optional argument. The next() built-in function is then unnecessary. The only line that changes in the translation is the line VAR1 = next(itr, arg) which will be replaced by this if arg is None: VAR1 = itr.next() else: VAR1 = itr.next(arg) If "continue EXPR2" is used and EXPR2 does not evaluate to None, and the iterator's next() method does not support the optional argument, a TypeError exception will be raised, which is the same behavior as above. This proposal is more compatible (no new method name, no new built-in needed) but less future-proof; in some sense it was a mistake to call this method next() instead of __next__(), since *all* other operations corresponding to function pointers in the C type structure have names with leading and trailing underscores. Acknowledgements See Acknowledgements of PEP 340. References TBD. Copyright This document has been placed in the public domain.