PEP: 279
Title: Enhanced Generators
Version: $Revision$
Last-Modified: $Date$
Author: python@rcn.com (Raymond D. Hettinger)
Status: Draft
Type: Standards Track
Created: 30-Jan-2002
Python-Version: 2.3
Post-History:


Abstract

    This PEP introduces four orthogonal (not mutually exclusive) ideas
    for enhancing the generators introduced in Python version 2.2 [1].
    The goal is to increase the convenience, utility, and power
    of generators.


Rationale

    Python 2.2 introduced the concept of an iterable interface as proposed
    in PEP 234 [4].  The iter() factory function was provided as common
    calling convention and deep changes were made to use iterators as a
    unifying theme throughout Python.  The unification came in the form of
    establishing a common iterable interface for mappings, sequences,
    and file objects.

    Generators, as proposed in PEP 255 [1], were introduced as a means for
    making it easier to create iterators, especially ones with a complex
    internal execution or variable states.  When I created new programs,
    generators were often the tool of choice for creating an iterator.

    However, when updating existing programs, I found that the tool had
    another use, one that improved program function as well as structure.
    Those programs exhibited a pattern of creating large lists and then
    looping over them.  As data sizes increased, the programs encountered
    scalability limitations owing to excessive memory consumption (and
    malloc time) for the intermediate lists.  Generators were found to be
    directly substitutable for the lists while eliminating the memory
    issues through lazy evaluation a.k.a. just in time manufacturing.

    Python itself encountered similar issues.  As a result, xrange() and
    xreadlines() were introduced.  And, in the case of file objects and
    mappings, lazy evaluation became the norm.  Generators provide a tool
    to program memory conserving for-loops whenever complete evaluation is
    not desired because of memory restrictions or availability of data.

    The next steps in the evolution of generators are:

    1. Add a new builtin function, indexed() which was made possible
       once iterators and generators became available.  It provides
       all iterables with the same advantage that iteritem() affords
       to dictionaries -- a compact, readable, reliable index notation.
       
    2. Establish a generator alternative to list comprehensions [3]
       to provide a simple way to convert a list comprehensions into
       generators whenever memory issues arise.

    3. Add a generator method to enable exceptions to be passed to a
       generator.  Currently, there is no clean method for triggering
       exceptions from outside the generator.  Also, generator exception
       passing helps mitigate the try/finally prohibition for generators.

    4. [Proposal 4 is now deferred until Python 2.4]
       Extend the syntax of the 'yield' keyword to enable generator
       parameter passing.  The resulting increase in power simplifies
       the creation of consumer streams which have a complex execution
       state and/or variable state. 

    All of the suggestions are designed to take advantage of the
    existing implementation and require little additional effort to
    incorporate.  Each is backward compatible and requires no new
    keywords.  The first three generator tools go into Python 2.3 when
    generators become final and are not imported from __future__.
    The fourth proposal should be considered deferred and will be
    proposed for Python 2.4 after the Python community has more
    experience with generators.



Reference Implementation

    There is not currently a CPython implementation; however, a simulation
    module written in pure Python is available on SourceForge [7].  The
    simulation covers every feature proposed in this PEP and is meant
    to allow direct experimentation with the proposals.

    There is also a module [8] with working source code for all of the
    examples used in this PEP.  It serves as a test suite for the simulator
    and it documents how each of the new features works in practice.

    The authors and implementers of PEP 255 [1] were contacted to provide
    their assessment of whether these enhancements were going to be
    straight-forward to implement and require only minor modification
    of the existing generator code.  Neil felt the assertion was correct.
    Ka-Ping thought so also.  GvR said he could believe that it was true.
    Tim did not have an opportunity to give an assessment.

    

Specification for a new builtin:

    def indexed(collection, cnt=0, limit=None):
        'Generates an indexed series:  (0,seqn[0]), (1,seqn[1]) ...'
        gen = iter(collection)
        while limit is None or cnt<limit:
            yield (cnt, gen.next())
            cnt += 1


    Note A: PEP 212 Loop Counter Iteration [2] discussed several
    proposals for achieving indexing.  Some of the proposals only work
    for lists unlike the above function which works for any generator,
    xrange, sequence, or iterable object.  Also, those proposals were
    presented and evaluated in the world prior to Python 2.2 which did
    not include generators.  As a result, the non-generator version in
    PEP 212 had the disadvantage of consuming memory with a giant list
    of tuples.  The generator version presented here is fast and light,
    works with all iterables, and allows users to abandon the sequence
    in mid-stream.

    There are other PEPs which touch on related issues:  integer iterators,
    integer for-loops, and one for modifying the arguments to range and
    xrange.  The indexed() proposal does not preclude the other proposals
    and it still meets an important need even if those are adopted -- the need
    to count items in any iterable.  The other proposals give a means of
    producing an index but not the corresponding value.  This is especially
    problematic if a sequence is given which doesn't support random access
    such as a file object, generator, or sequence defined with __getitem__.


    Note B:  Almost all of the PEP reviewers welcomed the function but were
    divided as to whether there should be any builtins.  The main argument
    for a separate module was to slow the rate of language inflation.  The
    main argument for a builtin was that the function is destined to be
    part of a core programming style, applicable to any object with an
    iterable interface.  Just as zip() solves the problem of looping
    over multiple sequences, the indexed() function solves the loop
    counter problem.
    
    If only one builtin is allowed, then indexed() is the most important
    general purpose tool, solving the broadest class of problems while
    improving program brevity, clarity and reliability.
    

    Commentary from GvR:  filter and map should die and be subsumed into list
        comprehensions, not grow more variants. I'd rather introduce builtins
        that do iterator algebra (e.g. the iterzip that I've often used as
        an example).

    Commentary from Ka-Ping Yee:  I'm also quite happy with everything  you
        proposed ... and the extra builtins (really 'indexed' in particular)
        are things I have wanted for a long time.
        
    Commentary from Neil Schemenauer:  The new builtins sound okay.  Guido
        may be concerned with increasing the number of builtins too much.  You
        might be better off selling them as part of a module.  If you use a
        module then you can add lots of useful functions (Haskell has lots of
        them that we could steal).
    
    Author response:  Prior to these comments, four builtins were proposed.
        After the comments, xmap xfilter and xzip were withdrawn.  The one
        that remains is vital for the language and is proposed by itself.
        
        I still secretly covet xzip() a.k.a. iterzip() but think that it will
        happen on its own someday.



Specification for Generator Comprehensions:

    If a list comprehension starts with a 'yield' keyword, then
    express the comprehension with a generator.  For example:

        g = [yield (len(line),line)  for line in file  if len(line)>5]
        print g.next()

    This would be implemented as if it had been written:

        def __temp(self):
            for line in file:
                if len(line) > 5:
                    yield (len(line), line)
        g = __temp()
        print g.next()

    Note A: There is some discussion about whether the enclosing brackets
    should be part of the syntax for generator comprehensions.  On the
    plus side, it neatly parallels list comprehensions and would be
    immediately recognizable as a similar form with similar internal
    syntax (taking maximum advantage of what people already know).
    More importantly, it sets off the generator comprehension from the
    rest of the function so as to not suggest that the enclosing
    function is a generator (currently the only cue that a function is
    really a generator is the presence of the yield keyword).  On the
    minus side, the brackets may falsely suggest that the whole
    expression returns a list.  Most of the feedback received to date
    indicates that brackets are helpful and not misleading. Unfortunately,
    the one dissent is from GvR.

    A key advantage of the generator comprehension syntax is that it
    makes it trivially easy to transform existing list comprehension
    code to a generator by adding yield.  Likewise, it can be converted
    back to a list by deleting yield.  This makes it easy to scale-up
    programs from small datasets to ones large enough to warrant 
    just in time evaluation.


    Note B: List comprehensions expose their looping variable and
    leave that variable in the enclosing scope.  The code, [str(i) for
    i in range(8)] leaves 'i' set to 7 in the scope where the
    comprehension appears.  This behavior is by design and reflects an
    intent to duplicate the result of coding a for-loop instead of a
    list comprehension.  Further, the variable 'i' is in a defined and
    potentially useful state on the line immediately following the
    list comprehension.

    In contrast, generator comprehensions do not expose the looping
    variable to the enclosing scope.  The code, [yield str(i) for i in
    range(8)] leaves 'i' untouched in the scope where the
    comprehension appears.  This is also by design and reflects an
    intent to duplicate the result of coding a generator directly
    instead of a generator comprehension.  Further, the variable 'i'
    is not in a defined state on the line immediately following the
    list comprehension.  It does not come into existence until
    iteration starts (possibly never).


    Commentary from GvR:  Cute hack, but I think the use of the [] syntax
        strongly suggests that it would return a list, not an iterator. I
        also think that this is trying to turn Python into a functional
        language, where most algorithms use lazy infinite sequences, and I
        just don't think that's where its future lies. 

    Commentary from Ka-Ping Yee:  I am very happy with the things you have
        proposed in this PEP.  I feel quite positive about generator
        comprehensions and have no reservations.  So a +1 on that.

    Commentary from Neil Schemenauer:  I'm -0 on the generator list
        comprehensions.  They don't seem to add much.  You could easily use
        a nested generator to do the same thing.  They smell like lambda.

    Author response:  This may be before its time in that some people still
        don't like list comprehensions and half of this PEP's reviewers did
        not have any use for generators in any form.  What I like best about
        generator comprehensions is that I can design using list
        comprehensions and then easily switch to a generator (by adding
        yield) in response to scalability requirements (when the list
        comprehension produces too large of an intermediate result).



Specification for Generator Exception Passing:

    Add a .throw(exception) method to the generator interface:

        def logger():
        start = time.time()
        log = []
            try:
                while 1:0
            log.append( time.time() - start )
                    yield log[-1]
            except WriteLog:
                return log

        g = logger()
        for i in [10,20,40,80,160]:
        testsuite(i)
        g.next()
        g.throw(WriteLog)

    There is no existing work-around for triggering an exception
    inside a generator.  This is a true deficiency.  It is the only
    case in Python where active code cannot be excepted to or through.    

    Generator exception passing also helps address an intrinsic limitation
    on generators, the prohibition against their using try/finally to
    trigger clean-up code [1].  Without .throw(), the current work-around
    forces the resolution or clean-up code to be moved outside the generator.
                

    Note A: The name of the throw method was selected for several
    reasons.  Raise is a keyword and so cannot be used as a method
    name.  Unlike raise which immediately raises an exception from the
    current execution point, throw will first return to the generator
    and then raise the exception.  The word throw is suggestive of
    putting the exception in another location.  The word throw is
    already associated with exceptions in other languages.

    Alternative method names were considered: resolve(), signal(),
    genraise(), raiseinto(), and flush().  None of these seem to fit
    as well as throw().


    Note B: The throw syntax should exactly match raise's syntax:
                
        throw([expression, [expression, [expression]]])
                
    Accordingly, it should be implemented to handle all of the following:
                
        raise string                    g.throw(string)
        raise string, data              g.throw(string,data)
        raise class, instance           g.throw(class,instance)
        raise instance                  g.throw(instance)
        raise                           g.throw()


    Commentary from GvR:  I'm not convinced that the cleanup problem that
        this is trying to solve exists in practice. I've never felt the need
        to put yield inside a try/except. I think the PEP doesn't make enough 
        of a case that this is useful.  

    Commentary from Ka-Ping Yee:  I agree that the exception issue needs to
        be resolved and [that] you have suggested a fine solution.

    Commentary from Neil Schemenauer:  The exception passing idea is one I
        hadn't thought of before and looks interesting.  If we enable the
        passing of values back, then we should add this feature too.

    Author response:  If the sole use of generators is to simplify writing
        iterators for lazy producers, then the odds of needing generator 
        exception passing are very slim.  If, on the other hand, generators
        are used to write lazy consumers, create coroutines, generate output
        streams, or simply for their marvelous capability for restarting a
        previously frozen state, THEN the need to raise exceptions will
        come up almost every time.

        I'm no judge of what is truly Pythonic, but am still astonished
        that there can exist blocks of code that can't be excepted to or
        through, that the try/finally combination is blocked, and that the
        only work-around is to rewrite as a class and move the exception
        code out of the function or method being excepted.



Specification for Generator Parameter Passing [Deferred Proposal]

    1. Allow 'yield' to assign a value as in:

        def mygen():
            while 1:
                x = yield None
                print x

    2. Let the .next() method take a value to pass to the generator as in:

        g = mygen()
        g.next()       # runs the generator until the first 'yield'
        g.next(1)      # '1' is bound to 'x' in mygen(), then printed
        g.next(2)      # '2' is bound to 'x' in mygen(), then printed

    The control flow of 'yield' and 'next' is unchanged by this proposal.
    The only change is that a value can be sent into the generator.
    By analogy, consider the quality improvement from GOSUB (which had
    no argument passing mechanism) to modern procedure calls (which can
    pass in arguments and return values).

    Most of the underlying machinery is already in place, only the
    communication needs to be added by modifying the parse syntax to
    accept the new 'x = yield expr' syntax and by allowing the .next()
    method to accept an optional argument.

    Yield is more than just a simple iterator creator.  It does
    something else truly wonderful -- it suspends execution and saves
    state.  It is good for a lot more than writing iterators.  This
    proposal further expands its capability by making it easier to
    share data with the generator.

    The .next(arg) mechanism is especially useful for:
        1. Sending data to any generator
        2. Writing lazy consumers with complex execution states
        3. Writing co-routines (as demonstrated in Dr. Mertz's article [5])

    The proposal is a clear improvement over the existing alternative
    of passing data via global variables.  It is also much simpler,
    more readable and easier to debug than an approach involving the
    threading module with its attendant mutexes, semaphores, and data
    queues.  A class-based approach competes well when there are no
    complex execution states or variable states.  However, when the
    complexity increases, generators with parameter passing are much simpler
    because they automatically save state (unlike classes which must
    explicitly save the variable and execution state in instance variables).

    Note A:  This proposal changes 'yield' from a statement to an
    expression with binding and precedence similar to lambda.
                

    Example of a Complex Consumer

    The encoder for arithmetic compression sends a series of
    fractional values to a complex, lazy consumer.  That consumer
    makes computations based on previous inputs and only writes out
    when certain conditions have been met.  After the last fraction is
    received, it has a procedure for flushing any unwritten data.


    Example of a Consumer Stream

        def filelike(packagename, appendOrOverwrite):
            cum = []
            if appendOrOverwrite == 'w+':
                cum.extend(packages[packagename])
            try:
                while 1:
                    dat = yield None
                    cum.append(dat)
            except FlushStream:
                packages[packagename] = cum

        ostream = filelike('mydest','w')   # Analogous to file.open(name,flag)
        ostream.next()                     # Advance to the first yield
        ostream.next(firstdat)             # Analogous to file.write(dat)
        ostream.next(seconddat)
        ostream.throw(FlushStream)         # This feature proposed above


    Example of a Complex Consumer

    Loop over the picture files in a directory, shrink them
    one at a time to thumbnail size using PIL [6], and send them to a
    lazy consumer.  That consumer is responsible for creating a large
    blank image, accepting thumbnails one at a time and placing them
    in a 5 by 3 grid format onto the blank image.  Whenever the grid is
    full, it writes-out the large image as an index print.  A
    FlushStream exception indicates that no more thumbnails are
    available and that the partial index print should be written out
    if there are one or more thumbnails on it.


    Example of a Producer and Consumer Used Together in a Pipe-like Fashion

        'Analogy to Linux style pipes:  source | upper | sink'
        sink = sinkgen()
        sink.next()
        for word in source():
            sink.next(word.upper())


    Commentary from GvR:  We discussed this at length when we were hashing
        out generators and coroutines, and found that there's always a problem
        with this: the argument to the first next() call has to be thrown away,
        because it doesn't correspond to a yield statement. This looks ugly
        (note that the example code has a dummy call to next() to get the  
        generator going). But there may be useful examples that can only be
        programmed (elegantly) with this feature, so I'm reserving judgment.
        I can believe that it's easy to implement.  

    Commentary from Ka-Ping Yee:  I also think there is a lot of power to be
        gained from generator argument passing.

    Commentary from Neil Schemenauer:  I like the idea of being able to pass
        values back into a generator.  I originally pitched this idea to Guido
        but in the end we decided against it (at least for the initial
        implementation).  There was a few issues to work out but I can't seem
        to remember what they were.  My feeling is that we need to wait until
        the Python community has more experience with generators before adding
        this feature.  Maybe for 2.4 but not for 2.3.  In the mean time you
        can work around this limitation by making your generator a method.
        Values can be passed back by mutating the instance.

    Author response:  Okay, consider this part of the proposal deferred
        until 2.4.



Restartability

    [Discussion of restartability deleted]
                
    Commentary from GvR:  The PEP then goes on to discuss restartable
        iterators.  I think this is an evil idea obtained from reading too
        much about C++ STL  iterators. It should definitely be a separate
        PEP if the author wants me to take this seriously.  

    Commentary from Ka-Ping Yee:  I have less of an opinion on restartability
        since i have not yet had to really run into that issue.  It seems
        reasonable that it might be good idea, though perhaps YAGNI will apply
        here until I experience the need for it first-hand.

    Author response:  Over thirty reviewers responded, only one was interested
        in restartability on the theory that it made life easier for beginners
        and that it made lazy evaluation more substitutable for full
        evaluation.  I was never sold on it myself.  Consider it retracted.

                

References

    [1] PEP 255 Simple Generators
        http://python.sourceforge.net/peps/pep-0255.html

    [2] PEP 212 Loop Counter Iteration
        http://python.sourceforge.net/peps/pep-0212.html

    [3] PEP 202 List Comprehensions
        http://python.sourceforge.net/peps/pep-0202.html

    [4] PEP 234 Iterators
        http://python.sourceforge.net/peps/pep-0234.html

    [5] Dr. David Mertz's draft column for Charming Python.
        http://gnosis.cx/publish/programming/charming_python_b5.txt

    [6] PIL, the Python Imaging Library can be found at:
        http://www.pythonware.com/products/pil/

    [7] A pure Python simulation of every feature in this PEP is at:
        http://sourceforge.net/tracker/download.php?group_id=5470&atid=305470&file_id=17348&aid=513752

    [8] The full, working source code for each of the examples in this PEP
        along with other examples and tests is at:
        http://sourceforge.net/tracker/download.php?group_id=5470&atid=305470&file_id=17412&aid=513756



Copyright

    This document has been placed in the public domain.



Local Variables:
mode: indented-text
indent-tabs-mode: nil
fill-column: 70
End: