python-peps/pep-0279.txt

PEP: 279
Title: Enhanced Generators
Version: $Revision$
Last-Modified: $Date$
Author: python@rcn.com (Raymond D. Hettinger)
Status: Accepted
Type: Standards Track
Created: 30-Jan-2002
Python-Version: 2.3
Post-History:


Abstract

    This PEP introduces a new built-in function, enumerate() to
    simplify a commonly used looping idiom.  It provides all iterable
    collections with the same advantage that iteritems() affords to
    dictionaries -- a compact, readable, reliable index notation.


Rationale

    Python 2.2 introduced the concept of an iterable interface as
    proposed in PEP 234 [3].  The iter() factory function was provided
    as common calling convention and deep changes were made to use
    iterators as a unifying theme throughout Python.  The unification
    came in the form of establishing a common iterable interface for
    mappings, sequences, and file objects.

    Generators, as proposed in PEP 255 [1], were introduced as a means
    for making it easier to create iterators, especially ones with
    complex internal execution or variable states.  The availability
    of generators makes it possible to improve on the loop counter
    ideas in PEP 212 [2].  Those ideas provided a clean syntax for
    iteration with indices and values, but did not apply to all
    iterable objects.  Also, that approach did not have the memory
    friendly benefit provided by generators which do not evaluate the
    entire sequence all at once.

    The new proposal is to add a built-in function, enumerate() which
    was made possible once iterators and generators became available.
    It provides all iterables with the same advantage that iteritems()
    affords to dictionaries -- a compact, readable, reliable index
    notation.  Like zip(), it is expected to become a commonly used
    looping idiom.

    This suggestion is designed to take advantage of the existing
    implementation and require little additional effort to
    incorporate.  It is backwards compatible and requires no new
    keywords.  The proposal will go into Python 2.3 when generators
    become final and are not imported from __future__.


BDFL Pronouncements

    The new built-in function is ACCEPTED.  


Specification for a new built-in:

    def enumerate(collection):
        'Generates an indexed series:  (0,coll[0]), (1,coll[1]) ...'     
        i = 0
        it = iter(collection)
        while 1:
            yield (i, it.next())
            i += 1

    Note A: PEP 212 Loop Counter Iteration [2] discussed several
    proposals for achieving indexing.  Some of the proposals only work
    for lists unlike the above function which works for any generator,
    xrange, sequence, or iterable object.  Also, those proposals were
    presented and evaluated in the world prior to Python 2.2 which did
    not include generators.  As a result, the non-generator version in
    PEP 212 had the disadvantage of consuming memory with a giant list
    of tuples.  The generator version presented here is fast and
    light, works with all iterables, and allows users to abandon the
    sequence in mid-stream with no loss of computation effort.

    There are other PEPs which touch on related issues: integer
    iterators, integer for-loops, and one for modifying the arguments
    to range and xrange.  The enumerate() proposal does not preclude
    the other proposals and it still meets an important need even if
    those are adopted -- the need to count items in any iterable.  The
    other proposals give a means of producing an index but not the
    corresponding value.  This is especially problematic if a sequence
    is given which doesn't support random access such as a file
    object, generator, or sequence defined with __getitem__.

    Note B: Almost all of the PEP reviewers welcomed the function but
    were divided as to whether there should be any built-ins.  The
    main argument for a separate module was to slow the rate of
    language inflation.  The main argument for a built-in was that the
    function is destined to be part of a core programming style,
    applicable to any object with an iterable interface.  Just as
    zip() solves the problem of looping over multiple sequences, the
    enumerate() function solves the loop counter problem.

    If only one built-in is allowed, then enumerate() is the most
    important general purpose tool, solving the broadest class of
    problems while improving program brevity, clarity and reliability.

    Note C:  Various alternative names were discussed:

        iterindexed()-- five syllables is a mouthful
        index()      -- nice verb but could be confused the .index() method
        indexed()    -- widely liked however adjectives should be avoided
        indexer()    -- noun did not read well in a for-loop
        count()      -- direct and explicit but often used in other contexts
        itercount()  -- direct, explicit and hated by more than one person
        iteritems()  -- conflicts with key:value concept for dictionaries
        itemize()    -- confusing because amap.items() != list(itemize(amap))
        enum()       -- pithy; less clear than enumerate; too similar to enum
                        in other languages where it has a different meaning

    All of the names involving 'count' had the further disadvantage of
    implying that the count would begin from one instead of zero.

    All of the names involving 'index' clashed with usage in database
    languages where indexing implies a sorting operation rather than
    linear sequencing.

    Note D: This function was originally proposed with optional start
    and stop arguments.  GvR pointed out that the function call
    enumerate(seqn,4,6) had an alternate, plausible interpretation as
    a slice that would return the fourth and fifth elements of the
    sequence.  To avoid the ambiguity, the optional arguments were
    dropped even though it meant losing flexibility as a loop counter.
    That flexibility was most important for the common case of
    counting from one, as in:
        
        for linenum, line in enumerate(source,1):  print linenum, line

    Comments from GvR:  filter and map should die and be subsumed into list
        comprehensions, not grow more variants. I'd rather introduce
        built-ins that do iterator algebra (e.g. the iterzip that I've
        often used as an example).

        I like the idea of having some way to iterate over a sequence
        and its index set in parallel.  It's fine for this to be a
        built-in.

        I don't like the name "indexed"; adjectives do not make good
        function names.  Maybe iterindexed()?

    Comments from Ka-Ping Yee:  I'm also quite happy with everything  you
        proposed ... and the extra built-ins (really 'indexed' in
        particular) are things I have wanted for a long time.

    Comments from Neil Schemenauer:  The new built-ins sound okay.  Guido
        may be concerned with increasing the number of built-ins too
        much.  You might be better off selling them as part of a
        module.  If you use a module then you can add lots of useful
        functions (Haskell has lots of them that we could steal).

    Comments for Magnus Lie Hetland:  I think indexed would be a useful and
        natural built-in function. I would certainly use it a lot.  I
        like indexed() a lot; +1. I'm quite happy to have it make PEP
        281 obsolete. Adding a separate module for iterator utilities
        seems like a good idea.

    Comments from the Community:  The response to the enumerate() proposal
        has been close to 100% favorable.  Almost everyone loves the
        idea.

    Author response:  Prior to these comments, four built-ins were proposed.
        After the comments, xmap xfilter and xzip were withdrawn.  The
        one that remains is vital for the language and is proposed by
        itself.  Indexed() is trivially easy to implement and can be
        documented in minutes.  More importantly, it is useful in
        everyday programming which does not otherwise involve explicit
        use of generators.

        Though withdrawn from the proposal, I still secretly covet
        xzip() a.k.a. iterzip() but think that it will happen on its
        own someday.


References

    [1] PEP 255 Simple Generators
        http://python.sourceforge.net/peps/pep-0255.html

    [2] PEP 212 Loop Counter Iteration
        http://python.sourceforge.net/peps/pep-0212.html

    [3] PEP 234 Iterators
        http://python.sourceforge.net/peps/pep-0234.html


Copyright

    This document has been placed in the public domain.



Local Variables:
mode: indented-text
indent-tabs-mode: nil
fill-column: 70
End: