python-peps/pep-0234.txt

PEP: 234
Title: Iterators
Version: $Revision$
Author: ping@lfw.org (Ka-Ping Yee)
Status: Draft
Type: Standards Track
Python-Version: 2.1
Created: 30-Jan-2001
Post-History:

Abstract

    This document proposes an iteration interface that objects can
    provide to control the behaviour of 'for' loops.  Looping is
    customized by providing a method that produces an iterator object.
    The iterator should be a callable object that returns the next
    item in the sequence each time it is called, raising an exception
    when no more items are available.


Copyright

    This document is in the public domain.


Sequence Iterators

    A new field named 'sq_iter' for requesting an iterator is added
    to the PySequenceMethods table.  Upon an attempt to iterate over
    an object with a loop such as

        for item in sequence:
            ...body...

    the interpreter looks for the 'sq_iter' of the 'sequence' object.
    If the method exists, it is called to get an iterator; it should
    return a callable object.  If the method does not exist, the
    interpreter produces a built-in iterator object in the following
    manner (described in Python here, but implemented in the core):

        def make_iterator(sequence):
            def iterator(sequence=sequence, index=[0]):
                item = sequence[index[0]]
                index[0] += 1
                return item
            return iterator

    To execute the above 'for' loop, the interpreter would proceed as
    follows, where 'iterator' is the iterator that was obtained:

        while 1:
            try:
                item = iterator()
            except IndexError:
                break
            ...body...

    (Note that the 'break' above doesn't translate to a "real" Python
    break, since it would go to the 'else:' clause of the loop whereas
    a "real" break in the body would skip the 'else:' clause.)

    The list() and tuple() built-in functions would be updated to use
    this same iterator logic to retrieve the items in their argument.

    List and tuple objects would implement the 'sq_iter' method by
    calling the built-in make_iterator() routine just described.

    Instance objects would implement the 'sq_iter' method as follows:

        if hasattr(self, '__iter__'):
            return self.__iter__()
        elif hasattr(self, '__getitem__'):
            return make_iterator(self)
        else:
            raise TypeError, thing.__class__.__name__ + \
                ' instance does not support iteration'

    Extension objects can implement 'sq_iter' however they wish, as
    long as they return a callable object.


Mapping Iterators

    An additional proposal from Guido is to provide special syntax
    for iterating over mappings.  The loop:

        for key:value in mapping:

    would bind both 'key' and 'value' to a key-value pair from the
    mapping on each iteration.  Tim Peters suggested that similarly,

        for key: in mapping:

    could iterate over just the keys and

        for :value in mapping:

    could iterate over just the values.

    The syntax is unambiguous since the new colon is currently not
    permitted in this position in the grammar.

    This behaviour would be provided by additional methods in the
    PyMappingMethods table: 'mp_iteritems', 'mp_iterkeys', and
    'mp_itervalues' respectively.  'mp_iteritems' is expected to
    produce a callable object that returns a (key, value) tuple;
    'mp_iterkeys' and 'mp_itervalues' are expected to produce a
    callable object that returns a single key or value.

    The implementations of these methods on instance objects would
    then check for and call the '__iteritems__', '__iterkeys__',
    and '__itervalues__' methods respectively.

    When 'mp_iteritems', 'mp_iterkeys', or 'mp_itervalues' is missing,
    the default behaviour is to do make_iterator(mapping.items()),
    make_iterator(mapping.keys()), or make_iterator(mapping.values())
    respectively, using the definition of make_iterator() above.


Indexing Sequences

    The special syntax described above can be applied to sequences
    as well, to provide the long-hoped-for ability to obtain the
    indices of a sequence without the strange-looking 'range(len(x))'
    expression.

        for index:item in sequence:

    causes 'index' to be bound to the index of each item as 'item' is
    bound to the items of the sequence in turn, and

        for index: in sequence:

    simply causes 'index' to start at 0 and increment until an attempt
    to get sequence[index] produces an IndexError.  For completeness,

        for :item in sequence:

    is equivalent to

        for item in sequence:

    In each case we try to request an appropriate iterator from the
    sequence.  In summary:

        for k:v in x    looks for mp_iteritems, then sq_iter
        for k: in x     looks for mp_iterkeys, then sq_iter
        for :v in x     looks for mp_itervalues, then sq_iter
        for v in x      looks for sq_iter

    If we fall back to sq_iter in the first two cases, we generate
    indices for k as needed, by starting at 0 and incrementing.

    The implementation of the mp_iter* methods on instance objects
    then checks for methods in the following order:

        mp_iteritems    __iteritems__, __iter__, items, __getitem__
        mp_iterkeys     __iterkeys__, __iter__, keys, __getitem__
        mp_itervalues   __itervalues__, __iter__, values, __getitem__
        sq_iter         __iter__, __getitem__

    If a __iteritems__, __iterkeys__, or __itervalues__ method is
    found, we just call it and use the resulting iterator.  If a
    mp_* function finds no such method but finds __iter__ instead,
    we generate indices as needed.

    Upon finding an items(), keys(), or values() method, we use
    make_iterator(x.items()), make_iterator(x.keys()), or
    make_iterator(x.values()) respectively.  Upon finding a
    __getitem__ method, we use it and generate indices as needed.

    For example, the complete implementation of the mp_iteritems
    method for instances can be roughly described as follows:

        def mp_iteritems(thing):
            if hasattr(thing, '__iteritems__'):
                return thing.__iteritems__()
            if hasattr(thing, '__iter__'):
                def iterator(sequence=thing, index=[0]):
                    item = (index[0], sequence.__iter__())
                    index[0] += 1
                    return item
                return iterator
            if hasattr(thing, 'items'):
                return make_iterator(thing.items())
            if hasattr(thing, '__getitem__'):
                def iterator(sequence=thing, index=[0]):
                    item = (index[0], sequence[index[0]])
                    index[0] += 1
                    return item
                return iterator
            raise TypeError, thing.__class__.__name__ + \
                ' instance does not support iteration over items'


Examples

    Here is a class written in Python that represents the sequence of
    lines in a file.

        class FileLines:
            def __init__(self, filename):
                self.file = open(filename)
            def __iter__(self):
                def iter(self=self):
                    line = self.file.readline()
                    if line: return line
                    else: raise IndexError
                return iter

        for line in FileLines('spam.txt'):
            print line

    And here's an interactive session demonstrating the proposed new
    looping syntax:

        >>> for i:item in ['a', 'b', 'c']:
        ...     print i, item
        ...
        0 a
        1 b
        2 c
        >>> for i: in 'abcdefg':        # just the indices, please
        ...     print i,
        ... print
        ...
        0 1 2 3 4 5 6
        >>> for k:v in os.environ:      # os.environ is an instance, but
        ...     print k, v              # this still works because we fall
        ...                             # back to calling items()
        MAIL /var/spool/mail/ping
        HOME /home/ping
        DISPLAY :0.0
        TERM xterm
        .
        .
        .


Rationale

    If all the parts of the proposal are included, this addresses many
    concerns in a consistent and flexible fashion.  Among its chief
    virtues are the following three -- no, four -- no, five -- points:

    1. It provides an extensible iterator interface.

    2. It resolves the endless "i indexing sequence" debate.

    3. It allows performance enhancements to dictionary iteration.

    4. It allows one to provide an interface for just iteration
       without pretending to provide random access to elements.

    5. It is backward-compatible with all existing user-defined
       classes and extension objects that emulate sequences and
       mappings, even mappings that only implement a subset of
       {__getitem__, keys, values, items}.


Errors

    Errors that occur during sq_iter, mp_iter*, or the __iter*__
    methods are allowed to propagate normally to the surface.

    An attempt to do

        for item in dict:

    over a dictionary object still produces:

        TypeError: loop over non-sequence

    An attempt to iterate over an instance that provides neither
    __iter__ nor __getitem__ produces:

        TypeError: instance does not support iteration

    Similarly, an attempt to do mapping-iteration over an instance
    that doesn't provide the right methods should produce one of the
    following errors:

        TypeError: instance does not support iteration over items
        TypeError: instance does not support iteration over keys
        TypeError: instance does not support iteration over values

    It's an error for the iterator produced by __iteritems__ or
    mp_iteritems to return an object whose length is not 2:

        TypeError: item iterator did not return a 2-tuple


Open Issues

    We could introduce a new exception type such as IteratorExit just
    for terminating loops rather than using IndexError.  In this case,
    the implementation of make_iterator() would catch and translate an
    IndexError into an IteratorExit for backward compatibility.

    We could provide access to the logic that calls either 'sq_item'
    or make_iterator() with an iter() function in the built-in module
    (just as the getattr() function provides access to 'tp_getattr').
    One possible motivation for this is to make it easier for the
    implementation of __iter__ to delegate iteration to some other
    sequence.  Presumably we would then have to consider adding
    iteritems(), iterkeys(), and itervalues() as well.

    An alternative way to let __iter__ delegate iteration to another
    sequence is for it to return another sequence.  Upon detecting
    that the object returned by __iter__ is not callable, the
    interpreter could repeat the process of looking for an iterator
    on the new object.  However, this process seems potentially
    convoluted and likely to produce more confusing error messages.

    If we decide to add "freezing" ability to lists and dictionaries,
    it is suggested that the implementation of make_iterator
    automatically freeze any list or dictionary argument for the
    duration of the loop, and produce an error complaining about any
    attempt to modify it during iteration.  Since it is relatively
    rare to actually want to modify it during iteration, this is
    likely to catch mistakes earlier.  If a programmer wants to
    modify a list or dictionary during iteration, they should
    explicitly make a copy to iterate over using x[:], x.clone(),
    x.keys(), x.values(), or x.items().

    For consistency with the 'key in dict' expression, we could
    support 'for key in dict' as equivalent to 'for key: in dict'.


BDFL Pronouncements

    The "parallel expression" to 'for key:value in mapping':

        if key:value in mapping:

    is infeasible since the first colon ends the "if" condition.
    The following compromise is technically feasible:

        if (key:value) in mapping:

    but the BDFL has pronounced a solid -1 on this.

    The BDFL gave a +0.5 to:

        for key:value in mapping:
        for index:item in sequence:

    and a +0.2 to the variations where the part before or after
    the first colon is missing.



Local Variables:
mode: indented-text
indent-tabs-mode: nil
End: