357 lines
12 KiB
Plaintext
357 lines
12 KiB
Plaintext
PEP: 234
|
||
Title: Iterators
|
||
Version: $Revision$
|
||
Author: ping@lfw.org (Ka-Ping Yee)
|
||
Status: Draft
|
||
Type: Standards Track
|
||
Python-Version: 2.1
|
||
Created: 30-Jan-2001
|
||
Post-History:
|
||
|
||
Abstract
|
||
|
||
This document proposes an iteration interface that objects can
|
||
provide to control the behaviour of 'for' loops. Looping is
|
||
customized by providing a method that produces an iterator object.
|
||
The iterator should be a callable object that returns the next
|
||
item in the sequence each time it is called, raising an exception
|
||
when no more items are available.
|
||
|
||
|
||
Copyright
|
||
|
||
This document is in the public domain.
|
||
|
||
|
||
Sequence Iterators
|
||
|
||
A new field named 'sq_iter' for requesting an iterator is added
|
||
to the PySequenceMethods table. Upon an attempt to iterate over
|
||
an object with a loop such as
|
||
|
||
for item in sequence:
|
||
...body...
|
||
|
||
the interpreter looks for the 'sq_iter' of the 'sequence' object.
|
||
If the method exists, it is called to get an iterator; it should
|
||
return a callable object. If the method does not exist, the
|
||
interpreter produces a built-in iterator object in the following
|
||
manner (described in Python here, but implemented in the core):
|
||
|
||
def make_iterator(sequence):
|
||
def iterator(sequence=sequence, index=[0]):
|
||
item = sequence[index[0]]
|
||
index[0] += 1
|
||
return item
|
||
return iterator
|
||
|
||
To execute the above 'for' loop, the interpreter would proceed as
|
||
follows, where 'iterator' is the iterator that was obtained:
|
||
|
||
while 1:
|
||
try:
|
||
item = iterator()
|
||
except IndexError:
|
||
break
|
||
...body...
|
||
|
||
(Note that the 'break' above doesn't translate to a "real" Python
|
||
break, since it would go to the 'else:' clause of the loop whereas
|
||
a "real" break in the body would skip the 'else:' clause.)
|
||
|
||
The list() and tuple() built-in functions would be updated to use
|
||
this same iterator logic to retrieve the items in their argument.
|
||
|
||
List and tuple objects would implement the 'sq_iter' method by
|
||
calling the built-in make_iterator() routine just described.
|
||
|
||
Instance objects would implement the 'sq_iter' method as follows:
|
||
|
||
if hasattr(self, '__iter__'):
|
||
return self.__iter__()
|
||
elif hasattr(self, '__getitem__'):
|
||
return make_iterator(self)
|
||
else:
|
||
raise TypeError, thing.__class__.__name__ + \
|
||
' instance does not support iteration'
|
||
|
||
Extension objects can implement 'sq_iter' however they wish, as
|
||
long as they return a callable object.
|
||
|
||
|
||
Mapping Iterators
|
||
|
||
An additional proposal from Guido is to provide special syntax
|
||
for iterating over mappings. The loop:
|
||
|
||
for key:value in mapping:
|
||
|
||
would bind both 'key' and 'value' to a key-value pair from the
|
||
mapping on each iteration. Tim Peters suggested that similarly,
|
||
|
||
for key: in mapping:
|
||
|
||
could iterate over just the keys and
|
||
|
||
for :value in mapping:
|
||
|
||
could iterate over just the values.
|
||
|
||
The syntax is unambiguous since the new colon is currently not
|
||
permitted in this position in the grammar.
|
||
|
||
This behaviour would be provided by additional methods in the
|
||
PyMappingMethods table: 'mp_iteritems', 'mp_iterkeys', and
|
||
'mp_itervalues' respectively. 'mp_iteritems' is expected to
|
||
produce a callable object that returns a (key, value) tuple;
|
||
'mp_iterkeys' and 'mp_itervalues' are expected to produce a
|
||
callable object that returns a single key or value.
|
||
|
||
The implementations of these methods on instance objects would
|
||
then check for and call the '__iteritems__', '__iterkeys__',
|
||
and '__itervalues__' methods respectively.
|
||
|
||
When 'mp_iteritems', 'mp_iterkeys', or 'mp_itervalues' is missing,
|
||
the default behaviour is to do make_iterator(mapping.items()),
|
||
make_iterator(mapping.keys()), or make_iterator(mapping.values())
|
||
respectively, using the definition of make_iterator() above.
|
||
|
||
|
||
Indexing Sequences
|
||
|
||
The special syntax described above can be applied to sequences
|
||
as well, to provide the long-hoped-for ability to obtain the
|
||
indices of a sequence without the strange-looking 'range(len(x))'
|
||
expression.
|
||
|
||
for index:item in sequence:
|
||
|
||
causes 'index' to be bound to the index of each item as 'item' is
|
||
bound to the items of the sequence in turn, and
|
||
|
||
for index: in sequence:
|
||
|
||
simply causes 'index' to start at 0 and increment until an attempt
|
||
to get sequence[index] produces an IndexError. For completeness,
|
||
|
||
for :item in sequence:
|
||
|
||
is equivalent to
|
||
|
||
for item in sequence:
|
||
|
||
In each case we try to request an appropriate iterator from the
|
||
sequence. In summary:
|
||
|
||
for k:v in x looks for mp_iteritems, then sq_iter
|
||
for k: in x looks for mp_iterkeys, then sq_iter
|
||
for :v in x looks for mp_itervalues, then sq_iter
|
||
for v in x looks for sq_iter
|
||
|
||
If we fall back to sq_iter in the first two cases, we generate
|
||
indices for k as needed, by starting at 0 and incrementing.
|
||
|
||
The implementation of the mp_iter* methods on instance objects
|
||
then checks for methods in the following order:
|
||
|
||
mp_iteritems __iteritems__, __iter__, items, __getitem__
|
||
mp_iterkeys __iterkeys__, __iter__, keys, __getitem__
|
||
mp_itervalues __itervalues__, __iter__, values, __getitem__
|
||
sq_iter __iter__, __getitem__
|
||
|
||
If a __iteritems__, __iterkeys__, or __itervalues__ method is
|
||
found, we just call it and use the resulting iterator. If a
|
||
mp_* function finds no such method but finds __iter__ instead,
|
||
we generate indices as needed.
|
||
|
||
Upon finding an items(), keys(), or values() method, we use
|
||
make_iterator(x.items()), make_iterator(x.keys()), or
|
||
make_iterator(x.values()) respectively. Upon finding a
|
||
__getitem__ method, we use it and generate indices as needed.
|
||
|
||
For example, the complete implementation of the mp_iteritems
|
||
method for instances can be roughly described as follows:
|
||
|
||
def mp_iteritems(thing):
|
||
if hasattr(thing, '__iteritems__'):
|
||
return thing.__iteritems__()
|
||
if hasattr(thing, '__iter__'):
|
||
def iterator(sequence=thing, index=[0]):
|
||
item = (index[0], sequence.__iter__())
|
||
index[0] += 1
|
||
return item
|
||
return iterator
|
||
if hasattr(thing, 'items'):
|
||
return make_iterator(thing.items())
|
||
if hasattr(thing, '__getitem__'):
|
||
def iterator(sequence=thing, index=[0]):
|
||
item = (index[0], sequence[index[0]])
|
||
index[0] += 1
|
||
return item
|
||
return iterator
|
||
raise TypeError, thing.__class__.__name__ + \
|
||
' instance does not support iteration over items'
|
||
|
||
|
||
Examples
|
||
|
||
Here is a class written in Python that represents the sequence of
|
||
lines in a file.
|
||
|
||
class FileLines:
|
||
def __init__(self, filename):
|
||
self.file = open(filename)
|
||
def __iter__(self):
|
||
def iter(self=self):
|
||
line = self.file.readline()
|
||
if line: return line
|
||
else: raise IndexError
|
||
return iter
|
||
|
||
for line in FileLines('spam.txt'):
|
||
print line
|
||
|
||
And here's an interactive session demonstrating the proposed new
|
||
looping syntax:
|
||
|
||
>>> for i:item in ['a', 'b', 'c']:
|
||
... print i, item
|
||
...
|
||
0 a
|
||
1 b
|
||
2 c
|
||
>>> for i: in 'abcdefg': # just the indices, please
|
||
... print i,
|
||
... print
|
||
...
|
||
0 1 2 3 4 5 6
|
||
>>> for k:v in os.environ: # os.environ is an instance, but
|
||
... print k, v # this still works because we fall
|
||
... # back to calling items()
|
||
MAIL /var/spool/mail/ping
|
||
HOME /home/ping
|
||
DISPLAY :0.0
|
||
TERM xterm
|
||
.
|
||
.
|
||
.
|
||
|
||
|
||
Rationale
|
||
|
||
If all the parts of the proposal are included, this addresses many
|
||
concerns in a consistent and flexible fashion. Among its chief
|
||
virtues are the following three -- no, four -- no, five -- points:
|
||
|
||
1. It provides an extensible iterator interface.
|
||
|
||
2. It resolves the endless "i indexing sequence" debate.
|
||
|
||
3. It allows performance enhancements to dictionary iteration.
|
||
|
||
4. It allows one to provide an interface for just iteration
|
||
without pretending to provide random access to elements.
|
||
|
||
5. It is backward-compatible with all existing user-defined
|
||
classes and extension objects that emulate sequences and
|
||
mappings, even mappings that only implement a subset of
|
||
{__getitem__, keys, values, items}.
|
||
|
||
|
||
Errors
|
||
|
||
Errors that occur during sq_iter, mp_iter*, or the __iter*__
|
||
methods are allowed to propagate normally to the surface.
|
||
|
||
An attempt to do
|
||
|
||
for item in dict:
|
||
|
||
over a dictionary object still produces:
|
||
|
||
TypeError: loop over non-sequence
|
||
|
||
An attempt to iterate over an instance that provides neither
|
||
__iter__ nor __getitem__ produces:
|
||
|
||
TypeError: instance does not support iteration
|
||
|
||
Similarly, an attempt to do mapping-iteration over an instance
|
||
that doesn't provide the right methods should produce one of the
|
||
following errors:
|
||
|
||
TypeError: instance does not support iteration over items
|
||
TypeError: instance does not support iteration over keys
|
||
TypeError: instance does not support iteration over values
|
||
|
||
It's an error for the iterator produced by __iteritems__ or
|
||
mp_iteritems to return an object whose length is not 2:
|
||
|
||
TypeError: item iterator did not return a 2-tuple
|
||
|
||
|
||
Open Issues
|
||
|
||
We could introduce a new exception type such as IteratorExit just
|
||
for terminating loops rather than using IndexError. In this case,
|
||
the implementation of make_iterator() would catch and translate an
|
||
IndexError into an IteratorExit for backward compatibility.
|
||
|
||
We could provide access to the logic that calls either 'sq_item'
|
||
or make_iterator() with an iter() function in the built-in module
|
||
(just as the getattr() function provides access to 'tp_getattr').
|
||
One possible motivation for this is to make it easier for the
|
||
implementation of __iter__ to delegate iteration to some other
|
||
sequence. Presumably we would then have to consider adding
|
||
iteritems(), iterkeys(), and itervalues() as well.
|
||
|
||
An alternative way to let __iter__ delegate iteration to another
|
||
sequence is for it to return another sequence. Upon detecting
|
||
that the object returned by __iter__ is not callable, the
|
||
interpreter could repeat the process of looking for an iterator
|
||
on the new object. However, this process seems potentially
|
||
convoluted and likely to produce more confusing error messages.
|
||
|
||
If we decide to add "freezing" ability to lists and dictionaries,
|
||
it is suggested that the implementation of make_iterator
|
||
automatically freeze any list or dictionary argument for the
|
||
duration of the loop, and produce an error complaining about any
|
||
attempt to modify it during iteration. Since it is relatively
|
||
rare to actually want to modify it during iteration, this is
|
||
likely to catch mistakes earlier. If a programmer wants to
|
||
modify a list or dictionary during iteration, they should
|
||
explicitly make a copy to iterate over using x[:], x.clone(),
|
||
x.keys(), x.values(), or x.items().
|
||
|
||
For consistency with the 'key in dict' expression, we could
|
||
support 'for key in dict' as equivalent to 'for key: in dict'.
|
||
|
||
|
||
BDFL Pronouncements
|
||
|
||
The "parallel expression" to 'for key:value in mapping':
|
||
|
||
if key:value in mapping:
|
||
|
||
is infeasible since the first colon ends the "if" condition.
|
||
The following compromise is technically feasible:
|
||
|
||
if (key:value) in mapping:
|
||
|
||
but the BDFL has pronounced a solid -1 on this.
|
||
|
||
The BDFL gave a +0.5 to:
|
||
|
||
for key:value in mapping:
|
||
for index:item in sequence:
|
||
|
||
and a +0.2 to the variations where the part before or after
|
||
the first colon is missing.
|
||
|
||
|
||
|
||
Local Variables:
|
||
mode: indented-text
|
||
indent-tabs-mode: nil
|
||
End:
|