465 lines
19 KiB
Plaintext
465 lines
19 KiB
Plaintext
PEP: 323
|
||
Title: Copyable Iterators
|
||
Version: $Revision$
|
||
Last-Modified: $Date$
|
||
Author: Alex Martelli <aleaxit@yahoo.com>
|
||
Status: Draft
|
||
Type: Standards Track
|
||
Content-Type: text/plain
|
||
Created: 25-Oct-2003
|
||
Python-Version: 2.4
|
||
Post-History: 29-Oct-2003
|
||
|
||
|
||
Abstract
|
||
|
||
This PEP suggests that some iterator types should support shallow
|
||
copies of their instances by exposing a __copy__ method which meets
|
||
some specific requirements, and indicates how code using an iterator
|
||
might exploit such a __copy__ method when present.
|
||
|
||
|
||
Motivation
|
||
|
||
In Python up to 2.3, most built-in iterator types don't let the user
|
||
copy their instances. User-coded iterators that do let their clients
|
||
call copy.copy on their instances may, or may not, happen to return,
|
||
as a result of the copy, a separate iterator object that may be
|
||
iterated upon independently from the original.
|
||
|
||
Currently, "support" for copy.copy in a user-coded iterator type is
|
||
almost invariably "accidental" -- i.e., the standard machinery of the
|
||
copy method in Python's standard library's copy module does build and
|
||
return a copy. However, the copy will be independently iterable with
|
||
respect to the original only if calling .next() on an instance of that
|
||
class happens to change instance state solely by rebinding some
|
||
attributes to new values, and not by mutating some attributes'
|
||
existing values.
|
||
|
||
For example, an iterator whose "index" state is held as an integer
|
||
attribute will probably give usable copies, since (integers being
|
||
immutable) .next() presumably just rebinds that attribute. On the
|
||
other hand, another iterator whose "index" state is held as a list
|
||
attribute will probably mutate the same list object when .next()
|
||
executes, and therefore copies of such an iterator will not be
|
||
iterable separately and independently from the original.
|
||
|
||
Given this existing situation, copy.copy(it) on some iterator object
|
||
isn't very useful, nor, therefore, is it at all widely used. However,
|
||
there are many cases in which being able to get a "snapshot" of an
|
||
iterator, as a "bookmark", so as to be able to keep iterating along
|
||
the sequence but later iterate again on the same sequence from the
|
||
bookmark onwards, is useful. To support such "bookmarking", module
|
||
itertools, in 2.4, has grown a 'tee' function, to be used as:
|
||
|
||
it, bookmark = itertools.tee(it)
|
||
|
||
The previous value of 'it' must not be used again, which is why this
|
||
typical usage idiom rebinds the name. After this call, 'it' and
|
||
'bookmark' are independently-iterable iterators on the same underlying
|
||
sequence as the original value of 'it': this satisfies application
|
||
needs for "iterator copying".
|
||
|
||
However, when itertools.tee can make no hypotheses about the nature of
|
||
the iterator it is passed as an argument, it must save in memory all
|
||
items through which one of the two 'teed' iterators, but not yet both,
|
||
have stepped. This can be quite costly in terms of memory, if the two
|
||
iterators get very far from each other in their stepping; indeed, in
|
||
some cases it may be preferable to make a list from the iterator so as
|
||
to be able to step repeatedly through the subsequence, or, if that is
|
||
too costy in terms of memory, save items to disk, again in order to be
|
||
able to iterate through them repeatedly.
|
||
|
||
This PEP proposes another idea that will, in some important cases,
|
||
allow itertools.tee to do its job with minimal cost in terms of
|
||
memory; user code may also occasionally be able to exploit the idea in
|
||
order to decide whether to copy an iterator, make a list from it, or
|
||
use an auxiliary disk file.
|
||
|
||
The key consideration is that some important iterators, such as those
|
||
which built-in function iter builds over sequences, would be
|
||
intrinsically easy to copy: just get another reference to the same
|
||
sequence, and a copy of the integer index. However, in Python 2.3,
|
||
those iterators don't expose the state, and don't support copy.copy.
|
||
|
||
The purpose of this PEP, therefore, is to have those iterator types
|
||
expose a suitable __copy__ method. Similarly, user-coded iterator
|
||
types that can provide copies of their instances, suitable for
|
||
separate and independent iteration, with limited costs in time and
|
||
space, should also expose a suitable __copy__ method. While
|
||
copy.copy also supports other ways to let a type control the way
|
||
its instances are copied, it is suggested, for simplicity, that
|
||
iterator types that support copying always do so by exposing a
|
||
__copy__ method, and not in the other ways copy.copy supports.
|
||
|
||
Having iterators expose a suitable __copy__ when feasible will afford
|
||
easy optimization of itertools.tee and similar user code, as in:
|
||
|
||
def tee(it):
|
||
it = iter(it)
|
||
try: copier = it.__copy__
|
||
except AttributeError:
|
||
# non-copyable iterator, do all the needed hard work
|
||
# [snipped!]
|
||
else:
|
||
return it, copier()
|
||
|
||
Note that this function does NOT call "copy.copy(it)", which (even
|
||
after this PEP is implemented) might well still "just happen to
|
||
succeed". for some iterator type that is implemented as a user-coded
|
||
class. without really supplying an adequate "independently iterable"
|
||
copy object as its result.
|
||
|
||
|
||
Specification
|
||
|
||
Any iterator type X may expose a method __copy__ that is callable
|
||
without arguments on any instance x of X. The method should be
|
||
exposed if and only if the iterator type can provide copyability with
|
||
reasonably little computational and memory effort. Furthermore, the
|
||
new object y returned by method __copy__ should be a new instance
|
||
of X that is iterable independently and separately from x, stepping
|
||
along the same "underlying sequence" of items.
|
||
|
||
For example, suppose a class Iter essentially duplicated the
|
||
functionality of the iter builtin for iterating on a sequence:
|
||
|
||
class Iter(object):
|
||
|
||
def __init__(self, sequence):
|
||
self.sequence = sequence
|
||
self.index = 0
|
||
|
||
def __iter__(self):
|
||
return self
|
||
|
||
def next(self):
|
||
try: result = self.sequence[self.index]
|
||
except IndexError: raise StopIteration
|
||
self.index += 1
|
||
return result
|
||
|
||
To make this Iter class compliant with this PEP, the following
|
||
addition to the body of class Iter would suffice:
|
||
|
||
def __copy__(self):
|
||
result = self.__class__(sequence)
|
||
result.index = self.index
|
||
return result
|
||
|
||
Note that __copy__, in this case, does not even try to copy the
|
||
sequence; if the sequence is altered while either or both of the
|
||
original and copied iterators are still stepping on it, the iteration
|
||
behavior is quite likely to go awry anyway -- it is not __copy__'s
|
||
responsibility to change this normal Python behavior for iterators
|
||
which iterate on mutable sequences (that might, perhaps, be the
|
||
specification for a __deepcopy__ method of iterators, which, however,
|
||
this PEP does not deal with).
|
||
|
||
Consider also a "random iterator", which provides a nonterminating
|
||
sequence of results from some method of a random instance, called
|
||
with given arguments:
|
||
|
||
class RandomIterator(object):
|
||
|
||
def __init__(self, bound_method, *args):
|
||
self.call = bound_method
|
||
self.args = args
|
||
|
||
def __iter__(self):
|
||
return self
|
||
|
||
def next(self):
|
||
return self.call(*self.args)
|
||
|
||
def __copy__(self):
|
||
import copy, new
|
||
im_self = copy.copy(self.call.im_self)
|
||
method = new.instancemethod(self.call.im_func, im_self)
|
||
return self.__class__(method, *self.args)
|
||
|
||
This iterator type is slightly more general than its name implies, as
|
||
it supports calls to any bound method (or other callable, but if the
|
||
callable is not a bound method, then method __copy__ will fail). But
|
||
the use case is for the purpose of generating random streams, as in:
|
||
|
||
import random
|
||
|
||
def show5(it):
|
||
for i, result in enumerate(it):
|
||
print '%6.3f'%result,
|
||
if i==4: break
|
||
print
|
||
|
||
normit = RandomIterator(random.Random().gauss, 0, 1)
|
||
show5(normit)
|
||
copit = normit.__copy__()
|
||
show5(normit)
|
||
show5(copit)
|
||
|
||
which will display some output such as:
|
||
|
||
-0.536 1.936 -1.182 -1.690 -1.184
|
||
0.666 -0.701 1.214 0.348 1.373
|
||
0.666 -0.701 1.214 0.348 1.373
|
||
|
||
the key point being that the second and third lines are equal, because
|
||
the normit and copit iterators will step along the same "underlying
|
||
sequence". (As an aside, note that to get a copy of self.call.im_self
|
||
we must use copy.copy, NOT try getting at a __copy__ method directly,
|
||
because for example instances of random.Random support copying via
|
||
__getstate__ and __setstate__, NOT via __copy__; indeed, using
|
||
copy.copy is the normal way to get a shallow copy of any object --
|
||
copyable iterators are different because of the already-mentioned
|
||
uncertainty about the result of copy.copy supporting these "copyable
|
||
iterator" specs).
|
||
|
||
|
||
Details
|
||
|
||
Besides adding to the Python docs a recommendation that user-coded
|
||
iterator types support a __copy__ method (if and only if it can be
|
||
implemented with small costs in memory and runtime, and produce an
|
||
independently-iterable copy of an iterator object), this PEP's
|
||
implementation will specifically include the addition of copyability
|
||
to the iterators over sequences that built-in iter returns, and also
|
||
to the iterators over a dictionary returned by the methods __iter__,
|
||
iterkeys, itervalues, and iteritems of built-in type dict.
|
||
|
||
Iterators produced by generator functions will not be copyable.
|
||
However, iterators produced by the new "generator expressions" of
|
||
Python 2.4 (PEP 289 [3]) should be copyable if their underlying
|
||
iterator[s] are; the strict limitations on what is possible in a
|
||
generator expression, compared to the much vaster generality of a
|
||
generator, should make that feasible. Similarly, the iterators
|
||
produced by the built-in function enumerate, and certain functions
|
||
suppiled by module itertools, should be copyable if the underlying
|
||
iterators are.
|
||
|
||
The implementation of this PEP will also include the optimization of
|
||
the new itertools.tee function mentioned in the Motivation section.
|
||
|
||
|
||
Rationale
|
||
|
||
The main use case for (shallow) copying of an iterator is the same as
|
||
for the function itertools.tee (new in 2.4). User code will not
|
||
directly attempt to copy an iterator, because it would have to deal
|
||
separately with uncopyable cases; calling itertools.tee will
|
||
internally perform the copy when appropriate, and implicitly fallback
|
||
to a maximally efficient non-copying strategy for iterators that are
|
||
not copyable. (Occasionally, user code may want more direct control,
|
||
specifically in order to deal with non-copyable iterators by other
|
||
strategies, such as making a list or saving the sequence to disk).
|
||
|
||
A tee'd iterator may serve as a "reference point", allowing processing
|
||
of a sequence to continue or resume from a known point, while the
|
||
other independent iterator can be freely advanced to "explore" a
|
||
further part of the sequence as needed. A simple example: a generator
|
||
function which, given an iterator of numbers (assumed to be positive),
|
||
returns a corresponding iterator, each of whose items is the fraction
|
||
of the total corresponding to each corresponding item of the input
|
||
iterator. The caller may pass the total as a value, if known in
|
||
advance; otherwise, the iterator returned by calling this generator
|
||
function will first compute the total.
|
||
|
||
def fractions(numbers, total=None):
|
||
if total is None:
|
||
numbers, aux = itertools.tee(numbers)
|
||
total = sum(aux)
|
||
total = float(total)
|
||
for item in numbers:
|
||
yield item / total
|
||
|
||
The ability to tee the numbers iterator allows this generator to
|
||
precompute the total, if needed, without necessarily requiring
|
||
O(N) auxiliary memory if the numbers iterator is copyable.
|
||
|
||
As another example of "iterator bookmarking", consider a stream of
|
||
numbers with an occasional string as a "postfix operator" now and
|
||
then. By far most frequent such operator is a '+', whereupon we must
|
||
sum all previous numbers (since the last previous operator if any, or
|
||
else since the start) and yield the result. Sometimes we find a '*'
|
||
instead, which is the same except that the previous numbers must
|
||
instead be multiplied, not summed.
|
||
|
||
def filter_weird_stream(stream):
|
||
it = iter(stream)
|
||
while True:
|
||
it, bookmark = itertools.tee(it)
|
||
total = 0
|
||
for item in it:
|
||
if item=='+':
|
||
yield total
|
||
break
|
||
elif item=='*':
|
||
product = 1
|
||
for item in bookmark:
|
||
if item=='*':
|
||
yield product
|
||
break
|
||
else:
|
||
product *= item
|
||
else:
|
||
total += item
|
||
|
||
Similar use cases of itertools.tee can support such tasks as
|
||
"undo" on a stream of commands represented by an iterator,
|
||
"backtracking" on the parse of a stream of tokens, and so on.
|
||
(Of course, in each case, one should also consider simpler
|
||
possibilities such as saving relevant portions of the sequence
|
||
into lists while stepping on the sequence with just one iterator,
|
||
depending on the details of one's task).
|
||
|
||
|
||
Here is an example, in pure Python, of how the 'enumerate'
|
||
built-in could be extended to support __copy__ if its underlying
|
||
iterator also supported __copy__:
|
||
|
||
class enumerate(object):
|
||
|
||
def __init__(self, it):
|
||
self.it = iter(it)
|
||
self.i = -1
|
||
|
||
def __iter__(self):
|
||
return self
|
||
|
||
def next(self):
|
||
self.i += 1
|
||
return self.i, self.it.next()
|
||
|
||
def __copy__(self):
|
||
result = self.__class__.new()
|
||
result.it = self.it.__copy__()
|
||
result.i = self.i
|
||
return result
|
||
|
||
|
||
Here is an example of the kind of "fragility" produced by "accidental
|
||
copyability" of an iterator -- the reason why one must NOT use
|
||
copy.copy expecting, if it succeeds, to receive as a result an
|
||
iterator which is iterable-on independently from the original. Here
|
||
is an iterator class that iterates (in preorder) on "trees" which, for
|
||
simplicity, are just nested lists -- any item that's a list is treated
|
||
as a subtree, any other item as a leaf.
|
||
|
||
class ListreeIter(object):
|
||
|
||
def __init__(self, tree):
|
||
self.tree = [tree]
|
||
self.indx = [-1]
|
||
|
||
def __iter__(self):
|
||
return self
|
||
|
||
def next(self):
|
||
if not self.indx:
|
||
raise StopIteration
|
||
self.indx[-1] += 1
|
||
try:
|
||
result = self.tree[-1][self.indx[-1]]
|
||
except IndexError:
|
||
self.tree.pop()
|
||
self.indx.pop()
|
||
return self.next()
|
||
if type(result) is not list:
|
||
return result
|
||
self.tree.append(result)
|
||
self.indx.append(-1)
|
||
return self.next()
|
||
|
||
Now, for example, the following code:
|
||
|
||
import copy
|
||
x = [ [1,2,3], [4, 5, [6, 7, 8], 9], 10, 11, [12] ]
|
||
|
||
print 'showing all items:',
|
||
it = ListreeIter(x)
|
||
for i in it:
|
||
print i,
|
||
if i==6: cop = copy.copy(it)
|
||
print
|
||
|
||
print 'showing items >6 again:'
|
||
for i in cop: print i,
|
||
print
|
||
|
||
does NOT work as intended -- the "cop" iterator gets consumed, and
|
||
exhausted, step by step as the original "it" iterator is, because
|
||
the accidental (rather than deliberate) copying performed by
|
||
copy.copy shares, rather than duplicating the "index" list, which
|
||
is the mutable attribute it.indx (a list of numerical indices).
|
||
Thus, this "client code" of the iterator, which attemps to iterate
|
||
twice over a portion of the sequence via a copy.copy on the
|
||
iterator, is NOT correct.
|
||
|
||
Some correct solutions include using itertools.tee, i.e., changing
|
||
the first for loop into:
|
||
|
||
for i in it:
|
||
print i,
|
||
if i==6:
|
||
it, cop = itertools.tee(it)
|
||
break
|
||
for i in it: print i,
|
||
|
||
(note that we MUST break the loop in two, otherwise we'd still
|
||
be looping on the ORIGINAL value of it, which must NOT be used
|
||
further after the call to tee!!!); or making a list, i.e.:
|
||
|
||
for i in it:
|
||
print i,
|
||
if i==6:
|
||
cop = lit = list(it)
|
||
break
|
||
for i in lit: print i,
|
||
|
||
(again, the loop must be broken in two, since iterator 'it'
|
||
gets exhausted by the call list(it)).
|
||
|
||
Finally, all of these solutions would work if Listiter supplied
|
||
a suitable __copy__ method, as this PEP recommends:
|
||
|
||
def __copy__(self):
|
||
result = self.__class__.new()
|
||
result.tree = copy.copy(self.tree)
|
||
result.indx = copy.copy(self.indx)
|
||
return result
|
||
|
||
There is no need to get any "deeper" in the copy, but the two
|
||
mutable "index state" attributes must indeed be copied in order
|
||
to achieve a "proper" (independently iterable) iterator-copy.
|
||
|
||
The recommended solution is to have class Listiter supply this
|
||
__copy__ method AND have client code use itertools.tee (with
|
||
the split-in-two-parts loop as shown above). This will make
|
||
client code maximally tolerant of different iterator types it
|
||
might be using AND achieve good performance for tee'ing of this
|
||
specific iterator type at the same time.
|
||
|
||
|
||
References
|
||
|
||
[1] Discussion on python-dev starting at post:
|
||
http://mail.python.org/pipermail/python-dev/2003-October/038969.html
|
||
|
||
[2] Online documentation for the copy module of the standard library:
|
||
http://www.python.org/doc/current/lib/module-copy.html
|
||
|
||
[3] PEP 289, Generator Expressions, Hettinger
|
||
http://www.python.org/peps/pep-0289.html
|
||
|
||
Copyright
|
||
|
||
This document has been placed in the public domain.
|
||
|
||
|
||
|
||
Local Variables:
|
||
mode: indented-text
|
||
indent-tabs-mode: nil
|
||
sentence-end-double-space: t
|
||
fill-column: 70
|
||
End:
|