422 lines
18 KiB
Plaintext
422 lines
18 KiB
Plaintext
PEP: 288
|
||
Title: Generators Attributes and Exceptions
|
||
Version: $Revision$
|
||
Last-Modified: $Date$
|
||
Author: python@rcn.com (Raymond D. Hettinger)
|
||
Status: Deferred
|
||
Type: Standards Track
|
||
Created: 21-Mar-2002
|
||
Python-Version: 2.4
|
||
Post-History:
|
||
|
||
|
||
Abstract
|
||
|
||
This PEP introduces ideas for enhancing the generators introduced
|
||
in Python version 2.2 [1]. The goal is to increase the
|
||
convenience, utility, and power of generators by providing a
|
||
mechanism for passing data into a generator and for triggering
|
||
exceptions inside a generator.
|
||
|
||
These mechanisms were first proposed along with two other
|
||
generator tools in PEP 279 [7]. They were split-off to this
|
||
separate PEP to allow the ideas more time to mature and for
|
||
alternatives to be considered.
|
||
|
||
|
||
Rationale
|
||
|
||
Python 2.2 introduced the concept of an iterable interface as
|
||
proposed in PEP 234 [2]. The iter() factory function was provided
|
||
as common calling convention and deep changes were made to use
|
||
iterators as a unifying theme throughout Python. The unification
|
||
came in the form of establishing a common iterable interface for
|
||
mappings, sequences, and file objects.
|
||
|
||
Generators, as proposed in PEP 255 [1], were introduced as a means for
|
||
making it easier to create iterators, especially ones with complex
|
||
internal execution or variable states.
|
||
|
||
The next step in the evolution of generators is to extend the
|
||
syntax of the 'yield' keyword to enable generator parameter
|
||
passing. The resulting increase in power simplifies the creation
|
||
of consumer streams which have a complex execution state and/or
|
||
variable state.
|
||
|
||
A better alternative being considered is to allow generators to
|
||
accept attribute assignments. This allows data to be passed in a
|
||
standard Python fashion.
|
||
|
||
A related evolutionary step is to add a generator method to enable
|
||
exceptions to be passed to a generator. Currently, there is no
|
||
clean method for triggering exceptions from outside the generator.
|
||
Also, generator exception passing helps mitigate the try/finally
|
||
prohibition for generators.
|
||
|
||
These suggestions are designed to take advantage of the existing
|
||
implementation and require little additional effort to
|
||
incorporate. They are backwards compatible and require no new
|
||
keywords. They are being recommended for Python version 2.4.
|
||
|
||
|
||
Reference Implementation
|
||
|
||
There is not currently a CPython implementation; however, a
|
||
simulation module written in pure Python is available on
|
||
SourceForge [5]. The simulation is meant to allow direct
|
||
experimentation with the proposal.
|
||
|
||
There is also a module [6] with working source code for all of the
|
||
examples used in this PEP. It serves as a test suite for the
|
||
simulator and it documents how each of the feature works in
|
||
practice.
|
||
|
||
The authors and implementers of PEP 255 [1] were contacted to
|
||
provide their assessment of whether the enhancement was going to
|
||
be straight-forward to implement and require only minor
|
||
modification of the existing generator code. Neil felt the
|
||
assertion was correct. Ka-Ping thought so also. GvR said he
|
||
could believe that it was true. Tim did not have an opportunity
|
||
to give an assessment.
|
||
|
||
|
||
Specification for Generator Parameter Passing
|
||
|
||
1. Allow 'yield' to assign a value as in:
|
||
|
||
def mygen():
|
||
while 1:
|
||
x = yield None
|
||
print x
|
||
|
||
2. Let the .next() method take a value to pass to the generator as in:
|
||
|
||
g = mygen()
|
||
g.next() # runs the generator until the first 'yield'
|
||
g.next(1) # '1' is bound to 'x' in mygen(), then printed
|
||
g.next(2) # '2' is bound to 'x' in mygen(), then printed
|
||
|
||
The control flow of 'yield' and 'next' is unchanged by this
|
||
proposal. The only change is that a value can be sent into the
|
||
generator. By analogy, consider the quality improvement from
|
||
GOSUB (which had no argument passing mechanism) to modern
|
||
procedure calls (which can pass in arguments and return values).
|
||
|
||
Most of the underlying machinery is already in place, only the
|
||
communication needs to be added by modifying the parse syntax to
|
||
accept the new 'x = yield expr' syntax and by allowing the .next()
|
||
method to accept an optional argument.
|
||
|
||
Yield is more than just a simple iterator creator. It does
|
||
something else truly wonderful -- it suspends execution and saves
|
||
state. It is good for a lot more than writing iterators. This
|
||
proposal further expands its capability by making it easier to
|
||
share data with the generator.
|
||
|
||
The .next(arg) mechanism is especially useful for:
|
||
1. Sending data to any generator
|
||
2. Writing lazy consumers with complex execution states
|
||
3. Writing co-routines (as demonstrated in Dr. Mertz's articles [3])
|
||
|
||
The proposal is a clear improvement over the existing alternative
|
||
of passing data via global variables. It is also much simpler,
|
||
more readable and easier to debug than an approach involving the
|
||
threading module with its attendant mutexes, semaphores, and data
|
||
queues. A class-based approach competes well when there are no
|
||
complex execution states or variable states. However, when the
|
||
complexity increases, generators with parameter passing are much
|
||
simpler because they automatically save state (unlike classes
|
||
which must explicitly save the variable and execution state in
|
||
instance variables).
|
||
|
||
Note A: This proposal changes 'yield' from a statement to an
|
||
expression with binding and precedence similar to lambda.
|
||
|
||
|
||
Examples
|
||
|
||
Example of a Complex Consumer
|
||
|
||
The encoder for arithmetic compression sends a series of
|
||
fractional values to a complex, lazy consumer. That consumer
|
||
makes computations based on previous inputs and only writes out
|
||
when certain conditions have been met. After the last fraction is
|
||
received, it has a procedure for flushing any unwritten data.
|
||
|
||
|
||
Example of a Consumer Stream
|
||
|
||
def filelike(packagename, appendOrOverwrite):
|
||
cum = []
|
||
if appendOrOverwrite == 'w+':
|
||
cum.extend(packages[packagename])
|
||
try:
|
||
while 1:
|
||
dat = yield None
|
||
cum.append(dat)
|
||
except FlushStream:
|
||
packages[packagename] = cum
|
||
|
||
ostream = filelike('mydest','w') # Analogous to file.open(name,flag)
|
||
ostream.next() # Advance to the first yield
|
||
ostream.next(firstdat) # Analogous to file.write(dat)
|
||
ostream.next(seconddat)
|
||
ostream.throw(FlushStream) # Throw is proposed below
|
||
|
||
|
||
Example of a Complex Consumer
|
||
|
||
Loop over the picture files in a directory, shrink them one at a
|
||
time to thumbnail size using PIL [4], and send them to a lazy
|
||
consumer. That consumer is responsible for creating a large blank
|
||
image, accepting thumbnails one at a time and placing them in a 5
|
||
by 3 grid format onto the blank image. Whenever the grid is full,
|
||
it writes-out the large image as an index print. A FlushStream
|
||
exception indicates that no more thumbnails are available and that
|
||
the partial index print should be written out if there are one or
|
||
more thumbnails on it.
|
||
|
||
|
||
Example of a Producer and Consumer Used Together in a Pipe-like Fashion
|
||
|
||
'Analogy to Linux style pipes: source | upper | sink'
|
||
sink = sinkgen()
|
||
sink.next()
|
||
for word in source():
|
||
sink.next(word.upper())
|
||
|
||
|
||
Comments
|
||
|
||
Comments from GvR: We discussed this at length when we were hashing
|
||
out generators and coroutines, and found that there's always a
|
||
problem with this: the argument to the first next() call has
|
||
to be thrown away, because it doesn't correspond to a yield
|
||
statement. This looks ugly (note that the example code has a
|
||
dummy call to next() to get the generator going). But there
|
||
may be useful examples that can only be programmed (elegantly)
|
||
with this feature, so I'm reserving judgment. I can believe
|
||
that it's easy to implement.
|
||
|
||
Comments from Ka-Ping Yee: I also think there is a lot of power to be
|
||
gained from generator argument passing.
|
||
|
||
Comments from Neil Schemenauer: I like the idea of being able to pass
|
||
values back into a generator. I originally pitched this idea
|
||
to Guido but in the end we decided against it (at least for
|
||
the initial implementation). There was a few issues to work
|
||
out but I can't seem to remember what they were. My feeling
|
||
is that we need to wait until the Python community has more
|
||
experience with generators before adding this feature. Maybe
|
||
for 2.4 but not for 2.3. In the mean time you can work around
|
||
this limitation by making your generator a method. Values can
|
||
be passed back by mutating the instance.
|
||
|
||
Comments for Magnus Lie Hetland: I like the generator parameter
|
||
passing mechanism. Although I see no need to defer it,
|
||
deferral seems to be the most likely scenario, and in the
|
||
meantime I guess the functionality can be emulated either by
|
||
implementing the generator as a method, or by passing a
|
||
parameter with the exception passing mechanism.
|
||
|
||
Author response: Okay, consider this proposal deferred until version 2.4
|
||
so the idea can fully mature. I am currently teasing out two
|
||
alternatives which may eliminate the issue with the initial
|
||
next() call not having a corresponding yield.
|
||
|
||
|
||
Alternative 1: Submit
|
||
|
||
Instead of next(arg), use a separate method, submit(arg).
|
||
Submit would behave just like next() except that on the first
|
||
call, it will call next() twice. The word 'submit' has the
|
||
further advantage of being explicit in its intent. It also
|
||
allows checking for the proper number of arguments (next
|
||
always has zero and submit always has one). Using this
|
||
alternative, the call to the consumer stream looks like this:
|
||
|
||
ostream = filelike('mydest','w')
|
||
ostream.submit(firstdat) # No call to next is needed
|
||
ostream.submit(seconddat)
|
||
ostream.throw(FlushStream) # Throw is proposed below
|
||
|
||
|
||
Alternative 2: Generator Attributes
|
||
|
||
Instead of generator parameter passing, enable writable
|
||
generator attributes: g.data=firstdat; g.next(). The code on
|
||
the receiving end is written knowing that the attribute is set
|
||
from the very beginning. This solves the problem because the
|
||
first next call does not need to be associated with a yield
|
||
statement.
|
||
|
||
This solution uses a standard Python tool, object attributes,
|
||
in a standard way. It is also explicit in its intention and
|
||
provides some error checking (the receiving code raises an
|
||
AttributeError if the expected field has not be set before the
|
||
call).
|
||
|
||
The one unclean part of this approach is that the generator
|
||
needs some way to reference itself (something parallel to the
|
||
use of the function name in a recursive function or to the use
|
||
of 'self' in a method). The only way I can think of is to
|
||
introduce a new system variable, __self__, in any function
|
||
that employs a yield statement. Using this alternative, the
|
||
code for the consumer stream looks like this:
|
||
|
||
def filelike(packagename, appendOrOverwrite):
|
||
cum = []
|
||
if appendOrOverwrite == 'w+':
|
||
cum.extend(packages[packagename])
|
||
try:
|
||
while 1:
|
||
cum.append(__self__.dat)
|
||
yield None
|
||
except FlushStream:
|
||
packages[packagename] = cum
|
||
|
||
ostream = filelike('mydest','w')
|
||
ostream.dat = firstdat; ostream.next()
|
||
ostream.dat = firstdat; ostream.next()
|
||
ostream.throw(FlushStream) # Throw is proposed in PEP 279
|
||
|
||
|
||
Specification for Generator Exception Passing:
|
||
|
||
Add a .throw(exception) method to the generator interface:
|
||
|
||
def logger():
|
||
start = time.time()
|
||
log = []
|
||
try:
|
||
while 1:
|
||
log.append( time.time() - start )
|
||
yield log[-1]
|
||
except WriteLog:
|
||
writelog(log)
|
||
|
||
g = logger()
|
||
for i in [10,20,40,80,160]:
|
||
testsuite(i)
|
||
g.next()
|
||
g.throw(WriteLog)
|
||
|
||
There is no existing work-around for triggering an exception
|
||
inside a generator. This is a true deficiency. It is the only
|
||
case in Python where active code cannot be excepted to or through.
|
||
|
||
Generator exception passing also helps address an intrinsic
|
||
limitation on generators, the prohibition against their using
|
||
try/finally to trigger clean-up code [1]. Without .throw(), the
|
||
current work-around forces the resolution or clean-up code to be
|
||
moved outside the generator.
|
||
|
||
Note A: The name of the throw method was selected for several
|
||
reasons. Raise is a keyword and so cannot be used as a method
|
||
name. Unlike raise which immediately raises an exception from the
|
||
current execution point, throw will first return to the generator
|
||
and then raise the exception. The word throw is suggestive of
|
||
putting the exception in another location. The word throw is
|
||
already associated with exceptions in other languages.
|
||
|
||
Alternative method names were considered: resolve(), signal(),
|
||
genraise(), raiseinto(), and flush(). None of these seem to fit
|
||
as well as throw().
|
||
|
||
Note B: The throw syntax should exactly match raise's syntax:
|
||
|
||
throw([expression, [expression, [expression]]])
|
||
|
||
Accordingly, it should be implemented to handle all of the following:
|
||
|
||
raise string g.throw(string)
|
||
raise string, data g.throw(string,data)
|
||
raise class, instance g.throw(class,instance)
|
||
raise instance g.throw(instance)
|
||
raise g.throw()
|
||
|
||
|
||
Comments from GvR: I'm not convinced that the cleanup problem that
|
||
this is trying to solve exists in practice. I've never felt
|
||
the need to put yield inside a try/except. I think the PEP
|
||
doesn't make enough of a case that this is useful.
|
||
|
||
This one gets a big fat -1 until there's a good motivational
|
||
section.
|
||
|
||
Comments from Ka-Ping Yee: I agree that the exception issue needs to
|
||
be resolved and [that] you have suggested a fine solution.
|
||
|
||
Comments from Neil Schemenauer: The exception passing idea is one I
|
||
hadn't thought of before and looks interesting. If we enable
|
||
the passing of values back, then we should add this feature
|
||
too.
|
||
|
||
Comments for Magnus Lie Hetland: Even though I cannot speak for the
|
||
ease of implementation, I vote +1 for the exception passing
|
||
mechanism.
|
||
|
||
Comments from the Community: The response has been mostly favorable. One
|
||
negative comment from GvR is shown above. The other was from
|
||
Martin von Loewis who was concerned that it could be difficult
|
||
to implement and is withholding his support until a working
|
||
patch is available. To probe Martin's comment, I checked with
|
||
the implementers of the original generator PEP for an opinion
|
||
on the ease of implementation. They felt that implementation
|
||
would be straight-forward and could be grafted onto the
|
||
existing implementation without disturbing its internals.
|
||
|
||
Author response: When the sole use of generators is to simplify writing
|
||
iterators for lazy producers, then the odds of needing
|
||
generator exception passing are slim. If, on the other hand,
|
||
generators are used to write lazy consumers, create
|
||
coroutines, generate output streams, or simply for their
|
||
marvelous capability for restarting a previously frozen state,
|
||
THEN the need to raise exceptions will come up frequently.
|
||
|
||
I'm no judge of what is truly Pythonic, but am still
|
||
astonished that there can exist blocks of code that can't be
|
||
excepted to or through, that the try/finally combination is
|
||
blocked, and that the only work-around is to rewrite as a
|
||
class and move the exception code out of the function or
|
||
method being excepted.
|
||
|
||
|
||
References
|
||
|
||
[1] PEP 255 Simple Generators
|
||
http://www.python.org/peps/pep-0255.html
|
||
|
||
[2] PEP 234 Iterators
|
||
http://www.python.org/peps/pep-0234.html
|
||
|
||
[3] Dr. David Mertz's draft column for Charming Python.
|
||
http://gnosis.cx/publish/programming/charming_python_b5.txt
|
||
http://gnosis.cx/publish/programming/charming_python_b7.txt
|
||
|
||
[4] PIL, the Python Imaging Library can be found at:
|
||
http://www.pythonware.com/products/pil/
|
||
|
||
[5] A pure Python simulation of every feature in this PEP is at:
|
||
http://sourceforge.net/tracker/download.php?group_id=5470&atid=305470&file_id=17348&aid=513752
|
||
|
||
[6] The full, working source code for each of the examples in this PEP
|
||
along with other examples and tests is at:
|
||
http://sourceforge.net/tracker/download.php?group_id=5470&atid=305470&file_id=17412&aid=513756
|
||
|
||
[7] PEP 279 Enhanced Generators
|
||
http://www.python.org/peps/pep-0279.html
|
||
|
||
|
||
Copyright
|
||
|
||
This document has been placed in the public domain.
|
||
|
||
|
||
|
||
Local Variables:
|
||
mode: indented-text
|
||
indent-tabs-mode: nil
|
||
fill-column: 70
|
||
End:
|