iSharkFly-Open
/
python-peps
mirror of https://github.com/python/peps.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Specify generator enhancements. Change keyword to 'with'.

This commit is contained in:
Guido van Rossum 2005-06-01 15:13:37 +00:00
parent 0c4151b242
commit a942512355
1 changed files with 236 additions and 117 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

353
pep-0343.txt
View File

@ -1,5 +1,5 @@
PEP: 343
Title: Anonymous Block Redux
Title: Anonymous Block Redux and Generator Enhancements
Version: $Revision$
Last-Modified: $Date$
Author: Guido van Rossum
@ -11,17 +11,13 @@ Post-History:
Introduction
After a lot of discussion about PEP 340 and alternatives, I've
decided to withdraw PEP 340 and propose a slight variant on
PEP 310.
Evolutionary Note
After ample discussion on python-dev, I'll add back a mechanism
After a lot of discussion about PEP 340 and alternatives, I
decided to withdraw PEP 340 and proposed a slight variant on PEP
310. After more discussion, I have added back a mechanism
for raising an exception in a suspended generator using a throw()
method, and a close() method which throws a new GeneratorExit
exception. Until I get a chance to update the PEP, see reference
[2]. I'm also leaning towards 'with' as the keyword.
exception; these additions were first proposed in [2] and
universally approved of. I'm also changing the keyword to 'with'.
Motivation and Summary
@ -53,7 +49,7 @@ Motivation and Summary
with VAR = EXPR:
BLOCK
which roughly translates into
which roughly translates into this:
VAR = EXPR
VAR.__enter__()
@ -74,7 +70,7 @@ Motivation and Summary
goto (a break, continue or return), BLOCK2 is *not* reached. The
magic added by the with-statement at the end doesn't affect this.
(You may ask, what if a bug in the __exit__ method causes an
(You may ask, what if a bug in the __exit__() method causes an
exception? Then all is lost -- but this is no worse than with
other exceptions; the nature of exceptions is that they can happen
*anywhere*, and you just have to live with that. Even if you
@ -89,16 +85,18 @@ Motivation and Summary
Inspired by a counter-proposal to PEP 340 by Phillip Eby I tried
to create a decorator that would turn a suitable generator into an
object with the necessary __entry__ and __exit__ methods. Here I
ran into a snag: while it wasn't too hard for the locking example,
it was impossible to do this for the opening example. The idea
was to define the template like this:
object with the necessary __enter__() and __exit__() methods.
Here I ran into a snag: while it wasn't too hard for the locking
example, it was impossible to do this for the opening example.
The idea was to define the template like this:
@with_template
def opening(filename):
f = open(filename)
yield f
f.close()
try:
yield f
finally:
f.close()
and used it like this:
@ -106,21 +104,21 @@ Motivation and Summary
...read data from f...
The problem is that in PEP 310, the result of calling EXPR is
assigned directly to VAR, and then VAR's __exit__ method is called
upon exit from BLOCK1. But here, VAR clearly needs to receive the
opened file, and that would mean that __exit__ would have to be a
method on the file.
assigned directly to VAR, and then VAR's __exit__() method is
called upon exit from BLOCK1. But here, VAR clearly needs to
receive the opened file, and that would mean that __exit__() would
have to be a method on the file.
While this can be solved using a proxy class, this is awkward and
made me realize that a slightly different translation would make
writing the desired decorator a piece of cake: let VAR receive the
result from calling the __enter__ method, and save the value of
EXPR to call its __exit__ method later. Then the decorator can
return an instance of a wrapper class whose __enter__ method calls
the generator's next() method and returns whatever next() returns;
the wrapper instance's __exit__ method calls next() again but
expects it to raise StopIteration. (Details below in the section
Optional Generator Decorator.)
result from calling the __enter__() method, and save the value of
EXPR to call its __exit__() method later. Then the decorator can
return an instance of a wrapper class whose __enter__() method
calls the generator's next() method and returns whatever next()
returns; the wrapper instance's __exit__() method calls next()
again but expects it to raise StopIteration. (Details below in
the section Optional Generator Decorator.)
So now the final hurdle was that the PEP 310 syntax:
@ -128,41 +126,61 @@ Motivation and Summary
BLOCK1
would be deceptive, since VAR does *not* receive the value of
EXPR. Given PEP 340, it was an easy step to:
EXPR. Borrowing from PEP 340, it was an easy step to:
with EXPR as VAR:
BLOCK1
or, using an alternate keyword that has been proposed a number of
times:
Additional discussion showed that people really liked being able
to "see" the exception in the generator, even if it was only to
log it; the generator is not allowed to yield another value, since
the with-statement should not be usable as a loop (raising a
different exception is marginally acceptable). To enable this, a
new throw() method for generators is proposed, which takes three
arguments representing an exception in the usual fashion (type,
value, traceback) and raises it at the point where the generator
is suspended.
do EXPR as VAR:
BLOCK1
Once we have this, it is a small step to proposing another
generator method, close(), which calls throw() with a special
exception, GeneratorExit. This tells the generator to exit, and
from there it's another small step to proposing that close() be
called automatically when the generator is garbage-collected.
Then, finally, we can allow a yield-statement inside a try-finally
statement, since we can now guarantee that the finally-clause will
(eventually) be executed. The usual cautions about finalization
apply -- the process may be terminated abruptly without finalizing
any objects, and objects may be kept alive forever by cycles or
memory leaks in the application (as opposed to cycles or leaks in
the Python implementation, which are taken care of by GC).
Note that we're not guaranteeing that the finally-clause is
executed immediately after the generator object becomes unused,
even though this is how it will work in CPython. This is similar
to auto-closing files: while a reference-counting implementation
like CPython deallocates an object as soon as the last reference
to it goes away, implementations that use other GC algorithms do
not make the same guarantee. This applies to Jython, IronPython,
and probably to Python running on Parrot.
Use Cases
See the Examples section near the end.
Specification
Specification: The 'with' Statement
A new statement is proposed with the syntax:
do EXPR as VAR:
with EXPR as VAR:
BLOCK
Here, 'do' and 'as' are new keywords; EXPR is an arbitrary
Here, 'with' and 'as' are new keywords; EXPR is an arbitrary
expression (but not an expression-list) and VAR is an arbitrary
assignment target (which may be a comma-separated list).
The "as VAR" part is optional.
The choice of the 'do' keyword is provisional; an alternative
under consideration is 'with'.
A yield-statement is illegal inside BLOCK. This is because the
do-statement is translated into a try/finally statement, and yield
is illegal in a try/finally statement.
The translation of the above statement is:
abc = EXPR
@ -209,94 +227,184 @@ Specification
non-local goto should be considered unexceptional for the purposes
of a database transaction roll-back decision.
Optional Generator Decorator
Specification: Generator Enhancements
Let a generator object be the iterator produced by calling a
generator function. Below, 'g' always refers to a generator
object.
New syntax: yield allowed inside try-finally
The syntax for generator functions is extended to allow a
yield-statement inside a try-finally statement.
New generator method: throw(type, value, traceback)
g.throw(type, value, traceback) causes the specified exception to
be thrown at the point where the generator g is currently
suspended (i.e. at a yield-statement, or at the start of its
function body if next() has not been called yet). If the
generator catches the exception and yields another value, that is
the return value of g.throw(). If it doesn't catch the exception,
the throw() appears to raise the same exception passed it (it
"falls through"). If the generator raises another exception (this
includes the StopIteration produced when it returns) that
exception is raised by the throw() call. In summary, throw()
behaves like next() except it raises an exception at the
suspension point. If the generator is already in the closed
state, throw() just raises the exception it was passed without
executing any of the generator's code.
The effect of raising the exception is exactly as if the
statement:
raise type, value, traceback
was executed at the suspension point. The type argument should
not be None.
New standard exception: GeneratorExit
A new standard exception is defined, GeneratorExit, inheriting
from Exception. A generator should handle this by re-raising it
or by raising StopIteration.
New generator method: close()
g.close() is defined by the following pseudo-code:
def close(self):
try:
self.throw(GeneratorExit, GeneratorExit(), None)
except (GeneratorExit, StopIteration):
pass
else:
raise TypeError("generator ignored GeneratorExit")
# Other exceptions are not caught
New generator method: __del__()
g.__del__() is an alias for g.close(). This will be called when
the generator object is garbage-collected (in CPython, this is
when its reference count goes to zero). If close() raises an
exception, a traceback for the exception is printed to sys.stderr
and further ignored; it is not propagated back to the place that
triggered the garbage collection. This is consistent with the
handling of exceptions in __del__() methods on class instances.
If the generator object participates in a cycle, g.__del__() may
not be called. This is the behavior of CPython's current garbage
collector. The reason for the restriction is that the GC code
needs to "break" a cycle at an arbitrary point in order to collect
it, and from then on no Python code should be allowed to see the
objects that formed the cycle, as they may be in an invalid state.
Objects "hanging off" a cycle are not subject to this restriction.
Note that it is unlikely to see a generator object participate in
a cycle in practice. However, storing a generator object in a
global variable creates a cycle via the generator frame's
f_globals pointer. Another way to create a cycle would be to
store a reference to the generator object in a data structure that
is passed to the generator as an argument. Neither of these cases
are very likely given the typical pattern of generator use.
Generator Decorator
It is possible to write a decorator that makes it possible to use
a generator that yields exactly once to control a do-statement.
a generator that yields exactly once to control a with-statement.
Here's a sketch of such a decorator:
class Wrapper(object):
def __init__(self, gen):
self.gen = gen
self.state = "initial"
def __enter__(self):
assert self.state == "initial"
self.state = "entered"
try:
return self.gen.next()
except StopIteration:
self.state = "error"
raise RuntimeError("template generator didn't yield")
def __exit__(self, *args):
assert self.state == "entered"
self.state = "exited"
try:
self.gen.next()
except StopIteration:
return
else:
self.state = "error"
raise RuntimeError("template generator didn't stop")
raise RuntimeError("generator didn't yield")
def do_template(func):
def __exit__(self, type, value, traceback):
if type is None:
try:
self.gen.next()
except StopIteration:
return
else:
raise RuntimeError("generator didn't stop")
else:
try:
self.gen.throw(type, value, traceback)
except (type, StopIteration):
return
else:
raise RuntimeError("generator caught exception")
def with_template(func):
def helper(*args, **kwds):
return Wrapper(func(*args, **kwds))
return helper
This decorator could be used as follows:
@do_template
@with_template
def opening(filename):
f = open(filename) # IOError here is untouched by Wrapper
yield f
f.close() # Ditto for errors here (however unlikely)
A robust implementation of such a decorator should be made part of
the standard library.
A robust implementation of this decorator should be made part of
the standard library, but not necessarily as a built-in function.
(I'm not sure which exception it should raise for errors;
RuntimeError is used above as an example only.)
Other Optional Extensions
Optional Extensions
It would be possible to endow certain objects, like files,
sockets, and locks, with __enter__ and __exit__ methods so that
instead of writing:
sockets, and locks, with __enter__() and __exit__() methods so
that instead of writing:
do locking(myLock):
with locking(myLock):
BLOCK
one could write simply:
do myLock:
with myLock:
BLOCK
I think we should be careful with this; it could lead to mistakes
like:
f = open(filename)
do f:
with f:
BLOCK1
do f:
with f:
BLOCK2
which does not do what one might think (f is closed when BLOCK2 is
entered).
which does not do what one might think (f is closed before BLOCK2
is entered).
OTOH such mistakes are easily diagnosed.
Examples
Several of these examples contain "yield None". If PEP 342 is
accepted, these can be changed to just "yield".
(Note: several of these examples contain "yield None". If PEP 342
is accepted, these can be changed to just "yield".)
1. A template for ensuring that a lock, acquired at the start of a
block, is released when the block is left:
@do_template
@with_template
def locking(lock):
lock.acquire()
yield None
lock.release()
try:
yield None
finally:
lock.release()
Used as follows:
do locking(myLock):
with locking(myLock):
# Code here executes with myLock held. The lock is
# guaranteed to be released when the block is left (even
# if via return or by an uncaught exception).
@ -304,29 +412,32 @@ Examples
2. A template for opening a file that ensures the file is closed
when the block is left:
@do_template
@with_template
def opening(filename, mode="r"):
f = open(filename, mode)
yield f
f.close()
try:
yield f
finally:
f.close()
Used as follows:
do opening("/etc/passwd") as f:
with opening("/etc/passwd") as f:
for line in f:
print line.rstrip()
3. A template for committing or rolling back a database
transaction; this is written as a class rather than as a
decorator since it requires access to the exception information:
decorator since it requires access to the exception
information:
class transactional:
def __init__(self, db):
self.db = db
def __enter__(self):
self.db.begin()
def __exit__(self, *args):
if args and args[0] is not None:
def __exit__(self, type, value, tb):
if type is not None:
self.db.rollback()
else:
self.db.commit()
@ -338,47 +449,51 @@ Examples
self.lock = lock
def __enter__(self):
self.lock.acquire()
def __exit__(self, *args):
def __exit__(self, type, value, tb):
self.lock.release()
(This example is easily modified to implement the other
examples; it shows how much simpler generators are for the same
purpose.)
examples; it shows the relative advantage of using a generator
template.)
5. Redirect stdout temporarily:
@do_template
@with_template
def redirecting_stdout(new_stdout):
save_stdout = sys.stdout
sys.stdout = new_stdout
yield None
sys.stdout = save_stdout
try:
yield None
finally:
sys.stdout = save_stdout
Used as follows:
do opening(filename, "w") as f:
do redirecting_stdout(f):
with opening(filename, "w") as f:
with redirecting_stdout(f):
print "Hello world"
This isn't thread-safe, of course, but neither is doing this
same dance manually. In a single-threaded program (e.g., a
script) it is a totally fine way of doing things.
same dance manually. In single-threaded programs (for example,
in scripts) it is a popular way of doing things.
6. A variant on opening() that also returns an error condition:
@do_template
@with_template
def opening_w_error(filename, mode="r"):
try:
f = open(filename, mode)
except IOError, err:
yield None, err
else:
yield f, None
f.close()
try:
yield f, None
finally:
f.close()
Used as follows:
do opening_w_error("/etc/passwd", "a") as f, err:
with opening_w_error("/etc/passwd", "a") as f, err:
if err:
print "IOError:", err
else:
@ -389,7 +504,7 @@ Examples
import signal
do signal.blocking():
with signal.blocking():
# code executed without worrying about signals
An optional argument might be a list of signals to be blocked;
@ -401,19 +516,21 @@ Examples
import decimal
@do_template
def with_extra_precision(places=2):
@with_template
def extra_precision(places=2):
c = decimal.getcontext()
saved_prec = c.prec
c.prec += places
yield None
c.prec = saved_prec
try:
yield None
finally:
c.prec = saved_prec
Sample usage (adapted from the Python Library Reference):
def sin(x):
"Return the sine of x as measured in radians."
do with_extra_precision():
with extra_precision():
i, lasts, s, fact, num, sign = 1, 0, x, 1, x, 1
while s != lasts:
lasts = s
@ -423,31 +540,33 @@ Examples
sign *= -1
s += num / fact * sign
# The "+s" rounds back to the original precision,
# so this must be outside the do-statement:
# so this must be outside the with-statement:
return +s
9. Here's a more general Decimal-context-switching template:
@do_template
def with_decimal_context(newctx=None):
@with_template
def decimal_context(newctx=None):
oldctx = decimal.getcontext()
if newctx is None:
newctx = oldctx.copy()
decimal.setcontext(newctx)
yield newctx
decimal.setcontext(oldctx)
try:
yield newctx
finally:
decimal.setcontext(oldctx)
Sample usage (adapted from the previous one):
Sample usage:
def sin(x):
do with_decimal_context() as ctx:
with decimal_context() as ctx:
ctx.prec += 2
# Rest of algorithm the same
# Rest of algorithm the same as above
return +s
(Nick Coghlan has proposed to add __enter__ and __exit__
(Nick Coghlan has proposed to add __enter__() and __exit__()
methods to the decimal.Context class so that this example can
be simplified to "do decimal.getcontext() as ctx: ...".)
be simplified to "with decimal.getcontext() as ctx: ...".)
References