582 lines
21 KiB
Plaintext
582 lines
21 KiB
Plaintext
PEP: 343
|
|
Title: Anonymous Block Redux and Generator Enhancements
|
|
Version: $Revision$
|
|
Last-Modified: $Date$
|
|
Author: Guido van Rossum
|
|
Status: Accepted
|
|
Type: Standards Track
|
|
Content-Type: text/plain
|
|
Created: 13-May-2005
|
|
Post-History: 2-Jun-2005
|
|
|
|
Introduction
|
|
|
|
After a lot of discussion about PEP 340 and alternatives, I
|
|
decided to withdraw PEP 340 and proposed a slight variant on PEP
|
|
310. After more discussion, I have added back a mechanism for
|
|
raising an exception in a suspended generator using a throw()
|
|
method, and a close() method which throws a new GeneratorExit
|
|
exception; these additions were first proposed on python-dev in
|
|
[2] and universally approved of. I'm also changing the keyword to
|
|
'with'.
|
|
|
|
On-line discussion of this PEP should take place in the Python
|
|
Wiki [3].
|
|
|
|
If this PEP is approved, the following PEPs will be rejected due
|
|
to overlap:
|
|
|
|
- PEP 310, Reliable Acquisition/Release Pairs. This is the
|
|
original with-statement proposal.
|
|
|
|
- PEP 319, Python Synchronize/Asynchronize Block. Its use cases
|
|
can be covered by the current PEP by providing suitable
|
|
with-statement controllers: for 'synchronize' we can use the
|
|
"locking" template from example 1; for 'asynchronize' we can use
|
|
a similar "unlocking" template. I don't think having an
|
|
"anonymous" lock associated with a code block is all that
|
|
important; in fact it may be better to always be explicit about
|
|
the mutex being used.
|
|
|
|
(PEP 340 and PEP 346 have already been withdrawn.)
|
|
|
|
Motivation and Summary
|
|
|
|
PEP 340, Anonymous Block Statements, combined many powerful ideas:
|
|
using generators as block templates, adding exception handling and
|
|
finalization to generators, and more. Besides praise it received
|
|
a lot of opposition from people who didn't like the fact that it
|
|
was, under the covers, a (potential) looping construct. This
|
|
meant that break and continue in a block-statement would break or
|
|
continue the block-statement, even if it was used as a non-looping
|
|
resource management tool.
|
|
|
|
But the final blow came when I read Raymond Chen's rant about
|
|
flow-control macros[1]. Raymond argues convincingly that hiding
|
|
flow control in macros makes your code inscrutable, and I find
|
|
that his argument applies to Python as well as to C. I realized
|
|
that PEP 340 templates can hide all sorts of control flow; for
|
|
example, its example 4 (auto_retry()) catches exceptions and
|
|
repeats the block up to three times.
|
|
|
|
However, the with-statement of PEP 310 does *not* hide control
|
|
flow, in my view: while a finally-suite temporarily suspends the
|
|
control flow, in the end, the control flow resumes as if the
|
|
finally-suite wasn't there at all.
|
|
|
|
Remember, PEP 310 proposes rougly this syntax (the "VAR =" part is
|
|
optional):
|
|
|
|
with VAR = EXPR:
|
|
BLOCK
|
|
|
|
which roughly translates into this:
|
|
|
|
VAR = EXPR
|
|
VAR.__enter__()
|
|
try:
|
|
BLOCK
|
|
finally:
|
|
VAR.__exit__()
|
|
|
|
Now consider this example:
|
|
|
|
with f = opening("/etc/passwd"):
|
|
BLOCK1
|
|
BLOCK2
|
|
|
|
Here, just as if the first line was "if True" instead, we know
|
|
that if BLOCK1 completes without an exception, BLOCK2 will be
|
|
reached; and if BLOCK1 raises an exception or executes a non-local
|
|
goto (a break, continue or return), BLOCK2 is *not* reached. The
|
|
magic added by the with-statement at the end doesn't affect this.
|
|
|
|
(You may ask, what if a bug in the __exit__() method causes an
|
|
exception? Then all is lost -- but this is no worse than with
|
|
other exceptions; the nature of exceptions is that they can happen
|
|
*anywhere*, and you just have to live with that. Even if you
|
|
write bug-free code, a KeyboardInterrupt exception can still cause
|
|
it to exit between any two virtual machine opcodes.)
|
|
|
|
This argument almost led me to endorse PEP 310, but I had one idea
|
|
left from the PEP 340 euphoria that I wasn't ready to drop: using
|
|
generators as "templates" for abstractions like acquiring and
|
|
releasing a lock or opening and closing a file is a powerful idea,
|
|
as can be seen by looking at the examples in that PEP.
|
|
|
|
Inspired by a counter-proposal to PEP 340 by Phillip Eby I tried
|
|
to create a decorator that would turn a suitable generator into an
|
|
object with the necessary __enter__() and __exit__() methods.
|
|
Here I ran into a snag: while it wasn't too hard for the locking
|
|
example, it was impossible to do this for the opening example.
|
|
The idea was to define the template like this:
|
|
|
|
@contextmanager
|
|
def opening(filename):
|
|
f = open(filename)
|
|
try:
|
|
yield f
|
|
finally:
|
|
f.close()
|
|
|
|
and used it like this:
|
|
|
|
with f = opening(filename):
|
|
...read data from f...
|
|
|
|
The problem is that in PEP 310, the result of calling EXPR is
|
|
assigned directly to VAR, and then VAR's __exit__() method is
|
|
called upon exit from BLOCK1. But here, VAR clearly needs to
|
|
receive the opened file, and that would mean that __exit__() would
|
|
have to be a method on the file.
|
|
|
|
While this can be solved using a proxy class, this is awkward and
|
|
made me realize that a slightly different translation would make
|
|
writing the desired decorator a piece of cake: let VAR receive the
|
|
result from calling the __enter__() method, and save the value of
|
|
EXPR to call its __exit__() method later. Then the decorator can
|
|
return an instance of a wrapper class whose __enter__() method
|
|
calls the generator's next() method and returns whatever next()
|
|
returns; the wrapper instance's __exit__() method calls next()
|
|
again but expects it to raise StopIteration. (Details below in
|
|
the section Optional Generator Decorator.)
|
|
|
|
So now the final hurdle was that the PEP 310 syntax:
|
|
|
|
with VAR = EXPR:
|
|
BLOCK1
|
|
|
|
would be deceptive, since VAR does *not* receive the value of
|
|
EXPR. Borrowing from PEP 340, it was an easy step to:
|
|
|
|
with EXPR as VAR:
|
|
BLOCK1
|
|
|
|
Additional discussion showed that people really liked being able
|
|
to "see" the exception in the generator, even if it was only to
|
|
log it; the generator is not allowed to yield another value, since
|
|
the with-statement should not be usable as a loop (raising a
|
|
different exception is marginally acceptable). To enable this, a
|
|
new throw() method for generators is proposed, which takes one to
|
|
three arguments representing an exception in the usual fashion
|
|
(type, value, traceback) and raises it at the point where the
|
|
generator is suspended.
|
|
|
|
Once we have this, it is a small step to proposing another
|
|
generator method, close(), which calls throw() with a special
|
|
exception, GeneratorExit. This tells the generator to exit, and
|
|
from there it's another small step to proposing that close() be
|
|
called automatically when the generator is garbage-collected.
|
|
|
|
Then, finally, we can allow a yield-statement inside a try-finally
|
|
statement, since we can now guarantee that the finally-clause will
|
|
(eventually) be executed. The usual cautions about finalization
|
|
apply -- the process may be terminated abruptly without finalizing
|
|
any objects, and objects may be kept alive forever by cycles or
|
|
memory leaks in the application (as opposed to cycles or leaks in
|
|
the Python implementation, which are taken care of by GC).
|
|
|
|
Note that we're not guaranteeing that the finally-clause is
|
|
executed immediately after the generator object becomes unused,
|
|
even though this is how it will work in CPython. This is similar
|
|
to auto-closing files: while a reference-counting implementation
|
|
like CPython deallocates an object as soon as the last reference
|
|
to it goes away, implementations that use other GC algorithms do
|
|
not make the same guarantee. This applies to Jython, IronPython,
|
|
and probably to Python running on Parrot.
|
|
|
|
Use Cases
|
|
|
|
See the Examples section near the end.
|
|
|
|
Specification: The 'with' Statement
|
|
|
|
A new statement is proposed with the syntax:
|
|
|
|
with EXPR as VAR:
|
|
BLOCK
|
|
|
|
Here, 'with' and 'as' are new keywords; EXPR is an arbitrary
|
|
expression (but not an expression-list) and VAR is a single
|
|
assignment target. It can *not* be a comma-separated sequence of
|
|
variables, but it *can* be a *parenthesized* comma-separated
|
|
sequence of variables. (This restriction makes a future extension
|
|
possible of the syntax to have multiple comma-separated resources,
|
|
each with its own optional as-clause.)
|
|
|
|
The "as VAR" part is optional.
|
|
|
|
The translation of the above statement is:
|
|
|
|
abc = EXPR
|
|
exc = (None, None, None)
|
|
VAR = abc.__enter__()
|
|
try:
|
|
try:
|
|
BLOCK
|
|
except:
|
|
exc = sys.exc_info()
|
|
raise
|
|
finally:
|
|
abc.__exit__(*exc)
|
|
|
|
Here, the variables 'abc' and 'exc' are internal variables and not
|
|
accessible to the user; they will most likely be implemented as
|
|
special registers or stack positions.
|
|
|
|
If the "as VAR" part of the syntax is omitted, the "VAR =" part of
|
|
the translation is omitted (but abc.__enter__() is still called).
|
|
|
|
The calling convention for abc.__exit__() is as follows. If the
|
|
finally-suite was reached through normal completion of BLOCK or
|
|
through a non-local goto (a break, continue or return statement in
|
|
BLOCK), abc.__exit__() is called with three None arguments. If
|
|
the finally-suite was reached through an exception raised in
|
|
BLOCK, abc.__exit__() is called with three arguments representing
|
|
the exception type, value, and traceback.
|
|
|
|
The motivation for this API to __exit__(), as opposed to the
|
|
argument-less __exit__() from PEP 310, was given by the
|
|
transactional() use case, example 3 below. The template in that
|
|
example must commit or roll back the transaction depending on
|
|
whether an exception occurred or not. Rather than just having a
|
|
boolean flag indicating whether an exception occurred, we pass the
|
|
complete exception information, for the benefit of an
|
|
exception-logging facility for example. Relying on sys.exc_info()
|
|
to get at the exception information was rejected; sys.exc_info()
|
|
has very complex semantics and it is perfectly possible that it
|
|
returns the exception information for an exception that was caught
|
|
ages ago. It was also proposed to add an additional boolean to
|
|
distinguish between reaching the end of BLOCK and a non-local
|
|
goto. This was rejected as too complex and unnecessary; a
|
|
non-local goto should be considered unexceptional for the purposes
|
|
of a database transaction roll-back decision.
|
|
|
|
Generator Decorator
|
|
|
|
If PEP 342 is accepted, it will be possible to write a decorator
|
|
that makes it possible to use a generator that yields exactly once
|
|
to control a with-statement. Here's a sketch of such a decorator:
|
|
|
|
class ContextWrapper(object):
|
|
|
|
def __init__(self, gen):
|
|
self.gen = gen
|
|
|
|
def __enter__(self):
|
|
try:
|
|
return self.gen.next()
|
|
except StopIteration:
|
|
raise RuntimeError("generator didn't yield")
|
|
|
|
def __exit__(self, type, value, traceback):
|
|
if type is None:
|
|
try:
|
|
self.gen.next()
|
|
except StopIteration:
|
|
return
|
|
else:
|
|
raise RuntimeError("generator didn't stop")
|
|
else:
|
|
try:
|
|
self.gen.throw(type, value, traceback)
|
|
except (type, StopIteration):
|
|
return
|
|
else:
|
|
raise RuntimeError("generator caught exception")
|
|
|
|
def contextmanager(func):
|
|
def helper(*args, **kwds):
|
|
return ContextWrapper(func(*args, **kwds))
|
|
return helper
|
|
|
|
This decorator could be used as follows:
|
|
|
|
@contextmanager
|
|
def opening(filename):
|
|
f = open(filename) # IOError is untouched by ContextWrapper
|
|
try:
|
|
yield f
|
|
finally:
|
|
f.close() # Ditto for errors here (however unlikely)
|
|
|
|
A robust implementation of this decorator should be made part of
|
|
the standard library, but not necessarily as a built-in function.
|
|
(I'm not sure which exception it should raise for errors;
|
|
RuntimeError is used above as an example only.)
|
|
|
|
Optional Extensions
|
|
|
|
It would be possible to endow certain objects, like files,
|
|
sockets, and locks, with __enter__() and __exit__() methods so
|
|
that instead of writing:
|
|
|
|
with locking(myLock):
|
|
BLOCK
|
|
|
|
one could write simply:
|
|
|
|
with myLock:
|
|
BLOCK
|
|
|
|
I think we should be careful with this; it could lead to mistakes
|
|
like:
|
|
|
|
f = open(filename)
|
|
with f:
|
|
BLOCK1
|
|
with f:
|
|
BLOCK2
|
|
|
|
which does not do what one might think (f is closed before BLOCK2
|
|
is entered).
|
|
|
|
OTOH such mistakes are easily diagnosed; for example, the
|
|
contextmanager decorator above raises RuntimeError when the second
|
|
with-statement calls f.__enter__() again.
|
|
|
|
Resolved Open Issues
|
|
|
|
Discussion on python-dev revealed some open issues. I list them
|
|
here, with my preferred resolution and its motivation. The PEP
|
|
has been accepted without these being challenged, so the issues
|
|
are now resolved.
|
|
|
|
1. The __exit__() method of the contextmanager decorator class
|
|
catches StopIteration and considers it equivalent to re-raising
|
|
the exception passed to throw(). Is allowing StopIteration
|
|
right here?
|
|
|
|
This is so that a generator doing cleanup depending on the
|
|
exception thrown (like the transactional() example below) can
|
|
*catch* the exception thrown if it wants to and doesn't have to
|
|
worry about re-raising it. I find this more convenient for the
|
|
generator writer. Against this was brought in that the
|
|
generator *appears* to suppress an exception that it cannot
|
|
suppress: the transactional() example would be more clear
|
|
according to this view if it re-raised the original exception
|
|
after the call to db.rollback(). I personally would find the
|
|
requirement to re-raise the exception an annoyance in a
|
|
generator used as a with-template, since all the code after
|
|
yield is used for is cleanup, and it is invoked from a
|
|
finally-clause (the one implicit in the with-statement) which
|
|
re-raises the original exception anyway.
|
|
|
|
Examples
|
|
|
|
(Note: several of these examples contain "yield None". If PEP 342
|
|
is accepted, these can be changed to just "yield".)
|
|
|
|
1. A template for ensuring that a lock, acquired at the start of a
|
|
block, is released when the block is left:
|
|
|
|
@contextmanager
|
|
def locking(lock):
|
|
lock.acquire()
|
|
try:
|
|
yield None
|
|
finally:
|
|
lock.release()
|
|
|
|
Used as follows:
|
|
|
|
with locking(myLock):
|
|
# Code here executes with myLock held. The lock is
|
|
# guaranteed to be released when the block is left (even
|
|
# if via return or by an uncaught exception).
|
|
|
|
PEP 319 gives a use case for also having an unlocking()
|
|
template; this can be written very similarly (just swap the
|
|
acquire() and release() calls).
|
|
|
|
2. A template for opening a file that ensures the file is closed
|
|
when the block is left:
|
|
|
|
@contextmanager
|
|
def opening(filename, mode="r"):
|
|
f = open(filename, mode)
|
|
try:
|
|
yield f
|
|
finally:
|
|
f.close()
|
|
|
|
Used as follows:
|
|
|
|
with opening("/etc/passwd") as f:
|
|
for line in f:
|
|
print line.rstrip()
|
|
|
|
3. A template for committing or rolling back a database
|
|
transaction:
|
|
|
|
@contextmanager
|
|
def transactional(db):
|
|
db.begin()
|
|
try:
|
|
yield None
|
|
except:
|
|
db.rollback()
|
|
else:
|
|
db.commit()
|
|
|
|
4. Example 1 rewritten without a generator:
|
|
|
|
class locking:
|
|
def __init__(self, lock):
|
|
self.lock = lock
|
|
def __enter__(self):
|
|
self.lock.acquire()
|
|
def __exit__(self, type, value, tb):
|
|
self.lock.release()
|
|
|
|
(This example is easily modified to implement the other
|
|
examples; it shows the relative advantage of using a generator
|
|
template.)
|
|
|
|
5. Redirect stdout temporarily:
|
|
|
|
@contextmanager
|
|
def redirecting_stdout(new_stdout):
|
|
save_stdout = sys.stdout
|
|
sys.stdout = new_stdout
|
|
try:
|
|
yield None
|
|
finally:
|
|
sys.stdout = save_stdout
|
|
|
|
Used as follows:
|
|
|
|
with opening(filename, "w") as f:
|
|
with redirecting_stdout(f):
|
|
print "Hello world"
|
|
|
|
This isn't thread-safe, of course, but neither is doing this
|
|
same dance manually. In single-threaded programs (for example,
|
|
in scripts) it is a popular way of doing things.
|
|
|
|
6. A variant on opening() that also returns an error condition:
|
|
|
|
@contextmanager
|
|
def opening_w_error(filename, mode="r"):
|
|
try:
|
|
f = open(filename, mode)
|
|
except IOError, err:
|
|
yield None, err
|
|
else:
|
|
try:
|
|
yield f, None
|
|
finally:
|
|
f.close()
|
|
|
|
Used as follows:
|
|
|
|
with opening_w_error("/etc/passwd", "a") as (f, err):
|
|
if err:
|
|
print "IOError:", err
|
|
else:
|
|
f.write("guido::0:0::/:/bin/sh\n")
|
|
|
|
7. Another useful example would be an operation that blocks
|
|
signals. The use could be like this:
|
|
|
|
import signal
|
|
|
|
with signal.blocking():
|
|
# code executed without worrying about signals
|
|
|
|
An optional argument might be a list of signals to be blocked;
|
|
by default all signals are blocked. The implementation is left
|
|
as an exercise to the reader.
|
|
|
|
8. Another use for this feature is the Decimal context. Here's a
|
|
simple example, after one posted by Michael Chermside:
|
|
|
|
import decimal
|
|
|
|
@contextmanager
|
|
def extra_precision(places=2):
|
|
c = decimal.getcontext()
|
|
saved_prec = c.prec
|
|
c.prec += places
|
|
try:
|
|
yield None
|
|
finally:
|
|
c.prec = saved_prec
|
|
|
|
Sample usage (adapted from the Python Library Reference):
|
|
|
|
def sin(x):
|
|
"Return the sine of x as measured in radians."
|
|
with extra_precision():
|
|
i, lasts, s, fact, num, sign = 1, 0, x, 1, x, 1
|
|
while s != lasts:
|
|
lasts = s
|
|
i += 2
|
|
fact *= i * (i-1)
|
|
num *= x * x
|
|
sign *= -1
|
|
s += num / fact * sign
|
|
# The "+s" rounds back to the original precision,
|
|
# so this must be outside the with-statement:
|
|
return +s
|
|
|
|
9. Here's a more general Decimal-context-switching template:
|
|
|
|
@contextmanager
|
|
def decimal_context(newctx=None):
|
|
oldctx = decimal.getcontext()
|
|
if newctx is None:
|
|
newctx = oldctx.copy()
|
|
decimal.setcontext(newctx)
|
|
try:
|
|
yield newctx
|
|
finally:
|
|
decimal.setcontext(oldctx)
|
|
|
|
Sample usage:
|
|
|
|
def sin(x):
|
|
with decimal_context() as ctx:
|
|
ctx.prec += 2
|
|
# Rest of algorithm the same as above
|
|
return +s
|
|
|
|
(Nick Coghlan has proposed to add __enter__() and __exit__()
|
|
methods to the decimal.Context class so that this example can
|
|
be simplified to "with decimal.getcontext() as ctx: ...".)
|
|
|
|
10. A generic "object-closing" template:
|
|
|
|
@contextmanager
|
|
def closing(obj):
|
|
try:
|
|
yield obj
|
|
finally:
|
|
obj.close()
|
|
|
|
This can be used to deterministically close anything with a
|
|
close method, be it file, generator, or something else:
|
|
|
|
# emulate opening():
|
|
with closing(open("argument.txt")) as contradiction:
|
|
for line in contradiction:
|
|
print line
|
|
|
|
# deterministically finalize a generator:
|
|
with closing(some_gen()) as data:
|
|
for datum in data:
|
|
process(datum)
|
|
|
|
|
|
References
|
|
|
|
[1] http://blogs.msdn.com/oldnewthing/archive/2005/01/06/347666.aspx
|
|
|
|
[2] http://mail.python.org/pipermail/python-dev/2005-May/053885.html
|
|
|
|
[3] http://wiki.python.org/moin/WithStatement
|
|
|
|
Copyright
|
|
|
|
This document has been placed in the public domain.
|