python-peps/pep-0343.txt

582 lines
21 KiB
Plaintext
Raw Normal View History

2005-05-13 20:08:20 -04:00
PEP: 343
Title: Anonymous Block Redux and Generator Enhancements
2005-05-13 20:08:20 -04:00
Version: $Revision$
Last-Modified: $Date$
Author: Guido van Rossum
Status: Accepted
2005-05-13 20:08:20 -04:00
Type: Standards Track
Content-Type: text/plain
Created: 13-May-2005
2005-06-06 13:15:17 -04:00
Post-History: 2-Jun-2005
2005-05-13 20:08:20 -04:00
Introduction
After a lot of discussion about PEP 340 and alternatives, I
decided to withdraw PEP 340 and proposed a slight variant on PEP
2005-06-03 05:32:57 -04:00
310. After more discussion, I have added back a mechanism for
raising an exception in a suspended generator using a throw()
2005-05-31 16:27:15 -04:00
method, and a close() method which throws a new GeneratorExit
2005-06-03 05:32:57 -04:00
exception; these additions were first proposed on python-dev in
[2] and universally approved of. I'm also changing the keyword to
'with'.
2005-05-31 16:27:15 -04:00
2005-06-06 13:15:17 -04:00
On-line discussion of this PEP should take place in the Python
Wiki [3].
If this PEP is approved, the following PEPs will be rejected due
to overlap:
- PEP 310, Reliable Acquisition/Release Pairs. This is the
original with-statement proposal.
- PEP 319, Python Synchronize/Asynchronize Block. Its use cases
can be covered by the current PEP by providing suitable
with-statement controllers: for 'synchronize' we can use the
"locking" template from example 1; for 'asynchronize' we can use
a similar "unlocking" template. I don't think having an
"anonymous" lock associated with a code block is all that
important; in fact it may be better to always be explicit about
the mutex being used.
(PEP 340 and PEP 346 have already been withdrawn.)
2005-05-13 20:08:20 -04:00
Motivation and Summary
PEP 340, Anonymous Block Statements, combined many powerful ideas:
using generators as block templates, adding exception handling and
finalization to generators, and more. Besides praise it received
a lot of opposition from people who didn't like the fact that it
2005-05-14 20:45:42 -04:00
was, under the covers, a (potential) looping construct. This
meant that break and continue in a block-statement would break or
continue the block-statement, even if it was used as a non-looping
resource management tool.
But the final blow came when I read Raymond Chen's rant about
flow-control macros[1]. Raymond argues convincingly that hiding
flow control in macros makes your code inscrutable, and I find
that his argument applies to Python as well as to C. I realized
that PEP 340 templates can hide all sorts of control flow; for
example, its example 4 (auto_retry()) catches exceptions and
repeats the block up to three times.
However, the with-statement of PEP 310 does *not* hide control
flow, in my view: while a finally-suite temporarily suspends the
control flow, in the end, the control flow resumes as if the
finally-suite wasn't there at all.
Remember, PEP 310 proposes rougly this syntax (the "VAR =" part is
optional):
with VAR = EXPR:
BLOCK
which roughly translates into this:
VAR = EXPR
VAR.__enter__()
try:
BLOCK
finally:
VAR.__exit__()
Now consider this example:
with f = opening("/etc/passwd"):
BLOCK1
BLOCK2
Here, just as if the first line was "if True" instead, we know
that if BLOCK1 completes without an exception, BLOCK2 will be
reached; and if BLOCK1 raises an exception or executes a non-local
goto (a break, continue or return), BLOCK2 is *not* reached. The
magic added by the with-statement at the end doesn't affect this.
(You may ask, what if a bug in the __exit__() method causes an
exception? Then all is lost -- but this is no worse than with
other exceptions; the nature of exceptions is that they can happen
*anywhere*, and you just have to live with that. Even if you
write bug-free code, a KeyboardInterrupt exception can still cause
it to exit between any two virtual machine opcodes.)
This argument almost led me to endorse PEP 310, but I had one idea
left from the PEP 340 euphoria that I wasn't ready to drop: using
generators as "templates" for abstractions like acquiring and
releasing a lock or opening and closing a file is a powerful idea,
as can be seen by looking at the examples in that PEP.
Inspired by a counter-proposal to PEP 340 by Phillip Eby I tried
to create a decorator that would turn a suitable generator into an
object with the necessary __enter__() and __exit__() methods.
Here I ran into a snag: while it wasn't too hard for the locking
example, it was impossible to do this for the opening example.
The idea was to define the template like this:
@contextmanager
def opening(filename):
f = open(filename)
try:
yield f
finally:
f.close()
and used it like this:
with f = opening(filename):
...read data from f...
The problem is that in PEP 310, the result of calling EXPR is
assigned directly to VAR, and then VAR's __exit__() method is
called upon exit from BLOCK1. But here, VAR clearly needs to
receive the opened file, and that would mean that __exit__() would
have to be a method on the file.
While this can be solved using a proxy class, this is awkward and
made me realize that a slightly different translation would make
writing the desired decorator a piece of cake: let VAR receive the
result from calling the __enter__() method, and save the value of
EXPR to call its __exit__() method later. Then the decorator can
return an instance of a wrapper class whose __enter__() method
calls the generator's next() method and returns whatever next()
returns; the wrapper instance's __exit__() method calls next()
again but expects it to raise StopIteration. (Details below in
the section Optional Generator Decorator.)
So now the final hurdle was that the PEP 310 syntax:
with VAR = EXPR:
BLOCK1
would be deceptive, since VAR does *not* receive the value of
EXPR. Borrowing from PEP 340, it was an easy step to:
with EXPR as VAR:
BLOCK1
Additional discussion showed that people really liked being able
to "see" the exception in the generator, even if it was only to
log it; the generator is not allowed to yield another value, since
the with-statement should not be usable as a loop (raising a
different exception is marginally acceptable). To enable this, a
new throw() method for generators is proposed, which takes one to
three arguments representing an exception in the usual fashion
(type, value, traceback) and raises it at the point where the
generator is suspended.
Once we have this, it is a small step to proposing another
generator method, close(), which calls throw() with a special
exception, GeneratorExit. This tells the generator to exit, and
from there it's another small step to proposing that close() be
called automatically when the generator is garbage-collected.
Then, finally, we can allow a yield-statement inside a try-finally
statement, since we can now guarantee that the finally-clause will
(eventually) be executed. The usual cautions about finalization
apply -- the process may be terminated abruptly without finalizing
any objects, and objects may be kept alive forever by cycles or
memory leaks in the application (as opposed to cycles or leaks in
the Python implementation, which are taken care of by GC).
Note that we're not guaranteeing that the finally-clause is
executed immediately after the generator object becomes unused,
even though this is how it will work in CPython. This is similar
to auto-closing files: while a reference-counting implementation
like CPython deallocates an object as soon as the last reference
to it goes away, implementations that use other GC algorithms do
not make the same guarantee. This applies to Jython, IronPython,
and probably to Python running on Parrot.
2005-05-13 20:08:20 -04:00
Use Cases
See the Examples section near the end.
Specification: The 'with' Statement
2005-05-13 20:08:20 -04:00
A new statement is proposed with the syntax:
2005-05-13 20:08:20 -04:00
with EXPR as VAR:
2005-05-13 20:08:20 -04:00
BLOCK
Here, 'with' and 'as' are new keywords; EXPR is an arbitrary
expression (but not an expression-list) and VAR is a single
assignment target. It can *not* be a comma-separated sequence of
variables, but it *can* be a *parenthesized* comma-separated
sequence of variables. (This restriction makes a future extension
possible of the syntax to have multiple comma-separated resources,
each with its own optional as-clause.)
2005-05-13 20:08:20 -04:00
The "as VAR" part is optional.
The translation of the above statement is:
abc = EXPR
2005-05-14 13:36:15 -04:00
exc = (None, None, None)
VAR = abc.__enter__()
2005-05-13 20:08:20 -04:00
try:
try:
BLOCK
except:
exc = sys.exc_info()
raise
2005-05-13 20:08:20 -04:00
finally:
2005-05-14 13:36:15 -04:00
abc.__exit__(*exc)
2005-05-13 20:08:20 -04:00
Here, the variables 'abc' and 'exc' are internal variables and not
accessible to the user; they will most likely be implemented as
special registers or stack positions.
2005-05-13 20:08:20 -04:00
If the "as VAR" part of the syntax is omitted, the "VAR =" part of
the translation is omitted (but abc.__enter__() is still called).
2005-05-14 20:45:42 -04:00
The calling convention for abc.__exit__() is as follows. If the
finally-suite was reached through normal completion of BLOCK or
2005-05-14 13:36:15 -04:00
through a non-local goto (a break, continue or return statement in
BLOCK), abc.__exit__() is called with three None arguments. If
the finally-suite was reached through an exception raised in
BLOCK, abc.__exit__() is called with three arguments representing
the exception type, value, and traceback.
2005-05-13 20:08:20 -04:00
2005-05-14 01:18:36 -04:00
The motivation for this API to __exit__(), as opposed to the
argument-less __exit__() from PEP 310, was given by the
transactional() use case, example 3 below. The template in that
example must commit or roll back the transaction depending on
whether an exception occurred or not. Rather than just having a
boolean flag indicating whether an exception occurred, we pass the
2005-05-14 20:45:42 -04:00
complete exception information, for the benefit of an
2005-05-14 01:18:36 -04:00
exception-logging facility for example. Relying on sys.exc_info()
2005-05-14 20:45:42 -04:00
to get at the exception information was rejected; sys.exc_info()
2005-05-14 01:18:36 -04:00
has very complex semantics and it is perfectly possible that it
returns the exception information for an exception that was caught
ages ago. It was also proposed to add an additional boolean to
distinguish between reaching the end of BLOCK and a non-local
goto. This was rejected as too complex and unnecessary; a
non-local goto should be considered unexceptional for the purposes
of a database transaction roll-back decision.
Generator Decorator
2005-05-13 20:08:20 -04:00
If PEP 342 is accepted, it will be possible to write a decorator
that makes it possible to use a generator that yields exactly once
to control a with-statement. Here's a sketch of such a decorator:
2005-05-13 20:08:20 -04:00
class ContextWrapper(object):
def __init__(self, gen):
self.gen = gen
def __enter__(self):
try:
return self.gen.next()
except StopIteration:
raise RuntimeError("generator didn't yield")
def __exit__(self, type, value, traceback):
if type is None:
try:
self.gen.next()
except StopIteration:
return
else:
raise RuntimeError("generator didn't stop")
else:
try:
self.gen.throw(type, value, traceback)
except (type, StopIteration):
return
else:
raise RuntimeError("generator caught exception")
def contextmanager(func):
def helper(*args, **kwds):
return ContextWrapper(func(*args, **kwds))
return helper
2005-05-13 20:08:20 -04:00
This decorator could be used as follows:
@contextmanager
def opening(filename):
f = open(filename) # IOError is untouched by ContextWrapper
try:
yield f
finally:
f.close() # Ditto for errors here (however unlikely)
2005-05-13 20:08:20 -04:00
A robust implementation of this decorator should be made part of
the standard library, but not necessarily as a built-in function.
(I'm not sure which exception it should raise for errors;
RuntimeError is used above as an example only.)
2005-05-13 20:08:20 -04:00
Optional Extensions
It would be possible to endow certain objects, like files,
sockets, and locks, with __enter__() and __exit__() methods so
that instead of writing:
with locking(myLock):
BLOCK
one could write simply:
with myLock:
BLOCK
I think we should be careful with this; it could lead to mistakes
like:
f = open(filename)
with f:
BLOCK1
with f:
BLOCK2
which does not do what one might think (f is closed before BLOCK2
is entered).
2005-06-02 11:13:55 -04:00
OTOH such mistakes are easily diagnosed; for example, the
contextmanager decorator above raises RuntimeError when the second
2005-06-02 11:13:55 -04:00
with-statement calls f.__enter__() again.
Resolved Open Issues
Discussion on python-dev revealed some open issues. I list them
here, with my preferred resolution and its motivation. The PEP
has been accepted without these being challenged, so the issues
are now resolved.
1. The __exit__() method of the contextmanager decorator class
catches StopIteration and considers it equivalent to re-raising
the exception passed to throw(). Is allowing StopIteration
right here?
This is so that a generator doing cleanup depending on the
exception thrown (like the transactional() example below) can
*catch* the exception thrown if it wants to and doesn't have to
worry about re-raising it. I find this more convenient for the
generator writer. Against this was brought in that the
generator *appears* to suppress an exception that it cannot
suppress: the transactional() example would be more clear
according to this view if it re-raised the original exception
after the call to db.rollback(). I personally would find the
requirement to re-raise the exception an annoyance in a
generator used as a with-template, since all the code after
yield is used for is cleanup, and it is invoked from a
finally-clause (the one implicit in the with-statement) which
re-raises the original exception anyway.
2005-05-13 20:08:20 -04:00
Examples
(Note: several of these examples contain "yield None". If PEP 342
is accepted, these can be changed to just "yield".)
2005-05-13 20:08:20 -04:00
1. A template for ensuring that a lock, acquired at the start of a
block, is released when the block is left:
@contextmanager
2005-05-13 20:08:20 -04:00
def locking(lock):
lock.acquire()
try:
yield None
finally:
lock.release()
2005-05-13 20:08:20 -04:00
Used as follows:
with locking(myLock):
2005-05-13 20:08:20 -04:00
# Code here executes with myLock held. The lock is
# guaranteed to be released when the block is left (even
# if via return or by an uncaught exception).
PEP 319 gives a use case for also having an unlocking()
template; this can be written very similarly (just swap the
acquire() and release() calls).
2005-05-13 20:08:20 -04:00
2. A template for opening a file that ensures the file is closed
when the block is left:
@contextmanager
2005-05-13 20:08:20 -04:00
def opening(filename, mode="r"):
f = open(filename, mode)
try:
yield f
finally:
f.close()
2005-05-13 20:08:20 -04:00
Used as follows:
with opening("/etc/passwd") as f:
2005-05-13 20:08:20 -04:00
for line in f:
print line.rstrip()
3. A template for committing or rolling back a database
transaction:
@contextmanager
def transactional(db):
db.begin()
try:
yield None
except:
db.rollback()
else:
db.commit()
2005-05-13 20:08:20 -04:00
4. Example 1 rewritten without a generator:
class locking:
def __init__(self, lock):
self.lock = lock
def __enter__(self):
self.lock.acquire()
def __exit__(self, type, value, tb):
self.lock.release()
2005-05-13 20:08:20 -04:00
(This example is easily modified to implement the other
examples; it shows the relative advantage of using a generator
template.)
2005-05-13 20:08:20 -04:00
5. Redirect stdout temporarily:
@contextmanager
2005-05-13 20:08:20 -04:00
def redirecting_stdout(new_stdout):
save_stdout = sys.stdout
sys.stdout = new_stdout
try:
yield None
finally:
sys.stdout = save_stdout
2005-05-13 20:08:20 -04:00
Used as follows:
with opening(filename, "w") as f:
with redirecting_stdout(f):
2005-05-13 20:08:20 -04:00
print "Hello world"
This isn't thread-safe, of course, but neither is doing this
same dance manually. In single-threaded programs (for example,
in scripts) it is a popular way of doing things.
2005-05-13 20:08:20 -04:00
6. A variant on opening() that also returns an error condition:
@contextmanager
2005-05-13 20:08:20 -04:00
def opening_w_error(filename, mode="r"):
try:
f = open(filename, mode)
except IOError, err:
yield None, err
else:
try:
yield f, None
finally:
f.close()
2005-05-13 20:08:20 -04:00
Used as follows:
with opening_w_error("/etc/passwd", "a") as (f, err):
2005-05-13 20:08:20 -04:00
if err:
print "IOError:", err
else:
f.write("guido::0:0::/:/bin/sh\n")
7. Another useful example would be an operation that blocks
signals. The use could be like this:
import signal
with signal.blocking():
2005-05-13 20:08:20 -04:00
# code executed without worrying about signals
An optional argument might be a list of signals to be blocked;
by default all signals are blocked. The implementation is left
as an exercise to the reader.
8. Another use for this feature is the Decimal context. Here's a
simple example, after one posted by Michael Chermside:
import decimal
@contextmanager
def extra_precision(places=2):
c = decimal.getcontext()
saved_prec = c.prec
c.prec += places
try:
yield None
finally:
c.prec = saved_prec
Sample usage (adapted from the Python Library Reference):
def sin(x):
"Return the sine of x as measured in radians."
with extra_precision():
i, lasts, s, fact, num, sign = 1, 0, x, 1, x, 1
while s != lasts:
lasts = s
i += 2
fact *= i * (i-1)
num *= x * x
sign *= -1
s += num / fact * sign
# The "+s" rounds back to the original precision,
# so this must be outside the with-statement:
return +s
2005-05-18 10:24:27 -04:00
9. Here's a more general Decimal-context-switching template:
2005-05-17 22:53:26 -04:00
@contextmanager
def decimal_context(newctx=None):
2005-05-18 10:24:27 -04:00
oldctx = decimal.getcontext()
if newctx is None:
newctx = oldctx.copy()
decimal.setcontext(newctx)
try:
yield newctx
finally:
decimal.setcontext(oldctx)
2005-05-18 10:24:27 -04:00
Sample usage:
2005-05-18 10:24:27 -04:00
def sin(x):
with decimal_context() as ctx:
2005-05-18 10:24:27 -04:00
ctx.prec += 2
# Rest of algorithm the same as above
2005-05-18 10:24:27 -04:00
return +s
(Nick Coghlan has proposed to add __enter__() and __exit__()
2005-05-18 10:24:27 -04:00
methods to the decimal.Context class so that this example can
be simplified to "with decimal.getcontext() as ctx: ...".)
2005-05-17 22:53:26 -04:00
2005-06-02 11:13:55 -04:00
10. A generic "object-closing" template:
@contextmanager
2005-06-02 11:13:55 -04:00
def closing(obj):
try:
yield obj
finally:
obj.close()
This can be used to deterministically close anything with a
close method, be it file, generator, or something else:
# emulate opening():
with closing(open("argument.txt")) as contradiction:
for line in contradiction:
print line
# deterministically finalize a generator:
with closing(some_gen()) as data:
for datum in data:
process(datum)
References
[1] http://blogs.msdn.com/oldnewthing/archive/2005/01/06/347666.aspx
2005-05-31 16:27:15 -04:00
[2] http://mail.python.org/pipermail/python-dev/2005-May/053885.html
2005-06-06 13:15:17 -04:00
[3] http://wiki.python.org/moin/WithStatement
2005-05-13 20:08:20 -04:00
Copyright
This document has been placed in the public domain.