402 lines
14 KiB
Plaintext
402 lines
14 KiB
Plaintext
PEP: 343
|
|
Title: Anonymous Block Redux
|
|
Version: $Revision$
|
|
Last-Modified: $Date$
|
|
Author: Guido van Rossum
|
|
Status: Draft
|
|
Type: Standards Track
|
|
Content-Type: text/plain
|
|
Created: 13-May-2005
|
|
Post-History:
|
|
|
|
Introduction
|
|
|
|
After a lot of discussion about PEP 340 and alternatives, I've
|
|
decided to withdraw PEP 340 and propose a slight variant on
|
|
PEP 310.
|
|
|
|
Motivation and Summary
|
|
|
|
PEP 340, Anonymous Block Statements, combined many powerful ideas:
|
|
using generators as block templates, adding exception handling and
|
|
finalization to generators, and more. Besides praise it received
|
|
a lot of opposition from people who didn't like the fact that it
|
|
was, under the covers, a (potential) looping construct. This
|
|
meant that break and continue in a block-statement would break or
|
|
continue the block-statement, even if it was used as a non-looping
|
|
resource management tool.
|
|
|
|
But the final blow came when I read Raymond Chen's rant about
|
|
flow-control macros[1]. Raymond argues convincingly that hiding
|
|
flow control in macros makes your code inscrutable, and I find
|
|
that his argument applies to Python as well as to C. I realized
|
|
that PEP 340 templates can hide all sorts of control flow; for
|
|
example, its example 4 (auto_retry()) catches exceptions and
|
|
repeats the block up to three times.
|
|
|
|
However, the with-statement of PEP 310 does *not* hide control
|
|
flow, in my view: while a finally-suite temporarily suspends the
|
|
control flow, in the end, the control flow resumes as if the
|
|
finally-suite wasn't there at all.
|
|
|
|
Remember, PEP 310 proposes rougly this syntax (the "VAR =" part is
|
|
optional):
|
|
|
|
with VAR = EXPR:
|
|
BLOCK
|
|
|
|
which roughly translates into
|
|
|
|
VAR = EXPR
|
|
VAR.__enter__()
|
|
try:
|
|
BLOCK
|
|
finally:
|
|
VAR.__exit__()
|
|
|
|
Now consider this example:
|
|
|
|
with f = opening("/etc/passwd"):
|
|
BLOCK1
|
|
BLOCK2
|
|
|
|
Here, just as if the first line was "if True" instead, we know
|
|
that if BLOCK1 completes without an exception, BLOCK2 will be
|
|
reached; and if BLOCK1 raises an exception or executes a non-local
|
|
goto (a break, continue or return), BLOCK2 is *not* reached. The
|
|
magic added by the with-statement at the end doesn't affect this.
|
|
|
|
(You may ask, what if a bug in the __exit__ method causes an
|
|
exception? Then all is lost -- but this is no worse than with
|
|
other exceptions; the nature of exceptions is that they can happen
|
|
*anywhere*, and you just have to live with that. Even if you
|
|
write bug-free code, a KeyboardInterrupt exception can still cause
|
|
it to exit between any two virtual machine opcodes.)
|
|
|
|
This argument almost led me to endorse PEP 310, but I had one idea
|
|
left from the PEP 340 euphoria that I wasn't ready to drop: using
|
|
generators as "templates" for abstractions like acquiring and
|
|
releasing a lock or opening and closing a file is a powerful idea,
|
|
as can be seen by looking at the examples in that PEP.
|
|
|
|
Inspired by a counter-proposal to PEP 340 by Phillip Eby I tried
|
|
to create a decorator that would turn a suitable generator into an
|
|
object with the necessary __entry__ and __exit__ methods. Here I
|
|
ran into a snag: while it wasn't too hard for the locking example,
|
|
it was impossible to do this for the opening example. The idea
|
|
was to define the template like this:
|
|
|
|
@with_template
|
|
def opening(filename):
|
|
f = open(filename)
|
|
yield f
|
|
f.close()
|
|
|
|
and used it like this:
|
|
|
|
with f = opening(filename):
|
|
...read data from f...
|
|
|
|
The problem is that in PEP 310, the result of calling EXPR is
|
|
assigned directly to VAR, and then VAR's __exit__ method is called
|
|
upon exit from BLOCK1. But here, VAR clearly needs to receive the
|
|
opened file, and that would mean that __exit__ would have to be a
|
|
method on the file.
|
|
|
|
While this can be solved using a proxy class, this is awkward and
|
|
made me realize that a slightly different translation would make
|
|
writing the desired decorator a piece of cake: let VAR receive the
|
|
result from calling the __enter__ method, and save the value of
|
|
EXPR to call its __exit__ method later. Then the decorator can
|
|
return an instance of a wrapper class whose __enter__ method calls
|
|
the generator's next() method and returns whatever next() returns;
|
|
the wrapper instance's __exit__ method calls next() again but
|
|
expects it to raise StopIteration. (Details below in the section
|
|
Optional Generator Decorator.)
|
|
|
|
So now the final hurdle was that the PEP 310 syntax:
|
|
|
|
with VAR = EXPR:
|
|
BLOCK1
|
|
|
|
would be deceptive, since VAR does *not* receive the value of
|
|
EXPR. Given PEP 340, it was an easy step to:
|
|
|
|
with EXPR as VAR:
|
|
BLOCK1
|
|
|
|
or, using an alternate keyword that has been proposed a number of
|
|
times:
|
|
|
|
do EXPR as VAR:
|
|
BLOCK1
|
|
|
|
Use Cases
|
|
|
|
See the Examples section near the end.
|
|
|
|
Specification
|
|
|
|
A new statement is proposed with the syntax:
|
|
|
|
do EXPR as VAR:
|
|
BLOCK
|
|
|
|
Here, 'do' and 'as' are new keywords; EXPR is an arbitrary
|
|
expression (but not an expression-list) and VAR is an arbitrary
|
|
assignment target (which may be a comma-separated list).
|
|
|
|
The "as VAR" part is optional.
|
|
|
|
The choice of the 'do' keyword is provisional; an alternative
|
|
under consideration is 'with'.
|
|
|
|
A yield-statement is illegal inside BLOCK. This is because the
|
|
do-statement is translated into a try/finally statement, and yield
|
|
is illegal in a try/finally statement.
|
|
|
|
The translation of the above statement is:
|
|
|
|
abc = EXPR
|
|
exc = (None, None, None)
|
|
try:
|
|
VAR = abc.__enter__()
|
|
try:
|
|
BLOCK
|
|
except:
|
|
exc = sys.exc_info()
|
|
raise
|
|
finally:
|
|
abc.__exit__(*exc)
|
|
|
|
Here, the variables 'abc' and 'exc' are internal variables and not
|
|
accessible to the user; they will most likely be implemented as
|
|
special registers or stack positions.
|
|
|
|
If the "as VAR" part of the syntax is omitted, the "VAR =" part of
|
|
the translation is omitted (but abc.__enter__() is still called).
|
|
|
|
The calling convention for abc.__exit__() is as follows. If the
|
|
finally-suite was reached through normal completion of BLOCK or
|
|
through a non-local goto (a break, continue or return statement in
|
|
BLOCK), abc.__exit__() is called with three None arguments. If
|
|
the finally-suite was reached through an exception raised in
|
|
BLOCK, abc.__exit__() is called with three arguments representing
|
|
the exception type, value, and traceback.
|
|
|
|
The motivation for this API to __exit__(), as opposed to the
|
|
argument-less __exit__() from PEP 310, was given by the
|
|
transactional() use case, example 3 below. The template in that
|
|
example must commit or roll back the transaction depending on
|
|
whether an exception occurred or not. Rather than just having a
|
|
boolean flag indicating whether an exception occurred, we pass the
|
|
complete exception information, for the benefit of an
|
|
exception-logging facility for example. Relying on sys.exc_info()
|
|
to get at the exception information was rejected; sys.exc_info()
|
|
has very complex semantics and it is perfectly possible that it
|
|
returns the exception information for an exception that was caught
|
|
ages ago. It was also proposed to add an additional boolean to
|
|
distinguish between reaching the end of BLOCK and a non-local
|
|
goto. This was rejected as too complex and unnecessary; a
|
|
non-local goto should be considered unexceptional for the purposes
|
|
of a database transaction roll-back decision.
|
|
|
|
Optional Generator Decorator
|
|
|
|
It is possible to write a decorator that makes it possible to use
|
|
a generator that yields exactly once to control a do-statement.
|
|
Here's a sketch of such a decorator:
|
|
|
|
class Wrapper(object):
|
|
def __init__(self, gen):
|
|
self.gen = gen
|
|
self.state = "initial"
|
|
def __enter__(self):
|
|
assert self.state == "initial"
|
|
self.state = "entered"
|
|
try:
|
|
return self.gen.next()
|
|
except StopIteration:
|
|
self.state = "error"
|
|
raise RuntimeError("template generator didn't yield")
|
|
def __exit__(self, *args):
|
|
assert self.state == "entered"
|
|
self.state = "exited"
|
|
try:
|
|
self.gen.next()
|
|
except StopIteration:
|
|
return
|
|
else:
|
|
self.state = "error"
|
|
raise RuntimeError("template generator didn't stop")
|
|
|
|
def do_template(func):
|
|
def helper(*args, **kwds):
|
|
return Wrapper(func(*args, **kwds))
|
|
return helper
|
|
|
|
This decorator could be used as follows:
|
|
|
|
@do_template
|
|
def opening(filename):
|
|
f = open(filename) # IOError here is untouched by Wrapper
|
|
yield f
|
|
f.close() # Ditto for errors here (however unlikely)
|
|
|
|
A robust implementation of such a decorator should be made part of
|
|
the standard library.
|
|
|
|
Other Optional Extensions
|
|
|
|
It would be possible to endow certain objects, like files,
|
|
sockets, and locks, with __enter__ and __exit__ methods so that
|
|
instead of writing:
|
|
|
|
do locking(myLock):
|
|
BLOCK
|
|
|
|
one could write simply:
|
|
|
|
do myLock:
|
|
BLOCK
|
|
|
|
I think we should be careful with this; it could lead to mistakes
|
|
like:
|
|
|
|
f = open(filename)
|
|
do f:
|
|
BLOCK1
|
|
do f:
|
|
BLOCK2
|
|
|
|
which does not do what one might think (f is closed when BLOCK2 is
|
|
entered).
|
|
|
|
Examples
|
|
|
|
Several of these examples contain "yield None". If PEP 342 is
|
|
accepted, these can be changed to just "yield".
|
|
|
|
1. A template for ensuring that a lock, acquired at the start of a
|
|
block, is released when the block is left:
|
|
|
|
@do_template
|
|
def locking(lock):
|
|
lock.acquire()
|
|
yield None
|
|
lock.release()
|
|
|
|
Used as follows:
|
|
|
|
do locking(myLock):
|
|
# Code here executes with myLock held. The lock is
|
|
# guaranteed to be released when the block is left (even
|
|
# if via return or by an uncaught exception).
|
|
|
|
2. A template for opening a file that ensures the file is closed
|
|
when the block is left:
|
|
|
|
@do_template
|
|
def opening(filename, mode="r"):
|
|
f = open(filename, mode)
|
|
yield f
|
|
f.close()
|
|
|
|
Used as follows:
|
|
|
|
do opening("/etc/passwd") as f:
|
|
for line in f:
|
|
print line.rstrip()
|
|
|
|
3. A template for committing or rolling back a database
|
|
transaction; this is written as a class rather than as a
|
|
decorator since it requires access to the exception information:
|
|
|
|
class transactional:
|
|
def __init__(self, db):
|
|
self.db = db
|
|
def __enter__(self):
|
|
self.db.begin()
|
|
def __exit__(self, *args):
|
|
if args and args[0] is not None:
|
|
self.db.rollback()
|
|
else:
|
|
self.db.commit()
|
|
|
|
4. Example 1 rewritten without a generator:
|
|
|
|
class locking:
|
|
def __init__(self, lock):
|
|
self.lock = lock
|
|
def __enter__(self):
|
|
self.lock.acquire()
|
|
def __exit__(self, *args):
|
|
self.lock.release()
|
|
|
|
(This example is easily modified to implement the other
|
|
examples; it shows how much simpler generators are for the same
|
|
purpose.)
|
|
|
|
5. Redirect stdout temporarily:
|
|
|
|
@do_template
|
|
def redirecting_stdout(new_stdout):
|
|
save_stdout = sys.stdout
|
|
sys.stdout = new_stdout
|
|
yield None
|
|
sys.stdout = save_stdout
|
|
|
|
Used as follows:
|
|
|
|
do opening(filename, "w") as f:
|
|
do redirecting_stdout(f):
|
|
print "Hello world"
|
|
|
|
This isn't thread-safe, of course, but neither is doing this
|
|
same dance manually. In a single-threaded program (e.g., a
|
|
script) it is a totally fine way of doing things.
|
|
|
|
6. A variant on opening() that also returns an error condition:
|
|
|
|
@do_template
|
|
def opening_w_error(filename, mode="r"):
|
|
try:
|
|
f = open(filename, mode)
|
|
except IOError, err:
|
|
yield None, err
|
|
else:
|
|
yield f, None
|
|
f.close()
|
|
|
|
Used as follows:
|
|
|
|
do opening_w_error("/etc/passwd", "a") as f, err:
|
|
if err:
|
|
print "IOError:", err
|
|
else:
|
|
f.write("guido::0:0::/:/bin/sh\n")
|
|
|
|
7. Another useful example would be an operation that blocks
|
|
signals. The use could be like this:
|
|
|
|
import signal
|
|
|
|
do signal.blocking():
|
|
# code executed without worrying about signals
|
|
|
|
An optional argument might be a list of signals to be blocked;
|
|
by default all signals are blocked. The implementation is left
|
|
as an exercise to the reader.
|
|
|
|
8. Another use for this feature is the Decimal context. It's left
|
|
as an exercise for the reader. (Mail it to me if you'd like to
|
|
see it here.)
|
|
|
|
References
|
|
|
|
[1] http://blogs.msdn.com/oldnewthing/archive/2005/01/06/347666.aspx
|
|
|
|
Copyright
|
|
|
|
This document has been placed in the public domain.
|