python-peps/pep-0343.txt

PEP: 343
Title: Anonymous Block Redux
Version: $Revision$
Last-Modified: $Date$
Author: Guido van Rossum
Status: Draft
Type: Standards Track
Content-Type: text/plain
Created: 13-May-2005
Post-History:

Introduction

    After a lot of discussion about PEP 340 and alternatives, I've
    decided to withdraw PEP 340 and propose a slight variant on
    PEP 310.

Motivation and Summary

    PEP 340, Anonymous Block Statements, combined many powerful ideas:
    using generators as block templates, adding exception handling and
    finalization to generators, and more.  Besides praise it received
    a lot of opposition from people who didn't like the fact that it
    was, under the covers, a (potential) looping construct.  This
    meant that break and continue in a block-statement would break or
    continue the block-statement, even if it was used as a non-looping
    resource management tool.

    But the final blow came when I read Raymond Chen's rant about
    flow-control macros[1].  Raymond argues convincingly that hiding
    flow control in macros makes your code inscrutable, and I find
    that his argument applies to Python as well as to C.  I realized
    that PEP 340 templates can hide all sorts of control flow; for
    example, its example 4 (auto_retry()) catches exceptions and
    repeats the block up to three times.

    However, the with-statement of PEP 310 does *not* hide control
    flow, in my view: while a finally-suite temporarily suspends the
    control flow, in the end, the control flow resumes as if the
    finally-suite wasn't there at all.

    Remember, PEP 310 proposes rougly this syntax (the "VAR =" part is
    optional):

        with VAR = EXPR:
            BLOCK

    which roughly translates into

        VAR = EXPR
        VAR.__enter__()
        try:
            BLOCK
        finally:
            VAR.__exit__()

    Now consider this example:

        with f = opening("/etc/passwd"):
            BLOCK1
        BLOCK2

    Here, just as if the first line was "if True" instead, we know
    that if BLOCK1 completes without an exception, BLOCK2 will be
    reached; and if BLOCK1 raises an exception or executes a non-local
    goto (a break, continue or return), BLOCK2 is *not* reached.  The
    magic added by the with-statement at the end doesn't affect this.

    (You may ask, what if a bug in the __exit__ method causes an
    exception?  Then all is lost -- but this is no worse than with
    other exceptions; the nature of exceptions is that they can happen
    *anywhere*, and you just have to live with that.  Even if you
    write bug-free code, a KeyboardInterrupt exception can still cause
    it to exit between any two virtual machine opcodes.)

    This argument almost led me to endorse PEP 310, but I had one idea
    left from the PEP 340 euphoria that I wasn't ready to drop: using
    generators as "templates" for abstractions like acquiring and
    releasing a lock or opening and closing a file is a powerful idea,
    as can be seen by looking at the examples in that PEP.

    Inspired by a counter-proposal to PEP 340 by Phillip Eby I tried
    to create a decorator that would turn a suitable generator into an
    object with the necessary __entry__ and __exit__ methods.  Here I
    ran into a snag: while it wasn't too hard for the locking example,
    it was impossible to do this for the opening example.  The idea
    was to define the template like this:

        @with_template
        def opening(filename):
            f = open(filename)
            yield f
            f.close()

    and used it like this:

        with f = opening(filename):
            ...read data from f...

    The problem is that in PEP 310, the result of calling EXPR is
    assigned directly to VAR, and then VAR's __exit__ method is called
    upon exit from BLOCK1.  But here, VAR clearly needs to receive the
    opened file, and that would mean that __exit__ would have to be a
    method on the file.

    While this can be solved using a proxy class, this is awkward and
    made me realize that a slightly different translation would make
    writing the desired decorator a piece of cake: let VAR receive the
    result from calling the __enter__ method, and save the value of
    EXPR to call its __exit__ method later.  Then the decorator can
    return an instance of a wrapper class whose __enter__ method calls
    the generator's next() method and returns whatever next() returns;
    the wrapper instance's __exit__ method calls next() again but
    expects it to raise StopIteration.  (Details below in the section
    Optional Generator Decorator.)

    So now the final hurdle was that the PEP 310 syntax:

        with VAR = EXPR:
            BLOCK1

    would be deceptive, since VAR does *not* receive the value of
    EXPR.  Given PEP 340, it was an easy step to:

        with EXPR as VAR:
            BLOCK1

    or, using an alternate keyword that has been proposed a number of
    times:

        do EXPR as VAR:
            BLOCK1

Use Cases

    See the Examples section near the end.

Specification

    A new statement is proposed with the syntax:

        do EXPR as VAR:
            BLOCK

    Here, 'do' and 'as' are new keywords; EXPR is an arbitrary
    expression (but not an expression-list) and VAR is an arbitrary
    assignment target (which may be a comma-separated list).

    The "as VAR" part is optional.

    The choice of the 'do' keyword is provisional; an alternative
    under consideration is 'with'.

    A yield-statement is illegal inside BLOCK.  This is because the
    do-statement is translated into a try/finally statement, and yield
    is illegal in a try/finally statement.

    The translation of the above statement is:

        abc = EXPR
        exc = (None, None, None)
        VAR = abc.__enter__()
        try:
            try:
                BLOCK
            except:
                exc = sys.exc_info()
                raise
        finally:
            abc.__exit__(*exc)

    Here, the variables 'abc' and 'exc' are internal variables and not
    accessible to the user; they will most likely be implemented as
    special registers or stack positions.

    If the "as VAR" part of the syntax is omitted, the "VAR =" part of
    the translation is omitted (but abc.__enter__() is still called).

    The calling convention for abc.__exit__() is as follows.  If the
    finally-suite was reached through normal completion of BLOCK or
    through a non-local goto (a break, continue or return statement in
    BLOCK), abc.__exit__() is called with three None arguments.  If
    the finally-suite was reached through an exception raised in
    BLOCK, abc.__exit__() is called with three arguments representing
    the exception type, value, and traceback.

    The motivation for this API to __exit__(), as opposed to the
    argument-less __exit__() from PEP 310, was given by the
    transactional() use case, example 3 below.  The template in that
    example must commit or roll back the transaction depending on
    whether an exception occurred or not.  Rather than just having a
    boolean flag indicating whether an exception occurred, we pass the
    complete exception information, for the benefit of an
    exception-logging facility for example.  Relying on sys.exc_info()
    to get at the exception information was rejected; sys.exc_info()
    has very complex semantics and it is perfectly possible that it
    returns the exception information for an exception that was caught
    ages ago.  It was also proposed to add an additional boolean to
    distinguish between reaching the end of BLOCK and a non-local
    goto.  This was rejected as too complex and unnecessary; a
    non-local goto should be considered unexceptional for the purposes
    of a database transaction roll-back decision.

Optional Generator Decorator

    It is possible to write a decorator that makes it possible to use
    a generator that yields exactly once to control a do-statement.
    Here's a sketch of such a decorator:

        class Wrapper(object):
           def __init__(self, gen):
               self.gen = gen
               self.state = "initial"
           def __enter__(self):
               assert self.state == "initial"
               self.state = "entered"
               try:
                   return self.gen.next()
               except StopIteration:
                   self.state = "error"
                   raise RuntimeError("template generator didn't yield")
           def __exit__(self, *args):
               assert self.state == "entered"
               self.state = "exited"
               try:
                   self.gen.next()
               except StopIteration:
                   return
               else:
                   self.state = "error"
                   raise RuntimeError("template generator didn't stop")

        def do_template(func):
           def helper(*args, **kwds):
               return Wrapper(func(*args, **kwds))
           return helper

    This decorator could be used as follows:

        @do_template
        def opening(filename):
           f = open(filename) # IOError here is untouched by Wrapper
           yield f
           f.close() # Ditto for errors here (however unlikely)

    A robust implementation of such a decorator should be made part of
    the standard library.

Other Optional Extensions

    It would be possible to endow certain objects, like files,
    sockets, and locks, with __enter__ and __exit__ methods so that
    instead of writing:

        do locking(myLock):
            BLOCK

    one could write simply:

        do myLock:
            BLOCK

    I think we should be careful with this; it could lead to mistakes
    like:

        f = open(filename)
        do f:
            BLOCK1
        do f:
            BLOCK2

    which does not do what one might think (f is closed when BLOCK2 is
    entered).

Examples

    Several of these examples contain "yield None".  If PEP 342 is
    accepted, these can be changed to just "yield".

    1. A template for ensuring that a lock, acquired at the start of a
       block, is released when the block is left:

        @do_template
        def locking(lock):
            lock.acquire()
            yield None
            lock.release()

       Used as follows:

        do locking(myLock):
            # Code here executes with myLock held.  The lock is
            # guaranteed to be released when the block is left (even
            # if via return or by an uncaught exception).

    2. A template for opening a file that ensures the file is closed
       when the block is left:

        @do_template
        def opening(filename, mode="r"):
            f = open(filename, mode)
            yield f
            f.close()

       Used as follows:

        do opening("/etc/passwd") as f:
            for line in f:
                print line.rstrip()

    3. A template for committing or rolling back a database
       transaction; this is written as a class rather than as a
       decorator since it requires access to the exception information:

        class transactional:
            def __init__(self, db):
                self.db = db
            def __enter__(self):
                self.db.begin()
            def __exit__(self, *args):
                if args and args[0] is not None:
                    self.db.rollback()
                else:
                    self.db.commit()

    4. Example 1 rewritten without a generator:

        class locking:
           def __init__(self, lock):
               self.lock = lock
           def __enter__(self):
               self.lock.acquire()
           def __exit__(self, *args):
               self.lock.release()

       (This example is easily modified to implement the other
       examples; it shows how much simpler generators are for the same
       purpose.)

    5. Redirect stdout temporarily:

        @do_template
        def redirecting_stdout(new_stdout):
            save_stdout = sys.stdout
            sys.stdout = new_stdout
            yield None
            sys.stdout = save_stdout

       Used as follows:

        do opening(filename, "w") as f:
            do redirecting_stdout(f):
                print "Hello world"

       This isn't thread-safe, of course, but neither is doing this
       same dance manually.  In a single-threaded program (e.g., a
       script) it is a totally fine way of doing things.

    6. A variant on opening() that also returns an error condition:

        @do_template
        def opening_w_error(filename, mode="r"):
            try:
                f = open(filename, mode)
            except IOError, err:
                yield None, err
            else:
                yield f, None
                f.close()

       Used as follows:

        do opening_w_error("/etc/passwd", "a") as f, err:
            if err:
                print "IOError:", err
            else:
                f.write("guido::0:0::/:/bin/sh\n")

    7. Another useful example would be an operation that blocks
       signals.  The use could be like this:

        import signal

        do signal.blocking():
            # code executed without worrying about signals

       An optional argument might be a list of signals to be blocked;
       by default all signals are blocked.  The implementation is left
       as an exercise to the reader.

    8. Another use for this feature is the Decimal context.  Here's a
       simple example, after one posted by Michael Chermside:

        import decimal

        @do_template
        def with_extra_precision(places=2):
            c = decimal.getcontext()
            saved_prec = c.prec
            c.prec += places
            yield None
            c.prec = saved_prec

       Sample usage (adapted from the Python Library Reference):

        def sin(x):
            "Return the sine of x as measured in radians."
            do with_extra_precision():
                i, lasts, s, fact, num, sign = 1, 0, x, 1, x, 1
                while s != lasts:
                    lasts = s
                    i += 2
                    fact *= i * (i-1)
                    num *= x * x
                    sign *= -1
                    s += num / fact * sign
            # The "+s" rounds back to the original precision,
            # so this must be outside the do-statement:
            return +s

    9. Here's a more general Decimal-context-switching template:

        @do_template
        def with_decimal_context(newctx=None):
            oldctx = decimal.getcontext()
            if newctx is None:
                newctx = oldctx.copy()
            decimal.setcontext(newctx)
            yield newctx
            decimal.setcontext(oldctx)

       Sample usage (adapted from the previous one):

        def sin(x):
            do with_decimal_context() as ctx:
                ctx.prec += 2
                # Rest of algorithm the same
            return +s

       (Nick Coghlan has proposed to add __enter__ and __exit__
       methods to the decimal.Context class so that this example can
       be simplified to "do decimal.getcontext() as ctx: ...".)

References

    [1] http://blogs.msdn.com/oldnewthing/archive/2005/01/06/347666.aspx

Copyright

    This document has been placed in the public domain.