python-peps/pep-0343.txt

PEP: 343
Title: Anonymous Block Redux and Generator Enhancements
Version: $Revision$
Last-Modified: $Date$
Author: Guido van Rossum
Status: Draft
Type: Standards Track
Content-Type: text/plain
Created: 13-May-2005
Post-History: 2-Jun-2005

Introduction

    After a lot of discussion about PEP 340 and alternatives, I
    decided to withdraw PEP 340 and proposed a slight variant on PEP
    310.  After more discussion, I have added back a mechanism for
    raising an exception in a suspended generator using a throw()
    method, and a close() method which throws a new GeneratorExit
    exception; these additions were first proposed on python-dev in
    [2] and universally approved of.  I'm also changing the keyword to
    'with'.

    On-line discussion of this PEP should take place in the Python
    Wiki [3].

    If this PEP is approved, the following PEPs will be rejected due
    to overlap:

    - PEP 288, Generators Attributes and Exceptions.  The current PEP
      covers its second half, generator exceptions (in fact the
      throw() method name was taken from this PEP).  I'm not in favor
      of generator attributes, since they can easily be handled by
      making the generator a method or by passing a mutable argument.

    - PEP 310, Reliable Acquisition/Release Pairs.  This is the
      original with-statement proposal.

    - PEP 325, Resource-Release Support for Generators.  The current
      PEP covers this (in fact the close() method name was taken from
      this PEP).

    - PEP 319, Python Synchronize/Asynchronize Block.  Its use cases
      can be covered by the current PEP by providing suitable
      with-statement controllers: for 'synchronize' we can use the
      "locking" template from example 1; for 'asynchronize' we can use
      a similar "unlocking" template.  I don't think having an
      "anonymous" lock associated with a code block is all that
      important; in fact it may be better to always be explicit about
      the mutex being used.

    (PEP 340 and PEP 346 have already been withdrawn.)

Motivation and Summary

    PEP 340, Anonymous Block Statements, combined many powerful ideas:
    using generators as block templates, adding exception handling and
    finalization to generators, and more.  Besides praise it received
    a lot of opposition from people who didn't like the fact that it
    was, under the covers, a (potential) looping construct.  This
    meant that break and continue in a block-statement would break or
    continue the block-statement, even if it was used as a non-looping
    resource management tool.

    But the final blow came when I read Raymond Chen's rant about
    flow-control macros[1].  Raymond argues convincingly that hiding
    flow control in macros makes your code inscrutable, and I find
    that his argument applies to Python as well as to C.  I realized
    that PEP 340 templates can hide all sorts of control flow; for
    example, its example 4 (auto_retry()) catches exceptions and
    repeats the block up to three times.

    However, the with-statement of PEP 310 does *not* hide control
    flow, in my view: while a finally-suite temporarily suspends the
    control flow, in the end, the control flow resumes as if the
    finally-suite wasn't there at all.

    Remember, PEP 310 proposes rougly this syntax (the "VAR =" part is
    optional):

        with VAR = EXPR:
            BLOCK

    which roughly translates into this:

        VAR = EXPR
        VAR.__enter__()
        try:
            BLOCK
        finally:
            VAR.__exit__()

    Now consider this example:

        with f = opening("/etc/passwd"):
            BLOCK1
        BLOCK2

    Here, just as if the first line was "if True" instead, we know
    that if BLOCK1 completes without an exception, BLOCK2 will be
    reached; and if BLOCK1 raises an exception or executes a non-local
    goto (a break, continue or return), BLOCK2 is *not* reached.  The
    magic added by the with-statement at the end doesn't affect this.

    (You may ask, what if a bug in the __exit__() method causes an
    exception?  Then all is lost -- but this is no worse than with
    other exceptions; the nature of exceptions is that they can happen
    *anywhere*, and you just have to live with that.  Even if you
    write bug-free code, a KeyboardInterrupt exception can still cause
    it to exit between any two virtual machine opcodes.)

    This argument almost led me to endorse PEP 310, but I had one idea
    left from the PEP 340 euphoria that I wasn't ready to drop: using
    generators as "templates" for abstractions like acquiring and
    releasing a lock or opening and closing a file is a powerful idea,
    as can be seen by looking at the examples in that PEP.

    Inspired by a counter-proposal to PEP 340 by Phillip Eby I tried
    to create a decorator that would turn a suitable generator into an
    object with the necessary __enter__() and __exit__() methods.
    Here I ran into a snag: while it wasn't too hard for the locking
    example, it was impossible to do this for the opening example.
    The idea was to define the template like this:

        @with_template
        def opening(filename):
            f = open(filename)
            try:
                yield f
            finally:
                f.close()

    and used it like this:

        with f = opening(filename):
            ...read data from f...

    The problem is that in PEP 310, the result of calling EXPR is
    assigned directly to VAR, and then VAR's __exit__() method is
    called upon exit from BLOCK1.  But here, VAR clearly needs to
    receive the opened file, and that would mean that __exit__() would
    have to be a method on the file.

    While this can be solved using a proxy class, this is awkward and
    made me realize that a slightly different translation would make
    writing the desired decorator a piece of cake: let VAR receive the
    result from calling the __enter__() method, and save the value of
    EXPR to call its __exit__() method later.  Then the decorator can
    return an instance of a wrapper class whose __enter__() method
    calls the generator's next() method and returns whatever next()
    returns; the wrapper instance's __exit__() method calls next()
    again but expects it to raise StopIteration.  (Details below in
    the section Optional Generator Decorator.)

    So now the final hurdle was that the PEP 310 syntax:

        with VAR = EXPR:
            BLOCK1

    would be deceptive, since VAR does *not* receive the value of
    EXPR.  Borrowing from PEP 340, it was an easy step to:

        with EXPR as VAR:
            BLOCK1

    Additional discussion showed that people really liked being able
    to "see" the exception in the generator, even if it was only to
    log it; the generator is not allowed to yield another value, since
    the with-statement should not be usable as a loop (raising a
    different exception is marginally acceptable).  To enable this, a
    new throw() method for generators is proposed, which takes three
    arguments representing an exception in the usual fashion (type,
    value, traceback) and raises it at the point where the generator
    is suspended.

    Once we have this, it is a small step to proposing another
    generator method, close(), which calls throw() with a special
    exception, GeneratorExit.  This tells the generator to exit, and
    from there it's another small step to proposing that close() be
    called automatically when the generator is garbage-collected.

    Then, finally, we can allow a yield-statement inside a try-finally
    statement, since we can now guarantee that the finally-clause will
    (eventually) be executed.  The usual cautions about finalization
    apply -- the process may be terminated abruptly without finalizing
    any objects, and objects may be kept alive forever by cycles or
    memory leaks in the application (as opposed to cycles or leaks in
    the Python implementation, which are taken care of by GC).

    Note that we're not guaranteeing that the finally-clause is
    executed immediately after the generator object becomes unused,
    even though this is how it will work in CPython.  This is similar
    to auto-closing files: while a reference-counting implementation
    like CPython deallocates an object as soon as the last reference
    to it goes away, implementations that use other GC algorithms do
    not make the same guarantee.  This applies to Jython, IronPython,
    and probably to Python running on Parrot.

Use Cases

    See the Examples section near the end.

Specification: The 'with' Statement

    A new statement is proposed with the syntax:

        with EXPR as VAR:
            BLOCK

    Here, 'with' and 'as' are new keywords; EXPR is an arbitrary
    expression (but not an expression-list) and VAR is an arbitrary
    assignment target (which may be a comma-separated list).

    The "as VAR" part is optional.

    The translation of the above statement is:

        abc = EXPR
        exc = (None, None, None)
        VAR = abc.__enter__()
        try:
            try:
                BLOCK
            except:
                exc = sys.exc_info()
                raise
        finally:
            abc.__exit__(*exc)

    Here, the variables 'abc' and 'exc' are internal variables and not
    accessible to the user; they will most likely be implemented as
    special registers or stack positions.

    If the "as VAR" part of the syntax is omitted, the "VAR =" part of
    the translation is omitted (but abc.__enter__() is still called).

    The calling convention for abc.__exit__() is as follows.  If the
    finally-suite was reached through normal completion of BLOCK or
    through a non-local goto (a break, continue or return statement in
    BLOCK), abc.__exit__() is called with three None arguments.  If
    the finally-suite was reached through an exception raised in
    BLOCK, abc.__exit__() is called with three arguments representing
    the exception type, value, and traceback.

    The motivation for this API to __exit__(), as opposed to the
    argument-less __exit__() from PEP 310, was given by the
    transactional() use case, example 3 below.  The template in that
    example must commit or roll back the transaction depending on
    whether an exception occurred or not.  Rather than just having a
    boolean flag indicating whether an exception occurred, we pass the
    complete exception information, for the benefit of an
    exception-logging facility for example.  Relying on sys.exc_info()
    to get at the exception information was rejected; sys.exc_info()
    has very complex semantics and it is perfectly possible that it
    returns the exception information for an exception that was caught
    ages ago.  It was also proposed to add an additional boolean to
    distinguish between reaching the end of BLOCK and a non-local
    goto.  This was rejected as too complex and unnecessary; a
    non-local goto should be considered unexceptional for the purposes
    of a database transaction roll-back decision.

Specification: Generator Enhancements

    Let a generator object be the iterator produced by calling a
    generator function.  Below, 'g' always refers to a generator
    object.

  New syntax: yield allowed inside try-finally

    The syntax for generator functions is extended to allow a
    yield-statement inside a try-finally statement.

  New generator method: throw(type, value, traceback)

    g.throw(type, value, traceback) causes the specified exception to
    be thrown at the point where the generator g is currently
    suspended (i.e. at a yield-statement, or at the start of its
    function body if next() has not been called yet).  If the
    generator catches the exception and yields another value, that is
    the return value of g.throw().  If it doesn't catch the exception,
    the throw() appears to raise the same exception passed it (it
    "falls through").  If the generator raises another exception (this
    includes the StopIteration produced when it returns) that
    exception is raised by the throw() call.  In summary, throw()
    behaves like next() except it raises an exception at the
    suspension point.  If the generator is already in the closed
    state, throw() just raises the exception it was passed without
    executing any of the generator's code.

    The effect of raising the exception is exactly as if the
    statement:

        raise type, value, traceback

    was executed at the suspension point.  The type argument should
    not be None.

  New standard exception: GeneratorExit

    A new standard exception is defined, GeneratorExit, inheriting
    from Exception.  A generator should handle this by re-raising it
    or by raising StopIteration.

  New generator method: close()

    g.close() is defined by the following pseudo-code:

        def close(self):
            try:
                self.throw(GeneratorExit, GeneratorExit(), None)
            except (GeneratorExit, StopIteration):
                pass
            else:
                raise RuntimeError("generator ignored GeneratorExit")
            # Other exceptions are not caught

  New generator method: __del__()

    g.__del__() is an alias for g.close().  This will be called when
    the generator object is garbage-collected (in CPython, this is
    when its reference count goes to zero).  If close() raises an
    exception, a traceback for the exception is printed to sys.stderr
    and further ignored; it is not propagated back to the place that
    triggered the garbage collection.  This is consistent with the
    handling of exceptions in __del__() methods on class instances.

    If the generator object participates in a cycle, g.__del__() may
    not be called.  This is the behavior of CPython's current garbage
    collector.  The reason for the restriction is that the GC code
    needs to "break" a cycle at an arbitrary point in order to collect
    it, and from then on no Python code should be allowed to see the
    objects that formed the cycle, as they may be in an invalid state.
    Objects "hanging off" a cycle are not subject to this restriction.
    Note that it is unlikely to see a generator object participate in
    a cycle in practice.  However, storing a generator object in a
    global variable creates a cycle via the generator frame's
    f_globals pointer.  Another way to create a cycle would be to
    store a reference to the generator object in a data structure that
    is passed to the generator as an argument.  Neither of these cases
    are very likely given the typical pattern of generator use.

Generator Decorator

    It is possible to write a decorator that makes it possible to use
    a generator that yields exactly once to control a with-statement.
    Here's a sketch of such a decorator:

        class Wrapper(object):

           def __init__(self, gen):
               self.gen = gen

           def __enter__(self):
               try:
                   return self.gen.next()
               except StopIteration:
                   raise RuntimeError("generator didn't yield")

           def __exit__(self, type, value, traceback):
               if type is None:
                   try:
                       self.gen.next()
                   except StopIteration:
                       return
                   else:
                       raise RuntimeError("generator didn't stop")
               else:
                   try:
                       self.gen.throw(type, value, traceback)
                   except (type, StopIteration):
                       return
                   else:
                       raise RuntimeError("generator caught exception")

        def with_template(func):
           def helper(*args, **kwds):
               return Wrapper(func(*args, **kwds))
           return helper

    This decorator could be used as follows:

        @with_template
        def opening(filename):
           f = open(filename) # IOError here is untouched by Wrapper
           try:
               yield f
           finally:
               f.close() # Ditto for errors here (however unlikely)

    A robust implementation of this decorator should be made part of
    the standard library, but not necessarily as a built-in function.
    (I'm not sure which exception it should raise for errors;
    RuntimeError is used above as an example only.)

Optional Extensions

    It would be possible to endow certain objects, like files,
    sockets, and locks, with __enter__() and __exit__() methods so
    that instead of writing:

        with locking(myLock):
            BLOCK

    one could write simply:

        with myLock:
            BLOCK

    I think we should be careful with this; it could lead to mistakes
    like:

        f = open(filename)
        with f:
            BLOCK1
        with f:
            BLOCK2

    which does not do what one might think (f is closed before BLOCK2
    is entered).

    OTOH such mistakes are easily diagnosed; for example, the
    with_template decorator above raises RuntimeError when the second
    with-statement calls f.__enter__() again.

Open Issues

    Discussion on python-dev has revealed some open issues.  I list
    them here, with my preferred resolution and its motivation.  The
    PEP as currently written reflects this preferred resolution.

    1. What exception should be raised by close() when the generator
       yields another value as a response to the GeneratorExit
       exception?

       I originally chose TypeError because it represents gross
       misbehavior of the generator function, which should be fixed by
       changing the code.  But the with_template decorator class uses
       RuntimeError for similar offenses.  Arguably they should all
       use the same exception.  I'd rather not introduce a new
       exception class just for this purpose, since it's not an
       exception that I want people to catch: I want it to turn into a
       traceback which is seen by the programmer who then fixes the
       code.  So now I believe they should both raise RuntimeError.
       There are some precedents for that: it's raised by the core
       Python code in situations where endless recursion is detected,
       and for uninitialized objects (and for a variety of
       miscellaneous conditions).

    2. Both the generator close() method and the __exit__() method of
       the with_template decorator class catch StopIteration and
       consider it equivalent to re-raising the exception passed to
       throw().  Is allowing StopIteration right here?

       This is so that a generator doing cleanup depending on the
       exception thrown (like the transactional() example below) can
       *catch* the exception thrown if it wants to and doesn't have to
       worry about re-raising it.  I find this more convenient for the
       generator writer.  Against this was brought in that the
       generator *appears* to suppress an exception that it cannot
       suppress: the transactional() example would be more clear
       according to this view if it re-raised the original exception
       after the call to db.rollback().  I personally would find the
       requirement to re-raise the exception an annoyance in a
       generator used as a with-template, since all the code after
       yield is used for is cleanup, and it is invoked from a
       finally-clause (the one implicit in the with-statement) which
       re-raises the original exception anyway.

Examples

    (Note: several of these examples contain "yield None".  If PEP 342
    is accepted, these can be changed to just "yield".)

    1. A template for ensuring that a lock, acquired at the start of a
       block, is released when the block is left:

        @with_template
        def locking(lock):
            lock.acquire()
            try:
                yield None
            finally:
                lock.release()

       Used as follows:

        with locking(myLock):
            # Code here executes with myLock held.  The lock is
            # guaranteed to be released when the block is left (even
            # if via return or by an uncaught exception).

       PEP 319 gives a use case for also having an unlocking()
       template; this can be written very similarly (just swap the
       acquire() and release() calls).

    2. A template for opening a file that ensures the file is closed
       when the block is left:

        @with_template
        def opening(filename, mode="r"):
            f = open(filename, mode)
            try:
                yield f
            finally:
                f.close()

       Used as follows:

        with opening("/etc/passwd") as f:
            for line in f:
                print line.rstrip()

    3. A template for committing or rolling back a database
       transaction:

        @with_template
        def transactional(db):
            db.begin()
            try:
                yield None
            except:
                db.rollback()
            else:
                db.commit()

    4. Example 1 rewritten without a generator:

        class locking:
           def __init__(self, lock):
               self.lock = lock
           def __enter__(self):
               self.lock.acquire()
           def __exit__(self, type, value, tb):
               self.lock.release()

       (This example is easily modified to implement the other
       examples; it shows the relative advantage of using a generator
       template.)

    5. Redirect stdout temporarily:

        @with_template
        def redirecting_stdout(new_stdout):
            save_stdout = sys.stdout
            sys.stdout = new_stdout
            try:
                yield None
            finally:
                sys.stdout = save_stdout

       Used as follows:

        with opening(filename, "w") as f:
            with redirecting_stdout(f):
                print "Hello world"

       This isn't thread-safe, of course, but neither is doing this
       same dance manually.  In single-threaded programs (for example,
       in scripts) it is a popular way of doing things.

    6. A variant on opening() that also returns an error condition:

        @with_template
        def opening_w_error(filename, mode="r"):
            try:
                f = open(filename, mode)
            except IOError, err:
                yield None, err
            else:
                try:
                    yield f, None
                finally:
                    f.close()

       Used as follows:

        with opening_w_error("/etc/passwd", "a") as f, err:
            if err:
                print "IOError:", err
            else:
                f.write("guido::0:0::/:/bin/sh\n")

    7. Another useful example would be an operation that blocks
       signals.  The use could be like this:

        import signal

        with signal.blocking():
            # code executed without worrying about signals

       An optional argument might be a list of signals to be blocked;
       by default all signals are blocked.  The implementation is left
       as an exercise to the reader.

    8. Another use for this feature is the Decimal context.  Here's a
       simple example, after one posted by Michael Chermside:

        import decimal

        @with_template
        def extra_precision(places=2):
            c = decimal.getcontext()
            saved_prec = c.prec
            c.prec += places
            try:
                yield None
            finally:
                c.prec = saved_prec

       Sample usage (adapted from the Python Library Reference):

        def sin(x):
            "Return the sine of x as measured in radians."
            with extra_precision():
                i, lasts, s, fact, num, sign = 1, 0, x, 1, x, 1
                while s != lasts:
                    lasts = s
                    i += 2
                    fact *= i * (i-1)
                    num *= x * x
                    sign *= -1
                    s += num / fact * sign
            # The "+s" rounds back to the original precision,
            # so this must be outside the with-statement:
            return +s

    9. Here's a more general Decimal-context-switching template:

        @with_template
        def decimal_context(newctx=None):
            oldctx = decimal.getcontext()
            if newctx is None:
                newctx = oldctx.copy()
            decimal.setcontext(newctx)
            try:
                yield newctx
            finally:
                decimal.setcontext(oldctx)

       Sample usage:

        def sin(x):
            with decimal_context() as ctx:
                ctx.prec += 2
                # Rest of algorithm the same as above
            return +s

       (Nick Coghlan has proposed to add __enter__() and __exit__()
       methods to the decimal.Context class so that this example can
       be simplified to "with decimal.getcontext() as ctx: ...".)

    10. A generic "object-closing" template:

        @with_template
        def closing(obj):
            try:
                yield obj
            finally:
                obj.close()

        This can be used to deterministically close anything with a
        close method, be it file, generator, or something else:

        # emulate opening():
        with closing(open("argument.txt")) as contradiction:
           for line in contradiction:
               print line

        # deterministically finalize a generator:
        with closing(some_gen()) as data:
           for datum in data:
               process(datum)


References

    [1] http://blogs.msdn.com/oldnewthing/archive/2005/01/06/347666.aspx

    [2] http://mail.python.org/pipermail/python-dev/2005-May/053885.html

    [3] http://wiki.python.org/moin/WithStatement

Copyright

    This document has been placed in the public domain.