python-peps/pep-0343.txt

PEP: 343
Title: The "with" Statement
Version: $Revision$
Last-Modified: $Date$
Author: Guido van Rossum, Nick Coghlan
Status: Accepted
Type: Standards Track
Content-Type: text/plain
Created: 13-May-2005
Post-History: 2-Jun-2005

Abstract

    This PEP adds a new statement "with" to the Python language to make
    it possible to factor out standard uses of try/finally statements.

    The PEP has been approved in principle by the BDFL, but there are
    still a couple of implementation details to be worked out (see the
    section on Open Issues).

Author's Note

    This PEP was originally written in first person by Guido, and
    subsequently updated by Nick Coghlan to reflect later discussion
    on python-dev. Any first person references are from Guido's
    original.

Introduction

    After a lot of discussion about PEP 340 and alternatives, I
    decided to withdraw PEP 340 and proposed a slight variant on PEP
    310.  After more discussion, I have added back a mechanism for
    raising an exception in a suspended generator using a throw()
    method, and a close() method which throws a new GeneratorExit
    exception; these additions were first proposed on python-dev in
    [2] and universally approved of.  I'm also changing the keyword to
    'with'.

    On-line discussion of this PEP should take place in the Python
    Wiki [3].

    If this PEP is approved, the following PEPs will be rejected due
    to overlap:

    - PEP 310, Reliable Acquisition/Release Pairs.  This is the
      original with-statement proposal.

    - PEP 319, Python Synchronize/Asynchronize Block.  Its use cases
      can be covered by the current PEP by providing suitable
      with-statement controllers: for 'synchronize' we can use the
      "locking" template from example 1; for 'asynchronize' we can use
      a similar "unlocking" template.  I don't think having an
      "anonymous" lock associated with a code block is all that
      important; in fact it may be better to always be explicit about
      the mutex being used.

    (PEP 340 and PEP 346 have already been withdrawn.)

Motivation and Summary

    PEP 340, Anonymous Block Statements, combined many powerful ideas:
    using generators as block templates, adding exception handling and
    finalization to generators, and more.  Besides praise it received
    a lot of opposition from people who didn't like the fact that it
    was, under the covers, a (potential) looping construct.  This
    meant that break and continue in a block-statement would break or
    continue the block-statement, even if it was used as a non-looping
    resource management tool.

    But the final blow came when I read Raymond Chen's rant about
    flow-control macros[1].  Raymond argues convincingly that hiding
    flow control in macros makes your code inscrutable, and I find
    that his argument applies to Python as well as to C.  I realized
    that PEP 340 templates can hide all sorts of control flow; for
    example, its example 4 (auto_retry()) catches exceptions and
    repeats the block up to three times.

    However, the with-statement of PEP 310 does *not* hide control
    flow, in my view: while a finally-suite temporarily suspends the
    control flow, in the end, the control flow resumes as if the
    finally-suite wasn't there at all.

    Remember, PEP 310 proposes rougly this syntax (the "VAR =" part is
    optional):

        with VAR = EXPR:
            BLOCK

    which roughly translates into this:

        VAR = EXPR
        VAR.__enter__()
        try:
            BLOCK
        finally:
            VAR.__exit__()

    Now consider this example:

        with f = opening("/etc/passwd"):
            BLOCK1
        BLOCK2

    Here, just as if the first line was "if True" instead, we know
    that if BLOCK1 completes without an exception, BLOCK2 will be
    reached; and if BLOCK1 raises an exception or executes a non-local
    goto (a break, continue or return), BLOCK2 is *not* reached.  The
    magic added by the with-statement at the end doesn't affect this.

    (You may ask, what if a bug in the __exit__() method causes an
    exception?  Then all is lost -- but this is no worse than with
    other exceptions; the nature of exceptions is that they can happen
    *anywhere*, and you just have to live with that.  Even if you
    write bug-free code, a KeyboardInterrupt exception can still cause
    it to exit between any two virtual machine opcodes.)

    This argument almost led me to endorse PEP 310, but I had one idea
    left from the PEP 340 euphoria that I wasn't ready to drop: using
    generators as "templates" for abstractions like acquiring and
    releasing a lock or opening and closing a file is a powerful idea,
    as can be seen by looking at the examples in that PEP.

    Inspired by a counter-proposal to PEP 340 by Phillip Eby I tried
    to create a decorator that would turn a suitable generator into an
    object with the necessary __enter__() and __exit__() methods.
    Here I ran into a snag: while it wasn't too hard for the locking
    example, it was impossible to do this for the opening example.
    The idea was to define the template like this:

        @contextmanager
        def opening(filename):
            f = open(filename)
            try:
                yield f
            finally:
                f.close()

    and used it like this:

        with f = opening(filename):
            ...read data from f...

    The problem is that in PEP 310, the result of calling EXPR is
    assigned directly to VAR, and then VAR's __exit__() method is
    called upon exit from BLOCK1.  But here, VAR clearly needs to
    receive the opened file, and that would mean that __exit__() would
    have to be a method on the file.

    While this can be solved using a proxy class, this is awkward and
    made me realize that a slightly different translation would make
    writing the desired decorator a piece of cake: let VAR receive the
    result from calling the __enter__() method, and save the value of
    EXPR to call its __exit__() method later.  Then the decorator can
    return an instance of a wrapper class whose __enter__() method
    calls the generator's next() method and returns whatever next()
    returns; the wrapper instance's __exit__() method calls next()
    again but expects it to raise StopIteration.  (Details below in
    the section Optional Generator Decorator.)

    So now the final hurdle was that the PEP 310 syntax:

        with VAR = EXPR:
            BLOCK1

    would be deceptive, since VAR does *not* receive the value of
    EXPR.  Borrowing from PEP 340, it was an easy step to:

        with EXPR as VAR:
            BLOCK1

    Additional discussion showed that people really liked being able
    to "see" the exception in the generator, even if it was only to
    log it; the generator is not allowed to yield another value, since
    the with-statement should not be usable as a loop (raising a
    different exception is marginally acceptable).  To enable this, a
    new throw() method for generators is proposed, which takes one to
    three arguments representing an exception in the usual fashion
    (type, value, traceback) and raises it at the point where the
    generator is suspended.

    Once we have this, it is a small step to proposing another
    generator method, close(), which calls throw() with a special
    exception, GeneratorExit.  This tells the generator to exit, and
    from there it's another small step to proposing that close() be
    called automatically when the generator is garbage-collected.

    Then, finally, we can allow a yield-statement inside a try-finally
    statement, since we can now guarantee that the finally-clause will
    (eventually) be executed.  The usual cautions about finalization
    apply -- the process may be terminated abruptly without finalizing
    any objects, and objects may be kept alive forever by cycles or
    memory leaks in the application (as opposed to cycles or leaks in
    the Python implementation, which are taken care of by GC).

    Note that we're not guaranteeing that the finally-clause is
    executed immediately after the generator object becomes unused,
    even though this is how it will work in CPython.  This is similar
    to auto-closing files: while a reference-counting implementation
    like CPython deallocates an object as soon as the last reference
    to it goes away, implementations that use other GC algorithms do
    not make the same guarantee.  This applies to Jython, IronPython,
    and probably to Python running on Parrot.

Use Cases

    See the Examples section near the end.

Specification: The 'with' Statement

    A new statement is proposed with the syntax:

        with EXPR as VAR:
            BLOCK

    Here, 'with' and 'as' are new keywords; EXPR is an arbitrary
    expression (but not an expression-list) and VAR is a single
    assignment target.  It can *not* be a comma-separated sequence of
    variables, but it *can* be a *parenthesized* comma-separated
    sequence of variables.  (This restriction makes a future extension
    possible of the syntax to have multiple comma-separated resources,
    each with its own optional as-clause.)

    The "as VAR" part is optional.

    The translation of the above statement is:

        abc = (EXPR).__with__()
        exc = (None, None, None)
        VAR = abc.__enter__()
        try:
            try:
                BLOCK
            except:
                exc = sys.exc_info()
                raise
        finally:
            abc.__exit__(*exc)

    Here, the variables 'abc' and 'exc' are internal variables and not
    accessible to the user; they will most likely be implemented as
    special registers or stack positions.

    The call to the __with__() method serves a similar purpose to that
    of the __iter__() method of iterator and iterables. An object with
    with simple state requirements (such as threading.RLock) may provide
    its own __enter__() and __exit__() methods, and simply return
    'self' from its __with__ method. On the other hand, an object with
    more complex state requirements (such as decimal.Context) may
    return a distinct context manager object each time its __with__
    method is invoked.

    If the "as VAR" part of the syntax is omitted, the "VAR =" part of
    the translation is omitted (but abc.__enter__() is still called).

    The calling convention for abc.__exit__() is as follows.  If the
    finally-suite was reached through normal completion of BLOCK or
    through a non-local goto (a break, continue or return statement in
    BLOCK), abc.__exit__() is called with three None arguments.  If
    the finally-suite was reached through an exception raised in
    BLOCK, abc.__exit__() is called with three arguments representing
    the exception type, value, and traceback.

    The motivation for this API to __exit__(), as opposed to the
    argument-less __exit__() from PEP 310, was given by the
    transactional() use case, example 3 below.  The template in that
    example must commit or roll back the transaction depending on
    whether an exception occurred or not.  Rather than just having a
    boolean flag indicating whether an exception occurred, we pass the
    complete exception information, for the benefit of an
    exception-logging facility for example.  Relying on sys.exc_info()
    to get at the exception information was rejected; sys.exc_info()
    has very complex semantics and it is perfectly possible that it
    returns the exception information for an exception that was caught
    ages ago.  It was also proposed to add an additional boolean to
    distinguish between reaching the end of BLOCK and a non-local
    goto.  This was rejected as too complex and unnecessary; a
    non-local goto should be considered unexceptional for the purposes
    of a database transaction roll-back decision.

Generator Decorator

    With PEP 342 accepted, it is possible to write a decorator
    that makes it possible to use a generator that yields exactly once
    to control a with-statement.  Here's a sketch of such a decorator:

        class GeneratorContext(object):

           def __init__(self, gen):
               self.gen = gen

           def __with__(self):
               return self

           def __enter__(self):
               try:
                   return self.gen.next()
               except StopIteration:
                   raise RuntimeError("generator didn't yield")

           def __exit__(self, type, value, traceback):
               if type is None:
                   try:
                       self.gen.next()
                   except StopIteration:
                       return
                   else:
                       raise RuntimeError("generator didn't stop")
               else:
                   try:
                       self.gen.throw(type, value, traceback)
                   except (type, StopIteration):
                       return
                   else:
                       raise RuntimeError("generator caught exception")

        def context(func):
           def helper(*args, **kwds):
               return GeneratorContext(func(*args, **kwds))
           return helper

    This decorator could be used as follows:

        @context
        def opening(filename):
           f = open(filename) # IOError is untouched by GeneratorContext
           try:
               yield f
           finally:
               f.close() # Ditto for errors here (however unlikely)

    A robust implementation of this decorator should be made part of
    the standard library. Refer to Open Issues regarding its name and
    location.

    Just as generator-iterator functions are very useful for writing
    __iter__() methods for iterables, generator-context functions will
    be very useful for writing __with__() methods for contexts. It is
    proposed that the invocation of the "context" decorator be
    considered implicit for generator functions used as __with__()
    methods (again, refer to the Open Issues section).

Optional Extensions

    It would be possible to endow certain objects, like files,
    sockets, and locks, with __enter__() and __exit__() methods so
    that instead of writing:

        with locking(myLock):
            BLOCK

    one could write simply:

        with myLock:
            BLOCK

    I think we should be careful with this; it could lead to mistakes
    like:

        f = open(filename)
        with f:
            BLOCK1
        with f:
            BLOCK2

    which does not do what one might think (f is closed before BLOCK2
    is entered).

    OTOH such mistakes are easily diagnosed; for example, the
    generator-context decorator above raises RuntimeError when a
    second  with-statement calls f.__enter__() again. A similar error
    can be raised if __enter__ is invoked on a closed file object.

Standard Terminology

    Discussions about iterators and iterables are aided by the standard
    terminology used to discuss them. The protocol used by the for
    statement is called the iterator protocol and an iterator is any
    object that properly implements that protocol. The term "iterable"
    then encompasses all objects with an __iter__() method that
    returns an iterator (this means that all iterators are iterables,
    but not all iterables are iterators).

    This PEP proposes that the protocol used by the with statement be
    known as the "context management protocol", and that objects that
    implement that protocol be known as "context managers". The term
    "context" then encompasses all objects with a __with__() method
    that returns a context manager (this means that all context managers
    are contexts, but not all contexts are context managers).

    The term "context" is based on the concept that the context object
    defines a context of execution for the code that forms the body
    of the with statement.

    In cases where the general term "context" would be ambiguous, it
    can be made explicit by expanding it to "manageable context".

Open Issues

    Discussion on python-dev revealed some open issues. These are listed
    here and will be resolved either by consensus on python-dev or by
    BDFL fiat.

    1. The name of the decorator used to convert a generator-iterator
       function into a generator-context function is still to be
       finalised.
       The proposal in this PEP is that it be called simply "context"
       with the following reasoning:
        - A "generator function" is an undecorated function containing
          the 'yield' keyword, and the objects produced by
          such functions are "generator-iterators". The term
          "generator" may refer to either a generator function or a
          generator-iterator depending on the situation.
        - A "generator context function" is a generator function to
          which the "context" decorator is applied and the objects
          produced by such functions are "generator-context-managers".
          The term "generator context" may refer to either a generator
          context function or a generator-context-manager depending on
          the situation.

    2. Should the decorator to convert a generator function into a
       generator context function be a builtin, or located elsewhere in
       the standard library? This PEP suggests that it should be a
       builtin, as generator context functions are the recommended way
       of writing new context managers.

    3. Should a generator function used to implement a __with__ method
       always be considered to be a generator context function, without
       requiring the context decorator? This PEP suggests that it
       should, as applying a decorator to a slot just looks strange,
       and omitting the decorator would be a source of obscure bugs.
       The __new__ slot provides some precedent for special casing of
       certain slots when processing slot methods.

Resolved Issues

    The following issues were resolved either by BDFL fiat, consensus on
    python-dev, or a simple lack of objection to proposals in the
    original version of this PEP.

    1. The __exit__() method of the GeneratorContext class
       catches StopIteration and considers it equivalent to re-raising
       the exception passed to throw().  Is allowing StopIteration
       right here?

       This is so that a generator doing cleanup depending on the
       exception thrown (like the transactional() example below) can
       *catch* the exception thrown if it wants to and doesn't have to
       worry about re-raising it.  I find this more convenient for the
       generator writer.  Against this was brought in that the
       generator *appears* to suppress an exception that it cannot
       suppress: the transactional() example would be more clear
       according to this view if it re-raised the original exception
       after the call to db.rollback().  I personally would find the
       requirement to re-raise the exception an annoyance in a
       generator used as a with-template, since all the code after
       yield is used for is cleanup, and it is invoked from a
       finally-clause (the one implicit in the with-statement) which
       re-raises the original exception anyway.

    2. What exception should GeneratorContext raise when the underlying
       generator-iterator misbehaves? The following quote is the reason
       behind Guido's choice of RuntimeError for both this and for the
       generator close() method in PEP 342 (from [8]):

       "I'd rather not introduce a new exception class just for this
       purpose, since it's not an exception that I want people to catch:
       I want it to turn into a traceback which is seen by the
       programmer who then fixes the code.  So now I believe they
       should both raise RuntimeError.
       There are some precedents for that: it's raised by the core
       Python code in situations where endless recursion is detected,
       and for uninitialized objects (and for a variety of
       miscellaneous conditions)."

    3. After this PEP was originally approved, a subsequent discussion
       on python-dev [4] settled on the term "context manager" for
       objects which provide __enter__ and __exit__ methods, and
       "context management protocol" for the protocol itself. With the
       addition of the __with__ method to the protocol, a natural
       extension is to call all objects which provide a __with__ method
       "contexts" (or "manageable contexts" in situations where the
       general term "context" would be ambiguous).
       This is now documented in the "Standard Terminology" section.

    4. The originally approved version of this PEP did not include a
       __with__ method - the method was only added to the PEP after
       Jason Orendorff pointed out the difficulty of writing
       appropriate __enter__ and __exit__ methods for decimal.Context
       [5]. This approach allows a class to define a native context
       manager using generator syntax. It also allows a class to use an
       existing independent context manager as its native context
       manager by applying the independent context manager to 'self' in
       its __with__ method. It even allows a class written in C to use
       a generator context manager written in Python.
       The __with__ method parallels the __iter__ method which forms
       part of the iterator protocol.

    5. The suggestion was made by Jason Orendorff that the __enter__
       and __exit__ methods could be removed from the context
       management protocol, and the protocol instead defined directly
       in terms of the enhanced generator interface described in PEP
       342 [6].
       Guido rejected this idea [7]. The following are some of benefits
       of keeping the __enter__ and __exit__ methods:
          - it makes it easy to implement a simple context manager in C
            without having to rely on a separate coroutine builder
          - it makes it easy to provide a low-overhead implementation
            for context managers which don't need to maintain any
            special state between the __enter__ and __exit__ methods
            (having to use a generator for these would impose
            unnecessary overhead without any compensating benefit)
          - it makes it possible to understand how the with statement
            works without having to first understand the mechanics of
            how generator context managers are implemented.

Examples

    (The generator based examples assume PEP 342 is implemented. Also,
    some of the examples are likely to be unnecessary in practice, as
    the appropriate objects, such as threading.RLock, will be able to
    be used directly in with statements)

    1. A template for ensuring that a lock, acquired at the start of a
       block, is released when the block is left:

        @context
        def locking(lock):
            lock.acquire()
            try:
                yield
            finally:
                lock.release()

       Used as follows:

        with locking(myLock):
            # Code here executes with myLock held.  The lock is
            # guaranteed to be released when the block is left (even
            # if via return or by an uncaught exception).

       PEP 319 gives a use case for also having an unlocking()
       template; this can be written very similarly (just swap the
       acquire() and release() calls).

    2. A template for opening a file that ensures the file is closed
       when the block is left:

        @context
        def opening(filename, mode="r"):
            f = open(filename, mode)
            try:
                yield f
            finally:
                f.close()

       Used as follows:

        with opening("/etc/passwd") as f:
            for line in f:
                print line.rstrip()

    3. A template for committing or rolling back a database
       transaction:

        @context
        def transactional(db):
            db.begin()
            try:
                yield None
            except:
                db.rollback()
            else:
                db.commit()

    4. Example 1 rewritten without a generator:

        class locking:
           def __init__(self, lock):
               self.lock = lock
           def __with__(self, lock):
               return self
           def __enter__(self):
               self.lock.acquire()
           def __exit__(self, type, value, tb):
               self.lock.release()

       (This example is easily modified to implement the other
       examples; it shows that is is easy to avoid the need for a
       generator if no special state needs to be preserved.)

    5. Redirect stdout temporarily:

        @context
        def redirecting_stdout(new_stdout):
            save_stdout = sys.stdout
            sys.stdout = new_stdout
            try:
                yield None
            finally:
                sys.stdout = save_stdout

       Used as follows:

        with opening(filename, "w") as f:
            with redirecting_stdout(f):
                print "Hello world"

       This isn't thread-safe, of course, but neither is doing this
       same dance manually.  In single-threaded programs (for example,
       in scripts) it is a popular way of doing things.

    6. A variant on opening() that also returns an error condition:

        @context
        def opening_w_error(filename, mode="r"):
            try:
                f = open(filename, mode)
            except IOError, err:
                yield None, err
            else:
                try:
                    yield f, None
                finally:
                    f.close()

       Used as follows:

        with opening_w_error("/etc/passwd", "a") as (f, err):
            if err:
                print "IOError:", err
            else:
                f.write("guido::0:0::/:/bin/sh\n")

    7. Another useful example would be an operation that blocks
       signals.  The use could be like this:

        import signal

        with signal.blocking():
            # code executed without worrying about signals

       An optional argument might be a list of signals to be blocked;
       by default all signals are blocked.  The implementation is left
       as an exercise to the reader.

    8. Another use for this feature is the Decimal context.  Here's a
       simple example, after one posted by Michael Chermside:

        import decimal

        @contextmanager
        def extra_precision(places=2):
            c = decimal.getcontext()
            saved_prec = c.prec
            c.prec += places
            try:
                yield None
            finally:
                c.prec = saved_prec

       Sample usage (adapted from the Python Library Reference):

        def sin(x):
            "Return the sine of x as measured in radians."
            with extra_precision():
                i, lasts, s, fact, num, sign = 1, 0, x, 1, x, 1
                while s != lasts:
                    lasts = s
                    i += 2
                    fact *= i * (i-1)
                    num *= x * x
                    sign *= -1
                    s += num / fact * sign
            # The "+s" rounds back to the original precision,
            # so this must be outside the with-statement:
            return +s

     9. Here's a proposed native context manager for decimal.Context:

         # This would be a new decimal.Context method
         def __with__(self):
             # We set the thread context to a copy of this context
             # to ensure that changes within the block are kept
             # local to the block. This also gives us thread safety
             # and supports nested usage of a given context.
             newctx = self.copy()
             oldctx = decimal.getcontext()
             decimal.setcontext(newctx)
             try:
                 yield newctx
             finally:
                 decimal.setcontext(oldctx)

        Sample usage:

         def sin(x):
             with decimal.getcontext() as ctx:
                 ctx.prec += 2
                 # Rest of sin calculation algorithm
                 # uses a precision 2 greater than normal
             return +s # Convert result to normal precision

         def sin(x):
             with decimal.ExtendedContext:
                 # Rest of sin calculation algorithm
                 # uses the Extended Context from the
                 # General Decimal Arithmetic Specification
             return +s # Convert result to normal context

    10. A generic "object-closing" template:

        @context
        def closing(obj):
            try:
                yield obj
            finally:
                try:
                    close = obj.close
                except AttributeError:
                    pass
                else:
                    close()

        This can be used to deterministically close anything with a
        close method, be it file, generator, or something else. It can
        even be used when the object isn't guaranteed to require
        closing (e.g., a function that accepts an arbitrary iterable):

        # emulate opening():
        with closing(open("argument.txt")) as contradiction:
           for line in contradiction:
               print line

        # deterministically finalize an iterator:
        with closing(iter(data_source)) as data:
           for datum in data:
               process(datum)

References

    [1] http://blogs.msdn.com/oldnewthing/archive/2005/01/06/347666.aspx

    [2] http://mail.python.org/pipermail/python-dev/2005-May/053885.html

    [3] http://wiki.python.org/moin/WithStatement

    [4]
    http://mail.python.org/pipermail/python-dev/2005-July/054658.html

    [5]
    http://mail.python.org/pipermail/python-dev/2005-October/056947.html

    [6]
    http://mail.python.org/pipermail/python-dev/2005-October/056969.html

    [7]
    http://mail.python.org/pipermail/python-dev/2005-October/057018.html

    [8]
    http://mail.python.org/pipermail/python-dev/2005-June/054064.html

Copyright

    This document has been placed in the public domain.