PEP: 343 Title: The "with" Statement Version: $Revision$ Last-Modified: $Date$ Author: Guido van Rossum, Nick Coghlan Status: Accepted Type: Standards Track Content-Type: text/plain Created: 13-May-2005 Post-History: 2-Jun-2005, 16-Oct-2005, 29-Oct-2005 Abstract This PEP adds a new statement "with" to the Python language to make it possible to factor out standard uses of try/finally statements. The PEP was approved in principle by the BDFL, but there were still a couple of implementation details to be worked out (see the section on Resolved Issues). It's still at Draft status until Guido gives a final blessing to the updated PEP. Author's Note This PEP was originally written in first person by Guido, and subsequently updated by Nick Coghlan to reflect later discussion on python-dev. Any first person references are from Guido's original. Introduction After a lot of discussion about PEP 340 and alternatives, I decided to withdraw PEP 340 and proposed a slight variant on PEP 310. After more discussion, I have added back a mechanism for raising an exception in a suspended generator using a throw() method, and a close() method which throws a new GeneratorExit exception; these additions were first proposed on python-dev in [2] and universally approved of. I'm also changing the keyword to 'with'. On-line discussion of this PEP should take place in the Python Wiki [3]. If this PEP is approved, the following PEPs will be rejected due to overlap: - PEP 310, Reliable Acquisition/Release Pairs. This is the original with-statement proposal. - PEP 319, Python Synchronize/Asynchronize Block. Its use cases can be covered by the current PEP by providing suitable with-statement controllers: for 'synchronize' we can use the "locking" template from example 1; for 'asynchronize' we can use a similar "unlocking" template. I don't think having an "anonymous" lock associated with a code block is all that important; in fact it may be better to always be explicit about the mutex being used. (PEP 340 and PEP 346 have already been withdrawn.) Motivation and Summary PEP 340, Anonymous Block Statements, combined many powerful ideas: using generators as block templates, adding exception handling and finalization to generators, and more. Besides praise it received a lot of opposition from people who didn't like the fact that it was, under the covers, a (potential) looping construct. This meant that break and continue in a block-statement would break or continue the block-statement, even if it was used as a non-looping resource management tool. But the final blow came when I read Raymond Chen's rant about flow-control macros[1]. Raymond argues convincingly that hiding flow control in macros makes your code inscrutable, and I find that his argument applies to Python as well as to C. I realized that PEP 340 templates can hide all sorts of control flow; for example, its example 4 (auto_retry()) catches exceptions and repeats the block up to three times. However, the with-statement of PEP 310 does *not* hide control flow, in my view: while a finally-suite temporarily suspends the control flow, in the end, the control flow resumes as if the finally-suite wasn't there at all. Remember, PEP 310 proposes rougly this syntax (the "VAR =" part is optional): with VAR = EXPR: BLOCK which roughly translates into this: VAR = EXPR VAR.__enter__() try: BLOCK finally: VAR.__exit__() Now consider this example: with f = open("/etc/passwd"): BLOCK1 BLOCK2 Here, just as if the first line was "if True" instead, we know that if BLOCK1 completes without an exception, BLOCK2 will be reached; and if BLOCK1 raises an exception or executes a non-local goto (a break, continue or return), BLOCK2 is *not* reached. The magic added by the with-statement at the end doesn't affect this. (You may ask, what if a bug in the __exit__() method causes an exception? Then all is lost -- but this is no worse than with other exceptions; the nature of exceptions is that they can happen *anywhere*, and you just have to live with that. Even if you write bug-free code, a KeyboardInterrupt exception can still cause it to exit between any two virtual machine opcodes.) This argument almost led me to endorse PEP 310, but I had one idea left from the PEP 340 euphoria that I wasn't ready to drop: using generators as "templates" for abstractions like acquiring and releasing a lock or opening and closing a file is a powerful idea, as can be seen by looking at the examples in that PEP. Inspired by a counter-proposal to PEP 340 by Phillip Eby I tried to create a decorator that would turn a suitable generator into an object with the necessary __enter__() and __exit__() methods. Here I ran into a snag: while it wasn't too hard for the locking example, it was impossible to do this for the opening example. The idea was to define the template like this: @contextmanager def opening(filename): f = open(filename) try: yield f finally: f.close() and used it like this: with f = opening(filename): ...read data from f... The problem is that in PEP 310, the result of calling EXPR is assigned directly to VAR, and then VAR's __exit__() method is called upon exit from BLOCK1. But here, VAR clearly needs to receive the opened file, and that would mean that __exit__() would have to be a method on the file. While this can be solved using a proxy class, this is awkward and made me realize that a slightly different translation would make writing the desired decorator a piece of cake: let VAR receive the result from calling the __enter__() method, and save the value of EXPR to call its __exit__() method later. Then the decorator can return an instance of a wrapper class whose __enter__() method calls the generator's next() method and returns whatever next() returns; the wrapper instance's __exit__() method calls next() again but expects it to raise StopIteration. (Details below in the section Optional Generator Decorator.) So now the final hurdle was that the PEP 310 syntax: with VAR = EXPR: BLOCK1 would be deceptive, since VAR does *not* receive the value of EXPR. Borrowing from PEP 340, it was an easy step to: with EXPR as VAR: BLOCK1 Additional discussion showed that people really liked being able to "see" the exception in the generator, even if it was only to log it; the generator is not allowed to yield another value, since the with-statement should not be usable as a loop (raising a different exception is marginally acceptable). To enable this, a new throw() method for generators is proposed, which takes one to three arguments representing an exception in the usual fashion (type, value, traceback) and raises it at the point where the generator is suspended. Once we have this, it is a small step to proposing another generator method, close(), which calls throw() with a special exception, GeneratorExit. This tells the generator to exit, and from there it's another small step to proposing that close() be called automatically when the generator is garbage-collected. Then, finally, we can allow a yield-statement inside a try-finally statement, since we can now guarantee that the finally-clause will (eventually) be executed. The usual cautions about finalization apply -- the process may be terminated abruptly without finalizing any objects, and objects may be kept alive forever by cycles or memory leaks in the application (as opposed to cycles or leaks in the Python implementation, which are taken care of by GC). Note that we're not guaranteeing that the finally-clause is executed immediately after the generator object becomes unused, even though this is how it will work in CPython. This is similar to auto-closing files: while a reference-counting implementation like CPython deallocates an object as soon as the last reference to it goes away, implementations that use other GC algorithms do not make the same guarantee. This applies to Jython, IronPython, and probably to Python running on Parrot. Use Cases See the Examples section near the end. Specification: The 'with' Statement A new statement is proposed with the syntax: with EXPR as VAR: BLOCK Here, 'with' and 'as' are new keywords; EXPR is an arbitrary expression (but not an expression-list) and VAR is a single assignment target. It can *not* be a comma-separated sequence of variables, but it *can* be a *parenthesized* comma-separated sequence of variables. (This restriction makes a future extension possible of the syntax to have multiple comma-separated resources, each with its own optional as-clause.) The "as VAR" part is optional. The translation of the above statement is: ctx = (EXPR).__context__() exit = ctx.__exit__ # Not calling it yet value = ctx.__enter__() exc = True try: try: VAR = value # Only if "as VAR" is present BLOCK except: # The exceptional case is handled here exc = False if not exit(*sys.exc_info()): raise # The exception is swallowed if exit() returns true finally: # The normal and non-local-goto cases are handled here if exc: exit(None, None, None) Here, the lowercase variables (ctx, exit, value, exc) are internal variables and not accessible to the user; they will most likely be implemented as special registers or stack positions. The details of the above translation are intended to prescribe the exact semantics. If any of the relevant methods are not found as expected, the interpreter will raise AttributeError, in the order that they are tried (__context__, __exit__, __enter__). Similarly, if any of the calls raises an exception, the effect is exactly as it would be in the above code. Finally, if BLOCK contains a break, continue or return statement, the __exit__() method is called with three None arguments just as if BLOCK completed normally. (I.e. these "pseudo-exceptions" are not seen as exceptions by __exit__().) The call to the __context__() method serves a similar purpose to that of the __iter__() method of iterator and iterables. An object with with simple state requirements (such as threading.RLock) may provide its own __enter__() and __exit__() methods, and simply return 'self' from its __context__ method. On the other hand, an object with more complex state requirements (such as decimal.Context) may return a distinct context manager object each time its __context__ method is invoked. If the "as VAR" part of the syntax is omitted, the "VAR =" part of the translation is omitted (but ctx.__enter__() is still called). The calling convention for ctx.__exit__() is as follows. If the finally-suite was reached through normal completion of BLOCK or through a non-local goto (a break, continue or return statement in BLOCK), ctx.__exit__() is called with three None arguments. If the finally-suite was reached through an exception raised in BLOCK, ctx.__exit__() is called with three arguments representing the exception type, value, and traceback. IMPORTANT: if ctx.__exit__() returns a "true" value, the exception is "swallowed". That is, if it returns "true", execution continues at the next statement after the with-statement, even if an exception happened inside the with-statement. However, if the with-statement was left via a non-local goto (break, continue or return), this non-local return is resumed when ctx.__exit__() returns regardless of the return value. The motivation for this detail is to make it possible for ctx.__exit__() to swallow exceptions, without making it too easy (since the default return value, None, is false and this causes the exception to be re-raised). The main use case for swallowing exceptions is to make it possible to write the @contextmanager decorator so thatn that a try/except block in a decorated generator behaves exactly as if the body of the generator were expanded in-line at the place of the with-statement. The motivation for passing the exception details to __exit__(), as opposed to the argument-less __exit__() from PEP 310, was given by the transactional() use case, example 3 below. The template in that example must commit or roll back the transaction depending on whether an exception occurred or not. Rather than just having a boolean flag indicating whether an exception occurred, we pass the complete exception information, for the benefit of an exception-logging facility for example. Relying on sys.exc_info() to get at the exception information was rejected; sys.exc_info() has very complex semantics and it is perfectly possible that it returns the exception information for an exception that was caught ages ago. It was also proposed to add an additional boolean to distinguish between reaching the end of BLOCK and a non-local goto. This was rejected as too complex and unnecessary; a non-local goto should be considered unexceptional for the purposes of a database transaction roll-back decision. Transition Plan In Python 2.5, the new syntax will only be recognized if a future statement is present: from __future__ import with_statement This will make both 'with' and 'as' keywords. Without the future statement, using 'with' or 'as' as an identifier will cause a Warning to be issued to stderr. In Python 2.6, the new syntax will always be recognized; 'with' and 'as' are always keywords. Generator Decorator With PEP 342 accepted, it is possible to write a decorator that makes it possible to use a generator that yields exactly once to control a with-statement. Here's a sketch of such a decorator: class GeneratorContextManager(object): def __init__(self, gen): self.gen = gen def __context__(self): return self def __enter__(self): try: return self.gen.next() except StopIteration: raise RuntimeError("generator didn't yield") def __exit__(self, type, value, traceback): if type is None: try: self.gen.next() except StopIteration: return else: raise RuntimeError("generator didn't stop") else: try: self.gen.throw(type, value, traceback) return True except StopIteration: return True def contextmanager(func): def helper(*args, **kwds): return GeneratorContextManager(func(*args, **kwds)) return helper This decorator could be used as follows: @contextmanager def opening(filename): f = open(filename) # IOError is untouched by GeneratorContext try: yield f finally: f.close() # Ditto for errors here (however unlikely) A robust builtin implementation of this decorator will be made part of the standard library. Just as generator-iterator functions are very useful for writing __iter__() methods for iterables, generator-context functions will be very useful for writing __context__() methods for contexts. These methods will still need to be decorated using the contextmanager decorator. To ensure an obvious error message if the decorator is left out, generator-iterator objects will NOT be given a native context - if you want to ensure a generator is closed promptly, use something similar to the duck-typed "closing" context manager in the examples. Optional Extensions It would be possible to endow certain objects, like files, sockets, and locks, with __enter__() and __exit__() methods so that instead of writing: with locking(myLock): BLOCK one could write simply: with myLock: BLOCK I think we should be careful with this; it could lead to mistakes like: f = open(filename) with f: BLOCK1 with f: BLOCK2 which does not do what one might think (f is closed before BLOCK2 is entered). OTOH such mistakes are easily diagnosed; for example, the generator-context decorator above raises RuntimeError when a second with-statement calls f.__enter__() again. A similar error can be raised if __enter__ is invoked on a closed file object. For Python 2.5, the following candidates have been identified for native context managers: - file - decimal.Context - thread.LockType - threading.Lock - threading.RLock - threading.Condition - threading.Semaphore and threading.BoundedSemaphore Standard Terminology Discussions about iterators and iterables are aided by the standard terminology used to discuss them. The protocol used by the for statement is called the iterator protocol and an iterator is any object that properly implements that protocol. The term "iterable" then encompasses all objects with an __iter__() method that returns an iterator (this means that all iterators are iterables, but not all iterables are iterators). This PEP proposes that the protocol used by the with statement be known as the "context management protocol", and that objects that implement that protocol be known as "context managers". The term "context" then encompasses all objects with a __context__() method that returns a context manager (this means that all context managers are contexts, but not all contexts are context managers). The term "context" is based on the concept that the context object defines a context of execution for the code that forms the body of the with statement. In cases where the general term "context" would be ambiguous, it can be made explicit by expanding it to "manageable context". Resolved Issues The following issues were resolved either by BDFL approval, consensus on python-dev, or a simple lack of objection to proposals in the original version of this PEP. 1. The __exit__() method of the GeneratorContextManager class catches StopIteration and considers it equivalent to re-raising the exception passed to throw(). Is allowing StopIteration right here? This is so that a generator doing cleanup depending on the exception thrown (like the transactional() example below) can *catch* the exception thrown if it wants to and doesn't have to worry about re-raising it. I find this more convenient for the generator writer. Against this was brought in that the generator *appears* to suppress an exception that it cannot suppress: the transactional() example would be more clear according to this view if it re-raised the original exception after the call to db.rollback(). I personally would find the requirement to re-raise the exception an annoyance in a generator used as a with-template, since all the code after yield is used for is cleanup, and it is invoked from a finally-clause (the one implicit in the with-statement) which re-raises the original exception anyway. 2. What exception should GeneratorContextManager raise when the underlying generator-iterator misbehaves? The following quote is the reason behind Guido's choice of RuntimeError for both this and for the generator close() method in PEP 342 (from [8]): "I'd rather not introduce a new exception class just for this purpose, since it's not an exception that I want people to catch: I want it to turn into a traceback which is seen by the programmer who then fixes the code. So now I believe they should both raise RuntimeError. There are some precedents for that: it's raised by the core Python code in situations where endless recursion is detected, and for uninitialized objects (and for a variety of miscellaneous conditions)." 3. After this PEP was originally approved, a subsequent discussion on python-dev [4] settled on the term "context manager" for objects which provide __enter__ and __exit__ methods, and "context management protocol" for the protocol itself. With the addition of the __context__ method to the protocol, a natural extension is to call all objects which provide a __context__ method "contexts" (or "manageable contexts" in situations where the general term "context" would be ambiguous). This is now documented in the "Standard Terminology" section. 4. The originally approved version of this PEP did not include a __context__ method - the method was only added to the PEP after Jason Orendorff pointed out the difficulty of writing appropriate __enter__ and __exit__ methods for decimal.Context [5]. This approach allows a class to define a native context manager using generator syntax. It also allows a class to use an existing independent context manager as its native context manager by applying the independent context manager to 'self' in its __context__ method. It even allows a class written in C to use a generator context manager written in Python. The __context__ method parallels the __iter__ method which forms part of the iterator protocol. An earlier version of this PEP called this the __with__ method. This was later changed to match the name of the protocol rather than the keyword for the statement [9]. 5. The suggestion was made by Jason Orendorff that the __enter__ and __exit__ methods could be removed from the context management protocol, and the protocol instead defined directly in terms of the enhanced generator interface described in PEP 342 [6]. Guido rejected this idea [7]. The following are some of benefits of keeping the __enter__ and __exit__ methods: - it makes it easy to implement a simple context manager in C without having to rely on a separate coroutine builder - it makes it easy to provide a low-overhead implementation for context managers which don't need to maintain any special state between the __enter__ and __exit__ methods (having to use a generator for these would impose unnecessary overhead without any compensating benefit) - it makes it possible to understand how the with statement works without having to first understand the mechanics of how generator context managers are implemented. 6. The decorator to make a context manager from a generator will be a builtin called "contextmanager". The shorter term "context" was considered too ambiguous and potentially confusing [9]. The different flavours of generators can then be described as: - A "generator function" is an undecorated function containing the 'yield' keyword, and the objects produced by such functions are "generator-iterators". The term "generator" may refer to either a generator function or a generator-iterator depending on the situation. - A "generator context function" is a generator function to which the "contextmanager" decorator is applied and the objects produced by such functions are "generator-context- managers". The term "generator context" may refer to either a generator context function or a generator-context-manager depending on the situation. 7. A generator function used to implement a __context__ method will need to be decorated with the contextmanager decorator in order to have the correct behaviour. Otherwise, you will get an AttributeError when using the class in a with statement, as normal generator-iterators will NOT have __enter__ or __exit__ methods. Getting deterministic closure of generators will require a separate context manager such as the closing example below. As Guido put it, "too much magic is bad for your health" [10]. 8. It is fine to raise AttributeError instead of TypeError if the relevant methods aren't present on a class involved in a with statement. The fact that the abstract object C API raises TypeError rather than AttributeError is an accident of history, rather than a deliberate design decision [11]. Examples The generator based examples rely on PEP 342. Also, some of the examples are likely to be unnecessary in practice, as the appropriate objects, such as threading.RLock, will be able to be used directly in with statements. The tense used in the names of the example context managers is not arbitrary. Past tense ("-ed") is used when the name refers to an action which is done in the __enter__ method and undone in the __exit__ method. Progressive tense ("-ing") is used when the name refers to an action which is to be done in the __exit__ method. 1. A template for ensuring that a lock, acquired at the start of a block, is released when the block is left: @contextmanager def locked(lock): lock.acquire() try: yield finally: lock.release() Used as follows: with locked(myLock): # Code here executes with myLock held. The lock is # guaranteed to be released when the block is left (even # if via return or by an uncaught exception). PEP 319 gives a use case for also having an unlocked() template; this can be written very similarly (just swap the acquire() and release() calls). 2. A template for opening a file that ensures the file is closed when the block is left: @contextmanager def opened(filename, mode="r"): f = open(filename, mode) try: yield f finally: f.close() Used as follows: with opened("/etc/passwd") as f: for line in f: print line.rstrip() 3. A template for committing or rolling back a database transaction: @contextmanager def transaction(db): db.begin() try: yield None except: db.rollback() raise else: db.commit() 4. Example 1 rewritten without a generator: class locked: def __init__(self, lock): self.lock = lock def __context__(self): return self def __enter__(self): self.lock.acquire() def __exit__(self, type, value, tb): self.lock.release() if type is not None: raise type, value, tb (This example is easily modified to implement the other relatively stateless examples; it shows that it is easy to avoid the need for a generator if no special state needs to be preserved.) 5. Redirect stdout temporarily: @contextmanager def stdout_redirected(new_stdout): save_stdout = sys.stdout sys.stdout = new_stdout try: yield None finally: sys.stdout = save_stdout Used as follows: with opened(filename, "w") as f: with stdout_redirected(f): print "Hello world" This isn't thread-safe, of course, but neither is doing this same dance manually. In single-threaded programs (for example, in scripts) it is a popular way of doing things. 6. A variant on opened() that also returns an error condition: @contextmanager def opened_w_error(filename, mode="r"): try: f = open(filename, mode) except IOError, err: yield None, err else: try: yield f, None finally: f.close() Used as follows: with opened_w_error("/etc/passwd", "a") as (f, err): if err: print "IOError:", err else: f.write("guido::0:0::/:/bin/sh\n") 7. Another useful example would be an operation that blocks signals. The use could be like this: import signal with signal.blocked(): # code executed without worrying about signals An optional argument might be a list of signals to be blocked; by default all signals are blocked. The implementation is left as an exercise to the reader. 8. Another use for this feature is the Decimal context. Here's a simple example, after one posted by Michael Chermside: import decimal @contextmanager def extra_precision(places=2): c = decimal.getcontext() saved_prec = c.prec c.prec += places try: yield None finally: c.prec = saved_prec Sample usage (adapted from the Python Library Reference): def sin(x): "Return the sine of x as measured in radians." with extra_precision(): i, lasts, s, fact, num, sign = 1, 0, x, 1, x, 1 while s != lasts: lasts = s i += 2 fact *= i * (i-1) num *= x * x sign *= -1 s += num / fact * sign # The "+s" rounds back to the original precision, # so this must be outside the with-statement: return +s 9. Here's a proposed native context manager for decimal.Context: # This would be a new decimal.Context method @contextmanager def __context__(self): # We set the thread context to a copy of this context # to ensure that changes within the block are kept # local to the block. This also gives us thread safety # and supports nested usage of a given context. newctx = self.copy() oldctx = decimal.getcontext() decimal.setcontext(newctx) try: yield newctx finally: decimal.setcontext(oldctx) Sample usage: def sin(x): with decimal.getcontext() as ctx: ctx.prec += 2 # Rest of sin calculation algorithm # uses a precision 2 greater than normal return +s # Convert result to normal precision def sin(x): with decimal.ExtendedContext: # Rest of sin calculation algorithm # uses the Extended Context from the # General Decimal Arithmetic Specification return +s # Convert result to normal context 10. A generic "object-closing" template: @contextmanager def closing(obj): try: yield obj finally: try: close = obj.close except AttributeError: pass else: close() This can be used to deterministically close anything with a close method, be it file, generator, or something else. It can even be used when the object isn't guaranteed to require closing (e.g., a function that accepts an arbitrary iterable): # emulate opening(): with closing(open("argument.txt")) as contradiction: for line in contradiction: print line # deterministically finalize an iterator: with closing(iter(data_source)) as data: for datum in data: process(datum) 11. Native contexts for objects with acquire/release methods: # This would be a new method of e.g., threading.RLock def __context__(self): return locked(self) def released(self): return unlocked(self) Sample usage: with my_lock: # Operations with the lock held with my_lock.released(): # Operations without the lock # e.g. blocking I/O # Lock is held again here 12. A "nested" context manager that automatically nests the supplied contexts from left-to-right to avoid excessive indentation: @contextmanager def nested(*contexts): exits = [] vars = [] try: try: for context in contexts: mgr = context.__context__() exit = mgr.__exit__ enter = mgr.__enter__ vars.append(enter()) exits.append(exit) yield vars except: exc = sys.exc_info() else: exc = (None, None, None) finally: while exits: exit = exits.pop() try: exit(*exc) except: exc = sys.exc_info() else: exc = (None, None, None) if exc != (None, None, None): raise Sample usage: with nested(a, b, c) as (x, y, z): # Perform operation Is equivalent to: with a as x: with b as y: with c as z: # Perform operation Reference Implementation There is no implementation at this time. This PEP was accepted by Guido at his EuroPython keynote, 27 June 2005. References [1] http://blogs.msdn.com/oldnewthing/archive/2005/01/06/347666.aspx [2] http://mail.python.org/pipermail/python-dev/2005-May/053885.html [3] http://wiki.python.org/moin/WithStatement [4] http://mail.python.org/pipermail/python-dev/2005-July/054658.html [5] http://mail.python.org/pipermail/python-dev/2005-October/056947.html [6] http://mail.python.org/pipermail/python-dev/2005-October/056969.html [7] http://mail.python.org/pipermail/python-dev/2005-October/057018.html [8] http://mail.python.org/pipermail/python-dev/2005-June/054064.html [9] http://mail.python.org/pipermail/python-dev/2005-October/057520.html [10] http://mail.python.org/pipermail/python-dev/2005-October/057535.html [11] http://mail.python.org/pipermail/python-dev/2005-October/057625.html [12] http://sourceforge.net/tracker/index.php?func=detail&aid=1223381&group_id=5470&atid=305470 [13] http://mail.python.org/pipermail/python-dev/2006-February/061903.html Copyright This document has been placed in the public domain.