Update PEP 343 to reflect post-acceptance python-dev discussions

pep-0343.txt

@@ -2,13 +2,29 @@ PEP: 343
 Title: Anonymous Block Redux and Generator Enhancements
 Version: $Revision$
 Last-Modified: $Date$
-Author: Guido van Rossum
+Author: Guido van Rossum, Nick Coghlan
 Status: Accepted
 Type: Standards Track
 Content-Type: text/plain
 Created: 13-May-2005
 Post-History: 2-Jun-2005
 
+Abstract
+
+    This PEP adds a new statement "with" to the Python language to make
+    it possible to factor out standard uses of try/finally statements.
+
+    The PEP has been approved in principle by the BDFL, but there are
+    still a couple of implementation details to be worked out (see the
+    section on Open Issues).
+
+Author's Note
+
+    This PEP was originally written in first person by Guido, and
+    subsequently updated by Nick Coghlan to reflect later discussion
+    on python-dev. Any first person references are from Guido's
+    original.
+
 Introduction
 
     After a lot of discussion about PEP 340 and alternatives, I
@@ -208,7 +224,7 @@ Specification: The 'with' Statement
 
     The translation of the above statement is:
 
-        abc = EXPR
+        abc = (EXPR).__with__()
         exc = (None, None, None)
         VAR = abc.__enter__()
         try:
@@ -224,6 +240,15 @@ Specification: The 'with' Statement
     accessible to the user; they will most likely be implemented as
     special registers or stack positions.
 
+    The call to the __with__() method serves a similar purpose to that
+    of the __iter__() method of iterator and iterables. An object with
+    with simple state requirements (such as threading.RLock) may provide
+    its own __enter__() and __exit__() methods, and simply return
+    'self' from its __with__ method. On the other hand, an object with
+    more complex state requirements (such as decimal.Context) may
+    return a distinct context manager object each time its __with__
+    method is invoked.
+
     If the "as VAR" part of the syntax is omitted, the "VAR =" part of
     the translation is omitted (but abc.__enter__() is still called).
 
@@ -254,15 +279,18 @@ Specification: The 'with' Statement
 
 Generator Decorator
 
-    If PEP 342 is accepted, it will be possible to write a decorator
+    With PEP 342 accepted, it is possible to write a decorator
     that makes it possible to use a generator that yields exactly once
     to control a with-statement.  Here's a sketch of such a decorator:
 
-        class ContextWrapper(object):
+        class GeneratorContext(object):
 
            def __init__(self, gen):
                self.gen = gen
 
+           def __with__(self):
+               return self
+
            def __enter__(self):
                try:
                    return self.gen.next()
@@ -285,25 +313,31 @@ Generator Decorator
                    else:
                        raise RuntimeError("generator caught exception")
 
-        def contextmanager(func):
+        def context(func):
            def helper(*args, **kwds):
-               return ContextWrapper(func(*args, **kwds))
+               return GeneratorContext(func(*args, **kwds))
            return helper
 
     This decorator could be used as follows:
 
-        @contextmanager
+        @context
         def opening(filename):
-           f = open(filename) # IOError is untouched by ContextWrapper
+           f = open(filename) # IOError is untouched by GeneratorContext
            try:
                yield f
            finally:
                f.close() # Ditto for errors here (however unlikely)
 
     A robust implementation of this decorator should be made part of
-    the standard library, but not necessarily as a built-in function.
+    the standard library. Refer to Open Issues regarding its name and
-    (I'm not sure which exception it should raise for errors;
+    location.
-    RuntimeError is used above as an example only.)
+
+    Just as generator-iterator functions are very useful for writing
+    __iter__() methods for iterables, generator-context functions will
+    be very useful for writing __with__() methods for contexts. It is
+    proposed that the invocation of the "context" decorator be
+    considered implicit for generator functions used as __with__()
+    methods (again, refer to the Open Issues section).
 
 Optional Extensions
 
@@ -332,17 +366,78 @@ Optional Extensions
     is entered).
 
     OTOH such mistakes are easily diagnosed; for example, the
-    contextmanager decorator above raises RuntimeError when the second
+    generator-context decorator above raises RuntimeError when a
-    with-statement calls f.__enter__() again.
+    second  with-statement calls f.__enter__() again. A similar error
+    can be raised if __enter__ is invoked on a closed file object.
 
-Resolved Open Issues
+Standard Terminology
 
-    Discussion on python-dev revealed some open issues.  I list them
+    Discussions about iterators and iterables are aided by the standard
-    here, with my preferred resolution and its motivation.  The PEP
+    terminology used to discuss them. The protocol used by the for
-    has been accepted without these being challenged, so the issues
+    statement is called the iterator protocol and an iterator is any
-    are now resolved.
+    object that properly implements that protocol. The term "iterable"
+    then encompasses all objects with an __iter__() method that
+    returns an iterator (this means that all iterators are iterables,
+    but not all iterables are iterators).
 
-    1. The __exit__() method of the contextmanager decorator class
+    This PEP proposes that the protocol used by the with statement be
+    known as the "context management protocol", and that objects that
+    implement that protocol be known as "context managers". The term
+    "context" then encompasses all objects with a __with__() method
+    that returns a context manager (this means that all context managers
+    are contexts, but not all contexts are context managers).
+
+    The term "context" is based on the concept that the context object
+    defines a context of execution for the code that forms the body
+    of the with statement.
+
+    In cases where the general term "context" would be ambiguous, it
+    can be made explicit by expanding it to "manageable context".
+
+Open Issues
+
+    Discussion on python-dev revealed some open issues. These are listed
+    here and will be resolved either by consensus on python-dev or by
+    BDFL fiat.
+
+    1. The name of the decorator used to convert a generator-iterator
+       function into a generator-context function is still to be
+       finalised.
+       The proposal in this PEP is that it be called simply "context"
+       with the following reasoning:
+        - A "generator function" is an undecorated function containing
+          the 'yield' keyword, and the objects produced by
+          such functions are "generator-iterators". The term
+          "generator" may refer to either a generator function or a
+          generator-iterator depending on the situation.
+        - A "generator context function" is a generator function to
+          which the "context" decorator is applied and the objects
+          produced by such functions are "generator-context-managers".
+          The term "generator context" may refer to either a generator
+          context function or a generator-context-manager depending on
+          the situation.
+
+    2. Should the decorator to convert a generator function into a
+       generator context function be a builtin, or located elsewhere in
+       the standard library? This PEP suggests that it should be a
+       builtin, as generator context functions are the recommended way
+       of writing new context managers.
+
+    3. Should a generator function used to implement a __with__ method
+       always be considered to be a generator context function, without
+       requiring the context decorator? This PEP suggests that it
+       should, as applying a decorator to a slot just looks strange,
+       and omitting the decorator would be a source of obscure bugs.
+       The __new__ slot provides some precedent for special casing of
+       certain slots when processing slot methods.
+
+Resolved Issues
+
+    The following issues were resolved either by BDFL fiat, consensus on
+    python-dev, or a simple lack of objection to proposals in the
+    original version of this PEP.
+
+    1. The __exit__() method of the GeneratorContext class
        catches StopIteration and considers it equivalent to re-raising
        the exception passed to throw().  Is allowing StopIteration
        right here?
@@ -362,19 +457,77 @@ Resolved Open Issues
        finally-clause (the one implicit in the with-statement) which
        re-raises the original exception anyway.
 
+    2. What exception should GeneratorContext raise when the underlying
+       generator-iterator misbehaves? The following quote is the reason
+       behind Guido's choice of RuntimeError for both this and for the
+       generator close() method in PEP 342 (from [8]):
+
+       "I'd rather not introduce a new exception class just for this
+       purpose, since it's not an exception that I want people to catch:
+       I want it to turn into a traceback which is seen by the
+       programmer who then fixes the code.  So now I believe they
+       should both raise RuntimeError.
+       There are some precedents for that: it's raised by the core
+       Python code in situations where endless recursion is detected,
+       and for uninitialized objects (and for a variety of
+       miscellaneous conditions)."
+
+    3. After this PEP was originally approved, a subsequent discussion
+       on python-dev [4] settled on the term "context manager" for
+       objects which provide __enter__ and __exit__ methods, and
+       "context management protocol" for the protocol itself. With the
+       addition of the __with__ method to the protocol, a natural
+       extension is to call all objects which provide a __with__ method
+       "contexts" (or "manageable contexts" in situations where the
+       general term "context" would be ambiguous).
+       This is now documented in the "Standard Terminology" section.
+
+    4. The originally approved version of this PEP did not include a
+       __with__ method - the method was only added to the PEP after
+       Jason Orendorff pointed out the difficulty of writing
+       appropriate __enter__ and __exit__ methods for decimal.Context
+       [5]. This approach allows a class to define a native context
+       manager using generator syntax. It also allows a class to use an
+       existing independent context manager as its native context
+       manager by applying the independent context manager to 'self' in
+       its __with__ method. It even allows a class written in C to use
+       a generator context manager written in Python.
+       The __with__ method parallels the __iter__ method which forms
+       part of the iterator protocol.
+
+    5. The suggestion was made by Jason Orendorff that the __enter__
+       and __exit__ methods could be removed from the context
+       management protocol, and the protocol instead defined directly
+       in terms of the enhanced generator interface described in PEP
+       342 [6].
+       Guido rejected this idea [7]. The following are some of benefits
+       of keeping the __enter__ and __exit__ methods:
+          - it makes it easy to implement a simple context manager in C
+            without having to rely on a separate coroutine builder
+          - it makes it easy to provide a low-overhead implementation
+            for context managers which don't need to maintain any
+            special state between the __enter__ and __exit__ methods
+            (having to use a generator for these would impose
+            unnecessary overhead without any compensating benefit)
+          - it makes it possible to understand how the with statement
+            works without having to first understand the mechanics of
+            how generator context managers are implemented.
+
 Examples
 
-    (Note: several of these examples contain "yield None".  If PEP 342
+    (The generator based examples assume PEP 342 is implemented. Also,
-    is accepted, these can be changed to just "yield".)
+    some of the examples are likely to be unnecessary in practice, as
+    the appropriate objects, such as threading.RLock, will be able to
+    be used directly in with statements)
 
     1. A template for ensuring that a lock, acquired at the start of a
        block, is released when the block is left:
 
-        @contextmanager
+        @context
         def locking(lock):
             lock.acquire()
             try:
-                yield None
+                yield
             finally:
                 lock.release()
 
@@ -392,7 +545,7 @@ Examples
     2. A template for opening a file that ensures the file is closed
        when the block is left:
 
-        @contextmanager
+        @context
         def opening(filename, mode="r"):
             f = open(filename, mode)
             try:
@@ -409,7 +562,7 @@ Examples
     3. A template for committing or rolling back a database
        transaction:
 
-        @contextmanager
+        @context
         def transactional(db):
             db.begin()
             try:
@@ -424,18 +577,20 @@ Examples
         class locking:
            def __init__(self, lock):
                self.lock = lock
+           def __with__(self, lock):
+               return self
            def __enter__(self):
                self.lock.acquire()
            def __exit__(self, type, value, tb):
                self.lock.release()
 
        (This example is easily modified to implement the other
-       examples; it shows the relative advantage of using a generator
+       examples; it shows that is is easy to avoid the need for a
-       template.)
+       generator if no special state needs to be preserved.)
 
     5. Redirect stdout temporarily:
 
-        @contextmanager
+        @context
         def redirecting_stdout(new_stdout):
             save_stdout = sys.stdout
             sys.stdout = new_stdout
@@ -456,7 +611,7 @@ Examples
 
     6. A variant on opening() that also returns an error condition:
 
-        @contextmanager
+        @context
         def opening_w_error(filename, mode="r"):
             try:
                 f = open(filename, mode)
@@ -520,54 +675,67 @@ Examples
             # so this must be outside the with-statement:
             return +s
 
-    9. Here's a more general Decimal-context-switching template:
+     9. Here's a proposed native context manager for decimal.Context:
 
-        @contextmanager
+         # This would be a new decimal.Context method
-        def decimal_context(newctx=None):
+         def __with__(self):
-            oldctx = decimal.getcontext()
+             # We set the thread context to a copy of this context
-            if newctx is None:
+             # to ensure that changes within the block are kept
-                newctx = oldctx.copy()
+             # local to the block. This also gives us thread safety
-            decimal.setcontext(newctx)
+             # and supports nested usage of a given context.
-            try:
+             newctx = self.copy()
-                yield newctx
+             oldctx = decimal.getcontext()
-            finally:
+             decimal.setcontext(newctx)
-                decimal.setcontext(oldctx)
+             try:
+                 yield newctx
+             finally:
+                 decimal.setcontext(oldctx)
 
-       Sample usage:
+        Sample usage:
 
-        def sin(x):
+         def sin(x):
-            with decimal_context() as ctx:
+             with decimal.getcontext() as ctx:
-                ctx.prec += 2
+                 ctx.prec += 2
-                # Rest of algorithm the same as above
+                 # Rest of sin calculation algorithm
-            return +s
+                 # uses a precision 2 greater than normal
+             return +s # Convert result to normal precision
 
-       (Nick Coghlan has proposed to add __enter__() and __exit__()
+         def sin(x):
-       methods to the decimal.Context class so that this example can
+             with decimal.ExtendedContext:
-       be simplified to "with decimal.getcontext() as ctx: ...".)
+                 # Rest of sin calculation algorithm
+                 # uses the Extended Context from the
+                 # General Decimal Arithmetic Specification
+             return +s # Convert result to normal context
 
     10. A generic "object-closing" template:
 
-        @contextmanager
+        @context
         def closing(obj):
             try:
                 yield obj
             finally:
-                obj.close()
+                try:
+                    close = obj.close
+                except AttributeError:
+                    pass
+                else:
+                    close()
 
         This can be used to deterministically close anything with a
-        close method, be it file, generator, or something else:
+        close method, be it file, generator, or something else. It can
+        even be used when the object isn't guaranteed to require
+        closing (e.g., a function that accepts an arbitrary iterable):
 
         # emulate opening():
         with closing(open("argument.txt")) as contradiction:
            for line in contradiction:
                print line
 
-        # deterministically finalize a generator:
+        # deterministically finalize an iterator:
-        with closing(some_gen()) as data:
+        with closing(iter(data_source)) as data:
            for datum in data:
                process(datum)
 
-
 References
 
     [1] http://blogs.msdn.com/oldnewthing/archive/2005/01/06/347666.aspx
@@ -576,6 +744,21 @@ References
 
     [3] http://wiki.python.org/moin/WithStatement
 
+    [4]
+    http://mail.python.org/pipermail/python-dev/2005-July/054658.html
+
+    [5]
+    http://mail.python.org/pipermail/python-dev/2005-October/056947.html
+
+    [6]
+    http://mail.python.org/pipermail/python-dev/2005-October/056969.html
+
+    [7]
+    http://mail.python.org/pipermail/python-dev/2005-October/057018.html
+
+    [8]
+    http://mail.python.org/pipermail/python-dev/2005-June/054064.html
+
 Copyright
 
     This document has been placed in the public domain.