Update with outcome of recent python-dev discussions

pep-0343.txt

@@ -7,17 +7,17 @@ Status: Draft
 Type: Standards Track
 Content-Type: text/plain
 Created: 13-May-2005
-Post-History: 2-Jun-2005
+Post-History: 2-Jun-2005, 16-Oct-2005, 29-Oct-2005
 
 Abstract
 
     This PEP adds a new statement "with" to the Python language to make
     it possible to factor out standard uses of try/finally statements.
 
-    The PEP has been approved in principle by the BDFL, but there are
+    The PEP was approved in principle by the BDFL, but there were
     still a couple of implementation details to be worked out (see the
-    section on Open Issues). It's been reverted to Draft status until
+    section on Resolved Issues). It's still at Draft status until
-    those issues have been settled to Guido's satisfaction.
+    Guido gives a final blessing to the updated PEP.
 
 Author's Note
 
@@ -225,7 +225,7 @@ Specification: The 'with' Statement
 
     The translation of the above statement is:
 
-        abc = (EXPR).__with__()
+        abc = (EXPR).__context__()
         exc = (None, None, None)
         VAR = abc.__enter__()
         try:
@@ -241,14 +241,18 @@ Specification: The 'with' Statement
     accessible to the user; they will most likely be implemented as
     special registers or stack positions.
 
-    The call to the __with__() method serves a similar purpose to that
+    The above translation is fairly literal - if any of the relevant
-    of the __iter__() method of iterator and iterables. An object with
+    methods are not found as expected, the interpreter will raise
-    with simple state requirements (such as threading.RLock) may provide
+    AttributeError.
-    its own __enter__() and __exit__() methods, and simply return
+
-    'self' from its __with__ method. On the other hand, an object with
+    The call to the __context__() method serves a similar purpose to
-    more complex state requirements (such as decimal.Context) may
+    that of the __iter__() method of iterator and iterables. An
-    return a distinct context manager object each time its __with__
+    object with with simple state requirements (such as
-    method is invoked.
+    threading.RLock) may provide its own __enter__() and __exit__()
+    methods, and simply return 'self' from its __context__ method. On
+    the other hand, an object with more complex state requirements
+    (such as decimal.Context) may return a distinct context manager
+    object each time its __context__ method is invoked.
 
     If the "as VAR" part of the syntax is omitted, the "VAR =" part of
     the translation is omitted (but abc.__enter__() is still called).
@@ -284,12 +288,12 @@ Generator Decorator
     that makes it possible to use a generator that yields exactly once
     to control a with-statement.  Here's a sketch of such a decorator:
 
-        class GeneratorContext(object):
+        class GeneratorContextManager(object):
 
            def __init__(self, gen):
                self.gen = gen
 
-           def __with__(self):
+           def __context__(self):
                return self
 
            def __enter__(self):
@@ -314,14 +318,14 @@ Generator Decorator
                    else:
                        raise RuntimeError("generator caught exception")
 
-        def context(func):
+        def contextmanager(func):
            def helper(*args, **kwds):
-               return GeneratorContext(func(*args, **kwds))
+               return GeneratorContextManager(func(*args, **kwds))
            return helper
 
     This decorator could be used as follows:
 
-        @context
+        @contextmanager
         def opening(filename):
            f = open(filename) # IOError is untouched by GeneratorContext
            try:
@@ -329,16 +333,18 @@ Generator Decorator
            finally:
                f.close() # Ditto for errors here (however unlikely)
 
-    A robust implementation of this decorator should be made part of
+    A robust builtin implementation of this decorator will be made
-    the standard library. Refer to Open Issues regarding its name and
+    part of the standard library.
-    location.
 
     Just as generator-iterator functions are very useful for writing
     __iter__() methods for iterables, generator-context functions will
-    be very useful for writing __with__() methods for contexts. It is
+    be very useful for writing __context__() methods for contexts.
-    proposed that the invocation of the "context" decorator be
+    These methods will still need to be decorated using the
-    considered implicit for generator functions used as __with__()
+    contextmanager decorator. To ensure an obvious error message if the
-    methods (again, refer to the Open Issues section).
+    decorator is left out, generator-iterator objects will NOT be given
+    a native context - if you want to ensure a generator is closed
+    promptly, use something similar to the duck-typed "closing" context
+    manager in the examples.
 
 Optional Extensions
 
@@ -371,6 +377,15 @@ Optional Extensions
     second  with-statement calls f.__enter__() again. A similar error
     can be raised if __enter__ is invoked on a closed file object.
 
+    For Python 2.5, the following candidates have been identified for
+    native context managers:
+        - file
+        - decimal.Context
+        - thread.LockType
+        - threading.Lock
+        - threading.RLock
+        - threading.Condition
+
 Standard Terminology
 
     Discussions about iterators and iterables are aided by the standard
@@ -384,7 +399,7 @@ Standard Terminology
     This PEP proposes that the protocol used by the with statement be
     known as the "context management protocol", and that objects that
     implement that protocol be known as "context managers". The term
-    "context" then encompasses all objects with a __with__() method
+    "context" then encompasses all objects with a __context__() method
     that returns a context manager (this means that all context managers
     are contexts, but not all contexts are context managers).
 
@@ -395,50 +410,13 @@ Standard Terminology
     In cases where the general term "context" would be ambiguous, it
     can be made explicit by expanding it to "manageable context".
 
-Open Issues
-
-    Discussion on python-dev revealed some open issues. These are listed
-    here and will be resolved either by consensus on python-dev or by
-    BDFL fiat.
-
-    1. The name of the decorator used to convert a generator-iterator
-       function into a generator-context function is still to be
-       finalised.
-       The proposal in this PEP is that it be called simply "context"
-       with the following reasoning:
-        - A "generator function" is an undecorated function containing
-          the 'yield' keyword, and the objects produced by
-          such functions are "generator-iterators". The term
-          "generator" may refer to either a generator function or a
-          generator-iterator depending on the situation.
-        - A "generator context function" is a generator function to
-          which the "context" decorator is applied and the objects
-          produced by such functions are "generator-context-managers".
-          The term "generator context" may refer to either a generator
-          context function or a generator-context-manager depending on
-          the situation.
-
-    2. Should the decorator to convert a generator function into a
-       generator context function be a builtin, or located elsewhere in
-       the standard library? This PEP suggests that it should be a
-       builtin, as generator context functions are the recommended way
-       of writing new context managers.
-
-    3. Should a generator function used to implement a __with__ method
-       always be considered to be a generator context function, without
-       requiring the context decorator? This PEP suggests that it
-       should, as applying a decorator to a slot just looks strange,
-       and omitting the decorator would be a source of obscure bugs.
-       The __new__ slot provides some precedent for special casing of
-       certain slots when processing slot methods.
-
 Resolved Issues
 
-    The following issues were resolved either by BDFL fiat, consensus on
+    The following issues were resolved either by BDFL approval,
-    python-dev, or a simple lack of objection to proposals in the
+    consensus on python-dev, or a simple lack of objection to
-    original version of this PEP.
+    proposals in the original version of this PEP.
 
-    1. The __exit__() method of the GeneratorContext class
+    1. The __exit__() method of the GeneratorContextManager class
        catches StopIteration and considers it equivalent to re-raising
        the exception passed to throw().  Is allowing StopIteration
        right here?
@@ -458,10 +436,10 @@ Resolved Issues
        finally-clause (the one implicit in the with-statement) which
        re-raises the original exception anyway.
 
-    2. What exception should GeneratorContext raise when the underlying
+    2. What exception should GeneratorContextManager raise when the
-       generator-iterator misbehaves? The following quote is the reason
+       underlying generator-iterator misbehaves? The following quote is
-       behind Guido's choice of RuntimeError for both this and for the
+       the reason behind Guido's choice of RuntimeError for both this
-       generator close() method in PEP 342 (from [8]):
+       and for the generator close() method in PEP 342 (from [8]):
 
        "I'd rather not introduce a new exception class just for this
        purpose, since it's not an exception that I want people to catch:
@@ -477,24 +455,27 @@ Resolved Issues
        on python-dev [4] settled on the term "context manager" for
        objects which provide __enter__ and __exit__ methods, and
        "context management protocol" for the protocol itself. With the
-       addition of the __with__ method to the protocol, a natural
+       addition of the __context__ method to the protocol, a natural
-       extension is to call all objects which provide a __with__ method
+       extension is to call all objects which provide a __context__
-       "contexts" (or "manageable contexts" in situations where the
+       method "contexts" (or "manageable contexts" in situations where
-       general term "context" would be ambiguous).
+       the general term "context" would be ambiguous).
        This is now documented in the "Standard Terminology" section.
 
     4. The originally approved version of this PEP did not include a
-       __with__ method - the method was only added to the PEP after
+       __context__ method - the method was only added to the PEP after
        Jason Orendorff pointed out the difficulty of writing
        appropriate __enter__ and __exit__ methods for decimal.Context
        [5]. This approach allows a class to define a native context
        manager using generator syntax. It also allows a class to use an
        existing independent context manager as its native context
        manager by applying the independent context manager to 'self' in
-       its __with__ method. It even allows a class written in C to use
+       its __context__ method. It even allows a class written in C to
-       a generator context manager written in Python.
+       use a generator context manager written in Python.
-       The __with__ method parallels the __iter__ method which forms
+       The __context__ method parallels the __iter__ method which forms
        part of the iterator protocol.
+       An earlier version of this PEP called this the __with__ method.
+       This was later changed to match the name of the protocol rather
+       than the keyword for the statement [9].
 
     5. The suggestion was made by Jason Orendorff that the __enter__
        and __exit__ methods could be removed from the context
@@ -514,18 +495,56 @@ Resolved Issues
             works without having to first understand the mechanics of
             how generator context managers are implemented.
 
+    6. The decorator to make a context manager from a generator will be
+       a builtin called "contextmanager". The shorter term "context" was
+       considered too ambiguous and potentially confusing [9].
+       The different flavours of generators can then be described as:
+        - A "generator function" is an undecorated function containing
+          the 'yield' keyword, and the objects produced by
+          such functions are "generator-iterators". The term
+          "generator" may refer to either a generator function or a
+          generator-iterator depending on the situation.
+        - A "generator context function" is a generator function to
+          which the "contextmanager" decorator is applied and the
+          objects produced by such functions are "generator-context-
+          managers". The term "generator context" may refer to either a
+          generator context function or a generator-context-manager
+          depending on the situation.
+
+    7. A generator function used to implement a __context__ method will
+       need to be decorated with the contextmanager decorator in order
+       to have the correct behaviour. Otherwise, you will get an
+       AttributeError when using the class in a with statement, as
+       normal generator-iterators will NOT have __enter__ or __exit__
+       methods.
+       Getting deterministic closure of generators will require a
+       separate context manager such as the closing example below.
+       As Guido put it, "too much magic is bad for your health" [10].
+
+    8. It is fine to raise AttributeError instead of TypeError if the
+       relevant methods aren't present on a class involved in a with
+       statement. The fact that the abstract object C API raises
+       TypeError rather than AttributeError is an accident of history,
+       rather than a deliberate design decision [11].
+
 Examples
 
-    (The generator based examples assume PEP 342 is implemented. Also,
+    The generator based examples rely on PEP 342. Also, some of the
-    some of the examples are likely to be unnecessary in practice, as
+    examples are likely to be unnecessary in practice, as the
-    the appropriate objects, such as threading.RLock, will be able to
+    appropriate objects, such as threading.RLock, will be able to be
-    be used directly in with statements)
+    used directly in with statements.
+
+    The tense used in the names of the example context managers is not
+    arbitrary. Past tense ("-ed") is used when the name refers to an
+    action which is done in the __enter__ method and undone in the
+    __exit__ method. Progressive tense  ("-ing") is used when the name
+    refers to an action which is to be done in the __exit__ method.
 
     1. A template for ensuring that a lock, acquired at the start of a
        block, is released when the block is left:
 
-        @context
+        @contextmanager
-        def locking(lock):
+        def locked(lock):
             lock.acquire()
             try:
                 yield
@@ -534,20 +553,20 @@ Examples
 
        Used as follows:
 
-        with locking(myLock):
+        with locked(myLock):
             # Code here executes with myLock held.  The lock is
             # guaranteed to be released when the block is left (even
             # if via return or by an uncaught exception).
 
-       PEP 319 gives a use case for also having an unlocking()
+       PEP 319 gives a use case for also having an unlocked()
        template; this can be written very similarly (just swap the
        acquire() and release() calls).
 
     2. A template for opening a file that ensures the file is closed
        when the block is left:
 
-        @context
+        @contextmanager
-        def opening(filename, mode="r"):
+        def opened(filename, mode="r"):
             f = open(filename, mode)
             try:
                 yield f
@@ -556,15 +575,15 @@ Examples
 
        Used as follows:
 
-        with opening("/etc/passwd") as f:
+        with opened("/etc/passwd") as f:
             for line in f:
                 print line.rstrip()
 
     3. A template for committing or rolling back a database
        transaction:
 
-        @context
+        @contextmanager
-        def transactional(db):
+        def transaction(db):
             db.begin()
             try:
                 yield None
@@ -575,10 +594,10 @@ Examples
 
     4. Example 1 rewritten without a generator:
 
-        class locking:
+        class locked:
            def __init__(self, lock):
                self.lock = lock
-           def __with__(self, lock):
+           def __context__(self):
                return self
            def __enter__(self):
                self.lock.acquire()
@@ -586,13 +605,14 @@ Examples
                self.lock.release()
 
        (This example is easily modified to implement the other
-       examples; it shows that is is easy to avoid the need for a
+       relatively stateless examples; it shows that it is easy to avoid
-       generator if no special state needs to be preserved.)
+       the need for a generator if no special state needs to be
+       preserved.)
 
     5. Redirect stdout temporarily:
 
-        @context
+        @contextmanager
-        def redirecting_stdout(new_stdout):
+        def stdout_redirected(new_stdout):
             save_stdout = sys.stdout
             sys.stdout = new_stdout
             try:
@@ -602,18 +622,18 @@ Examples
 
        Used as follows:
 
-        with opening(filename, "w") as f:
+        with opened(filename, "w") as f:
-            with redirecting_stdout(f):
+            with stdout_redirected(f):
                 print "Hello world"
 
        This isn't thread-safe, of course, but neither is doing this
        same dance manually.  In single-threaded programs (for example,
        in scripts) it is a popular way of doing things.
 
-    6. A variant on opening() that also returns an error condition:
+    6. A variant on opened() that also returns an error condition:
 
-        @context
+        @contextmanager
-        def opening_w_error(filename, mode="r"):
+        def opened_w_error(filename, mode="r"):
             try:
                 f = open(filename, mode)
             except IOError, err:
@@ -626,7 +646,7 @@ Examples
 
        Used as follows:
 
-        with opening_w_error("/etc/passwd", "a") as (f, err):
+        with opened_w_error("/etc/passwd", "a") as (f, err):
             if err:
                 print "IOError:", err
             else:
@@ -637,7 +657,7 @@ Examples
 
         import signal
 
-        with signal.blocking():
+        with signal.blocked():
             # code executed without worrying about signals
 
        An optional argument might be a list of signals to be blocked;
@@ -679,7 +699,8 @@ Examples
      9. Here's a proposed native context manager for decimal.Context:
 
          # This would be a new decimal.Context method
-         def __with__(self):
+         @contextmanager
+         def __context__(self):
              # We set the thread context to a copy of this context
              # to ensure that changes within the block are kept
              # local to the block. This also gives us thread safety
@@ -710,7 +731,7 @@ Examples
 
     10. A generic "object-closing" template:
 
-        @context
+        @contextmanager
         def closing(obj):
             try:
                 yield obj
@@ -737,6 +758,78 @@ Examples
            for datum in data:
                process(datum)
 
+     11. Native contexts for objects with acquire/release methods:
+
+         # This would be a new method of e.g., threading.RLock
+         def __context__(self):
+             return locked(self)
+
+         def released(self):
+             return unlocked(self)
+
+        Sample usage:
+
+         with my_lock:
+             # Operations with the lock held
+             with my_lock.released():
+                 # Operations without the lock
+                 # e.g. blocking I/O
+             # Lock is held again here
+
+     12. A "nested" context manager that automatically nests the
+         supplied contexts from left-to-right to avoid excessive
+         indentation:
+
+         class nested(object):
+             def __init__(*contexts):
+                 self.contexts = contexts
+                 self.entered = None
+
+             def __context__(self):
+                 return self
+
+             def __enter__(self):
+                 if self.entered is not None:
+                     raise RuntimeError("Context is not reentrant")
+                 self.entered = deque()
+                 vars = []
+                 try:
+                     for context in self.contexts:
+                         mgr = context.__context__()
+                         vars.append(mgr.__enter__())
+                         self.entered.appendleft(mgr)
+                 except:
+                     self.__exit__(*sys.exc_info())
+                     raise
+                 return vars
+
+             def __exit__(self, *exc_info):
+                 # Behave like nested with statements
+                 # first in, last out
+                 # New exceptions override old ones
+                 ex = exc_info
+                 for mgr in self.entered:
+                     try:
+                         mgr.__exit__(*ex)
+                     except:
+                         ex = sys.exc_info()
+                 self.entered = None
+                 if ex is not exc_info:
+                     raise ex[0], ex[1], ex[2]
+
+        Sample usage:
+
+          with nested(a, b, c) as (x, y, z):
+              # Perform operation
+
+        Is equivalent to:
+
+           with a as x:
+               with b as y:
+                   with c as z:
+                       # Perform operation
+
+
 References
 
     [1] http://blogs.msdn.com/oldnewthing/archive/2005/01/06/347666.aspx
@@ -760,6 +853,15 @@ References
     [8]
     http://mail.python.org/pipermail/python-dev/2005-June/054064.html
 
+    [9]
+    http://mail.python.org/pipermail/python-dev/2005-October/057520.html
+
+    [10]
+    http://mail.python.org/pipermail/python-dev/2005-October/057535.html
+
+    [11]
+    http://mail.python.org/pipermail/python-dev/2005-October/057625.html
+
 Copyright
 
     This document has been placed in the public domain.