iSharkFly-Open
/
python-peps
mirror of https://github.com/python/peps.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Remove the __context__ method from PEP 343, and update the terminology section (again). Record a couple of remaining open issues, and try to clarify the resolved issues sections by only giving a very brief overview of all the rejected options that aren't relevant any more.

This commit is contained in:
Nick Coghlan 2006-05-02 14:28:47 +00:00
parent 1fe4adbb71
commit 15948f5fe0
1 changed files with 217 additions and 267 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

484
pep-0343.txt
View File

@ -7,18 +7,16 @@ Status: Accepted
Type: Standards Track
Content-Type: text/plain
Created: 13-May-2005
Post-History: 2-Jun-2005, 16-Oct-2005, 29-Oct-2005, 23-Apr-2006
Post-History: 2-Jun-2005, 16-Oct-2005, 29-Oct-2005, 23-Apr-2006, 1-May-2006
Abstract
This PEP adds a new statement "with" to the Python language to make
it possible to factor out standard uses of try/finally statements.
The PEP was approved in principle by the BDFL, but there were
still a couple of implementation details to be worked out (see the
section on Resolved Issues). It's still at Draft status until
Guido gives a final blessing to the updated PEP.
In this PEP, context managers provide __enter__() and __exit__()
methods that are invoked on entry to and exit from the managed
context that forms the body of the with statement.
Author's Note
@ -29,13 +27,17 @@ Author's Note
Python's alpha release cycle revealed terminology problems in this
PEP and in the associated documentation and implementation [14].
So while the PEP is already accepted, this refers to the
implementation rather than the exact terminology.
So while the PEP is already accepted in principle, it won't really
be considered stable until the status becomes Final.
The current version of the PEP reflects the implementation and
documentation as at Python 2.5a2. The PEP will be updated to
reflect any changes made to the terminology prior to the final
Python 2.5 release.
The current version of the PEP reflects the discussions that
occurred on python-dev shortly after the release of Python 2.5a2.
The PEP will continue to be updated to reflect any changes made to
the details of the feature prior to the final Python 2.5 release.
Yes, the verb tense is messed up in a few places. We've been
working on this PEP for nearly a year now, so things that were
originally in the future are now in the past :)
Introduction
@ -48,11 +50,8 @@ Introduction
[2] and universally approved of. I'm also changing the keyword to
'with'.
On-line discussion of this PEP should take place in the Python
Wiki [3].
If this PEP is approved, the following PEPs will be rejected due
to overlap:
Following acceptance of this PEP, the following PEPs have been
rejected due to overlap:
- PEP 310, Reliable Acquisition/Release Pairs. This is the
original with-statement proposal.
@ -66,7 +65,11 @@ Introduction
important; in fact it may be better to always be explicit about
the mutex being used.
(PEP 340 and PEP 346 have already been withdrawn.)
PEP 340 and PEP 346 also overlapped with this PEP, but were
voluntarily withdrawn when this PEP was submitted.
Some discussion of earlier incarnations of this PEP took place on
the Python Wiki [3].
Motivation and Summary
@ -92,7 +95,7 @@ Motivation and Summary
control flow, in the end, the control flow resumes as if the
finally-suite wasn't there at all.
Remember, PEP 310 proposes rougly this syntax (the "VAR =" part is
Remember, PEP 310 proposes roughly this syntax (the "VAR =" part is
optional):
with VAR = EXPR:
@ -213,6 +216,9 @@ Motivation and Summary
not make the same guarantee. This applies to Jython, IronPython,
and probably to Python running on Parrot.
(The details of the changes made to generators can now be found in
PEP 342 rather than in the current PEP)
Use Cases
See the Examples section near the end.
@ -236,7 +242,7 @@ Specification: The 'with' Statement
The translation of the above statement is:
mgr = (EXPR).__context__()
mgr = (EXPR)
exit = mgr.__exit__ # Not calling it yet
value = mgr.__enter__()
exc = True
@ -260,9 +266,9 @@ Specification: The 'with' Statement
implemented as special registers or stack positions.
The details of the above translation are intended to prescribe the
exact semantics. If any of the relevant methods are not found as
expected, the interpreter will raise AttributeError, in the order
that they are tried (__context__, __exit__, __enter__).
exact semantics. If either of the relevant methods are not found
as expected, the interpreter will raise AttributeError, in the
order that they are tried (__exit__, __enter__).
Similarly, if any of the calls raises an exception, the effect is
exactly as it would be in the above code. Finally, if BLOCK
contains a break, continue or return statement, the __exit__()
@ -270,15 +276,6 @@ Specification: The 'with' Statement
completed normally. (I.e. these "pseudo-exceptions" are not seen
as exceptions by __exit__().)
The call to the __context__() method serves a similar purpose to
that of the __iter__() method of iterator and iterables. A context
specifier with simple state requirements (such as
threading.RLock) may provide its own __enter__() and __exit__()
methods, and simply return 'self' from its __context__ method. On
the other hand, a context specifier with more complex state
requirements (such as decimal.Context) may return a distinct
context manager each time its __context__ method is invoked.
If the "as VAR" part of the syntax is omitted, the "VAR =" part of
the translation is omitted (but mgr.__enter__() is still called).
@ -324,18 +321,19 @@ Specification: The 'with' Statement
of a database transaction roll-back decision.
To facilitate chaining of contexts in Python code that directly
manipulates context specifiers and managers, __exit__() methods
should *not* re-raise the error that is passed in to them, because
it is always the responsibility of the *caller* to do any reraising
in that case.
manipulates context managers, __exit__() methods should *not*
re-raise the error that is passed in to them. It is always the
responsibility of the *caller* of the __exit__() method to do any
reraising in that case.
That way, if the caller needs to tell whether the __exit__()
invocation *failed* (as opposed to successfully cleaning up before
propagating the original error), it can do so.
If __exit__() returns without an error, this can then be
interpreted as success of the __exit__() method itself (whether the
original error is to be propagated or suppressed).
interpreted as success of the __exit__() method itself (regardless
of whether or not the original error is to be propagated or
suppressed).
However, if __exit__() propagates an exception to its caller, this
means that __exit__() *itself* has failed. Thus, __exit__()
@ -343,20 +341,6 @@ Specification: The 'with' Statement
failed. (And allowing the original error to proceed isn't a
failure.)
Objects returned by __context__() methods should also provide a
__context__() method that returns self. This allows a program to
retrieve the context manager directly without breaking anything.
For example, the following should work just as well as the normal
case where the extra variable isn't used:
mgr = (EXPR).__context__()
with mgr as VAR:
BLOCK
The with statement implementation and examples like the nested()
function require this behaviour in order to be able to deal
transparently with both context specifiers and context managers.
Transition Plan
In Python 2.5, the new syntax will only be recognized if a future
@ -382,9 +366,6 @@ Generator Decorator
def __init__(self, gen):
self.gen = gen
def __context__(self):
return self
def __enter__(self):
try:
return self.gen.next()
@ -435,17 +416,7 @@ Generator Decorator
A robust implementation of this decorator will be made
part of the standard library.
Just as generator-iterator functions are very useful for writing
__iter__() methods for iterables, generator context functions will
be very useful for writing __context__() methods for context
specifiers. These methods will still need to be decorated using the
contextmanager decorator. To ensure an obvious error message if the
decorator is left out, generator-iterator objects will NOT be given
a native context - if you want to ensure a generator is closed
promptly, use something similar to the duck-typed "closing" context
manager in the examples.
Optional Extensions
Context Managers in the Standard Library
It would be possible to endow certain objects, like files,
sockets, and locks, with __enter__() and __exit__() methods so
@ -476,133 +447,110 @@ Optional Extensions
second with-statement calls f.__enter__() again. A similar error
can be raised if __enter__ is invoked on a closed file object.
For Python 2.5, the following candidates have been identified for
native context managers:
For Python 2.5, the following types have been identified as
context managers:
- file
- decimal.Context
- thread.LockType
- threading.Lock
- threading.RLock
- threading.Condition
- threading.Semaphore and threading.BoundedSemaphore
- threading.Semaphore
- threading.BoundedSemaphore
A context manager will also be added to the decimal module to
support using a local decimal arithmetic context within the body
of a with statement, automatically restoring the original context
when the with statement is exited.
Standard Terminology
Discussions about iterators and iterables are aided by the standard
terminology used to discuss them. The protocol used by the for
statement is called the iterator protocol and an iterator is any
object that properly implements that protocol. The term "iterable"
then encompasses all objects with an __iter__() method that
returns an iterator.
This PEP proposes that the protocol consisting of the __enter__()
and __exit__() methods, and a __context__() method that returns
self be known as the "context management protocol", and that
objects that implement that protocol be known as "context
managers".
and __exit__() methods be known as the "context management protocol",
and that objects that implement that protocol be known as "context
managers". [4]
The term "context specifier" then encompasses all objects with a
__context__() method that returns a context manager. The protocol
these objects implement is called the "context specification
protocol". This means that all context managers are context
specifiers, but not all context specifiers are context managers,
just as all iterators are iterables, but not all iterables are
iterators.
The code in the body of the with statement is a "managed context".
This term refers primarily to the code location, rather than to the
runtime environment established by the context manager.
These terms are based on the concept that the context specifier
defines a context of execution for the code that forms the body of
the with statement. The role of the context manager is to
translate the context specifier's stored state into an active
manipulation of the runtime environment to setup and tear down the
desired runtime context for the duration of the with statement.
For example, a synchronisation lock's context manager acquires the
lock when entering the with statement, and releases the lock when
leaving it. The runtime context established within the body of the
with statement is that the synchronisation lock is currently held.
The expression immediately following the with keyword in the
statement is a "context expression" as that expression provides the
main clue as to the runtime environment the context manager
establishes for the duration of the managed context.
The value assigned to the target list after the as keyword is the
"context entry value", as that value is returned as the result of
entering the context.
These terms are based on the idea that the context expression
provides a context manager to appropriately handle entry into the
managed context. The context manager may also provide a meaningful
context entry value and perform clean up operations on exit from
the managed context.
The general term "context" is unfortunately ambiguous. If necessary,
it can be made more explicit by using the terms "context specifier"
for objects providing a __context__() method and "runtime context"
for the runtime environment modifications made by the context
manager. When solely discussing use of the with statement, the
distinction between the two shouldn't matter as the context
specifier fully defines the changes made to the runtime context.
The distinction is more important when discussing the process of
implementing context specifiers and context managers.
it can be made more explicit by using the terms "context manager"
for the concrete object created by the context expression,
"managed context" for the code in the body of the with statement,
and "runtime context" or (preferebly) "runtime environment" for the
actual state modifications made by the context manager. When solely
discussing use of the with statement, the distinction between these
shouldn't matter too much as the context manager fully defines the
changes made to the runtime environment, and those changes apply for
the duration of the managed context. The distinction is more
important when discussing the process of implementing context
managers and the mechanics of the with statement itself.
Caching Context Managers
Many context managers (such as files and generator-based contexts)
will be single-use objects. Once the __exit__() method has been
called, the context manager will no longer be in a usable state
(e.g. the file has been closed, or the underlying generator has
finished execution).
Requiring a fresh manager object for each with statement is the
easiest way to avoid problems with multi-threaded code and nested
with statements trying to use the same context manager. It isn't
coincidental that all of the standard library context managers
that support reuse come from the threading module - they're all
already designed to deal with the problems created by threaded
and nested usage.
This means that in order to save a context manager with particular
initialisation arguments to be used in multiple with statements, it
will typically be necessary to store it in a zero-argument callable
that is then called in the context expression of each statement
rather than caching the context manager directly.
When this restriction does not apply, the documentation of the
affected context manager should make that clear.
Open Issues
1. After this PEP was originally approved, a subsequent discussion
on python-dev [4] settled on the term "context manager" for
objects which provide __enter__ and __exit__ methods, and
"context management protocol" for the protocol itself. With the
addition of the __context__ method to the protocol, the natural
adjustment is to call all objects which provide a __context__
method "context managers", and the objects with __enter__ and
__exit__ methods "contexts" (or "manageable contexts" in
situations where the general term "context" would be ambiguous).
1. Greg Ewing raised the question of whether or not the term
"context manager" was too generic and suggested "context guard"
as an alternative name.
As noted above, the Python 2.5 release cycle revealed problems
with the previously agreed terminology. The updated standard
terminology section has not yet met with consensus on
python-dev. It will be refined throughout the Python 2.5 release
cycle based on user feedback on the usability of the
documentation.
The first change made as a result of the current discussion is
replacement of the term "context object" with
"context specifier".
2. The original resolution was for the decorator to make a context
manager from a generator to be a builtin called "contextmanager".
The shorter term "context" was considered too ambiguous and
potentially confusing [9].
The different flavours of generators could then be described as:
- A "generator function" is an undecorated function containing
the 'yield' keyword, and the objects produced by
such functions are "generator-iterators". The term
"generator" may refer to either a generator function or a
generator-iterator depending on the situation.
- A "generator context function" is a generator function to
which the "contextmanager" decorator is applied and the
objects produced by such functions are "generator-context-
managers". The term "generator context" may refer to either
a generator context function or a generator-context-manager
depending on the situation.
In the Python 2.5 implementation, the decorator is actually part
of the standard library module contextlib. The ongoing
terminology review may lead to it being renamed
"contextlib.context" (with the existence of the underlying context
manager being an implementation detail).
2. In Python 2.5a2, the decorator in contextlib to create a
context manager from a generator function is called
@contextfactory. This made sense when the __context__()
method existed and the result of the factory function was
a managed context object.
With the elimination of the __context__() method, the
result of the factory function is once again a context
manager, suggesting the decorator should be renamed to
either @contextmanager or @managerfactory.
The PEP currently uses @contextmanager.
Resolved Issues
The following issues were resolved either by BDFL approval,
consensus on python-dev, or a simple lack of objection to
proposals in the original version of this PEP.
The following issues were resolved by BDFL approval (and a lack
of any major objections on python-dev).
1. The __exit__() method of the GeneratorContextManager class
catches StopIteration and considers it equivalent to re-raising
the exception passed to throw(). Is allowing StopIteration
right here?
This is so that a generator doing cleanup depending on the
exception thrown (like the transactional() example below) can
*catch* the exception thrown if it wants to and doesn't have to
worry about re-raising it. I find this more convenient for the
generator writer. Against this was brought in that the
generator *appears* to suppress an exception that it cannot
suppress: the transactional() example would be more clear
according to this view if it re-raised the original exception
after the call to db.rollback(). I personally would find the
requirement to re-raise the exception an annoyance in a
generator used as a with-template, since all the code after
yield is used for is cleanup, and it is invoked from a
finally-clause (the one implicit in the with-statement) which
re-raises the original exception anyway.
2. What exception should GeneratorContextManager raise when the
1. What exception should GeneratorContextManager raise when the
underlying generator-iterator misbehaves? The following quote is
the reason behind Guido's choice of RuntimeError for both this
and for the generator close() method in PEP 342 (from [8]):
@ -617,61 +565,32 @@ Resolved Issues
and for uninitialized objects (and for a variety of
miscellaneous conditions)."
3. See item 1 in open issues :)
4. The originally approved version of this PEP did not include a
__context__ method - the method was only added to the PEP after
Jason Orendorff pointed out the difficulty of writing
appropriate __enter__ and __exit__ methods for decimal.Context
[5]. This approach allows a class to define a native context
manager using generator syntax. It also allows a class to use an
existing independent context as its native context object by
applying the independent context to 'self' in its __context__
method. It even allows a class written in C to
use a generator context manager written in Python.
The __context__ method parallels the __iter__ method which forms
part of the iterator protocol.
An earlier version of this PEP called this the __with__ method.
This was later changed to match the name of the protocol rather
than the keyword for the statement [9].
5. The suggestion was made by Jason Orendorff that the __enter__
and __exit__ methods could be removed from the context
management protocol, and the protocol instead defined directly
in terms of the enhanced generator interface described in PEP
342 [6].
Guido rejected this idea [7]. The following are some of benefits
of keeping the __enter__ and __exit__ methods:
- it makes it easy to implement a simple context in C
without having to rely on a separate coroutine builder
- it makes it easy to provide a low-overhead implementation
for contexts that don't need to maintain any
special state between the __enter__ and __exit__ methods
(having to use a generator for these would impose
unnecessary overhead without any compensating benefit)
- it makes it possible to understand how the with statement
works without having to first understand the mechanics of
how generator context managers are implemented.
6. See item 2 in open issues :)
7. A generator function used to implement a __context__ method will
need to be decorated with the contextmanager decorator in order
to have the correct behaviour. Otherwise, you will get an
AttributeError when using the class in a with statement, as
normal generator-iterators will NOT have __enter__ or __exit__
methods.
Getting deterministic closure of generators will require a
separate context manager such as the closing example below.
As Guido put it, "too much magic is bad for your health" [10].
8. It is fine to raise AttributeError instead of TypeError if the
2. It is fine to raise AttributeError instead of TypeError if the
relevant methods aren't present on a class involved in a with
statement. The fact that the abstract object C API raises
TypeError rather than AttributeError is an accident of history,
rather than a deliberate design decision [11].
Rejected Options
For several months, the PEP prohibited suppression of exceptions
in order to avoid hidden flow control. Implementation
revealed this to be a right royal pain, so Guido restored the
ability [13].
Another aspect of the PEP that caused no end of questions and
terminology debates was providing a __context__() method that
was analogous to an iterable's __iter__() method [5, 7, 9].
The ongoing problems [10, 13] with explaining what it was and why
it was and how it was meant to work eventually lead to Guido
killing the concept outright [15] (and there was much rejoicing!).
The notion of using the PEP 342 generator API directly to define
the with statement was also briefly entertained [6], but quickly
dismissed as making it too difficult to write non-generator
based context managers.
Examples
The generator based examples rely on PEP 342. Also, some of the
@ -703,10 +622,6 @@ Examples
# guaranteed to be released when the block is left (even
# if via return or by an uncaught exception).
PEP 319 gives a use case for also having an unlocked()
context; this can be written very similarly (just swap the
acquire() and release() calls).
2. A template for opening a file that ensures the file is closed
when the block is left:
@ -743,14 +658,10 @@ Examples
class locked:
def __init__(self, lock):
self.lock = lock
def __context__(self):
return self
def __enter__(self):
self.lock.acquire()
def __exit__(self, type, value, tb):
self.lock.release()
if type is not None:
raise type, value, tb
(This example is easily modified to implement the other
relatively stateless examples; it shows that it is easy to avoid
@ -844,52 +755,59 @@ Examples
# so this must be outside the with-statement:
return +s
9. Here's a proposed native context manager for decimal.Context:
9. Here's a proposed context manager for the decimal module:
# This would be a new decimal.Context method
@contextmanager
def __context__(self):
def localcontext(ctx=None):
"""Set a new local decimal context for the block"""
# Default to using the current context
if ctx is None:
ctx = getcontext()
# We set the thread context to a copy of this context
# to ensure that changes within the block are kept
# local to the block. This also gives us thread safety
# and supports nested usage of a given context.
newctx = self.copy()
# local to the block.
newctx = ctx.copy()
oldctx = decimal.getcontext()
decimal.setcontext(newctx)
try:
yield newctx
finally:
# Always restore the original context
decimal.setcontext(oldctx)
Sample usage:
from decimal import localcontext, ExtendedContext
def sin(x):
with decimal.getcontext() as ctx:
with localcontext() as ctx:
ctx.prec += 2
# Rest of sin calculation algorithm
# uses a precision 2 greater than normal
return +s # Convert result to normal precision
def sin(x):
with decimal.ExtendedContext:
with localcontext(ExtendedContext):
# Rest of sin calculation algorithm
# uses the Extended Context from the
# General Decimal Arithmetic Specification
return +s # Convert result to normal context
10. A generic "object-closing" template:
10. A generic "object-closing" context manager:
@contextmanager
def closing(obj):
try:
yield obj
finally:
class closing(object):
def __init__(self, obj):
self.obj = obj
def __enter__(self):
return self.obj
def __exit__(self, *exc_info):
try:
close = obj.close
close_it = self.obj.close
except AttributeError:
pass
else:
close()
close_it()
This can be used to deterministically close anything with a
close method, be it file, generator, or something else. It
@ -907,20 +825,27 @@ Examples
for datum in data:
process(datum)
11. Native contexts for objects with acquire/release methods:
(Python 2.5's contextlib module contains a version
of this context manager)
# This would be a new method of e.g., threading.RLock
def __context__(self):
return locked(self)
11. PEP 319 gives a use case for also having a released()
context to temporarily release a previously acquired lock;
this can be written very similarly to the locked context
manager above by swapping the acquire() and release() calls.
def released(self):
return unlocked(self)
class released:
def __init__(self, lock):
self.lock = lock
def __enter__(self):
self.lock.release()
def __exit__(self, type, value, tb):
self.lock.acquire()
Sample usage:
with my_lock:
# Operations with the lock held
with my_lock.released():
with released(my_lock):
# Operations without the lock
# e.g. blocking I/O
# Lock is held again here
@ -973,56 +898,81 @@ Examples
with c as z:
# Perform operation
(Python 2.5's contextlib module contains a version
of this context manager)
Reference Implementation
This PEP was first accepted by Guido at his EuroPython
keynote, 27 June 2005.
It was accepted again later, with the __context__ method added.
The PEP was implemented for Python 2.5a1
The PEP was implemented in subversion for Python 2.5a1
The __context__() method will be removed in Python 2.5a3
Ackowledgements
Many people contributed to the ideas and concepts in this PEP,
including all those mentioned in the acknowledgements for PEP 340
and PEP 346.
Additional thanks goes to (in no meaningful order): Paul Moore,
Phillip J. Eby, Greg Ewing, Jason Orendorff, Michael Hudson,
Raymond Hettinger, Walter Dörwald, Aahz, Georg Brandl, Terry Reedy,
A.M. Kuchling, Brett Cannon, and all those that participated in the
discussions on python-dev.
References
[1] http://blogs.msdn.com/oldnewthing/archive/2005/01/06/347666.aspx
[1] Raymond Chen's article on hidden flow control
http://blogs.msdn.com/oldnewthing/archive/2005/01/06/347666.aspx
[2] http://mail.python.org/pipermail/python-dev/2005-May/053885.html
[2] Guido suggests some generator changes that ended up in PEP 342
http://mail.python.org/pipermail/python-dev/2005-May/053885.html
[3] http://wiki.python.org/moin/WithStatement
[3] Wiki discussion of PEP 343
http://wiki.python.org/moin/WithStatement
[4]
[4] Early draft of some documentation for the with statement
http://mail.python.org/pipermail/python-dev/2005-July/054658.html
[5]
[5] Proposal to add the __with__ method
http://mail.python.org/pipermail/python-dev/2005-October/056947.html
[6]
[6] Proposal to use the PEP 342 enhanced generator API directly
http://mail.python.org/pipermail/python-dev/2005-October/056969.html
[7]
[7] Guido lets me (Nick Coghlan) talk him into a bad idea ;)
http://mail.python.org/pipermail/python-dev/2005-October/057018.html
[8]
[8] Guido raises some exception handling questions
http://mail.python.org/pipermail/python-dev/2005-June/054064.html
[9]
[9] Guido answers some questions about the __context__ method
http://mail.python.org/pipermail/python-dev/2005-October/057520.html
[10]
[10] Guido answers more questions about the __context__ method
http://mail.python.org/pipermail/python-dev/2005-October/057535.html
[11]
[11] Guido says AttributeError is fine for missing special methods
http://mail.python.org/pipermail/python-dev/2005-October/057625.html
[12]
[12] Original PEP 342 implementation patch
http://sourceforge.net/tracker/index.php?func=detail&aid=1223381&group_id=5470&atid=305470
[13]
http://mail.python.org/pipermail/python-dev/2006-February/061903.html
[13] Guido restores the ability to suppress exceptions
http://mail.python.org/pipermail/python-dev/2006-February/061909.html
[14]
[14] A simple question kickstarts a thorough review of PEP 343
http://mail.python.org/pipermail/python-dev/2006-April/063859.html
[15] Guido kills the __context__() method
http://mail.python.org/pipermail/python-dev/2006-April/064632.html
[16] Greg propose 'context guard' instead of 'context manager'
http://mail.python.org/pipermail/python-dev/2006-May/064676.html
Copyright
This document has been placed in the public domain.