iSharkFly-Open
/
python-peps
mirror of https://github.com/python/peps.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Update PEP 343 to reflect post-acceptance python-dev discussions

This commit is contained in:
Nick Coghlan 2005-10-16 07:30:17 +00:00
parent fe551846ee
commit a4e8e16f80
1 changed files with 238 additions and 55 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

293
pep-0343.txt
View File

@ -2,13 +2,29 @@ PEP: 343
Title: Anonymous Block Redux and Generator Enhancements
Version: $Revision$
Last-Modified: $Date$
Author: Guido van Rossum
Author: Guido van Rossum, Nick Coghlan
Status: Accepted
Type: Standards Track
Content-Type: text/plain
Created: 13-May-2005
Post-History: 2-Jun-2005
Abstract
This PEP adds a new statement "with" to the Python language to make
it possible to factor out standard uses of try/finally statements.
The PEP has been approved in principle by the BDFL, but there are
still a couple of implementation details to be worked out (see the
section on Open Issues).
Author's Note
This PEP was originally written in first person by Guido, and
subsequently updated by Nick Coghlan to reflect later discussion
on python-dev. Any first person references are from Guido's
original.
Introduction
After a lot of discussion about PEP 340 and alternatives, I
@ -208,7 +224,7 @@ Specification: The 'with' Statement
The translation of the above statement is:
abc = EXPR
abc = (EXPR).__with__()
exc = (None, None, None)
VAR = abc.__enter__()
try:
@ -224,6 +240,15 @@ Specification: The 'with' Statement
accessible to the user; they will most likely be implemented as
special registers or stack positions.
The call to the __with__() method serves a similar purpose to that
of the __iter__() method of iterator and iterables. An object with
with simple state requirements (such as threading.RLock) may provide
its own __enter__() and __exit__() methods, and simply return
'self' from its __with__ method. On the other hand, an object with
more complex state requirements (such as decimal.Context) may
return a distinct context manager object each time its __with__
method is invoked.
If the "as VAR" part of the syntax is omitted, the "VAR =" part of
the translation is omitted (but abc.__enter__() is still called).
@ -254,15 +279,18 @@ Specification: The 'with' Statement
Generator Decorator
If PEP 342 is accepted, it will be possible to write a decorator
With PEP 342 accepted, it is possible to write a decorator
that makes it possible to use a generator that yields exactly once
to control a with-statement. Here's a sketch of such a decorator:
class ContextWrapper(object):
class GeneratorContext(object):
def __init__(self, gen):
self.gen = gen
def __with__(self):
return self
def __enter__(self):
try:
return self.gen.next()
@ -285,25 +313,31 @@ Generator Decorator
else:
raise RuntimeError("generator caught exception")
def contextmanager(func):
def context(func):
def helper(*args, **kwds):
return ContextWrapper(func(*args, **kwds))
return GeneratorContext(func(*args, **kwds))
return helper
This decorator could be used as follows:
@contextmanager
@context
def opening(filename):
f = open(filename) # IOError is untouched by ContextWrapper
f = open(filename) # IOError is untouched by GeneratorContext
try:
yield f
finally:
f.close() # Ditto for errors here (however unlikely)
A robust implementation of this decorator should be made part of
the standard library, but not necessarily as a built-in function.
(I'm not sure which exception it should raise for errors;
RuntimeError is used above as an example only.)
the standard library. Refer to Open Issues regarding its name and
location.
Just as generator-iterator functions are very useful for writing
__iter__() methods for iterables, generator-context functions will
be very useful for writing __with__() methods for contexts. It is
proposed that the invocation of the "context" decorator be
considered implicit for generator functions used as __with__()
methods (again, refer to the Open Issues section).
Optional Extensions
@ -332,17 +366,78 @@ Optional Extensions
is entered).
OTOH such mistakes are easily diagnosed; for example, the
contextmanager decorator above raises RuntimeError when the second
with-statement calls f.__enter__() again.
generator-context decorator above raises RuntimeError when a
second with-statement calls f.__enter__() again. A similar error
can be raised if __enter__ is invoked on a closed file object.
Resolved Open Issues
Standard Terminology
Discussion on python-dev revealed some open issues. I list them
here, with my preferred resolution and its motivation. The PEP
has been accepted without these being challenged, so the issues
are now resolved.
Discussions about iterators and iterables are aided by the standard
terminology used to discuss them. The protocol used by the for
statement is called the iterator protocol and an iterator is any
object that properly implements that protocol. The term "iterable"
then encompasses all objects with an __iter__() method that
returns an iterator (this means that all iterators are iterables,
but not all iterables are iterators).
1. The __exit__() method of the contextmanager decorator class
This PEP proposes that the protocol used by the with statement be
known as the "context management protocol", and that objects that
implement that protocol be known as "context managers". The term
"context" then encompasses all objects with a __with__() method
that returns a context manager (this means that all context managers
are contexts, but not all contexts are context managers).
The term "context" is based on the concept that the context object
defines a context of execution for the code that forms the body
of the with statement.
In cases where the general term "context" would be ambiguous, it
can be made explicit by expanding it to "manageable context".
Open Issues
Discussion on python-dev revealed some open issues. These are listed
here and will be resolved either by consensus on python-dev or by
BDFL fiat.
1. The name of the decorator used to convert a generator-iterator
function into a generator-context function is still to be
finalised.
The proposal in this PEP is that it be called simply "context"
with the following reasoning:
- A "generator function" is an undecorated function containing
the 'yield' keyword, and the objects produced by
such functions are "generator-iterators". The term
"generator" may refer to either a generator function or a
generator-iterator depending on the situation.
- A "generator context function" is a generator function to
which the "context" decorator is applied and the objects
produced by such functions are "generator-context-managers".
The term "generator context" may refer to either a generator
context function or a generator-context-manager depending on
the situation.
2. Should the decorator to convert a generator function into a
generator context function be a builtin, or located elsewhere in
the standard library? This PEP suggests that it should be a
builtin, as generator context functions are the recommended way
of writing new context managers.
3. Should a generator function used to implement a __with__ method
always be considered to be a generator context function, without
requiring the context decorator? This PEP suggests that it
should, as applying a decorator to a slot just looks strange,
and omitting the decorator would be a source of obscure bugs.
The __new__ slot provides some precedent for special casing of
certain slots when processing slot methods.
Resolved Issues
The following issues were resolved either by BDFL fiat, consensus on
python-dev, or a simple lack of objection to proposals in the
original version of this PEP.
1. The __exit__() method of the GeneratorContext class
catches StopIteration and considers it equivalent to re-raising
the exception passed to throw(). Is allowing StopIteration
right here?
@ -362,19 +457,77 @@ Resolved Open Issues
finally-clause (the one implicit in the with-statement) which
re-raises the original exception anyway.
2. What exception should GeneratorContext raise when the underlying
generator-iterator misbehaves? The following quote is the reason
behind Guido's choice of RuntimeError for both this and for the
generator close() method in PEP 342 (from [8]):
"I'd rather not introduce a new exception class just for this
purpose, since it's not an exception that I want people to catch:
I want it to turn into a traceback which is seen by the
programmer who then fixes the code. So now I believe they
should both raise RuntimeError.
There are some precedents for that: it's raised by the core
Python code in situations where endless recursion is detected,
and for uninitialized objects (and for a variety of
miscellaneous conditions)."
3. After this PEP was originally approved, a subsequent discussion
on python-dev [4] settled on the term "context manager" for
objects which provide __enter__ and __exit__ methods, and
"context management protocol" for the protocol itself. With the
addition of the __with__ method to the protocol, a natural
extension is to call all objects which provide a __with__ method
"contexts" (or "manageable contexts" in situations where the
general term "context" would be ambiguous).
This is now documented in the "Standard Terminology" section.
4. The originally approved version of this PEP did not include a
__with__ method - the method was only added to the PEP after
Jason Orendorff pointed out the difficulty of writing
appropriate __enter__ and __exit__ methods for decimal.Context
[5]. This approach allows a class to define a native context
manager using generator syntax. It also allows a class to use an
existing independent context manager as its native context
manager by applying the independent context manager to 'self' in
its __with__ method. It even allows a class written in C to use
a generator context manager written in Python.
The __with__ method parallels the __iter__ method which forms
part of the iterator protocol.
5. The suggestion was made by Jason Orendorff that the __enter__
and __exit__ methods could be removed from the context
management protocol, and the protocol instead defined directly
in terms of the enhanced generator interface described in PEP
342 [6].
Guido rejected this idea [7]. The following are some of benefits
of keeping the __enter__ and __exit__ methods:
- it makes it easy to implement a simple context manager in C
without having to rely on a separate coroutine builder
- it makes it easy to provide a low-overhead implementation
for context managers which don't need to maintain any
special state between the __enter__ and __exit__ methods
(having to use a generator for these would impose
unnecessary overhead without any compensating benefit)
- it makes it possible to understand how the with statement
works without having to first understand the mechanics of
how generator context managers are implemented.
Examples
(Note: several of these examples contain "yield None". If PEP 342
is accepted, these can be changed to just "yield".)
(The generator based examples assume PEP 342 is implemented. Also,
some of the examples are likely to be unnecessary in practice, as
the appropriate objects, such as threading.RLock, will be able to
be used directly in with statements)
1. A template for ensuring that a lock, acquired at the start of a
block, is released when the block is left:
@contextmanager
@context
def locking(lock):
lock.acquire()
try:
yield None
yield
finally:
lock.release()
@ -392,7 +545,7 @@ Examples
2. A template for opening a file that ensures the file is closed
when the block is left:
@contextmanager
@context
def opening(filename, mode="r"):
f = open(filename, mode)
try:
@ -409,7 +562,7 @@ Examples
3. A template for committing or rolling back a database
transaction:
@contextmanager
@context
def transactional(db):
db.begin()
try:
@ -424,18 +577,20 @@ Examples
class locking:
def __init__(self, lock):
self.lock = lock
def __with__(self, lock):
return self
def __enter__(self):
self.lock.acquire()
def __exit__(self, type, value, tb):
self.lock.release()
(This example is easily modified to implement the other
examples; it shows the relative advantage of using a generator
template.)
examples; it shows that is is easy to avoid the need for a
generator if no special state needs to be preserved.)
5. Redirect stdout temporarily:
@contextmanager
@context
def redirecting_stdout(new_stdout):
save_stdout = sys.stdout
sys.stdout = new_stdout
@ -456,7 +611,7 @@ Examples
6. A variant on opening() that also returns an error condition:
@contextmanager
@context
def opening_w_error(filename, mode="r"):
try:
f = open(filename, mode)
@ -520,54 +675,67 @@ Examples
# so this must be outside the with-statement:
return +s
9. Here's a more general Decimal-context-switching template:
9. Here's a proposed native context manager for decimal.Context:
@contextmanager
def decimal_context(newctx=None):
oldctx = decimal.getcontext()
if newctx is None:
newctx = oldctx.copy()
decimal.setcontext(newctx)
try:
yield newctx
finally:
decimal.setcontext(oldctx)
# This would be a new decimal.Context method
def __with__(self):
# We set the thread context to a copy of this context
# to ensure that changes within the block are kept
# local to the block. This also gives us thread safety
# and supports nested usage of a given context.
newctx = self.copy()
oldctx = decimal.getcontext()
decimal.setcontext(newctx)
try:
yield newctx
finally:
decimal.setcontext(oldctx)
Sample usage:
Sample usage:
def sin(x):
with decimal_context() as ctx:
ctx.prec += 2
# Rest of algorithm the same as above
return +s
def sin(x):
with decimal.getcontext() as ctx:
ctx.prec += 2
# Rest of sin calculation algorithm
# uses a precision 2 greater than normal
return +s # Convert result to normal precision
(Nick Coghlan has proposed to add __enter__() and __exit__()
methods to the decimal.Context class so that this example can
be simplified to "with decimal.getcontext() as ctx: ...".)
def sin(x):
with decimal.ExtendedContext:
# Rest of sin calculation algorithm
# uses the Extended Context from the
# General Decimal Arithmetic Specification
return +s # Convert result to normal context
10. A generic "object-closing" template:
@contextmanager
@context
def closing(obj):
try:
yield obj
finally:
obj.close()
try:
close = obj.close
except AttributeError:
pass
else:
close()
This can be used to deterministically close anything with a
close method, be it file, generator, or something else:
close method, be it file, generator, or something else. It can
even be used when the object isn't guaranteed to require
closing (e.g., a function that accepts an arbitrary iterable):
# emulate opening():
with closing(open("argument.txt")) as contradiction:
for line in contradiction:
print line
# deterministically finalize a generator:
with closing(some_gen()) as data:
# deterministically finalize an iterator:
with closing(iter(data_source)) as data:
for datum in data:
process(datum)
References
[1] http://blogs.msdn.com/oldnewthing/archive/2005/01/06/347666.aspx
@ -576,6 +744,21 @@ References
[3] http://wiki.python.org/moin/WithStatement
[4]
http://mail.python.org/pipermail/python-dev/2005-July/054658.html
[5]
http://mail.python.org/pipermail/python-dev/2005-October/056947.html
[6]
http://mail.python.org/pipermail/python-dev/2005-October/056969.html
[7]
http://mail.python.org/pipermail/python-dev/2005-October/057018.html
[8]
http://mail.python.org/pipermail/python-dev/2005-June/054064.html
Copyright
This document has been placed in the public domain.