iSharkFly-Open
/
python-peps
mirror of https://github.com/python/peps.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Edit PEP 419: fix grammar, spelling and formatting.

This commit is contained in:
Georg Brandl 2012-04-08 00:44:05 +02:00
parent a5a6357c66
commit 1a71214d4e
1 changed files with 253 additions and 242 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

495
pep-0419.txt
View File

@ -13,90 +13,93 @@ Python-Version: 3.3
Abstract Abstract
======== ========
This PEP proposes a way to protect python code from being interrupted inside This PEP proposes a way to protect Python code from being interrupted
finally statement or context manager. inside a finally clause or during context manager cleanup.
Rationale Rationale
========= =========
Python has two nice ways to do cleanup. One is a ``finally`` statement Python has two nice ways to do cleanup. One is a ``finally``
and the other is context manager (or ``with`` statement). Although, statement and the other is a context manager (usually called using a
neither of them is protected from ``KeyboardInterrupt`` or ``with`` statement). However, neither is protected from interruption
``generator.throw()``. For example:: by ``KeyboardInterrupt`` or ``GeneratorExit`` caused by
``generator.throw()``. For example::
lock.acquire() lock.acquire()
try: try:
print('starting') print('starting')
do_someting() do_something()
finally: finally:
print('finished') print('finished')
lock.release() lock.release()
If ``KeyboardInterrupt`` occurs just after ``print`` function is If ``KeyboardInterrupt`` occurs just after the second ``print()``
executed, lock will not be released. Similarly the following code call, the lock will not be released. Similarly, the following code
using ``with`` statement is affected:: using the ``with`` statement is affected::
from threading import Lock from threading import Lock
class MyLock: class MyLock:
def __init__(self): def __init__(self):
self._lock_impl = lock self._lock_impl = Lock()
def __enter__(self): def __enter__(self):
self._lock_impl.acquire() self._lock_impl.acquire()
print("LOCKED") print("LOCKED")
def __exit__(self): def __exit__(self):
print("UNLOCKING") print("UNLOCKING")
self._lock_impl.release() self._lock_impl.release()
lock = MyLock() lock = MyLock()
with lock: with lock:
do_something do_something
If ``KeyboardInterrupt`` occurs near any of the ``print`` statements, If ``KeyboardInterrupt`` occurs near any of the ``print()`` calls, the
lock will never be released. lock will never be released.
Coroutine Use Case Coroutine Use Case
------------------ ------------------
Similar case occurs with coroutines. Usually coroutine libraries want A similar case occurs with coroutines. Usually coroutine libraries
to interrupt coroutine with a timeout. There is a want to interrupt the coroutine with a timeout. The
``generator.throw()`` method for this use case, but there are no ``generator.throw()`` method works for this use case, but there is no
method to know is it currently yielded from inside a ``finally``. way of knowing if the coroutine is currently suspended from inside a
``finally`` clause.
Example that uses yield-based coroutines follows. Code looks An example that uses yield-based coroutines follows. The code looks
similar using any of the popular coroutine libraries Monocle [1]_, similar using any of the popular coroutine libraries Monocle [1]_,
Bluelet [2]_, or Twisted [3]_. :: Bluelet [2]_, or Twisted [3]_. ::
def run_locked() def run_locked():
yield connection.sendall('LOCK') yield connection.sendall('LOCK')
try: try:
yield do_something() yield do_something()
yield do_something_else() yield do_something_else()
finally: finally:
yield connection.sendall('UNLOCK') yield connection.sendall('UNLOCK')
with timeout(5): with timeout(5):
yield run_locked() yield run_locked()
In the example above ``yield something`` means pause executing current In the example above, ``yield something`` means to pause executing the
coroutine and execute coroutine ``something`` until it finished current coroutine and to execute coroutine ``something`` until it
execution. So that library keeps stack of generators itself. The finishes execution. Therefore the coroutine library itself needs to
``connection.sendall`` waits until socket is writable and does thing maintain a stack of generators. The ``connection.sendall()`` call waits
similar to what ``socket.sendall`` does. until the socket is writable and does a similar thing to what
``socket.sendall()`` does.
The ``with`` statement ensures that all that code is executed within 5 The ``with`` statement ensures that all code is executed within 5
seconds timeout. It does so by registering a callback in main loop, seconds timeout. It does so by registering a callback in the main
which calls ``generator.throw()`` to the top-most frame in the loop, which calls ``generator.throw()`` on the top-most frame in the
coroutine stack when timeout happens. coroutine stack when a timeout happens.
The ``greenlets`` extension works in similar way, except it doesn't The ``greenlets`` extension works in a similar way, except that it
need ``yield`` to enter new stack frame. Otherwise considerations are doesn't need ``yield`` to enter a new stack frame. Otherwise
similar. considerations are similar.
Specification Specification
@ -105,62 +108,63 @@ Specification
Frame Flag 'f_in_cleanup' Frame Flag 'f_in_cleanup'
------------------------- -------------------------
A new flag on frame object is proposed. It is set to ``True`` if this A new flag on the frame object is proposed. It is set to ``True`` if
frame is currently in the ``finally`` suite. Internally it must be this frame is currently executing a ``finally`` clause. Internally,
implemented as a counter of nested finally statements currently the flag must be implemented as a counter of nested finally statements
executed. currently being executed.
The internal counter is also incremented when entering ``WITH_SETUP`` The internal counter also needs to be incremented during execution of
bytecode and ``WITH_CLEANUP`` bytecode, and is decremented when the ``SETUP_WITH`` and ``WITH_CLEANUP`` bytecodes, and decremented
leaving that bytecode. This allows to protect ``__enter__`` and when execution for these bytecodes is finished. This allows to also
``__exit__`` methods too. protect ``__enter__()`` and ``__exit__()`` methods.
Function 'sys.setcleanuphook' Function 'sys.setcleanuphook'
----------------------------- -----------------------------
A new function for the ``sys`` module is proposed. This function sets A new function for the ``sys`` module is proposed. This function sets
a callback which is executed every time ``f_in_cleanup`` becomes a callback which is executed every time ``f_in_cleanup`` becomes
``False``. Callbacks gets ``frame`` as it's sole argument so it can false. Callbacks get a frame object as their sole argument, so that
get some evindence where it is called from. they can figure out where they are called from.
The setting is thread local and is stored inside ``PyThreadState`` The setting is thread local and must be stored in the
structure. ``PyThreadState`` structure.
Inspect Module Enhancements Inspect Module Enhancements
--------------------------- ---------------------------
Two new functions are proposed for ``inspect`` module: Two new functions are proposed for the ``inspect`` module:
``isframeincleanup`` and ``getcleanupframe``. ``isframeincleanup()`` and ``getcleanupframe()``.
``isframeincleanup`` given ``frame`` object or ``generator`` object as ``isframeincleanup()``, given a frame or generator object as its sole
sole argument returns the value of ``f_in_cleanup`` attribute of a argument, returns the value of the ``f_in_cleanup`` attribute of a
frame itself or of the ``gi_frame`` attribute of a generator. frame itself or of the ``gi_frame`` attribute of a generator.
``getcleanupframe`` given ``frame`` object as sole argument returns ``getcleanupframe()``, given a frame object as its sole argument,
the innermost frame which has true value of ``f_in_cleanup`` or returns the innermost frame which has a true value of
``None`` if no frames in the stack has the attribute set. It starts to ``f_in_cleanup``, or ``None`` if no frames in the stack have a nonzero
inspect from specified frame and walks to outer frames using value for that attribute. It starts to inspect from the specified
``f_back`` pointers, just like ``getouterframes`` does. frame and walks to outer frames using ``f_back`` pointers, just like
``getouterframes()`` does.
Example Example
======= =======
Example implementation of ``SIGINT`` handler that interrupts safely An example implementation of a SIGINT handler that interrupts safely
might look like:: might look like::
import inspect, sys, functools import inspect, sys, functools
def sigint_handler(sig, frame) def sigint_handler(sig, frame):
if inspect.getcleanupframe(frame) is None: if inspect.getcleanupframe(frame) is None:
raise KeyboardInterrupt() raise KeyboardInterrupt()
sys.setcleanuphook(functools.partial(sigint_handler, 0)) sys.setcleanuphook(functools.partial(sigint_handler, 0))
Coroutine example is out of scope of this document, because it's A coroutine example is out of scope of this document, because its
implemention depends very much on a trampoline (or main loop) used by implementation depends very much on a trampoline (or main loop) used
coroutine library. by coroutine library.
Unresolved Issues Unresolved Issues
@ -169,277 +173,284 @@ Unresolved Issues
Interruption Inside With Statement Expression Interruption Inside With Statement Expression
--------------------------------------------- ---------------------------------------------
Given the statement:: Given the statement ::
with open(filename): with open(filename):
do_something() do_something()
Python can be interrupted after ``open`` is called, but before Python can be interrupted after ``open()`` is called, but before the
``SETUP_WITH`` bytecode is executed. There are two possible decisions: ``SETUP_WITH`` bytecode is executed. There are two possible
decisions:
* Protect expression inside ``with`` statement. This would need * Protect ``with`` expressions. This would require another bytecode,
another bytecode, since currently there is no delimiter at the start since currently there is no way of recognizing the start of the
of ``with`` expression ``with`` expression.
* Let user write a wrapper if he considers it's important for his * Let the user write a wrapper if he considers it important for the
use-case. Safe wrapper code might look like the following:: use-case. A safe wrapper might look like this::
class FileWrapper(object): class FileWrapper(object):
def __init__(self, filename, mode): def __init__(self, filename, mode):
self.filename = filename self.filename = filename
self.mode = mode self.mode = mode
def __enter__(self): def __enter__(self):
self.file = open(self.filename, self.mode) self.file = open(self.filename, self.mode)
def __exit__(self): def __exit__(self):
self.file.close() self.file.close()
Alternatively it can be written using context manager:: Alternatively it can be written using the ``contextmanager()``
decorator::
@contextmanager @contextmanager
def open_wrapper(filename, mode): def open_wrapper(filename, mode):
file = open(filename, mode) file = open(filename, mode)
try: try:
yield file yield file
finally: finally:
file.close() file.close()
This code is safe, as first part of generator (before yield) is This code is safe, as the first part of the generator (before yield)
executed inside ``WITH_SETUP`` bytecode of caller is executed inside the ``SETUP_WITH`` bytecode of the caller.
Exception Propagation Exception Propagation
--------------------- ---------------------
Sometimes ``finally`` block or ``__enter__/__exit__`` method can be Sometimes a ``finally`` clause or an ``__enter__()``/``__exit__()``
exited with an exception. Usually it's not a problem, since more method can raise an exception. Usually this is not a problem, since
important exception like ``KeyboardInterrupt`` or ``SystemExit`` more important exceptions like ``KeyboardInterrupt`` or ``SystemExit``
should be thrown instead. But it may be nice to be able to keep should be raised instead. But it may be nice to be able to keep the
original exception inside a ``__context__`` attibute. So cleanup hook original exception inside a ``__context__`` attribute. So the cleanup
signature may grow an exception argument:: hook signature may grow an exception argument::
def sigint_handler(sig, frame) def sigint_handler(sig, frame)
if inspect.getcleanupframe(frame) is None: if inspect.getcleanupframe(frame) is None:
raise KeyboardInterrupt() raise KeyboardInterrupt()
sys.setcleanuphook(retry_sigint) sys.setcleanuphook(retry_sigint)
def retry_sigint(frame, exception=None): def retry_sigint(frame, exception=None):
if inspect.getcleanupframe(frame) is None: if inspect.getcleanupframe(frame) is None:
raise KeyboardInterrupt() from exception raise KeyboardInterrupt() from exception
.. note:: .. note::
No need to have three arguments like in ``__exit__`` method since There is no need to have three arguments like in the ``__exit__``
we have a ``__traceback__`` attribute in exception in Python 3.x method since there is a ``__traceback__`` attribute in exception in
Python 3.
Although, this will set ``__cause__`` for the exception, which is not However, this will set the ``__cause__`` for the exception, which is
exactly what's intended. So some hidden interpeter logic may be used not exactly what's intended. So some hidden interpreter logic may be
to put ``__context__`` attribute on every exception raised in cleanup used to put a ``__context__`` attribute on every exception raised in a
hook. cleanup hook.
Interruption Between Acquiring Resource and Try Block Interruption Between Acquiring Resource and Try Block
----------------------------------------------------- -----------------------------------------------------
Example from the first section is not totally safe. Let's look closer:: The example from the first section is not totally safe. Let's take a
closer look::
lock.acquire() lock.acquire()
try: try:
do_something() do_something()
finally: finally:
lock.release() lock.release()
There is no way it can be fixed without modifying the code. The actual There is no way the code can be fixed unmodified. The actual fix
fix of this code depends very much on use case. depends very much on the use case. Usually code can be fixed using a
``with`` statement::
Usually code can be fixed using a ``with`` statement:: with lock:
do_something()
with lock: However, for coroutines one usually can't use the ``with`` statement
do_something() because you need to ``yield`` for both the acquire and release
operations. So the code might be rewritten like this::
Although, for coroutines you usually can't use ``with`` statement try:
because you need to ``yield`` for both aquire and release operations. yield lock.acquire()
So code might be rewritten as following:: do_something()
finally:
yield lock.release()
try: The actual locking code might need more code to support this use case,
yield lock.acquire() but the implementation is usually trivial, like this: check if the
do_something() lock has been acquired and unlock if it is.
finally:
yield lock.release()
The actual lock code might need more code to support this use case,
but implementation is usually trivial, like check if lock has been
acquired and unlock if it is.
Setting Interruption Context Inside Finally Itself Setting Interruption Context Inside Finally Itself
-------------------------------------------------- --------------------------------------------------
Some coroutine libraries may need to set a timeout for the finally Some coroutine libraries may need to set a timeout for the finally
clause itself. For example:: clause itself. For example::
try: try:
do_something() do_something()
finally: finally:
with timeout(0.5): with timeout(0.5):
try: try:
yield do_slow_cleanup() yield do_slow_cleanup()
finally: finally:
yield do_fast_cleanup() yield do_fast_cleanup()
With current semantics timeout will either protect With current semantics, timeout will either protect the whole ``with``
the whole ``with`` block or nothing at all, depending on the block or nothing at all, depending on the implementation of each
implementation of a library. What the author is intended is to treat library. What the author intended is to treat ``do_slow_cleanup`` as
``do_slow_cleanup`` as an ordinary code, and ``do_fast_cleanup`` as a ordinary code, and ``do_fast_cleanup`` as a cleanup (a
cleanup (non-interruptible one). non-interruptible one).
Similar case might occur when using greenlets or tasklets. A similar case might occur when using greenlets or tasklets.
This case can be fixed by exposing ``f_in_cleanup`` as a counter, and This case can be fixed by exposing ``f_in_cleanup`` as a counter, and
by calling cleanup hook on each decrement. Corouting library may then by calling a cleanup hook on each decrement. A coroutine library may
remember the value at timeout start, and compare it on each hook then remember the value at timeout start, and compare it on each hook
execution. execution.
But in practice example is considered to be too obscure to take in But in practice, the example is considered to be too obscure to take
account. into account.
Alternative Python Implementations Support Alternative Python Implementations Support
========================================== ==========================================
We consider ``f_in_cleanup`` and implementation detail. The actual We consider ``f_in_cleanup`` an implementation detail. The actual
implementation may have some fake frame-like object passed to signal implementation may have some fake frame-like object passed to signal
handler, cleanup hook and returned from ``getcleanupframe``. The only handler, cleanup hook and returned from ``getcleanupframe()``. The
requirement is that ``inspect`` module functions work as expected on only requirement is that the ``inspect`` module functions work as
that objects. For this reason we also allow to pass a ``generator`` expected on these objects. For this reason, we also allow to pass a
object to a ``isframeincleanup`` function, this disables need to use generator object to the ``isframeincleanup()`` function, which removes
``gi_frame`` attribute. the need to use the ``gi_frame`` attribute.
It may need to be specified that ``getcleanupframe`` must return the It might be necessary to specify that ``getcleanupframe()`` must
same object that will be passed to cleanup hook at next invocation. return the same object that will be passed to cleanup hook at the next
invocation.
Alternative Names Alternative Names
================= =================
Original proposal had ``f_in_finally`` flag. The original intention The original proposal had a ``f_in_finally`` frame attribute, as the
was to protect ``finally`` clauses. But as it grew up to protecting original intention was to protect ``finally`` clauses. But as it grew
``__enter__`` and ``__exit__`` methods too, the ``f_in_cleanup`` up to protecting ``__enter__`` and ``__exit__`` methods too, the
method seems better. Although ``__enter__`` method is not a cleanup ``f_in_cleanup`` name seems better. Although the ``__enter__`` method
routine, it at least relates to cleanup done by context managers. is not a cleanup routine, it at least relates to cleanup done by
context managers.
``setcleanuphook``, ``isframeincleanup`` and ``getcleanupframe`` can ``setcleanuphook``, ``isframeincleanup`` and ``getcleanupframe`` can
be unobscured to ``set_cleanup_hook``, ``is_frame_in_cleanup`` and be unobscured to ``set_cleanup_hook``, ``is_frame_in_cleanup`` and
``get_cleanup_frame``, althought they follow convention of their ``get_cleanup_frame``, although they follow the naming convention of
respective modules. their respective modules.
Alternative Proposals Alternative Proposals
===================== =====================
Propagating 'f_in_cleanup' Flag Automatically Propagating 'f_in_cleanup' Flag Automatically
----------------------------------------------- ---------------------------------------------
This can make ``getcleanupframe`` unnecessary. But for yield based This can make ``getcleanupframe()`` unnecessary. But for yield-based
coroutines you need to propagate it yourself. Making it writable leads coroutines you need to propagate it yourself. Making it writable
to somewhat unpredictable behavior of ``setcleanuphook`` leads to somewhat unpredictable behavior of ``setcleanuphook()``.
Add Bytecodes 'INCR_CLEANUP', 'DECR_CLEANUP' Add Bytecodes 'INCR_CLEANUP', 'DECR_CLEANUP'
-------------------------------------------- --------------------------------------------
These bytecodes can be used to protect expression inside ``with`` These bytecodes can be used to protect the expression inside the
statement, as well as making counter increments more explicit and easy ``with`` statement, as well as making counter increments more explicit
to debug (visible inside a disassembly). Some middle ground might be and easy to debug (visible inside a disassembly). Some middle ground
chosen, like ``END_FINALLY`` and ``SETUP_WITH`` imlicitly decrements might be chosen, like ``END_FINALLY`` and ``SETUP_WITH`` implicitly
counter (``END_FINALLY`` is present at end of ``with`` suite). decrementing the counter (``END_FINALLY`` is present at end of every
``with`` suite).
Although, adding new bytecodes must be considered very carefully. However, adding new bytecodes must be considered very carefully.
Expose 'f_in_cleanup' as a Counter Expose 'f_in_cleanup' as a Counter
---------------------------------- ----------------------------------
The original intention was to expose minimum needed functionality. The original intention was to expose a minimum of needed
Although, as we consider frame flag ``f_in_cleanup`` as an functionality. However, as we consider the frame flag
implementation detail, we may expose it as a counter. ``f_in_cleanup`` an implementation detail, we may expose it as a
counter.
Similarly, if we have a counter we may need to have cleanup hook Similarly, if we have a counter we may need to have the cleanup hook
called on every counter decrement. It's unlikely have much performance called on every counter decrement. It's unlikely to have much
impact as nested finally clauses are unlikely common case. performance impact as nested finally clauses are an uncommon case.
Add code object flag 'CO_CLEANUP' Add code object flag 'CO_CLEANUP'
--------------------------------- ---------------------------------
As an alternative to set flag inside ``WITH_SETUP``, and As an alternative to set the flag inside the ``SETUP_WITH`` and
``WITH_CLEANUP`` bytecodes we can introduce a flag ``CO_CLEANUP``. ``WITH_CLEANUP`` bytecodes, we can introduce a flag ``CO_CLEANUP``.
When interpreter starts to execute code with ``CO_CLEANUP`` set, it When the interpreter starts to execute code with ``CO_CLEANUP`` set,
sets ``f_in_cleanup`` for the whole function body. This flag is set it sets ``f_in_cleanup`` for the whole function body. This flag is
for code object of ``__enter__`` and ``__exit__`` special methods. set for code objects of ``__enter__`` and ``__exit__`` special
Technically it might be set on functions called ``__enter__`` and methods. Technically it might be set on functions called
``__exit__``. ``__enter__`` and ``__exit__``.
This seems to be less clear solution. It also covers the case where This seems to be less clear solution. It also covers the case where
``__enter__`` and ``__exit__`` are called manually. This may be ``__enter__`` and ``__exit__`` are called manually. This may be
accepted either as feature or as a unnecessary side-effect (unlikely accepted either as a feature or as an unnecessary side-effect (or,
as a bug). though unlikely, as a bug).
It may also impose a problem when ``__enter__`` or ``__exit__`` It may also impose a problem when ``__enter__`` or ``__exit__``
function are implemented in C, as there usually no frame to check for functions are implemented in C, as there is no code object to check
``f_in_cleanup`` flag. for the ``f_in_cleanup`` flag.
Have Cleanup Callback on Frame Object Itself Have Cleanup Callback on Frame Object Itself
---------------------------------------------- --------------------------------------------
Frame may be extended to have ``f_cleanup_callback`` which is called The frame object may be extended to have a ``f_cleanup_callback``
when ``f_in_cleanup`` is reset to 0. It would help to register member which is called when ``f_in_cleanup`` is reset to 0. This
different callbacks to different coroutines. would help to register different callbacks to different coroutines.
Despite apparent beauty. This solution doesn't add anything. As there Despite its apparent beauty, this solution doesn't add anything, as
are two primary use cases: the two primary use cases are:
* Set callback in signal handler. The callback is inherently single * Setting the callback in a signal handler. The callback is
one for this case inherently a single one for this case.
* Use single callback per loop for coroutine use case. And in almost * Use a single callback per loop for the coroutine use case. Here, in
all cases there is only one loop per thread almost all cases, there is only one loop per thread.
No Cleanup Hook No Cleanup Hook
--------------- ---------------
Original proposal included no cleanup hook specification. As there are The original proposal included no cleanup hook specification, as there
few ways to achieve the same using current tools: are a few ways to achieve the same using current tools:
* Use ``sys.settrace`` and ``f_trace`` callback. It may impose some * Using ``sys.settrace()`` and the ``f_trace`` callback. This may
problem to debugging, and has big performance impact (although, impose some problem to debugging, and has a big performance impact
interrupting doesn't happen very often) (although interrupting doesn't happen very often).
* Sleep a bit more and try again. For coroutine library it's easy. For * Sleeping a bit more and trying again. For a coroutine library this
signals it may be achieved using ``alert``. is easy. For signals it may be achieved using ``signal.alert``.
Both methods are considered too impractical and a way to catch exit Both methods are considered too impractical and a way to catch exit
from ``finally`` statement is proposed. from ``finally`` clauses is proposed.
References References
========== ==========
.. [1] Monocle .. [1] Monocle
https://github.com/saucelabs/monocle https://github.com/saucelabs/monocle
.. [2] Bluelet .. [2] Bluelet
https://github.com/sampsyo/bluelet https://github.com/sampsyo/bluelet
.. [3] Twisted: inlineCallbacks .. [3] Twisted: inlineCallbacks
http://twistedmatrix.com/documents/8.1.0/api/twisted.internet.defer.html http://twistedmatrix.com/documents/8.1.0/api/twisted.internet.defer.html
.. [4] Original discussion .. [4] Original discussion
http://mail.python.org/pipermail/python-ideas/2012-April/014705.html http://mail.python.org/pipermail/python-ideas/2012-April/014705.html
Copyright Copyright