Publish pep-419
This commit is contained in:
Andrew Svetlov
parent
fda80aabbb
commit
a5a6357c66
|
|
@ -0,0 +1,459 @@
|
|||
PEP: 419
|
||||
Title: Protecting cleanup statements from interruptions
|
||||
Version: $Revision$
|
||||
Last-Modified: $Date$
|
||||
Author: Paul Colomiets <paul@colomiets.name>
|
||||
Status: Draft
|
||||
Type: Standards Track
|
||||
Content-Type: text/x-rst
|
||||
Created: 06-Apr-2012
|
||||
Python-Version: 3.3
|
||||
|
||||
|
||||
Abstract
|
||||
========
|
||||
|
||||
This PEP proposes a way to protect python code from being interrupted inside
|
||||
finally statement or context manager.
|
||||
|
||||
|
||||
Rationale
|
||||
=========
|
||||
|
||||
Python has two nice ways to do cleanup. One is a ``finally`` statement
|
||||
and the other is context manager (or ``with`` statement). Although,
|
||||
neither of them is protected from ``KeyboardInterrupt`` or
|
||||
``generator.throw()``. For example::
|
||||
|
||||
lock.acquire()
|
||||
try:
|
||||
print('starting')
|
||||
do_someting()
|
||||
finally:
|
||||
print('finished')
|
||||
lock.release()
|
||||
|
||||
If ``KeyboardInterrupt`` occurs just after ``print`` function is
|
||||
executed, lock will not be released. Similarly the following code
|
||||
using ``with`` statement is affected::
|
||||
|
||||
from threading import Lock
|
||||
|
||||
class MyLock:
|
||||
|
||||
def __init__(self):
|
||||
self._lock_impl = lock
|
||||
|
||||
def __enter__(self):
|
||||
self._lock_impl.acquire()
|
||||
print("LOCKED")
|
||||
|
||||
def __exit__(self):
|
||||
print("UNLOCKING")
|
||||
self._lock_impl.release()
|
||||
|
||||
lock = MyLock()
|
||||
with lock:
|
||||
do_something
|
||||
|
||||
If ``KeyboardInterrupt`` occurs near any of the ``print`` statements,
|
||||
lock will never be released.
|
||||
|
||||
|
||||
Coroutine Use Case
|
||||
------------------
|
||||
|
||||
Similar case occurs with coroutines. Usually coroutine libraries want
|
||||
to interrupt coroutine with a timeout. There is a
|
||||
``generator.throw()`` method for this use case, but there are no
|
||||
method to know is it currently yielded from inside a ``finally``.
|
||||
|
||||
Example that uses yield-based coroutines follows. Code looks
|
||||
similar using any of the popular coroutine libraries Monocle [1]_,
|
||||
Bluelet [2]_, or Twisted [3]_. ::
|
||||
|
||||
def run_locked()
|
||||
yield connection.sendall('LOCK')
|
||||
try:
|
||||
yield do_something()
|
||||
yield do_something_else()
|
||||
finally:
|
||||
yield connection.sendall('UNLOCK')
|
||||
|
||||
with timeout(5):
|
||||
yield run_locked()
|
||||
|
||||
In the example above ``yield something`` means pause executing current
|
||||
coroutine and execute coroutine ``something`` until it finished
|
||||
execution. So that library keeps stack of generators itself. The
|
||||
``connection.sendall`` waits until socket is writable and does thing
|
||||
similar to what ``socket.sendall`` does.
|
||||
|
||||
The ``with`` statement ensures that all that code is executed within 5
|
||||
seconds timeout. It does so by registering a callback in main loop,
|
||||
which calls ``generator.throw()`` to the top-most frame in the
|
||||
coroutine stack when timeout happens.
|
||||
|
||||
The ``greenlets`` extension works in similar way, except it doesn't
|
||||
need ``yield`` to enter new stack frame. Otherwise considerations are
|
||||
similar.
|
||||
|
||||
|
||||
Specification
|
||||
=============
|
||||
|
||||
Frame Flag 'f_in_cleanup'
|
||||
-------------------------
|
||||
|
||||
A new flag on frame object is proposed. It is set to ``True`` if this
|
||||
frame is currently in the ``finally`` suite. Internally it must be
|
||||
implemented as a counter of nested finally statements currently
|
||||
executed.
|
||||
|
||||
The internal counter is also incremented when entering ``WITH_SETUP``
|
||||
bytecode and ``WITH_CLEANUP`` bytecode, and is decremented when
|
||||
leaving that bytecode. This allows to protect ``__enter__`` and
|
||||
``__exit__`` methods too.
|
||||
|
||||
|
||||
Function 'sys.setcleanuphook'
|
||||
-----------------------------
|
||||
|
||||
A new function for the ``sys`` module is proposed. This function sets
|
||||
a callback which is executed every time ``f_in_cleanup`` becomes
|
||||
``False``. Callbacks gets ``frame`` as it's sole argument so it can
|
||||
get some evindence where it is called from.
|
||||
|
||||
The setting is thread local and is stored inside ``PyThreadState``
|
||||
structure.
|
||||
|
||||
|
||||
Inspect Module Enhancements
|
||||
---------------------------
|
||||
|
||||
Two new functions are proposed for ``inspect`` module:
|
||||
``isframeincleanup`` and ``getcleanupframe``.
|
||||
|
||||
``isframeincleanup`` given ``frame`` object or ``generator`` object as
|
||||
sole argument returns the value of ``f_in_cleanup`` attribute of a
|
||||
frame itself or of the ``gi_frame`` attribute of a generator.
|
||||
|
||||
``getcleanupframe`` given ``frame`` object as sole argument returns
|
||||
the innermost frame which has true value of ``f_in_cleanup`` or
|
||||
``None`` if no frames in the stack has the attribute set. It starts to
|
||||
inspect from specified frame and walks to outer frames using
|
||||
``f_back`` pointers, just like ``getouterframes`` does.
|
||||
|
||||
|
||||
Example
|
||||
=======
|
||||
|
||||
Example implementation of ``SIGINT`` handler that interrupts safely
|
||||
might look like::
|
||||
|
||||
import inspect, sys, functools
|
||||
|
||||
def sigint_handler(sig, frame)
|
||||
if inspect.getcleanupframe(frame) is None:
|
||||
raise KeyboardInterrupt()
|
||||
sys.setcleanuphook(functools.partial(sigint_handler, 0))
|
||||
|
||||
Coroutine example is out of scope of this document, because it's
|
||||
implemention depends very much on a trampoline (or main loop) used by
|
||||
coroutine library.
|
||||
|
||||
|
||||
Unresolved Issues
|
||||
=================
|
||||
|
||||
Interruption Inside With Statement Expression
|
||||
---------------------------------------------
|
||||
|
||||
Given the statement::
|
||||
|
||||
with open(filename):
|
||||
do_something()
|
||||
|
||||
Python can be interrupted after ``open`` is called, but before
|
||||
``SETUP_WITH`` bytecode is executed. There are two possible decisions:
|
||||
|
||||
* Protect expression inside ``with`` statement. This would need
|
||||
another bytecode, since currently there is no delimiter at the start
|
||||
of ``with`` expression
|
||||
|
||||
* Let user write a wrapper if he considers it's important for his
|
||||
use-case. Safe wrapper code might look like the following::
|
||||
|
||||
class FileWrapper(object):
|
||||
|
||||
def __init__(self, filename, mode):
|
||||
self.filename = filename
|
||||
self.mode = mode
|
||||
|
||||
def __enter__(self):
|
||||
self.file = open(self.filename, self.mode)
|
||||
|
||||
def __exit__(self):
|
||||
self.file.close()
|
||||
|
||||
Alternatively it can be written using context manager::
|
||||
|
||||
@contextmanager
|
||||
def open_wrapper(filename, mode):
|
||||
file = open(filename, mode)
|
||||
try:
|
||||
yield file
|
||||
finally:
|
||||
file.close()
|
||||
|
||||
This code is safe, as first part of generator (before yield) is
|
||||
executed inside ``WITH_SETUP`` bytecode of caller
|
||||
|
||||
|
||||
Exception Propagation
|
||||
---------------------
|
||||
|
||||
Sometimes ``finally`` block or ``__enter__/__exit__`` method can be
|
||||
exited with an exception. Usually it's not a problem, since more
|
||||
important exception like ``KeyboardInterrupt`` or ``SystemExit``
|
||||
should be thrown instead. But it may be nice to be able to keep
|
||||
original exception inside a ``__context__`` attibute. So cleanup hook
|
||||
signature may grow an exception argument::
|
||||
|
||||
def sigint_handler(sig, frame)
|
||||
if inspect.getcleanupframe(frame) is None:
|
||||
raise KeyboardInterrupt()
|
||||
sys.setcleanuphook(retry_sigint)
|
||||
|
||||
def retry_sigint(frame, exception=None):
|
||||
if inspect.getcleanupframe(frame) is None:
|
||||
raise KeyboardInterrupt() from exception
|
||||
|
||||
.. note::
|
||||
|
||||
No need to have three arguments like in ``__exit__`` method since
|
||||
we have a ``__traceback__`` attribute in exception in Python 3.x
|
||||
|
||||
Although, this will set ``__cause__`` for the exception, which is not
|
||||
exactly what's intended. So some hidden interpeter logic may be used
|
||||
to put ``__context__`` attribute on every exception raised in cleanup
|
||||
hook.
|
||||
|
||||
|
||||
Interruption Between Acquiring Resource and Try Block
|
||||
-----------------------------------------------------
|
||||
|
||||
Example from the first section is not totally safe. Let's look closer::
|
||||
|
||||
lock.acquire()
|
||||
try:
|
||||
do_something()
|
||||
finally:
|
||||
lock.release()
|
||||
|
||||
There is no way it can be fixed without modifying the code. The actual
|
||||
fix of this code depends very much on use case.
|
||||
|
||||
Usually code can be fixed using a ``with`` statement::
|
||||
|
||||
with lock:
|
||||
do_something()
|
||||
|
||||
Although, for coroutines you usually can't use ``with`` statement
|
||||
because you need to ``yield`` for both aquire and release operations.
|
||||
So code might be rewritten as following::
|
||||
|
||||
try:
|
||||
yield lock.acquire()
|
||||
do_something()
|
||||
finally:
|
||||
yield lock.release()
|
||||
|
||||
The actual lock code might need more code to support this use case,
|
||||
but implementation is usually trivial, like check if lock has been
|
||||
acquired and unlock if it is.
|
||||
|
||||
|
||||
Setting Interruption Context Inside Finally Itself
|
||||
--------------------------------------------------
|
||||
|
||||
Some coroutine libraries may need to set a timeout for the finally
|
||||
clause itself. For example::
|
||||
|
||||
try:
|
||||
do_something()
|
||||
finally:
|
||||
with timeout(0.5):
|
||||
try:
|
||||
yield do_slow_cleanup()
|
||||
finally:
|
||||
yield do_fast_cleanup()
|
||||
|
||||
With current semantics timeout will either protect
|
||||
the whole ``with`` block or nothing at all, depending on the
|
||||
implementation of a library. What the author is intended is to treat
|
||||
``do_slow_cleanup`` as an ordinary code, and ``do_fast_cleanup`` as a
|
||||
cleanup (non-interruptible one).
|
||||
|
||||
Similar case might occur when using greenlets or tasklets.
|
||||
|
||||
This case can be fixed by exposing ``f_in_cleanup`` as a counter, and
|
||||
by calling cleanup hook on each decrement. Corouting library may then
|
||||
remember the value at timeout start, and compare it on each hook
|
||||
execution.
|
||||
|
||||
But in practice example is considered to be too obscure to take in
|
||||
account.
|
||||
|
||||
|
||||
Alternative Python Implementations Support
|
||||
==========================================
|
||||
|
||||
We consider ``f_in_cleanup`` and implementation detail. The actual
|
||||
implementation may have some fake frame-like object passed to signal
|
||||
handler, cleanup hook and returned from ``getcleanupframe``. The only
|
||||
requirement is that ``inspect`` module functions work as expected on
|
||||
that objects. For this reason we also allow to pass a ``generator``
|
||||
object to a ``isframeincleanup`` function, this disables need to use
|
||||
``gi_frame`` attribute.
|
||||
|
||||
It may need to be specified that ``getcleanupframe`` must return the
|
||||
same object that will be passed to cleanup hook at next invocation.
|
||||
|
||||
|
||||
Alternative Names
|
||||
=================
|
||||
|
||||
Original proposal had ``f_in_finally`` flag. The original intention
|
||||
was to protect ``finally`` clauses. But as it grew up to protecting
|
||||
``__enter__`` and ``__exit__`` methods too, the ``f_in_cleanup``
|
||||
method seems better. Although ``__enter__`` method is not a cleanup
|
||||
routine, it at least relates to cleanup done by context managers.
|
||||
|
||||
``setcleanuphook``, ``isframeincleanup`` and ``getcleanupframe`` can
|
||||
be unobscured to ``set_cleanup_hook``, ``is_frame_in_cleanup`` and
|
||||
``get_cleanup_frame``, althought they follow convention of their
|
||||
respective modules.
|
||||
|
||||
|
||||
Alternative Proposals
|
||||
=====================
|
||||
|
||||
Propagating 'f_in_cleanup' Flag Automatically
|
||||
-----------------------------------------------
|
||||
|
||||
This can make ``getcleanupframe`` unnecessary. But for yield based
|
||||
coroutines you need to propagate it yourself. Making it writable leads
|
||||
to somewhat unpredictable behavior of ``setcleanuphook``
|
||||
|
||||
|
||||
Add Bytecodes 'INCR_CLEANUP', 'DECR_CLEANUP'
|
||||
--------------------------------------------
|
||||
|
||||
These bytecodes can be used to protect expression inside ``with``
|
||||
statement, as well as making counter increments more explicit and easy
|
||||
to debug (visible inside a disassembly). Some middle ground might be
|
||||
chosen, like ``END_FINALLY`` and ``SETUP_WITH`` imlicitly decrements
|
||||
counter (``END_FINALLY`` is present at end of ``with`` suite).
|
||||
|
||||
Although, adding new bytecodes must be considered very carefully.
|
||||
|
||||
|
||||
Expose 'f_in_cleanup' as a Counter
|
||||
----------------------------------
|
||||
|
||||
The original intention was to expose minimum needed functionality.
|
||||
Although, as we consider frame flag ``f_in_cleanup`` as an
|
||||
implementation detail, we may expose it as a counter.
|
||||
|
||||
Similarly, if we have a counter we may need to have cleanup hook
|
||||
called on every counter decrement. It's unlikely have much performance
|
||||
impact as nested finally clauses are unlikely common case.
|
||||
|
||||
|
||||
Add code object flag 'CO_CLEANUP'
|
||||
---------------------------------
|
||||
|
||||
As an alternative to set flag inside ``WITH_SETUP``, and
|
||||
``WITH_CLEANUP`` bytecodes we can introduce a flag ``CO_CLEANUP``.
|
||||
When interpreter starts to execute code with ``CO_CLEANUP`` set, it
|
||||
sets ``f_in_cleanup`` for the whole function body. This flag is set
|
||||
for code object of ``__enter__`` and ``__exit__`` special methods.
|
||||
Technically it might be set on functions called ``__enter__`` and
|
||||
``__exit__``.
|
||||
|
||||
This seems to be less clear solution. It also covers the case where
|
||||
``__enter__`` and ``__exit__`` are called manually. This may be
|
||||
accepted either as feature or as a unnecessary side-effect (unlikely
|
||||
as a bug).
|
||||
|
||||
It may also impose a problem when ``__enter__`` or ``__exit__``
|
||||
function are implemented in C, as there usually no frame to check for
|
||||
``f_in_cleanup`` flag.
|
||||
|
||||
|
||||
Have Cleanup Callback on Frame Object Itself
|
||||
----------------------------------------------
|
||||
|
||||
Frame may be extended to have ``f_cleanup_callback`` which is called
|
||||
when ``f_in_cleanup`` is reset to 0. It would help to register
|
||||
different callbacks to different coroutines.
|
||||
|
||||
Despite apparent beauty. This solution doesn't add anything. As there
|
||||
are two primary use cases:
|
||||
|
||||
* Set callback in signal handler. The callback is inherently single
|
||||
one for this case
|
||||
|
||||
* Use single callback per loop for coroutine use case. And in almost
|
||||
all cases there is only one loop per thread
|
||||
|
||||
|
||||
No Cleanup Hook
|
||||
---------------
|
||||
|
||||
Original proposal included no cleanup hook specification. As there are
|
||||
few ways to achieve the same using current tools:
|
||||
|
||||
* Use ``sys.settrace`` and ``f_trace`` callback. It may impose some
|
||||
problem to debugging, and has big performance impact (although,
|
||||
interrupting doesn't happen very often)
|
||||
|
||||
* Sleep a bit more and try again. For coroutine library it's easy. For
|
||||
signals it may be achieved using ``alert``.
|
||||
|
||||
Both methods are considered too impractical and a way to catch exit
|
||||
from ``finally`` statement is proposed.
|
||||
|
||||
|
||||
References
|
||||
==========
|
||||
|
||||
.. [1] Monocle
|
||||
https://github.com/saucelabs/monocle
|
||||
|
||||
.. [2] Bluelet
|
||||
https://github.com/sampsyo/bluelet
|
||||
|
||||
.. [3] Twisted: inlineCallbacks
|
||||
http://twistedmatrix.com/documents/8.1.0/api/twisted.internet.defer.html
|
||||
|
||||
.. [4] Original discussion
|
||||
http://mail.python.org/pipermail/python-ideas/2012-April/014705.html
|
||||
|
||||
|
||||
Copyright
|
||||
=========
|
||||
|
||||
This document has been placed in the public domain.
|
||||
|
||||
|
||||
|
||||
..
|
||||
Local Variables:
|
||||
mode: indented-text
|
||||
indent-tabs-mode: nil
|
||||
sentence-end-double-space: t
|
||||
fill-column: 70
|
||||
coding: utf-8
|
||||
End:
|
||||
Loading…
Reference in New Issue