python-peps/pep-0419/index.html

635 lines
51 KiB
HTML
Raw Permalink Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta name="color-scheme" content="light dark">
<title>PEP 419 Protecting cleanup statements from interruptions | peps.python.org</title>
<link rel="shortcut icon" href="../_static/py.png">
<link rel="canonical" href="https://peps.python.org/pep-0419/">
<link rel="stylesheet" href="../_static/style.css" type="text/css">
<link rel="stylesheet" href="../_static/mq.css" type="text/css">
<link rel="stylesheet" href="../_static/pygments.css" type="text/css" media="(prefers-color-scheme: light)" id="pyg-light">
<link rel="stylesheet" href="../_static/pygments_dark.css" type="text/css" media="(prefers-color-scheme: dark)" id="pyg-dark">
<link rel="alternate" type="application/rss+xml" title="Latest PEPs" href="https://peps.python.org/peps.rss">
<meta property="og:title" content='PEP 419 Protecting cleanup statements from interruptions | peps.python.org'>
<meta property="og:description" content="This PEP proposes a way to protect Python code from being interrupted inside a finally clause or during context manager cleanup.">
<meta property="og:type" content="website">
<meta property="og:url" content="https://peps.python.org/pep-0419/">
<meta property="og:site_name" content="Python Enhancement Proposals (PEPs)">
<meta property="og:image" content="https://peps.python.org/_static/og-image.png">
<meta property="og:image:alt" content="Python PEPs">
<meta property="og:image:width" content="200">
<meta property="og:image:height" content="200">
<meta name="description" content="This PEP proposes a way to protect Python code from being interrupted inside a finally clause or during context manager cleanup.">
<meta name="theme-color" content="#3776ab">
</head>
<body>
<svg xmlns="http://www.w3.org/2000/svg" style="display: none;">
<symbol id="svg-sun-half" viewBox="0 0 24 24" pointer-events="all">
<title>Following system colour scheme</title>
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none"
stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round">
<circle cx="12" cy="12" r="9"></circle>
<path d="M12 3v18m0-12l4.65-4.65M12 14.3l7.37-7.37M12 19.6l8.85-8.85"></path>
</svg>
</symbol>
<symbol id="svg-moon" viewBox="0 0 24 24" pointer-events="all">
<title>Selected dark colour scheme</title>
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none"
stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round">
<path stroke="none" d="M0 0h24v24H0z" fill="none"></path>
<path d="M12 3c.132 0 .263 0 .393 0a7.5 7.5 0 0 0 7.92 12.446a9 9 0 1 1 -8.313 -12.454z"></path>
</svg>
</symbol>
<symbol id="svg-sun" viewBox="0 0 24 24" pointer-events="all">
<title>Selected light colour scheme</title>
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none"
stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round">
<circle cx="12" cy="12" r="5"></circle>
<line x1="12" y1="1" x2="12" y2="3"></line>
<line x1="12" y1="21" x2="12" y2="23"></line>
<line x1="4.22" y1="4.22" x2="5.64" y2="5.64"></line>
<line x1="18.36" y1="18.36" x2="19.78" y2="19.78"></line>
<line x1="1" y1="12" x2="3" y2="12"></line>
<line x1="21" y1="12" x2="23" y2="12"></line>
<line x1="4.22" y1="19.78" x2="5.64" y2="18.36"></line>
<line x1="18.36" y1="5.64" x2="19.78" y2="4.22"></line>
</svg>
</symbol>
</svg>
<script>
document.documentElement.dataset.colour_scheme = localStorage.getItem("colour_scheme") || "auto"
</script>
<section id="pep-page-section">
<header>
<h1>Python Enhancement Proposals</h1>
<ul class="breadcrumbs">
<li><a href="https://www.python.org/" title="The Python Programming Language">Python</a> &raquo; </li>
<li><a href="../pep-0000/">PEP Index</a> &raquo; </li>
<li>PEP 419</li>
</ul>
<button id="colour-scheme-cycler" onClick="setColourScheme(nextColourScheme())">
<svg aria-hidden="true" class="colour-scheme-icon-when-auto"><use href="#svg-sun-half"></use></svg>
<svg aria-hidden="true" class="colour-scheme-icon-when-dark"><use href="#svg-moon"></use></svg>
<svg aria-hidden="true" class="colour-scheme-icon-when-light"><use href="#svg-sun"></use></svg>
<span class="visually-hidden">Toggle light / dark / auto colour theme</span>
</button>
</header>
<article>
<section id="pep-content">
<h1 class="page-title">PEP 419 Protecting cleanup statements from interruptions</h1>
<dl class="rfc2822 field-list simple">
<dt class="field-odd">Author<span class="colon">:</span></dt>
<dd class="field-odd">Paul Colomiets &lt;paul&#32;&#97;t&#32;colomiets.name&gt;</dd>
<dt class="field-even">Status<span class="colon">:</span></dt>
<dd class="field-even"><abbr title="Inactive draft that may be taken up again at a later time">Deferred</abbr></dd>
<dt class="field-odd">Type<span class="colon">:</span></dt>
<dd class="field-odd"><abbr title="Normative PEP with a new feature for Python, implementation change for CPython or interoperability standard for the ecosystem">Standards Track</abbr></dd>
<dt class="field-even">Created<span class="colon">:</span></dt>
<dd class="field-even">06-Apr-2012</dd>
<dt class="field-odd">Python-Version<span class="colon">:</span></dt>
<dd class="field-odd">3.3</dd>
</dl>
<hr class="docutils" />
<section id="contents">
<details><summary>Table of Contents</summary><ul class="simple">
<li><a class="reference internal" href="#abstract">Abstract</a></li>
<li><a class="reference internal" href="#pep-deferral">PEP Deferral</a></li>
<li><a class="reference internal" href="#rationale">Rationale</a><ul>
<li><a class="reference internal" href="#coroutine-use-case">Coroutine Use Case</a></li>
</ul>
</li>
<li><a class="reference internal" href="#specification">Specification</a><ul>
<li><a class="reference internal" href="#frame-flag-f-in-cleanup">Frame Flag f_in_cleanup</a></li>
<li><a class="reference internal" href="#function-sys-setcleanuphook">Function sys.setcleanuphook</a></li>
<li><a class="reference internal" href="#inspect-module-enhancements">Inspect Module Enhancements</a></li>
</ul>
</li>
<li><a class="reference internal" href="#example">Example</a></li>
<li><a class="reference internal" href="#unresolved-issues">Unresolved Issues</a><ul>
<li><a class="reference internal" href="#interruption-inside-with-statement-expression">Interruption Inside With Statement Expression</a></li>
<li><a class="reference internal" href="#exception-propagation">Exception Propagation</a></li>
<li><a class="reference internal" href="#interruption-between-acquiring-resource-and-try-block">Interruption Between Acquiring Resource and Try Block</a></li>
<li><a class="reference internal" href="#handling-eintr-inside-a-finally">Handling EINTR Inside a Finally</a></li>
<li><a class="reference internal" href="#setting-interruption-context-inside-finally-itself">Setting Interruption Context Inside Finally Itself</a></li>
<li><a class="reference internal" href="#modifying-keyboardinterrupt">Modifying KeyboardInterrupt</a></li>
</ul>
</li>
<li><a class="reference internal" href="#alternative-python-implementations-support">Alternative Python Implementations Support</a></li>
<li><a class="reference internal" href="#alternative-names">Alternative Names</a></li>
<li><a class="reference internal" href="#alternative-proposals">Alternative Proposals</a><ul>
<li><a class="reference internal" href="#propagating-f-in-cleanup-flag-automatically">Propagating f_in_cleanup Flag Automatically</a></li>
<li><a class="reference internal" href="#add-bytecodes-incr-cleanup-decr-cleanup">Add Bytecodes INCR_CLEANUP, DECR_CLEANUP</a></li>
<li><a class="reference internal" href="#expose-f-in-cleanup-as-a-counter">Expose f_in_cleanup as a Counter</a></li>
<li><a class="reference internal" href="#add-code-object-flag-co-cleanup">Add code object flag CO_CLEANUP</a></li>
<li><a class="reference internal" href="#have-cleanup-callback-on-frame-object-itself">Have Cleanup Callback on Frame Object Itself</a></li>
<li><a class="reference internal" href="#no-cleanup-hook">No Cleanup Hook</a></li>
</ul>
</li>
<li><a class="reference internal" href="#references">References</a></li>
<li><a class="reference internal" href="#copyright">Copyright</a></li>
</ul>
</details></section>
<section id="abstract">
<h2><a class="toc-backref" href="#abstract" role="doc-backlink">Abstract</a></h2>
<p>This PEP proposes a way to protect Python code from being interrupted
inside a finally clause or during context manager cleanup.</p>
</section>
<section id="pep-deferral">
<h2><a class="toc-backref" href="#pep-deferral" role="doc-backlink">PEP Deferral</a></h2>
<p>Further exploration of the concepts covered in this PEP has been deferred
for lack of a current champion interested in promoting the goals of the PEP
and collecting and incorporating feedback, and with sufficient available
time to do so effectively.</p>
</section>
<section id="rationale">
<h2><a class="toc-backref" href="#rationale" role="doc-backlink">Rationale</a></h2>
<p>Python has two nice ways to do cleanup. One is a <code class="docutils literal notranslate"><span class="pre">finally</span></code>
statement and the other is a context manager (usually called using a
<code class="docutils literal notranslate"><span class="pre">with</span></code> statement). However, neither is protected from interruption
by <code class="docutils literal notranslate"><span class="pre">KeyboardInterrupt</span></code> or <code class="docutils literal notranslate"><span class="pre">GeneratorExit</span></code> caused by
<code class="docutils literal notranslate"><span class="pre">generator.throw()</span></code>. For example:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">lock</span><span class="o">.</span><span class="n">acquire</span><span class="p">()</span>
<span class="k">try</span><span class="p">:</span>
<span class="nb">print</span><span class="p">(</span><span class="s1">&#39;starting&#39;</span><span class="p">)</span>
<span class="n">do_something</span><span class="p">()</span>
<span class="k">finally</span><span class="p">:</span>
<span class="nb">print</span><span class="p">(</span><span class="s1">&#39;finished&#39;</span><span class="p">)</span>
<span class="n">lock</span><span class="o">.</span><span class="n">release</span><span class="p">()</span>
</pre></div>
</div>
<p>If <code class="docutils literal notranslate"><span class="pre">KeyboardInterrupt</span></code> occurs just after the second <code class="docutils literal notranslate"><span class="pre">print()</span></code>
call, the lock will not be released. Similarly, the following code
using the <code class="docutils literal notranslate"><span class="pre">with</span></code> statement is affected:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">threading</span> <span class="kn">import</span> <span class="n">Lock</span>
<span class="k">class</span> <span class="nc">MyLock</span><span class="p">:</span>
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_lock_impl</span> <span class="o">=</span> <span class="n">Lock</span><span class="p">()</span>
<span class="k">def</span> <span class="fm">__enter__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_lock_impl</span><span class="o">.</span><span class="n">acquire</span><span class="p">()</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">&quot;LOCKED&quot;</span><span class="p">)</span>
<span class="k">def</span> <span class="fm">__exit__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">&quot;UNLOCKING&quot;</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_lock_impl</span><span class="o">.</span><span class="n">release</span><span class="p">()</span>
<span class="n">lock</span> <span class="o">=</span> <span class="n">MyLock</span><span class="p">()</span>
<span class="k">with</span> <span class="n">lock</span><span class="p">:</span>
<span class="n">do_something</span>
</pre></div>
</div>
<p>If <code class="docutils literal notranslate"><span class="pre">KeyboardInterrupt</span></code> occurs near any of the <code class="docutils literal notranslate"><span class="pre">print()</span></code> calls, the
lock will never be released.</p>
<section id="coroutine-use-case">
<h3><a class="toc-backref" href="#coroutine-use-case" role="doc-backlink">Coroutine Use Case</a></h3>
<p>A similar case occurs with coroutines. Usually coroutine libraries
want to interrupt the coroutine with a timeout. The
<code class="docutils literal notranslate"><span class="pre">generator.throw()</span></code> method works for this use case, but there is no
way of knowing if the coroutine is currently suspended from inside a
<code class="docutils literal notranslate"><span class="pre">finally</span></code> clause.</p>
<p>An example that uses yield-based coroutines follows. The code looks
similar using any of the popular coroutine libraries Monocle <a class="footnote-reference brackets" href="#id4" id="id1">[1]</a>,
Bluelet <a class="footnote-reference brackets" href="#id5" id="id2">[2]</a>, or Twisted <a class="footnote-reference brackets" href="#id6" id="id3">[3]</a>.</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">run_locked</span><span class="p">():</span>
<span class="k">yield</span> <span class="n">connection</span><span class="o">.</span><span class="n">sendall</span><span class="p">(</span><span class="s1">&#39;LOCK&#39;</span><span class="p">)</span>
<span class="k">try</span><span class="p">:</span>
<span class="k">yield</span> <span class="n">do_something</span><span class="p">()</span>
<span class="k">yield</span> <span class="n">do_something_else</span><span class="p">()</span>
<span class="k">finally</span><span class="p">:</span>
<span class="k">yield</span> <span class="n">connection</span><span class="o">.</span><span class="n">sendall</span><span class="p">(</span><span class="s1">&#39;UNLOCK&#39;</span><span class="p">)</span>
<span class="k">with</span> <span class="n">timeout</span><span class="p">(</span><span class="mi">5</span><span class="p">):</span>
<span class="k">yield</span> <span class="n">run_locked</span><span class="p">()</span>
</pre></div>
</div>
<p>In the example above, <code class="docutils literal notranslate"><span class="pre">yield</span> <span class="pre">something</span></code> means to pause executing the
current coroutine and to execute coroutine <code class="docutils literal notranslate"><span class="pre">something</span></code> until it
finishes execution. Therefore, the coroutine library itself needs to
maintain a stack of generators. The <code class="docutils literal notranslate"><span class="pre">connection.sendall()</span></code> call waits
until the socket is writable and does a similar thing to what
<code class="docutils literal notranslate"><span class="pre">socket.sendall()</span></code> does.</p>
<p>The <code class="docutils literal notranslate"><span class="pre">with</span></code> statement ensures that all code is executed within 5
seconds timeout. It does so by registering a callback in the main
loop, which calls <code class="docutils literal notranslate"><span class="pre">generator.throw()</span></code> on the top-most frame in the
coroutine stack when a timeout happens.</p>
<p>The <code class="docutils literal notranslate"><span class="pre">greenlets</span></code> extension works in a similar way, except that it
doesnt need <code class="docutils literal notranslate"><span class="pre">yield</span></code> to enter a new stack frame. Otherwise
considerations are similar.</p>
</section>
</section>
<section id="specification">
<h2><a class="toc-backref" href="#specification" role="doc-backlink">Specification</a></h2>
<section id="frame-flag-f-in-cleanup">
<h3><a class="toc-backref" href="#frame-flag-f-in-cleanup" role="doc-backlink">Frame Flag f_in_cleanup</a></h3>
<p>A new flag on the frame object is proposed. It is set to <code class="docutils literal notranslate"><span class="pre">True</span></code> if
this frame is currently executing a <code class="docutils literal notranslate"><span class="pre">finally</span></code> clause. Internally,
the flag must be implemented as a counter of nested finally statements
currently being executed.</p>
<p>The internal counter also needs to be incremented during execution of
the <code class="docutils literal notranslate"><span class="pre">SETUP_WITH</span></code> and <code class="docutils literal notranslate"><span class="pre">WITH_CLEANUP</span></code> bytecodes, and decremented
when execution for these bytecodes is finished. This allows to also
protect <code class="docutils literal notranslate"><span class="pre">__enter__()</span></code> and <code class="docutils literal notranslate"><span class="pre">__exit__()</span></code> methods.</p>
</section>
<section id="function-sys-setcleanuphook">
<h3><a class="toc-backref" href="#function-sys-setcleanuphook" role="doc-backlink">Function sys.setcleanuphook</a></h3>
<p>A new function for the <code class="docutils literal notranslate"><span class="pre">sys</span></code> module is proposed. This function sets
a callback which is executed every time <code class="docutils literal notranslate"><span class="pre">f_in_cleanup</span></code> becomes
false. Callbacks get a frame object as their sole argument, so that
they can figure out where they are called from.</p>
<p>The setting is thread local and must be stored in the
<code class="docutils literal notranslate"><span class="pre">PyThreadState</span></code> structure.</p>
</section>
<section id="inspect-module-enhancements">
<h3><a class="toc-backref" href="#inspect-module-enhancements" role="doc-backlink">Inspect Module Enhancements</a></h3>
<p>Two new functions are proposed for the <code class="docutils literal notranslate"><span class="pre">inspect</span></code> module:
<code class="docutils literal notranslate"><span class="pre">isframeincleanup()</span></code> and <code class="docutils literal notranslate"><span class="pre">getcleanupframe()</span></code>.</p>
<p><code class="docutils literal notranslate"><span class="pre">isframeincleanup()</span></code>, given a frame or generator object as its sole
argument, returns the value of the <code class="docutils literal notranslate"><span class="pre">f_in_cleanup</span></code> attribute of a
frame itself or of the <code class="docutils literal notranslate"><span class="pre">gi_frame</span></code> attribute of a generator.</p>
<p><code class="docutils literal notranslate"><span class="pre">getcleanupframe()</span></code>, given a frame object as its sole argument,
returns the innermost frame which has a true value of
<code class="docutils literal notranslate"><span class="pre">f_in_cleanup</span></code>, or <code class="docutils literal notranslate"><span class="pre">None</span></code> if no frames in the stack have a nonzero
value for that attribute. It starts to inspect from the specified
frame and walks to outer frames using <code class="docutils literal notranslate"><span class="pre">f_back</span></code> pointers, just like
<code class="docutils literal notranslate"><span class="pre">getouterframes()</span></code> does.</p>
</section>
</section>
<section id="example">
<h2><a class="toc-backref" href="#example" role="doc-backlink">Example</a></h2>
<p>An example implementation of a SIGINT handler that interrupts safely
might look like:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">inspect</span><span class="o">,</span> <span class="nn">sys</span><span class="o">,</span> <span class="nn">functools</span>
<span class="k">def</span> <span class="nf">sigint_handler</span><span class="p">(</span><span class="n">sig</span><span class="p">,</span> <span class="n">frame</span><span class="p">):</span>
<span class="k">if</span> <span class="n">inspect</span><span class="o">.</span><span class="n">getcleanupframe</span><span class="p">(</span><span class="n">frame</span><span class="p">)</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
<span class="k">raise</span> <span class="ne">KeyboardInterrupt</span><span class="p">()</span>
<span class="n">sys</span><span class="o">.</span><span class="n">setcleanuphook</span><span class="p">(</span><span class="n">functools</span><span class="o">.</span><span class="n">partial</span><span class="p">(</span><span class="n">sigint_handler</span><span class="p">,</span> <span class="mi">0</span><span class="p">))</span>
</pre></div>
</div>
<p>A coroutine example is out of scope of this document, because its
implementation depends very much on a trampoline (or main loop) used
by coroutine library.</p>
</section>
<section id="unresolved-issues">
<h2><a class="toc-backref" href="#unresolved-issues" role="doc-backlink">Unresolved Issues</a></h2>
<section id="interruption-inside-with-statement-expression">
<h3><a class="toc-backref" href="#interruption-inside-with-statement-expression" role="doc-backlink">Interruption Inside With Statement Expression</a></h3>
<p>Given the statement</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">filename</span><span class="p">):</span>
<span class="n">do_something</span><span class="p">()</span>
</pre></div>
</div>
<p>Python can be interrupted after <code class="docutils literal notranslate"><span class="pre">open()</span></code> is called, but before the
<code class="docutils literal notranslate"><span class="pre">SETUP_WITH</span></code> bytecode is executed. There are two possible
decisions:</p>
<ul>
<li>Protect <code class="docutils literal notranslate"><span class="pre">with</span></code> expressions. This would require another bytecode,
since currently there is no way of recognizing the start of the
<code class="docutils literal notranslate"><span class="pre">with</span></code> expression.</li>
<li>Let the user write a wrapper if he considers it important for the
use-case. A safe wrapper might look like this:<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">FileWrapper</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">filename</span><span class="p">,</span> <span class="n">mode</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">filename</span> <span class="o">=</span> <span class="n">filename</span>
<span class="bp">self</span><span class="o">.</span><span class="n">mode</span> <span class="o">=</span> <span class="n">mode</span>
<span class="k">def</span> <span class="fm">__enter__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">file</span> <span class="o">=</span> <span class="nb">open</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">filename</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">mode</span><span class="p">)</span>
<span class="k">def</span> <span class="fm">__exit__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">file</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
</pre></div>
</div>
<p>Alternatively it can be written using the <code class="docutils literal notranslate"><span class="pre">contextmanager()</span></code>
decorator:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="nd">@contextmanager</span>
<span class="k">def</span> <span class="nf">open_wrapper</span><span class="p">(</span><span class="n">filename</span><span class="p">,</span> <span class="n">mode</span><span class="p">):</span>
<span class="n">file</span> <span class="o">=</span> <span class="nb">open</span><span class="p">(</span><span class="n">filename</span><span class="p">,</span> <span class="n">mode</span><span class="p">)</span>
<span class="k">try</span><span class="p">:</span>
<span class="k">yield</span> <span class="n">file</span>
<span class="k">finally</span><span class="p">:</span>
<span class="n">file</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
</pre></div>
</div>
<p>This code is safe, as the first part of the generator (before yield)
is executed inside the <code class="docutils literal notranslate"><span class="pre">SETUP_WITH</span></code> bytecode of the caller.</p>
</li>
</ul>
</section>
<section id="exception-propagation">
<h3><a class="toc-backref" href="#exception-propagation" role="doc-backlink">Exception Propagation</a></h3>
<p>Sometimes a <code class="docutils literal notranslate"><span class="pre">finally</span></code> clause or an <code class="docutils literal notranslate"><span class="pre">__enter__()</span></code>/<code class="docutils literal notranslate"><span class="pre">__exit__()</span></code>
method can raise an exception. Usually this is not a problem, since
more important exceptions like <code class="docutils literal notranslate"><span class="pre">KeyboardInterrupt</span></code> or <code class="docutils literal notranslate"><span class="pre">SystemExit</span></code>
should be raised instead. But it may be nice to be able to keep the
original exception inside a <code class="docutils literal notranslate"><span class="pre">__context__</span></code> attribute. So the cleanup
hook signature may grow an exception argument:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">sigint_handler</span><span class="p">(</span><span class="n">sig</span><span class="p">,</span> <span class="n">frame</span><span class="p">)</span>
<span class="k">if</span> <span class="n">inspect</span><span class="o">.</span><span class="n">getcleanupframe</span><span class="p">(</span><span class="n">frame</span><span class="p">)</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
<span class="k">raise</span> <span class="ne">KeyboardInterrupt</span><span class="p">()</span>
<span class="n">sys</span><span class="o">.</span><span class="n">setcleanuphook</span><span class="p">(</span><span class="n">retry_sigint</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">retry_sigint</span><span class="p">(</span><span class="n">frame</span><span class="p">,</span> <span class="n">exception</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
<span class="k">if</span> <span class="n">inspect</span><span class="o">.</span><span class="n">getcleanupframe</span><span class="p">(</span><span class="n">frame</span><span class="p">)</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
<span class="k">raise</span> <span class="ne">KeyboardInterrupt</span><span class="p">()</span> <span class="kn">from</span> <span class="nn">exception</span>
</pre></div>
</div>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>There is no need to have three arguments like in the <code class="docutils literal notranslate"><span class="pre">__exit__</span></code>
method since there is a <code class="docutils literal notranslate"><span class="pre">__traceback__</span></code> attribute in exception in
Python 3.</p>
</div>
<p>However, this will set the <code class="docutils literal notranslate"><span class="pre">__cause__</span></code> for the exception, which is
not exactly whats intended. So some hidden interpreter logic may be
used to put a <code class="docutils literal notranslate"><span class="pre">__context__</span></code> attribute on every exception raised in a
cleanup hook.</p>
</section>
<section id="interruption-between-acquiring-resource-and-try-block">
<h3><a class="toc-backref" href="#interruption-between-acquiring-resource-and-try-block" role="doc-backlink">Interruption Between Acquiring Resource and Try Block</a></h3>
<p>The example from the first section is not totally safe. Lets take a
closer look:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">lock</span><span class="o">.</span><span class="n">acquire</span><span class="p">()</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">do_something</span><span class="p">()</span>
<span class="k">finally</span><span class="p">:</span>
<span class="n">lock</span><span class="o">.</span><span class="n">release</span><span class="p">()</span>
</pre></div>
</div>
<p>The problem might occur if the code is interrupted just after
<code class="docutils literal notranslate"><span class="pre">lock.acquire()</span></code> is executed but before the <code class="docutils literal notranslate"><span class="pre">try</span></code> block is
entered.</p>
<p>There is no way the code can be fixed unmodified. The actual fix
depends very much on the use case. Usually code can be fixed using a
<code class="docutils literal notranslate"><span class="pre">with</span></code> statement:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="k">with</span> <span class="n">lock</span><span class="p">:</span>
<span class="n">do_something</span><span class="p">()</span>
</pre></div>
</div>
<p>However, for coroutines one usually cant use the <code class="docutils literal notranslate"><span class="pre">with</span></code> statement
because you need to <code class="docutils literal notranslate"><span class="pre">yield</span></code> for both the acquire and release
operations. So the code might be rewritten like this:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="k">try</span><span class="p">:</span>
<span class="k">yield</span> <span class="n">lock</span><span class="o">.</span><span class="n">acquire</span><span class="p">()</span>
<span class="n">do_something</span><span class="p">()</span>
<span class="k">finally</span><span class="p">:</span>
<span class="k">yield</span> <span class="n">lock</span><span class="o">.</span><span class="n">release</span><span class="p">()</span>
</pre></div>
</div>
<p>The actual locking code might need more code to support this use case,
but the implementation is usually trivial, like this: check if the
lock has been acquired and unlock if it is.</p>
</section>
<section id="handling-eintr-inside-a-finally">
<h3><a class="toc-backref" href="#handling-eintr-inside-a-finally" role="doc-backlink">Handling EINTR Inside a Finally</a></h3>
<p>Even if a signal handler is prepared to check the <code class="docutils literal notranslate"><span class="pre">f_in_cleanup</span></code>
flag, <code class="docutils literal notranslate"><span class="pre">InterruptedError</span></code> might be raised in the cleanup handler,
because the respective system call returned an <code class="docutils literal notranslate"><span class="pre">EINTR</span></code> error. The
primary use cases are prepared to handle this:</p>
<ul class="simple">
<li>Posix mutexes never return <code class="docutils literal notranslate"><span class="pre">EINTR</span></code></li>
<li>Networking libraries are always prepared to handle <code class="docutils literal notranslate"><span class="pre">EINTR</span></code></li>
<li>Coroutine libraries are usually interrupted with the <code class="docutils literal notranslate"><span class="pre">throw()</span></code>
method, not with a signal</li>
</ul>
<p>The platform-specific function <code class="docutils literal notranslate"><span class="pre">siginterrupt()</span></code> might be used to
remove the need to handle <code class="docutils literal notranslate"><span class="pre">EINTR</span></code>. However, it may have hardly
predictable consequences, for example <code class="docutils literal notranslate"><span class="pre">SIGINT</span></code> a handler is never
called if the main thread is stuck inside an IO routine.</p>
<p>A better approach would be to have the code, which is usually used in
cleanup handlers, be prepared to handle <code class="docutils literal notranslate"><span class="pre">InterruptedError</span></code>
explicitly. An example of such code might be a file-based lock
implementation.</p>
<p><code class="docutils literal notranslate"><span class="pre">signal.pthread_sigmask</span></code> can be used to block signals inside
cleanup handlers which can be interrupted with <code class="docutils literal notranslate"><span class="pre">EINTR</span></code>.</p>
</section>
<section id="setting-interruption-context-inside-finally-itself">
<h3><a class="toc-backref" href="#setting-interruption-context-inside-finally-itself" role="doc-backlink">Setting Interruption Context Inside Finally Itself</a></h3>
<p>Some coroutine libraries may need to set a timeout for the finally
clause itself. For example:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="k">try</span><span class="p">:</span>
<span class="n">do_something</span><span class="p">()</span>
<span class="k">finally</span><span class="p">:</span>
<span class="k">with</span> <span class="n">timeout</span><span class="p">(</span><span class="mf">0.5</span><span class="p">):</span>
<span class="k">try</span><span class="p">:</span>
<span class="k">yield</span> <span class="n">do_slow_cleanup</span><span class="p">()</span>
<span class="k">finally</span><span class="p">:</span>
<span class="k">yield</span> <span class="n">do_fast_cleanup</span><span class="p">()</span>
</pre></div>
</div>
<p>With current semantics, timeout will either protect the whole <code class="docutils literal notranslate"><span class="pre">with</span></code>
block or nothing at all, depending on the implementation of each
library. What the author intended is to treat <code class="docutils literal notranslate"><span class="pre">do_slow_cleanup</span></code> as
ordinary code, and <code class="docutils literal notranslate"><span class="pre">do_fast_cleanup</span></code> as a cleanup (a
non-interruptible one).</p>
<p>A similar case might occur when using greenlets or tasklets.</p>
<p>This case can be fixed by exposing <code class="docutils literal notranslate"><span class="pre">f_in_cleanup</span></code> as a counter, and
by calling a cleanup hook on each decrement. A coroutine library may
then remember the value at timeout start, and compare it on each hook
execution.</p>
<p>But in practice, the example is considered to be too obscure to take
into account.</p>
</section>
<section id="modifying-keyboardinterrupt">
<h3><a class="toc-backref" href="#modifying-keyboardinterrupt" role="doc-backlink">Modifying KeyboardInterrupt</a></h3>
<p>It should be decided if the default <code class="docutils literal notranslate"><span class="pre">SIGINT</span></code> handler should be
modified to use the described mechanism. The initial proposition is
to keep old behavior, for two reasons:</p>
<ul class="simple">
<li>Most application do not care about cleanup on exit (either they do
not have external state, or they modify it in crash-safe way).</li>
<li>Cleanup may take too much time, not giving user a chance to
interrupt an application.</li>
</ul>
<p>The latter case can be fixed by allowing an unsafe break if a
<code class="docutils literal notranslate"><span class="pre">SIGINT</span></code> handler is called twice, but it seems not worth the
complexity.</p>
</section>
</section>
<section id="alternative-python-implementations-support">
<h2><a class="toc-backref" href="#alternative-python-implementations-support" role="doc-backlink">Alternative Python Implementations Support</a></h2>
<p>We consider <code class="docutils literal notranslate"><span class="pre">f_in_cleanup</span></code> an implementation detail. The actual
implementation may have some fake frame-like object passed to signal
handler, cleanup hook and returned from <code class="docutils literal notranslate"><span class="pre">getcleanupframe()</span></code>. The
only requirement is that the <code class="docutils literal notranslate"><span class="pre">inspect</span></code> module functions work as
expected on these objects. For this reason, we also allow to pass a
generator object to the <code class="docutils literal notranslate"><span class="pre">isframeincleanup()</span></code> function, which removes
the need to use the <code class="docutils literal notranslate"><span class="pre">gi_frame</span></code> attribute.</p>
<p>It might be necessary to specify that <code class="docutils literal notranslate"><span class="pre">getcleanupframe()</span></code> must
return the same object that will be passed to cleanup hook at the next
invocation.</p>
</section>
<section id="alternative-names">
<h2><a class="toc-backref" href="#alternative-names" role="doc-backlink">Alternative Names</a></h2>
<p>The original proposal had a <code class="docutils literal notranslate"><span class="pre">f_in_finally</span></code> frame attribute, as the
original intention was to protect <code class="docutils literal notranslate"><span class="pre">finally</span></code> clauses. But as it grew
up to protecting <code class="docutils literal notranslate"><span class="pre">__enter__</span></code> and <code class="docutils literal notranslate"><span class="pre">__exit__</span></code> methods too, the
<code class="docutils literal notranslate"><span class="pre">f_in_cleanup</span></code> name seems better. Although the <code class="docutils literal notranslate"><span class="pre">__enter__</span></code> method
is not a cleanup routine, it at least relates to cleanup done by
context managers.</p>
<p><code class="docutils literal notranslate"><span class="pre">setcleanuphook</span></code>, <code class="docutils literal notranslate"><span class="pre">isframeincleanup</span></code> and <code class="docutils literal notranslate"><span class="pre">getcleanupframe</span></code> can
be unobscured to <code class="docutils literal notranslate"><span class="pre">set_cleanup_hook</span></code>, <code class="docutils literal notranslate"><span class="pre">is_frame_in_cleanup</span></code> and
<code class="docutils literal notranslate"><span class="pre">get_cleanup_frame</span></code>, although they follow the naming convention of
their respective modules.</p>
</section>
<section id="alternative-proposals">
<h2><a class="toc-backref" href="#alternative-proposals" role="doc-backlink">Alternative Proposals</a></h2>
<section id="propagating-f-in-cleanup-flag-automatically">
<h3><a class="toc-backref" href="#propagating-f-in-cleanup-flag-automatically" role="doc-backlink">Propagating f_in_cleanup Flag Automatically</a></h3>
<p>This can make <code class="docutils literal notranslate"><span class="pre">getcleanupframe()</span></code> unnecessary. But for yield-based
coroutines you need to propagate it yourself. Making it writable
leads to somewhat unpredictable behavior of <code class="docutils literal notranslate"><span class="pre">setcleanuphook()</span></code>.</p>
</section>
<section id="add-bytecodes-incr-cleanup-decr-cleanup">
<h3><a class="toc-backref" href="#add-bytecodes-incr-cleanup-decr-cleanup" role="doc-backlink">Add Bytecodes INCR_CLEANUP, DECR_CLEANUP</a></h3>
<p>These bytecodes can be used to protect the expression inside the
<code class="docutils literal notranslate"><span class="pre">with</span></code> statement, as well as making counter increments more explicit
and easy to debug (visible inside a disassembly). Some middle ground
might be chosen, like <code class="docutils literal notranslate"><span class="pre">END_FINALLY</span></code> and <code class="docutils literal notranslate"><span class="pre">SETUP_WITH</span></code> implicitly
decrementing the counter (<code class="docutils literal notranslate"><span class="pre">END_FINALLY</span></code> is present at end of every
<code class="docutils literal notranslate"><span class="pre">with</span></code> suite).</p>
<p>However, adding new bytecodes must be considered very carefully.</p>
</section>
<section id="expose-f-in-cleanup-as-a-counter">
<h3><a class="toc-backref" href="#expose-f-in-cleanup-as-a-counter" role="doc-backlink">Expose f_in_cleanup as a Counter</a></h3>
<p>The original intention was to expose a minimum of needed
functionality. However, as we consider the frame flag
<code class="docutils literal notranslate"><span class="pre">f_in_cleanup</span></code> an implementation detail, we may expose it as a
counter.</p>
<p>Similarly, if we have a counter we may need to have the cleanup hook
called on every counter decrement. Its unlikely to have much
performance impact as nested finally clauses are an uncommon case.</p>
</section>
<section id="add-code-object-flag-co-cleanup">
<h3><a class="toc-backref" href="#add-code-object-flag-co-cleanup" role="doc-backlink">Add code object flag CO_CLEANUP</a></h3>
<p>As an alternative to set the flag inside the <code class="docutils literal notranslate"><span class="pre">SETUP_WITH</span></code> and
<code class="docutils literal notranslate"><span class="pre">WITH_CLEANUP</span></code> bytecodes, we can introduce a flag <code class="docutils literal notranslate"><span class="pre">CO_CLEANUP</span></code>.
When the interpreter starts to execute code with <code class="docutils literal notranslate"><span class="pre">CO_CLEANUP</span></code> set,
it sets <code class="docutils literal notranslate"><span class="pre">f_in_cleanup</span></code> for the whole function body. This flag is
set for code objects of <code class="docutils literal notranslate"><span class="pre">__enter__</span></code> and <code class="docutils literal notranslate"><span class="pre">__exit__</span></code> special
methods. Technically it might be set on functions called
<code class="docutils literal notranslate"><span class="pre">__enter__</span></code> and <code class="docutils literal notranslate"><span class="pre">__exit__</span></code>.</p>
<p>This seems to be less clear solution. It also covers the case where
<code class="docutils literal notranslate"><span class="pre">__enter__</span></code> and <code class="docutils literal notranslate"><span class="pre">__exit__</span></code> are called manually. This may be
accepted either as a feature or as an unnecessary side-effect (or,
though unlikely, as a bug).</p>
<p>It may also impose a problem when <code class="docutils literal notranslate"><span class="pre">__enter__</span></code> or <code class="docutils literal notranslate"><span class="pre">__exit__</span></code>
functions are implemented in C, as there is no code object to check
for the <code class="docutils literal notranslate"><span class="pre">f_in_cleanup</span></code> flag.</p>
</section>
<section id="have-cleanup-callback-on-frame-object-itself">
<h3><a class="toc-backref" href="#have-cleanup-callback-on-frame-object-itself" role="doc-backlink">Have Cleanup Callback on Frame Object Itself</a></h3>
<p>The frame object may be extended to have a <code class="docutils literal notranslate"><span class="pre">f_cleanup_callback</span></code>
member which is called when <code class="docutils literal notranslate"><span class="pre">f_in_cleanup</span></code> is reset to 0. This
would help to register different callbacks to different coroutines.</p>
<p>Despite its apparent beauty, this solution doesnt add anything, as
the two primary use cases are:</p>
<ul class="simple">
<li>Setting the callback in a signal handler. The callback is
inherently a single one for this case.</li>
<li>Use a single callback per loop for the coroutine use case. Here, in
almost all cases, there is only one loop per thread.</li>
</ul>
</section>
<section id="no-cleanup-hook">
<h3><a class="toc-backref" href="#no-cleanup-hook" role="doc-backlink">No Cleanup Hook</a></h3>
<p>The original proposal included no cleanup hook specification, as there
are a few ways to achieve the same using current tools:</p>
<ul class="simple">
<li>Using <code class="docutils literal notranslate"><span class="pre">sys.settrace()</span></code> and the <code class="docutils literal notranslate"><span class="pre">f_trace</span></code> callback. This may
impose some problem to debugging, and has a big performance impact
(although interrupting doesnt happen very often).</li>
<li>Sleeping a bit more and trying again. For a coroutine library this
is easy. For signals it may be achieved using <code class="docutils literal notranslate"><span class="pre">signal.alert</span></code>.</li>
</ul>
<p>Both methods are considered too impractical and a way to catch exit
from <code class="docutils literal notranslate"><span class="pre">finally</span></code> clauses is proposed.</p>
</section>
</section>
<section id="references">
<h2><a class="toc-backref" href="#references" role="doc-backlink">References</a></h2>
<aside class="footnote-list brackets">
<aside class="footnote brackets" id="id4" role="doc-footnote">
<dt class="label" id="id4">[<a href="#id1">1</a>]</dt>
<dd>Monocle
<a class="reference external" href="https://github.com/saucelabs/monocle">https://github.com/saucelabs/monocle</a></aside>
<aside class="footnote brackets" id="id5" role="doc-footnote">
<dt class="label" id="id5">[<a href="#id2">2</a>]</dt>
<dd>Bluelet
<a class="reference external" href="https://github.com/sampsyo/bluelet">https://github.com/sampsyo/bluelet</a></aside>
<aside class="footnote brackets" id="id6" role="doc-footnote">
<dt class="label" id="id6">[<a href="#id3">3</a>]</dt>
<dd>Twisted: inlineCallbacks
<a class="reference external" href="https://twisted.org/documents/8.1.0/api/twisted.internet.defer.html">https://twisted.org/documents/8.1.0/api/twisted.internet.defer.html</a></aside>
</aside>
<p>[4] Original discussion
<a class="reference external" href="https://mail.python.org/pipermail/python-ideas/2012-April/014705.html">https://mail.python.org/pipermail/python-ideas/2012-April/014705.html</a></p>
<p>[5] Implementation of PEP 419
<a class="reference external" href="https://github.com/python/cpython/issues/58935">https://github.com/python/cpython/issues/58935</a></p>
</section>
<section id="copyright">
<h2><a class="toc-backref" href="#copyright" role="doc-backlink">Copyright</a></h2>
<p>This document has been placed in the public domain.</p>
</section>
</section>
<hr class="docutils" />
<p>Source: <a class="reference external" href="https://github.com/python/peps/blob/main/peps/pep-0419.rst">https://github.com/python/peps/blob/main/peps/pep-0419.rst</a></p>
<p>Last modified: <a class="reference external" href="https://github.com/python/peps/commits/main/peps/pep-0419.rst">2023-09-09 17:39:29 GMT</a></p>
</article>
<nav id="pep-sidebar">
<h2>Contents</h2>
<ul>
<li><a class="reference internal" href="#abstract">Abstract</a></li>
<li><a class="reference internal" href="#pep-deferral">PEP Deferral</a></li>
<li><a class="reference internal" href="#rationale">Rationale</a><ul>
<li><a class="reference internal" href="#coroutine-use-case">Coroutine Use Case</a></li>
</ul>
</li>
<li><a class="reference internal" href="#specification">Specification</a><ul>
<li><a class="reference internal" href="#frame-flag-f-in-cleanup">Frame Flag f_in_cleanup</a></li>
<li><a class="reference internal" href="#function-sys-setcleanuphook">Function sys.setcleanuphook</a></li>
<li><a class="reference internal" href="#inspect-module-enhancements">Inspect Module Enhancements</a></li>
</ul>
</li>
<li><a class="reference internal" href="#example">Example</a></li>
<li><a class="reference internal" href="#unresolved-issues">Unresolved Issues</a><ul>
<li><a class="reference internal" href="#interruption-inside-with-statement-expression">Interruption Inside With Statement Expression</a></li>
<li><a class="reference internal" href="#exception-propagation">Exception Propagation</a></li>
<li><a class="reference internal" href="#interruption-between-acquiring-resource-and-try-block">Interruption Between Acquiring Resource and Try Block</a></li>
<li><a class="reference internal" href="#handling-eintr-inside-a-finally">Handling EINTR Inside a Finally</a></li>
<li><a class="reference internal" href="#setting-interruption-context-inside-finally-itself">Setting Interruption Context Inside Finally Itself</a></li>
<li><a class="reference internal" href="#modifying-keyboardinterrupt">Modifying KeyboardInterrupt</a></li>
</ul>
</li>
<li><a class="reference internal" href="#alternative-python-implementations-support">Alternative Python Implementations Support</a></li>
<li><a class="reference internal" href="#alternative-names">Alternative Names</a></li>
<li><a class="reference internal" href="#alternative-proposals">Alternative Proposals</a><ul>
<li><a class="reference internal" href="#propagating-f-in-cleanup-flag-automatically">Propagating f_in_cleanup Flag Automatically</a></li>
<li><a class="reference internal" href="#add-bytecodes-incr-cleanup-decr-cleanup">Add Bytecodes INCR_CLEANUP, DECR_CLEANUP</a></li>
<li><a class="reference internal" href="#expose-f-in-cleanup-as-a-counter">Expose f_in_cleanup as a Counter</a></li>
<li><a class="reference internal" href="#add-code-object-flag-co-cleanup">Add code object flag CO_CLEANUP</a></li>
<li><a class="reference internal" href="#have-cleanup-callback-on-frame-object-itself">Have Cleanup Callback on Frame Object Itself</a></li>
<li><a class="reference internal" href="#no-cleanup-hook">No Cleanup Hook</a></li>
</ul>
</li>
<li><a class="reference internal" href="#references">References</a></li>
<li><a class="reference internal" href="#copyright">Copyright</a></li>
</ul>
<br>
<a id="source" href="https://github.com/python/peps/blob/main/peps/pep-0419.rst">Page Source (GitHub)</a>
</nav>
</section>
<script src="../_static/colour_scheme.js"></script>
<script src="../_static/wrap_tables.js"></script>
<script src="../_static/sticky_banner.js"></script>
</body>
</html>