First version of the PEP 475
This commit is contained in:
Victor Stinner
parent
5130862086
commit
b09ad926df
|
|
@ -0,0 +1,413 @@
|
|||
PEP: 475
|
||||
Title: Retry system calls failing with EINTR
|
||||
Version: $Revision$
|
||||
Last-Modified: $Date$
|
||||
Author: Charles-François Natali <cf.natali@gmail.com>, Victor Stinner <victor.stinner@gmail.com>
|
||||
Status: Draft
|
||||
Type: Standards Track
|
||||
Content-Type: text/x-rst
|
||||
Created: 29-July-2014
|
||||
Python-Version: 3.5
|
||||
|
||||
|
||||
Abstract
|
||||
========
|
||||
|
||||
Retry system calls failing with the ``EINTR`` error and recompute
|
||||
timeout if needed. Deprecate the ``signal.siginterrupt()`` function.
|
||||
|
||||
|
||||
Rationale
|
||||
=========
|
||||
|
||||
Interrupted system calls
|
||||
------------------------
|
||||
|
||||
On POSIX systems, signals are common. Your program must be prepared to
|
||||
handle them. Examples of signals:
|
||||
|
||||
* The most common signal is ``SIGINT``, signal sent when CTRL+c is
|
||||
pressed. By default, Python raises a ``KeyboardInterrupt`` exception
|
||||
when this signal is received.
|
||||
* When running subprocesses, the ``SIGCHLD`` signal is sent when a
|
||||
child process exits.
|
||||
* Resizing the terminal sends the ``SIGWINCH`` signal to the
|
||||
applications running in the terminal.
|
||||
* Putting the application in background (ex: press CTRL-z and then
|
||||
type the ``bg`` command) sends the ``SIGCONT`` signal.
|
||||
|
||||
Writing a signal handler is difficult, only "signal-safe" functions
|
||||
can be called. For example, ``printf()`` and ``malloc()`` are not
|
||||
signal-safe. When a signal is sent to a process calling a system
|
||||
call, the system call fails with the ``EINTR`` error to give the
|
||||
program an opportunity to handle the signal without the restriction on
|
||||
signal safe functions.
|
||||
|
||||
If the signal handler was set with the ``SA_RESTART`` flag set, the C
|
||||
library retries some the system call instead of failing with
|
||||
``EINTR``. For example, ``read()`` is retried, whereas ``select()`` is
|
||||
not retried. The Python function ``signal.signal()`` clears the
|
||||
``SA_RESTART`` flag when setting the signal handler: all system calls
|
||||
should fail with ``EINTR`` in Python.
|
||||
|
||||
The problem is that handling ``EINTR`` should be done for all system
|
||||
calls. The problem is similar to handling errors in the C language
|
||||
which does not have exceptions: you must check all function returns to
|
||||
check for error, and usually duplicate the code checking for errors.
|
||||
Python does not have this issue, it uses exceptions to notify errors.
|
||||
|
||||
|
||||
Current status
|
||||
--------------
|
||||
|
||||
Currently in Python, the code to handle the ``InterruptedError``
|
||||
exception (``EINTR`` error) is duplicated on case by case. Only a few
|
||||
Python modules handle this exception, and fixes usually took several
|
||||
years to cover a whole module. Example of code retrying
|
||||
``file.read()`` on ``InterruptedError``::
|
||||
|
||||
while True:
|
||||
try:
|
||||
data = file.read()
|
||||
break
|
||||
except InterruptedError:
|
||||
continue
|
||||
|
||||
List of Python modules of the standard library which handle
|
||||
``InterruptedError``:
|
||||
|
||||
* ``asyncio``
|
||||
* ``asyncore``
|
||||
* ``io``, ``_pyio``
|
||||
* ``multiprocessing``
|
||||
* ``selectors``
|
||||
* ``socket``
|
||||
* ``socketserver``
|
||||
* ``subprocess``
|
||||
|
||||
|
||||
Use Case 1: Don't Bother of Signals
|
||||
-----------------------------------
|
||||
|
||||
In most cases, you don't want to be interrupted by signals and you
|
||||
don't expect to get ``InterruptedError`` exceptions. For example, do
|
||||
you really want to write such complex code for an "Hello World"
|
||||
example?
|
||||
|
||||
::
|
||||
|
||||
while True:
|
||||
try:
|
||||
print("Hello World")
|
||||
except InterruptedError:
|
||||
pass
|
||||
|
||||
``InterruptedError`` can happen in unexpected places. For example,
|
||||
``os.close()`` and ``FileIO.close()`` can raises ``InterruptedError``:
|
||||
see the article `close() and EINTR
|
||||
<http://alobbs.com/post/54503240599/close-and-eintr>`_.
|
||||
|
||||
The `Python issues related to EINTR`_ section below gives examples of
|
||||
bugs caused by "EINTR".
|
||||
|
||||
The expectation is that Python hides the ``InterruptedError``: retry
|
||||
system calls failing with the ``EINTR`` error.
|
||||
|
||||
|
||||
Use Case 2: Be notified of signals as soon as possible
|
||||
------------------------------------------------------
|
||||
|
||||
Sometimes, you expect some signals and you want to handle them as soon
|
||||
as possible. For example, you may want to quit immediatly a program
|
||||
using the ``CTRL+c`` keyboard shortcut.
|
||||
|
||||
Some signals are not interesting and should not interrupt the the
|
||||
application. There are two options to only interrupt an application
|
||||
on some signals:
|
||||
|
||||
* Raise an exception in the signal handler, like ``KeyboardInterrupt`` for
|
||||
``SIGINT``
|
||||
* Use a I/O multiplexing function like ``select()`` with the Python
|
||||
signal "wakeup" file descriptor: see the function
|
||||
``signal.set_wakeupfd()``.
|
||||
|
||||
|
||||
Proposition
|
||||
===========
|
||||
|
||||
If a system call fails with ``EINTR``, Python must call signal
|
||||
handlers: call ``PyErr_CheckSignals()``. If a signal handler raises
|
||||
an exception, the Python function fails with the exception.
|
||||
Otherwise, the system call is retried. If the system call takes a
|
||||
timeout parameter, the timeout is recomputed.
|
||||
|
||||
Modified functions
|
||||
------------------
|
||||
|
||||
Example of functions that need to be modified:
|
||||
|
||||
* ``os.read()``, ``io.FileIO.read()``, ``io.FileIO.readinto()``
|
||||
* ``os.write()``, ``io.FileIO.write()``
|
||||
* ``os.waitpid()``
|
||||
* ``socket.accept()``
|
||||
* ``socket.connect()``
|
||||
* ``socket.recv()``, ``socket.recv_into()``
|
||||
* ``socket.recv_from()``
|
||||
* ``socket.send()``
|
||||
* ``socket.sendto()``
|
||||
* ``time.sleep()``
|
||||
* ``select.select()``
|
||||
* ``select.poll()``
|
||||
* ``select.epoll.poll()``
|
||||
* ``select.devpoll.poll()``
|
||||
* ``select.kqueue.control()``
|
||||
* ``selectors.SelectSelector.select()`` and other selector classes
|
||||
|
||||
Note: The ``selector`` module already retries on ``InterruptedError``, but it
|
||||
doesn't recompute the timeout yet.
|
||||
|
||||
|
||||
Deprecate siginterrupt()
|
||||
------------------------
|
||||
|
||||
The function ``signal.siginterrupt()`` becomes useless with this PEP,
|
||||
it should be deprecated. When ``signal.siginterrupt(signum, False)``
|
||||
is used, some system calls don't fail with ``EINTR`` when a signal is
|
||||
received. Python cannot call its signal handler and interrupt the
|
||||
system call.
|
||||
|
||||
The function ``signal.siginterrupt()`` will be deprecated in Python
|
||||
3.5.
|
||||
|
||||
In Python 3.6, calling ``signal.siginterrupt(signum, False)`` will
|
||||
raise an exception, whereas ``signal.siginterrupt(signum, True)`` will
|
||||
only emit the deprecation warning.
|
||||
|
||||
|
||||
Backward Compatibility
|
||||
======================
|
||||
|
||||
Applications relying on the fact that system calls are interrupted
|
||||
with ``InterruptedError`` will hang. The authors of this PEP don't
|
||||
think that such application exist.
|
||||
|
||||
If such applications exist, they must be fixed to handle signals
|
||||
differently, to have a reliable behaviour on all platforms and all
|
||||
Python versions. For example, use a signal handle which raises an
|
||||
exception, or use a wakeup file descriptor.
|
||||
|
||||
Applications should not call ``signal.siginterrupt(signum, False)``
|
||||
anymore, since this call will raise an exception in Python 3.6.
|
||||
|
||||
For applications using event loops, ``signal.set_wakeup_fd()`` is the
|
||||
recommanded option to handle signals. The signal handler writes signal
|
||||
numbers into the file descriptor and the event loop is awaken to read
|
||||
them. The event loop can decide how to handle these signals without
|
||||
the restriction of signal handlers.
|
||||
|
||||
|
||||
Appendix
|
||||
========
|
||||
|
||||
Wakeup file descriptor
|
||||
----------------------
|
||||
|
||||
Since Python 3.3, ``signal.set_wakeup_fd()`` writes the signal number
|
||||
into the file descriptor, whereas it only wrote a null byte before.
|
||||
It becomes possible to handle different signals using the wakeup file
|
||||
descriptor.
|
||||
|
||||
Linux has a ``signalfd()`` which provides more information on each
|
||||
signal. For example, it's possible to know the pid and uid who sent
|
||||
the signal. This function is not exposed in Python yet (see the
|
||||
`issue 12304 <http://bugs.python.org/issue12304>`_).
|
||||
|
||||
On Unix, the ``asyncio`` module uses the wakeup file descriptor to
|
||||
wake up its event loop.
|
||||
|
||||
|
||||
Multithreading
|
||||
--------------
|
||||
|
||||
A C signal handler can be called from any thread, but the Python
|
||||
signal handler should only be called in the main thread.
|
||||
|
||||
Python has a ``PyErr_SetInterrupt()`` function which calls the
|
||||
``SIGINT`` signal handler to interrupt the Python main thread.
|
||||
|
||||
|
||||
Signals on Windows
|
||||
------------------
|
||||
|
||||
Control events
|
||||
^^^^^^^^^^^^^^
|
||||
|
||||
Windows uses "control events":
|
||||
|
||||
* ``CTRL_BREAK_EVENT``: Break (``SIGBREAK``)
|
||||
* ``CTRL_CLOSE_EVENT``: Close event
|
||||
* ``CTRL_C_EVENT``: CTRL+C (``SIGINT``)
|
||||
* ``CTRL_LOGOFF_EVENT``: Logoff
|
||||
* ``CTRL_SHUTDOWN_EVENT``: Shutdown
|
||||
|
||||
The `SetConsoleCtrlHandler() function
|
||||
<http://msdn.microsoft.com/en-us/library/windows/desktop/ms686016%28v=vs.85%29.aspx>`_
|
||||
can be used to install a control handler.
|
||||
|
||||
The ``CTRL_C_EVENT`` and ``CTRL_BREAK_EVENT`` events can be sent to a
|
||||
process using the `GenerateConsoleCtrlEvent() function
|
||||
<http://msdn.microsoft.com/en-us/library/windows/desktop/ms683155%28v=vs.85%29.aspx>`_.
|
||||
This function is exposed in Python as ``os.kill()``.
|
||||
|
||||
|
||||
Signals
|
||||
^^^^^^^
|
||||
|
||||
The following signals are supported on Windows:
|
||||
|
||||
* ``SIGABRT``
|
||||
* ``SIGBREAK`` (``CTRL_BREAK_EVENT``): signal only available on Windows
|
||||
* ``SIGFPE``
|
||||
* ``SIGILL``
|
||||
* ``SIGINT`` (``CTRL_C_EVENT``)
|
||||
* ``SIGSEGV``
|
||||
* ``SIGTERM``
|
||||
|
||||
|
||||
SIGINT
|
||||
^^^^^^
|
||||
|
||||
The default Python signal handler for ``SIGINT`` sets a Windows event
|
||||
object: ``sigint_event``.
|
||||
|
||||
``time.sleep()`` is implemented with ``WaitForSingleObjectEx()``, it
|
||||
waits for the ``sigint_event`` object using ``time.sleep()`` parameter
|
||||
as the timeout. So the sleep can be interrupted by ``SIGINT``.
|
||||
|
||||
``_winapi.WaitForMultipleObjects()`` automatically adds
|
||||
``sigint_event`` to the list of watched handles, so it can also be
|
||||
interrupted.
|
||||
|
||||
``PyOS_StdioReadline()`` also used ``sigint_event`` when ``fgets()``
|
||||
failed to check if Ctrl-C or Ctrl-Z was pressed.
|
||||
|
||||
|
||||
Links
|
||||
-----
|
||||
|
||||
Misc
|
||||
^^^^
|
||||
|
||||
* `glibc manual: Primitives Interrupted by Signals
|
||||
<http://www.gnu.org/software/libc/manual/html_node/Interrupted-Primitives.html>`_
|
||||
* `Bug #119097 for perl5: print returning EINTR in 5.14
|
||||
<https://rt.perl.org/Public/Bug/Display.html?id=119097>`_.
|
||||
|
||||
|
||||
Python issues related to EINTR
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
The main issue is: `handle EINTR in the stdlib
|
||||
<http://bugs.python.org/issue18885>`_.
|
||||
|
||||
Open issues:
|
||||
|
||||
* `Add a new signal.set_wakeup_socket() function
|
||||
<http://bugs.python.org/issue22018>`_
|
||||
* `signal.set_wakeup_fd(fd): set the fd to non-blocking mode
|
||||
<http://bugs.python.org/issue22042>`_
|
||||
* `Use a monotonic clock to compute timeouts
|
||||
<http://bugs.python.org/issue22043>`_
|
||||
* `sys.stdout.write on OS X is not EINTR safe
|
||||
<http://bugs.python.org/issue22007>`_
|
||||
* `platform.uname() not EINTR safe
|
||||
<http://bugs.python.org/issue21772>`_
|
||||
* `asyncore does not handle EINTR in recv, send, connect, accept,
|
||||
<http://bugs.python.org/issue11266>`_
|
||||
* `socket.create_connection() doesn't handle EINTR properly
|
||||
<http://bugs.python.org/issue20611>`_
|
||||
|
||||
Closed issues:
|
||||
|
||||
* `Interrupted system calls are not retried
|
||||
<http://bugs.python.org/issue9867>`_
|
||||
* `Solaris: EINTR exception in select/socket calls in telnetlib
|
||||
<http://bugs.python.org/issue1049450>`_
|
||||
* `subprocess: Popen.communicate() doesn't handle EINTR in some cases
|
||||
<http://bugs.python.org/issue12493>`_
|
||||
* `multiprocessing.util._eintr_retry doen't recalculate timeouts
|
||||
<http://bugs.python.org/issue12338>`_
|
||||
* `file readline, readlines & readall methods can lose data on EINTR
|
||||
<http://bugs.python.org/issue12268>`_
|
||||
* `multiprocessing BaseManager serve_client() does not check EINTR on recv
|
||||
<http://bugs.python.org/issue17097>`_
|
||||
* `selectors behaviour on EINTR undocumented
|
||||
<http://bugs.python.org/issue19849>`_
|
||||
* `asyncio: limit EINTR occurrences with SA_RESTART
|
||||
<http://bugs.python.org/issue19850>`_
|
||||
* `smtplib.py socket.create_connection() also doesn't handle EINTR properly
|
||||
<http://bugs.python.org/issue21602>`_
|
||||
* `Faulty RESTART/EINTR handling in Parser/myreadline.c
|
||||
<http://bugs.python.org/issue11650>`_
|
||||
* `test_httpservers intermittent failure, test_post and EINTR
|
||||
<http://bugs.python.org/issue3771>`_
|
||||
* `os.spawnv(P_WAIT, ...) on Linux doesn't handle EINTR
|
||||
<http://bugs.python.org/issue686667>`_
|
||||
* `asyncore fails when EINTR happens in pol
|
||||
<http://bugs.python.org/issue517554>`_
|
||||
* `file.write and file.read don't handle EINTR
|
||||
<http://bugs.python.org/issue10956>`_
|
||||
* `socket.readline() interface doesn't handle EINTR properly
|
||||
<http://bugs.python.org/issue1628205>`_
|
||||
* `subprocess is not EINTR-safe
|
||||
<http://bugs.python.org/issue1068268>`_
|
||||
* `SocketServer doesn't handle syscall interruption
|
||||
<http://bugs.python.org/issue7978>`_
|
||||
* `subprocess deadlock when read() is interrupted
|
||||
<http://bugs.python.org/issue17367>`_
|
||||
* `time.sleep(1): call PyErr_CheckSignals() if the sleep was interrupted
|
||||
<http://bugs.python.org/issue12462>`_
|
||||
* `siginterrupt with flag=False is reset when signal received
|
||||
<http://bugs.python.org/issue8354>`_
|
||||
* `need siginterrupt() on Linux - impossible to do timeouts
|
||||
<http://bugs.python.org/issue1089358>`_
|
||||
* `[Windows] Can not interrupt time.sleep()
|
||||
<http://bugs.python.org/issue581232>`_
|
||||
|
||||
Python issues related to signals
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
Open issues:
|
||||
|
||||
* `expose signalfd(2) in the signal module
|
||||
<http://bugs.python.org/issue12304>`_
|
||||
* `missing return in win32_kill?
|
||||
<http://bugs.python.org/issue14484>`_
|
||||
* `Interrupts are lost during readline PyOS_InputHook processing
|
||||
<http://bugs.python.org/issue3180>`_
|
||||
* `cannot catch KeyboardInterrupt when using curses getkey()
|
||||
<http://bugs.python.org/issue1687125>`_
|
||||
* `Deferred KeyboardInterrupt in interactive mode
|
||||
<http://bugs.python.org/issue16151>`_
|
||||
|
||||
Closed issues:
|
||||
|
||||
* `sys.interrupt_main()
|
||||
<http://bugs.python.org/issue753733>`_
|
||||
|
||||
|
||||
Copyright
|
||||
=========
|
||||
|
||||
This document has been placed in the public domain.
|
||||
|
||||
|
||||
|
||||
..
|
||||
Local Variables:
|
||||
mode: indented-text
|
||||
indent-tabs-mode: nil
|
||||
sentence-end-double-space: t
|
||||
fill-column: 70
|
||||
coding: utf-8
|
||||
End:
|
||||
Loading…
Reference in New Issue