From b09ad926df03ee8953aa9e2312dafa7e7d716201 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Thu, 24 Jul 2014 13:37:56 +0200 Subject: [PATCH] First version of the PEP 475 --- pep-0475.txt | 413 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 413 insertions(+) create mode 100644 pep-0475.txt diff --git a/pep-0475.txt b/pep-0475.txt new file mode 100644 index 000000000..01ef3c6d6 --- /dev/null +++ b/pep-0475.txt @@ -0,0 +1,413 @@ +PEP: 475 +Title: Retry system calls failing with EINTR +Version: $Revision$ +Last-Modified: $Date$ +Author: Charles-François Natali , Victor Stinner +Status: Draft +Type: Standards Track +Content-Type: text/x-rst +Created: 29-July-2014 +Python-Version: 3.5 + + +Abstract +======== + +Retry system calls failing with the ``EINTR`` error and recompute +timeout if needed. Deprecate the ``signal.siginterrupt()`` function. + + +Rationale +========= + +Interrupted system calls +------------------------ + +On POSIX systems, signals are common. Your program must be prepared to +handle them. Examples of signals: + +* The most common signal is ``SIGINT``, signal sent when CTRL+c is + pressed. By default, Python raises a ``KeyboardInterrupt`` exception + when this signal is received. +* When running subprocesses, the ``SIGCHLD`` signal is sent when a + child process exits. +* Resizing the terminal sends the ``SIGWINCH`` signal to the + applications running in the terminal. +* Putting the application in background (ex: press CTRL-z and then + type the ``bg`` command) sends the ``SIGCONT`` signal. + +Writing a signal handler is difficult, only "signal-safe" functions +can be called. For example, ``printf()`` and ``malloc()`` are not +signal-safe. When a signal is sent to a process calling a system +call, the system call fails with the ``EINTR`` error to give the +program an opportunity to handle the signal without the restriction on +signal safe functions. + +If the signal handler was set with the ``SA_RESTART`` flag set, the C +library retries some the system call instead of failing with +``EINTR``. For example, ``read()`` is retried, whereas ``select()`` is +not retried. The Python function ``signal.signal()`` clears the +``SA_RESTART`` flag when setting the signal handler: all system calls +should fail with ``EINTR`` in Python. + +The problem is that handling ``EINTR`` should be done for all system +calls. The problem is similar to handling errors in the C language +which does not have exceptions: you must check all function returns to +check for error, and usually duplicate the code checking for errors. +Python does not have this issue, it uses exceptions to notify errors. + + +Current status +-------------- + +Currently in Python, the code to handle the ``InterruptedError`` +exception (``EINTR`` error) is duplicated on case by case. Only a few +Python modules handle this exception, and fixes usually took several +years to cover a whole module. Example of code retrying +``file.read()`` on ``InterruptedError``:: + + while True: + try: + data = file.read() + break + except InterruptedError: + continue + +List of Python modules of the standard library which handle +``InterruptedError``: + +* ``asyncio`` +* ``asyncore`` +* ``io``, ``_pyio`` +* ``multiprocessing`` +* ``selectors`` +* ``socket`` +* ``socketserver`` +* ``subprocess`` + + +Use Case 1: Don't Bother of Signals +----------------------------------- + +In most cases, you don't want to be interrupted by signals and you +don't expect to get ``InterruptedError`` exceptions. For example, do +you really want to write such complex code for an "Hello World" +example? + +:: + + while True: + try: + print("Hello World") + except InterruptedError: + pass + +``InterruptedError`` can happen in unexpected places. For example, +``os.close()`` and ``FileIO.close()`` can raises ``InterruptedError``: +see the article `close() and EINTR +`_. + +The `Python issues related to EINTR`_ section below gives examples of +bugs caused by "EINTR". + +The expectation is that Python hides the ``InterruptedError``: retry +system calls failing with the ``EINTR`` error. + + +Use Case 2: Be notified of signals as soon as possible +------------------------------------------------------ + +Sometimes, you expect some signals and you want to handle them as soon +as possible. For example, you may want to quit immediatly a program +using the ``CTRL+c`` keyboard shortcut. + +Some signals are not interesting and should not interrupt the the +application. There are two options to only interrupt an application +on some signals: + +* Raise an exception in the signal handler, like ``KeyboardInterrupt`` for + ``SIGINT`` +* Use a I/O multiplexing function like ``select()`` with the Python + signal "wakeup" file descriptor: see the function + ``signal.set_wakeupfd()``. + + +Proposition +=========== + +If a system call fails with ``EINTR``, Python must call signal +handlers: call ``PyErr_CheckSignals()``. If a signal handler raises +an exception, the Python function fails with the exception. +Otherwise, the system call is retried. If the system call takes a +timeout parameter, the timeout is recomputed. + +Modified functions +------------------ + +Example of functions that need to be modified: + +* ``os.read()``, ``io.FileIO.read()``, ``io.FileIO.readinto()`` +* ``os.write()``, ``io.FileIO.write()`` +* ``os.waitpid()`` +* ``socket.accept()`` +* ``socket.connect()`` +* ``socket.recv()``, ``socket.recv_into()`` +* ``socket.recv_from()`` +* ``socket.send()`` +* ``socket.sendto()`` +* ``time.sleep()`` +* ``select.select()`` +* ``select.poll()`` +* ``select.epoll.poll()`` +* ``select.devpoll.poll()`` +* ``select.kqueue.control()`` +* ``selectors.SelectSelector.select()`` and other selector classes + +Note: The ``selector`` module already retries on ``InterruptedError``, but it +doesn't recompute the timeout yet. + + +Deprecate siginterrupt() +------------------------ + +The function ``signal.siginterrupt()`` becomes useless with this PEP, +it should be deprecated. When ``signal.siginterrupt(signum, False)`` +is used, some system calls don't fail with ``EINTR`` when a signal is +received. Python cannot call its signal handler and interrupt the +system call. + +The function ``signal.siginterrupt()`` will be deprecated in Python +3.5. + +In Python 3.6, calling ``signal.siginterrupt(signum, False)`` will +raise an exception, whereas ``signal.siginterrupt(signum, True)`` will +only emit the deprecation warning. + + +Backward Compatibility +====================== + +Applications relying on the fact that system calls are interrupted +with ``InterruptedError`` will hang. The authors of this PEP don't +think that such application exist. + +If such applications exist, they must be fixed to handle signals +differently, to have a reliable behaviour on all platforms and all +Python versions. For example, use a signal handle which raises an +exception, or use a wakeup file descriptor. + +Applications should not call ``signal.siginterrupt(signum, False)`` +anymore, since this call will raise an exception in Python 3.6. + +For applications using event loops, ``signal.set_wakeup_fd()`` is the +recommanded option to handle signals. The signal handler writes signal +numbers into the file descriptor and the event loop is awaken to read +them. The event loop can decide how to handle these signals without +the restriction of signal handlers. + + +Appendix +======== + +Wakeup file descriptor +---------------------- + +Since Python 3.3, ``signal.set_wakeup_fd()`` writes the signal number +into the file descriptor, whereas it only wrote a null byte before. +It becomes possible to handle different signals using the wakeup file +descriptor. + +Linux has a ``signalfd()`` which provides more information on each +signal. For example, it's possible to know the pid and uid who sent +the signal. This function is not exposed in Python yet (see the +`issue 12304 `_). + +On Unix, the ``asyncio`` module uses the wakeup file descriptor to +wake up its event loop. + + +Multithreading +-------------- + +A C signal handler can be called from any thread, but the Python +signal handler should only be called in the main thread. + +Python has a ``PyErr_SetInterrupt()`` function which calls the +``SIGINT`` signal handler to interrupt the Python main thread. + + +Signals on Windows +------------------ + +Control events +^^^^^^^^^^^^^^ + +Windows uses "control events": + +* ``CTRL_BREAK_EVENT``: Break (``SIGBREAK``) +* ``CTRL_CLOSE_EVENT``: Close event +* ``CTRL_C_EVENT``: CTRL+C (``SIGINT``) +* ``CTRL_LOGOFF_EVENT``: Logoff +* ``CTRL_SHUTDOWN_EVENT``: Shutdown + +The `SetConsoleCtrlHandler() function +`_ +can be used to install a control handler. + +The ``CTRL_C_EVENT`` and ``CTRL_BREAK_EVENT`` events can be sent to a +process using the `GenerateConsoleCtrlEvent() function +`_. +This function is exposed in Python as ``os.kill()``. + + +Signals +^^^^^^^ + +The following signals are supported on Windows: + +* ``SIGABRT`` +* ``SIGBREAK`` (``CTRL_BREAK_EVENT``): signal only available on Windows +* ``SIGFPE`` +* ``SIGILL`` +* ``SIGINT`` (``CTRL_C_EVENT``) +* ``SIGSEGV`` +* ``SIGTERM`` + + +SIGINT +^^^^^^ + +The default Python signal handler for ``SIGINT`` sets a Windows event +object: ``sigint_event``. + +``time.sleep()`` is implemented with ``WaitForSingleObjectEx()``, it +waits for the ``sigint_event`` object using ``time.sleep()`` parameter +as the timeout. So the sleep can be interrupted by ``SIGINT``. + +``_winapi.WaitForMultipleObjects()`` automatically adds +``sigint_event`` to the list of watched handles, so it can also be +interrupted. + +``PyOS_StdioReadline()`` also used ``sigint_event`` when ``fgets()`` +failed to check if Ctrl-C or Ctrl-Z was pressed. + + +Links +----- + +Misc +^^^^ + +* `glibc manual: Primitives Interrupted by Signals + `_ +* `Bug #119097 for perl5: print returning EINTR in 5.14 + `_. + + +Python issues related to EINTR +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The main issue is: `handle EINTR in the stdlib +`_. + +Open issues: + +* `Add a new signal.set_wakeup_socket() function + `_ +* `signal.set_wakeup_fd(fd): set the fd to non-blocking mode + `_ +* `Use a monotonic clock to compute timeouts + `_ +* `sys.stdout.write on OS X is not EINTR safe + `_ +* `platform.uname() not EINTR safe + `_ +* `asyncore does not handle EINTR in recv, send, connect, accept, + `_ +* `socket.create_connection() doesn't handle EINTR properly + `_ + +Closed issues: + +* `Interrupted system calls are not retried + `_ +* `Solaris: EINTR exception in select/socket calls in telnetlib + `_ +* `subprocess: Popen.communicate() doesn't handle EINTR in some cases + `_ +* `multiprocessing.util._eintr_retry doen't recalculate timeouts + `_ +* `file readline, readlines & readall methods can lose data on EINTR + `_ +* `multiprocessing BaseManager serve_client() does not check EINTR on recv + `_ +* `selectors behaviour on EINTR undocumented + `_ +* `asyncio: limit EINTR occurrences with SA_RESTART + `_ +* `smtplib.py socket.create_connection() also doesn't handle EINTR properly + `_ +* `Faulty RESTART/EINTR handling in Parser/myreadline.c + `_ +* `test_httpservers intermittent failure, test_post and EINTR + `_ +* `os.spawnv(P_WAIT, ...) on Linux doesn't handle EINTR + `_ +* `asyncore fails when EINTR happens in pol + `_ +* `file.write and file.read don't handle EINTR + `_ +* `socket.readline() interface doesn't handle EINTR properly + `_ +* `subprocess is not EINTR-safe + `_ +* `SocketServer doesn't handle syscall interruption + `_ +* `subprocess deadlock when read() is interrupted + `_ +* `time.sleep(1): call PyErr_CheckSignals() if the sleep was interrupted + `_ +* `siginterrupt with flag=False is reset when signal received + `_ +* `need siginterrupt() on Linux - impossible to do timeouts + `_ +* `[Windows] Can not interrupt time.sleep() + `_ + +Python issues related to signals +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Open issues: + +* `expose signalfd(2) in the signal module + `_ +* `missing return in win32_kill? + `_ +* `Interrupts are lost during readline PyOS_InputHook processing + `_ +* `cannot catch KeyboardInterrupt when using curses getkey() + `_ +* `Deferred KeyboardInterrupt in interactive mode + `_ + +Closed issues: + +* `sys.interrupt_main() + `_ + + +Copyright +========= + +This document has been placed in the public domain. + + + +.. + Local Variables: + mode: indented-text + indent-tabs-mode: nil + sentence-end-double-space: t + fill-column: 70 + coding: utf-8 + End: