From 98724124b94e86798d90bd3e8d9a79f5145c0f2f Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Tue, 6 Aug 2013 02:18:49 +0200 Subject: [PATCH] PEP 446: cleanup --- pep-0446.txt | 163 +++++++++++++++++++++++++-------------------------- 1 file changed, 81 insertions(+), 82 deletions(-) diff --git a/pep-0446.txt b/pep-0446.txt index 48a8f9e91..811877ff9 100644 --- a/pep-0446.txt +++ b/pep-0446.txt @@ -16,10 +16,9 @@ Abstract Leaking file descriptors in child processes causes various annoying issues and is a known major security vulnerability. This PEP proposes to make all file descriptors created by Python non-inheritable by default -to have a well defined and portable behaviour and reduce the risk of -these issues. This PEP fixes also a race condition -in multithreaded applications on operating systems supporting atomic -flags to create non-inheritable file descriptors. +to reduces the risk of these issues. This PEP fixes also a race +condition in multithreaded applications on operating systems supporting +atomic flags to create non-inheritable file descriptors. Rationale @@ -30,16 +29,16 @@ Inheritance of File Descriptors Each operating system handles the inheritance of file descriptors differently. Windows creates non-inheritable file descriptors by -default, whereas UNIX creates inheritable file descriptors. Python -prefers the POSIX API over the native Windows API to have a single code -base, and so creates inheritable file descriptors. +default, whereas UNIX creates inheritable file descriptors by default. +Python prefers the POSIX API over the native Windows API to have a +single code base, and so it creates inheritable file descriptors. There is one exception: ``os.pipe()`` creates non-inheritable pipes on -Windows, whereas it creates inheritable pipes on UNIX. The reason comes -from an implementation artifact: ``os.pipe()`` calls ``CreatePipe()`` on -Windows, whereas it calls ``pipe()`` on UNIX. The call to -``CreatePipe()`` was added in 1994, before the introduction of -``pipe()`` in the POSIX API in Windows 98. The `issue #4708 +Windows, whereas it creates inheritable pipes on UNIX. The reason is an +implementation artifact: ``os.pipe()`` calls ``CreatePipe()`` on Windows +(native API), whereas it calls ``pipe()`` on UNIX (POSIX API). The call +to ``CreatePipe()`` was added in Python in 1994, before the introduction +of ``pipe()`` in the POSIX API in Windows 98. The `issue #4708 `_ proposes to change ``os.pipe()`` on Windows to create inheritable pipes. @@ -51,29 +50,29 @@ On Windows, the native type of file objects are handles (C type ``HANDLE``). These handles have a ``HANDLE_FLAG_INHERIT`` flag which defines if a handle can be inherited in a child process or not. For the POSIX API, the C runtime (CRT) provides also file descriptors (C type -``int``). The handle of a file descriptor can be retrieved using -``_get_osfhandle(fd)``. A file descriptor can be created from a handle -using ``_open_osfhandle(handle)``. +``int``). The handle of a file descriptor can be get using the +function ``_get_osfhandle(fd)``. A file descriptor can be created from a +handle using the function ``_open_osfhandle(handle)``. -Handles are only inherited if their inheritable flag +Using `CreateProcess() +`_, +handles are only inherited if their inheritable flag (``HANDLE_FLAG_INHERIT``) is set and if the ``bInheritHandles`` -parameter of `CreateProcess() -`_ -is ``TRUE``. Using ``CreateProcess()``, all file descriptors except -standard streams (0, 1, 2) are closed in the child process, even if -``bInheritHandles`` is ``TRUE``. Using the ``spawnv()`` function, all -inheritable file descriptors are inherited in the child process. This -function uses the undocumented fields *cbReserved2* and *lpReserved2* of -the `STARTUPINFO +parameter of ``CreateProcess()`` is ``TRUE``; all file descriptors +except standard streams (0, 1, 2) are closed in the child process, even +if ``bInheritHandles`` is ``TRUE``. Using the ``spawnv()`` function, all +inheritable handles and all inheritable file descriptors are inherited +in the child process. This function uses the undocumented fields +*cbReserved2* and *lpReserved2* of the `STARTUPINFO `_ structure to pass an array of file descriptors. -To replace standard streams (stdin, stdout, stderr), the -``STARTF_USESTDHANDLES`` flag must be set in the *dwFlags* field of the -``STARTUPINFO`` structure and the *bInheritHandles* parameter of -``CreateProcess()`` must be set to ``TRUE``. So when at least one -standard stream is replaced, all inheritable handles are inherited by -the child process. +To replace standard streams (stdin, stdout, stderr) using +``CreateProcess()``, the ``STARTF_USESTDHANDLES`` flag must be set in +the *dwFlags* field of the ``STARTUPINFO`` structure and the +*bInheritHandles* parameter of ``CreateProcess()`` must be set to +``TRUE``. So when at least one standard stream is replaced, all +inheritable handles are inherited by the child process. See also: @@ -88,8 +87,8 @@ Inheritance of File Descriptors on UNIX POSIX provides a *close-on-exec* flag on file descriptors to close automatically a file descriptor when the C function ``execv()`` is -called. File descriptors with the *close-on-exec* flag unset are -inherited in the child process, file descriptros with the flag set are +called. File descriptors with the *close-on-exec* flag cleared are +inherited in the child process, file descriptors with the flag set are closed in the child process. The flag can be set in two syscalls (one to get current flags, a second @@ -113,8 +112,8 @@ the flag in a single syscall using ioctl():: The *close-on-exec* flag has no effect on ``fork()``: all file descriptors are inherited by the child process. The `Python issue #16500 "Add an atfork module" `_ proposes to -add a new ``atfork`` module to execute code at fork. It may be used to -close automatically file descriptors at fork. +add a new ``atfork`` module to execute code at fork, it may be used to +close automatically file descriptors. Issues with Inheritable File Descriptors @@ -124,15 +123,14 @@ Most of the time, inheritable file descriptors "leaked" in child processes are not noticed, because they don't cause major bugs. It does not mean that these bugs must not be fixed. -Two example of common issues with inherited file descriptors: +Two examples of common issues with inherited file descriptors: -* On Windows, a directory cannot be removed until all file handles open - in the directory are closed. It may explain why a temporary directory - cannot be removed. The same issue can be seen with files, except if - the file is temporary and was created with the ``FILE_SHARE_DELETE`` - flag (``O_TEMPORARY`` mode for ``open()``). +* On Windows, a directory cannot be removed before all file handles open + in the directory are closed. The same issue can be seen with files, + except if the file was created with the ``FILE_SHARE_DELETE`` flag + (``O_TEMPORARY`` mode for ``open()``). * If a listening socket is leaked in a child process, the socket address - cannot be reused until the parent and child processes terminated. For + cannot be reused before the parent and child processes terminated. For example, if a web server spawn a new program to handle a process, and the server restarts while the program is not done: the server cannot start because the TCP port is still in use. @@ -146,9 +144,9 @@ of the CERT. An untrusted child process can read sensitive data like passwords and take control of the parent process though leaked file descriptors. It is -for example a known vulnerability to escape from a chroot. With a leaked -listening socket, a child process can accept new connections to read -sensitive data. +for example a way to escape from a chroot. With a leaked listening +socket, a child process can accept new connections to read sensitive +data. Atomic Creation of non-inheritable File Descriptors @@ -156,22 +154,19 @@ Atomic Creation of non-inheritable File Descriptors In a multithreaded application, a inheritable file descriptor can be created just before a new program is spawn, before the file descriptor -is made non-inheritable. In this case, fhe file descriptor is leaked to +is made non-inheritable. In this case, the file descriptor is leaked to the child process. This race condition could be avoided if the file descriptor is created directly non-inheritable. FreeBSD, Linux, Mac OS X, Windows and many other operating systems support creating non-inheritable file descriptors with the inheritable -flag cleared atomically at the creating of the file descriptor. +flag cleared atomically at the creation of the file descriptor. -On Windows, since at least Windows XP, the `SECURITY_ATTRIBUTES -`_ -structure can be used to clear the ``HANDLE_FLAG_INHERIT`` flag: set -*bInheritHandle* field to ``FALSE``. This structure cannot be used with -sockets: a new ``WSA_FLAG_NO_HANDLE_INHERIT`` flag was added in Windows -7 SP1 and Windows Server 2008 R2 SP1 for ``WSASocket()``. If this flag -is used on an older Windows verison (ex: Windows XP SP3), -``WSASocket()`` fails with ``WSAEPROTOTYPE``. +A new ``WSA_FLAG_NO_HANDLE_INHERIT`` flag for ``WSASocket()`` was added +in Windows 7 SP1 and Windows Server 2008 R2 SP1 to create +non-inheritable sockets. If this flag is used on an older Windows +version (ex: Windows XP SP3), ``WSASocket()`` fails with +``WSAEPROTOTYPE``. On UNIX, new flags were added for files and sockets: @@ -221,14 +216,14 @@ Windows XP (2001) Seven SP1 (2011), 2008 R2 SP1 (201 Legend: * "Atomic File": first version of the operating system supporting - creating atomatically a non-inheritable file descriptor using + creating atomically a non-inheritable file descriptor using ``open()`` * "Atomic Socket": first version of the operating system supporting - creating atomatically a non-inheritable socket + creating atomically a non-inheritable socket * "X": not supported yet -Status in Python 3.3 +Status of Python 3.3 -------------------- Python 3.3 creates inheritable file descriptors on all platforms, except @@ -239,15 +234,15 @@ non-inheritable file descriptors were added to Python 3.3: ``os.O_CLOEXEC``, ``os.pipe2()`` and ``socket.SOCK_CLOEXEC``. On UNIX, the ``subprocess`` module closes all file descriptors in the -child process, except standard streams (0, 1, 2) and file descriptors of -the *pass_fds* parameter. If the *close_fds* parameter is set to -``False``, all inheritable file descriptors are inherited in the child -process. +child process by default, except standard streams (0, 1, 2) and file +descriptors of the *pass_fds* parameter. If the *close_fds* parameter is +set to ``False``, all inheritable file descriptors are inherited in the +child process. On Windows, the ``subprocess`` closes all handles and file descriptors in the child process by default. If at least one standard stream (stdin, stdout or stderr) is replaced (ex: redirected into a pipe), all -inheritable handles are inherited in the child process +inheritable handles are inherited in the child process. All inheritable file descriptors are inherited by the child process using the functions of the ``os.execv*()`` and ``os.spawn*()`` families. @@ -288,7 +283,7 @@ Non-inheritable File Descriptors -------------------------------- The following functions are modified to make newly created file -descriptors as non-inheritable by default: +descriptors non-inheritable by default: * ``asyncore.dispatcher.create_socket()`` * ``io.FileIO`` @@ -306,21 +301,25 @@ descriptors as non-inheritable by default: * ``socket.socket()`` * ``socket.socket.accept()`` * ``socket.socket.dup()`` - * ``socket.socket.fromfd`` + * ``socket.socket.fromfd()`` * ``socket.socketpair()`` +When available, atomic flags are used to make file descriptors +non-inheritable. The atomicity is not guaranteed because a fallback is +required when atomic flags are not available. + New Functions ------------- * ``os.get_inheritable(fd: int)``: return ``True`` if the file descriptor can be inherited by child processes, ``False`` otherwise. -* ``os.set_inheritable(fd: int, inheritable: bool)``: set the +* ``os.set_inheritable(fd: int, inheritable: bool)``: clear or set the inheritable flag of the specified file descriptor. These new functions are available on all platforms. -On Windows, these functions accept also "file descriptors" of sockets: +On Windows, these functions accept also file descriptors of sockets: the result of ``sockobj.fileno()``. @@ -329,7 +328,7 @@ Other Changes * On UNIX, subprocess makes file descriptors of the *pass_fds* parameter inheritable. The file descriptor is made inheritable in the child - process after the ``fork()`` and before ``execv()``, the inheritable + process after the ``fork()`` and before ``execv()``, so the inheritable flag of file descriptors is unchanged in the parent process. * ``os.dup2(fd, fd2)`` makes *fd2* inheritable if *fd2* is ``0`` @@ -342,23 +341,24 @@ Backward Compatibility This PEP break applications relying on inheritance of file descriptors. Developers are encouraged to reuse the high-level Python module -``subprocess`` which handle the inheritance of file descriptors in a +``subprocess`` which handles the inheritance of file descriptors in a portable way. Applications using the ``subprocess`` module with the *pass_fds* parameter or using ``os.dup2()`` to redirect standard streams should not be affected. -Python does no more conform to POSIX, since file descriptors are made -non-inheritable by default. Python was not designed to conform to POSIX, -Python is designed to develop portable applications. +Python does no more conform to POSIX, since file descriptors are now +made non-inheritable by default. Python was not designed to conform to +POSIX, but was designed to develop portable applications. -Previous Work -============= +Related Work +============ The programming languages Go, Perl and Ruby make newly created file -descriptors non-inheritable: since Go 1.0, Perl 1.0 and Ruby 2.0. +descriptors non-inheritable by default: since Go 1.0 (2009), Perl 1.0 +(1987) and Ruby 2.0 (2013). The SCons project overrides builtin functions ``file()`` and ``open()`` to make files non-inheritable on Windows: @@ -382,18 +382,17 @@ No special case for standard streams Functions handling file descriptors should not handle standard streams (file descriptors ``0``, ``1``, ``2``) differently. -This option does not work on Windows. On Windows, -``os.set_inheritable(fd, inheritable)`` (calling -``SetHandleInformation()`` to set or clear ``HANDLE_FLAG_INHERIT`` flag) -on file descriptor ``0`` (stdin), ``1`` (stdout) or ``2`` (stderr) fails -with ``OSError(87, 'invalid argument')``. If ``os.dup2(fd, fd2)`` would -always make *fd2* non-inheritable, the function would raise an exception -when used to redirect standard streams. +This option does not work on Windows. On Windows, calling +``SetHandleInformation()`` to set or clear ``HANDLE_FLAG_INHERIT`` flag +on standard streams (0, 1, 2) fails with the Windows error 87 (invalid +argument). If ``os.dup2(fd, fd2)`` would always make *fd2* +non-inheritable, the function would raise an exception when used to +redirect standard streams. Another option is to add a new *inheritable* parameter to ``os.dup2()``. This PEP has a special-case for ``os.dup2()`` to not break backward -compatibility on applications redirection standard streams before +compatibility on applications redirecting standard streams before calling the C function ``execv()``. Examples in the Python standard library: ``CGIHTTPRequestHandler.run_cgi()`` and ``pty.fork()`` use ``os.dup2()`` to redict stdin, stdout and stderr.