PEP 446: cleanup
This commit is contained in:
Victor Stinner
parent
21d9bf1202
commit
98724124b9
163
pep-0446.txt
163
pep-0446.txt
|
|
@ -16,10 +16,9 @@ Abstract
|
|||
Leaking file descriptors in child processes causes various annoying
|
||||
issues and is a known major security vulnerability. This PEP proposes to
|
||||
make all file descriptors created by Python non-inheritable by default
|
||||
to have a well defined and portable behaviour and reduce the risk of
|
||||
these issues. This PEP fixes also a race condition
|
||||
in multithreaded applications on operating systems supporting atomic
|
||||
flags to create non-inheritable file descriptors.
|
||||
to reduces the risk of these issues. This PEP fixes also a race
|
||||
condition in multithreaded applications on operating systems supporting
|
||||
atomic flags to create non-inheritable file descriptors.
|
||||
|
||||
|
||||
Rationale
|
||||
|
|
@ -30,16 +29,16 @@ Inheritance of File Descriptors
|
|||
|
||||
Each operating system handles the inheritance of file descriptors
|
||||
differently. Windows creates non-inheritable file descriptors by
|
||||
default, whereas UNIX creates inheritable file descriptors. Python
|
||||
prefers the POSIX API over the native Windows API to have a single code
|
||||
base, and so creates inheritable file descriptors.
|
||||
default, whereas UNIX creates inheritable file descriptors by default.
|
||||
Python prefers the POSIX API over the native Windows API to have a
|
||||
single code base, and so it creates inheritable file descriptors.
|
||||
|
||||
There is one exception: ``os.pipe()`` creates non-inheritable pipes on
|
||||
Windows, whereas it creates inheritable pipes on UNIX. The reason comes
|
||||
from an implementation artifact: ``os.pipe()`` calls ``CreatePipe()`` on
|
||||
Windows, whereas it calls ``pipe()`` on UNIX. The call to
|
||||
``CreatePipe()`` was added in 1994, before the introduction of
|
||||
``pipe()`` in the POSIX API in Windows 98. The `issue #4708
|
||||
Windows, whereas it creates inheritable pipes on UNIX. The reason is an
|
||||
implementation artifact: ``os.pipe()`` calls ``CreatePipe()`` on Windows
|
||||
(native API), whereas it calls ``pipe()`` on UNIX (POSIX API). The call
|
||||
to ``CreatePipe()`` was added in Python in 1994, before the introduction
|
||||
of ``pipe()`` in the POSIX API in Windows 98. The `issue #4708
|
||||
<http://bugs.python.org/issue4708>`_ proposes to change ``os.pipe()`` on
|
||||
Windows to create inheritable pipes.
|
||||
|
||||
|
|
@ -51,29 +50,29 @@ On Windows, the native type of file objects are handles (C type
|
|||
``HANDLE``). These handles have a ``HANDLE_FLAG_INHERIT`` flag which
|
||||
defines if a handle can be inherited in a child process or not. For the
|
||||
POSIX API, the C runtime (CRT) provides also file descriptors (C type
|
||||
``int``). The handle of a file descriptor can be retrieved using
|
||||
``_get_osfhandle(fd)``. A file descriptor can be created from a handle
|
||||
using ``_open_osfhandle(handle)``.
|
||||
``int``). The handle of a file descriptor can be get using the
|
||||
function ``_get_osfhandle(fd)``. A file descriptor can be created from a
|
||||
handle using the function ``_open_osfhandle(handle)``.
|
||||
|
||||
Handles are only inherited if their inheritable flag
|
||||
Using `CreateProcess()
|
||||
<http://msdn.microsoft.com/en-us/library/windows/desktop/ms682425%28v=vs.85%29.aspx>`_,
|
||||
handles are only inherited if their inheritable flag
|
||||
(``HANDLE_FLAG_INHERIT``) is set and if the ``bInheritHandles``
|
||||
parameter of `CreateProcess()
|
||||
<http://msdn.microsoft.com/en-us/library/windows/desktop/ms682425%28v=vs.85%29.aspx>`_
|
||||
is ``TRUE``. Using ``CreateProcess()``, all file descriptors except
|
||||
standard streams (0, 1, 2) are closed in the child process, even if
|
||||
``bInheritHandles`` is ``TRUE``. Using the ``spawnv()`` function, all
|
||||
inheritable file descriptors are inherited in the child process. This
|
||||
function uses the undocumented fields *cbReserved2* and *lpReserved2* of
|
||||
the `STARTUPINFO
|
||||
parameter of ``CreateProcess()`` is ``TRUE``; all file descriptors
|
||||
except standard streams (0, 1, 2) are closed in the child process, even
|
||||
if ``bInheritHandles`` is ``TRUE``. Using the ``spawnv()`` function, all
|
||||
inheritable handles and all inheritable file descriptors are inherited
|
||||
in the child process. This function uses the undocumented fields
|
||||
*cbReserved2* and *lpReserved2* of the `STARTUPINFO
|
||||
<http://msdn.microsoft.com/en-us/library/windows/desktop/ms686331%28v=vs.85%29.aspx>`_
|
||||
structure to pass an array of file descriptors.
|
||||
|
||||
To replace standard streams (stdin, stdout, stderr), the
|
||||
``STARTF_USESTDHANDLES`` flag must be set in the *dwFlags* field of the
|
||||
``STARTUPINFO`` structure and the *bInheritHandles* parameter of
|
||||
``CreateProcess()`` must be set to ``TRUE``. So when at least one
|
||||
standard stream is replaced, all inheritable handles are inherited by
|
||||
the child process.
|
||||
To replace standard streams (stdin, stdout, stderr) using
|
||||
``CreateProcess()``, the ``STARTF_USESTDHANDLES`` flag must be set in
|
||||
the *dwFlags* field of the ``STARTUPINFO`` structure and the
|
||||
*bInheritHandles* parameter of ``CreateProcess()`` must be set to
|
||||
``TRUE``. So when at least one standard stream is replaced, all
|
||||
inheritable handles are inherited by the child process.
|
||||
|
||||
See also:
|
||||
|
||||
|
|
@ -88,8 +87,8 @@ Inheritance of File Descriptors on UNIX
|
|||
|
||||
POSIX provides a *close-on-exec* flag on file descriptors to close
|
||||
automatically a file descriptor when the C function ``execv()`` is
|
||||
called. File descriptors with the *close-on-exec* flag unset are
|
||||
inherited in the child process, file descriptros with the flag set are
|
||||
called. File descriptors with the *close-on-exec* flag cleared are
|
||||
inherited in the child process, file descriptors with the flag set are
|
||||
closed in the child process.
|
||||
|
||||
The flag can be set in two syscalls (one to get current flags, a second
|
||||
|
|
@ -113,8 +112,8 @@ the flag in a single syscall using ioctl()::
|
|||
The *close-on-exec* flag has no effect on ``fork()``: all file
|
||||
descriptors are inherited by the child process. The `Python issue #16500
|
||||
"Add an atfork module" <http://bugs.python.org/issue16500>`_ proposes to
|
||||
add a new ``atfork`` module to execute code at fork. It may be used to
|
||||
close automatically file descriptors at fork.
|
||||
add a new ``atfork`` module to execute code at fork, it may be used to
|
||||
close automatically file descriptors.
|
||||
|
||||
|
||||
Issues with Inheritable File Descriptors
|
||||
|
|
@ -124,15 +123,14 @@ Most of the time, inheritable file descriptors "leaked" in child
|
|||
processes are not noticed, because they don't cause major bugs. It does
|
||||
not mean that these bugs must not be fixed.
|
||||
|
||||
Two example of common issues with inherited file descriptors:
|
||||
Two examples of common issues with inherited file descriptors:
|
||||
|
||||
* On Windows, a directory cannot be removed until all file handles open
|
||||
in the directory are closed. It may explain why a temporary directory
|
||||
cannot be removed. The same issue can be seen with files, except if
|
||||
the file is temporary and was created with the ``FILE_SHARE_DELETE``
|
||||
flag (``O_TEMPORARY`` mode for ``open()``).
|
||||
* On Windows, a directory cannot be removed before all file handles open
|
||||
in the directory are closed. The same issue can be seen with files,
|
||||
except if the file was created with the ``FILE_SHARE_DELETE`` flag
|
||||
(``O_TEMPORARY`` mode for ``open()``).
|
||||
* If a listening socket is leaked in a child process, the socket address
|
||||
cannot be reused until the parent and child processes terminated. For
|
||||
cannot be reused before the parent and child processes terminated. For
|
||||
example, if a web server spawn a new program to handle a process, and
|
||||
the server restarts while the program is not done: the server cannot
|
||||
start because the TCP port is still in use.
|
||||
|
|
@ -146,9 +144,9 @@ of the CERT.
|
|||
|
||||
An untrusted child process can read sensitive data like passwords and
|
||||
take control of the parent process though leaked file descriptors. It is
|
||||
for example a known vulnerability to escape from a chroot. With a leaked
|
||||
listening socket, a child process can accept new connections to read
|
||||
sensitive data.
|
||||
for example a way to escape from a chroot. With a leaked listening
|
||||
socket, a child process can accept new connections to read sensitive
|
||||
data.
|
||||
|
||||
|
||||
Atomic Creation of non-inheritable File Descriptors
|
||||
|
|
@ -156,22 +154,19 @@ Atomic Creation of non-inheritable File Descriptors
|
|||
|
||||
In a multithreaded application, a inheritable file descriptor can be
|
||||
created just before a new program is spawn, before the file descriptor
|
||||
is made non-inheritable. In this case, fhe file descriptor is leaked to
|
||||
is made non-inheritable. In this case, the file descriptor is leaked to
|
||||
the child process. This race condition could be avoided if the file
|
||||
descriptor is created directly non-inheritable.
|
||||
|
||||
FreeBSD, Linux, Mac OS X, Windows and many other operating systems
|
||||
support creating non-inheritable file descriptors with the inheritable
|
||||
flag cleared atomically at the creating of the file descriptor.
|
||||
flag cleared atomically at the creation of the file descriptor.
|
||||
|
||||
On Windows, since at least Windows XP, the `SECURITY_ATTRIBUTES
|
||||
<http://msdn.microsoft.com/en-us/library/windows/desktop/aa379560%28v=vs.85%29.aspx>`_
|
||||
structure can be used to clear the ``HANDLE_FLAG_INHERIT`` flag: set
|
||||
*bInheritHandle* field to ``FALSE``. This structure cannot be used with
|
||||
sockets: a new ``WSA_FLAG_NO_HANDLE_INHERIT`` flag was added in Windows
|
||||
7 SP1 and Windows Server 2008 R2 SP1 for ``WSASocket()``. If this flag
|
||||
is used on an older Windows verison (ex: Windows XP SP3),
|
||||
``WSASocket()`` fails with ``WSAEPROTOTYPE``.
|
||||
A new ``WSA_FLAG_NO_HANDLE_INHERIT`` flag for ``WSASocket()`` was added
|
||||
in Windows 7 SP1 and Windows Server 2008 R2 SP1 to create
|
||||
non-inheritable sockets. If this flag is used on an older Windows
|
||||
version (ex: Windows XP SP3), ``WSASocket()`` fails with
|
||||
``WSAEPROTOTYPE``.
|
||||
|
||||
On UNIX, new flags were added for files and sockets:
|
||||
|
||||
|
|
@ -221,14 +216,14 @@ Windows XP (2001) Seven SP1 (2011), 2008 R2 SP1 (201
|
|||
Legend:
|
||||
|
||||
* "Atomic File": first version of the operating system supporting
|
||||
creating atomatically a non-inheritable file descriptor using
|
||||
creating atomically a non-inheritable file descriptor using
|
||||
``open()``
|
||||
* "Atomic Socket": first version of the operating system supporting
|
||||
creating atomatically a non-inheritable socket
|
||||
creating atomically a non-inheritable socket
|
||||
* "X": not supported yet
|
||||
|
||||
|
||||
Status in Python 3.3
|
||||
Status of Python 3.3
|
||||
--------------------
|
||||
|
||||
Python 3.3 creates inheritable file descriptors on all platforms, except
|
||||
|
|
@ -239,15 +234,15 @@ non-inheritable file descriptors were added to Python 3.3:
|
|||
``os.O_CLOEXEC``, ``os.pipe2()`` and ``socket.SOCK_CLOEXEC``.
|
||||
|
||||
On UNIX, the ``subprocess`` module closes all file descriptors in the
|
||||
child process, except standard streams (0, 1, 2) and file descriptors of
|
||||
the *pass_fds* parameter. If the *close_fds* parameter is set to
|
||||
``False``, all inheritable file descriptors are inherited in the child
|
||||
process.
|
||||
child process by default, except standard streams (0, 1, 2) and file
|
||||
descriptors of the *pass_fds* parameter. If the *close_fds* parameter is
|
||||
set to ``False``, all inheritable file descriptors are inherited in the
|
||||
child process.
|
||||
|
||||
On Windows, the ``subprocess`` closes all handles and file descriptors
|
||||
in the child process by default. If at least one standard stream (stdin,
|
||||
stdout or stderr) is replaced (ex: redirected into a pipe), all
|
||||
inheritable handles are inherited in the child process
|
||||
inheritable handles are inherited in the child process.
|
||||
|
||||
All inheritable file descriptors are inherited by the child process
|
||||
using the functions of the ``os.execv*()`` and ``os.spawn*()`` families.
|
||||
|
|
@ -288,7 +283,7 @@ Non-inheritable File Descriptors
|
|||
--------------------------------
|
||||
|
||||
The following functions are modified to make newly created file
|
||||
descriptors as non-inheritable by default:
|
||||
descriptors non-inheritable by default:
|
||||
|
||||
* ``asyncore.dispatcher.create_socket()``
|
||||
* ``io.FileIO``
|
||||
|
|
@ -306,21 +301,25 @@ descriptors as non-inheritable by default:
|
|||
* ``socket.socket()``
|
||||
* ``socket.socket.accept()``
|
||||
* ``socket.socket.dup()``
|
||||
* ``socket.socket.fromfd``
|
||||
* ``socket.socket.fromfd()``
|
||||
* ``socket.socketpair()``
|
||||
|
||||
When available, atomic flags are used to make file descriptors
|
||||
non-inheritable. The atomicity is not guaranteed because a fallback is
|
||||
required when atomic flags are not available.
|
||||
|
||||
|
||||
New Functions
|
||||
-------------
|
||||
|
||||
* ``os.get_inheritable(fd: int)``: return ``True`` if the file
|
||||
descriptor can be inherited by child processes, ``False`` otherwise.
|
||||
* ``os.set_inheritable(fd: int, inheritable: bool)``: set the
|
||||
* ``os.set_inheritable(fd: int, inheritable: bool)``: clear or set the
|
||||
inheritable flag of the specified file descriptor.
|
||||
|
||||
These new functions are available on all platforms.
|
||||
|
||||
On Windows, these functions accept also "file descriptors" of sockets:
|
||||
On Windows, these functions accept also file descriptors of sockets:
|
||||
the result of ``sockobj.fileno()``.
|
||||
|
||||
|
||||
|
|
@ -329,7 +328,7 @@ Other Changes
|
|||
|
||||
* On UNIX, subprocess makes file descriptors of the *pass_fds* parameter
|
||||
inheritable. The file descriptor is made inheritable in the child
|
||||
process after the ``fork()`` and before ``execv()``, the inheritable
|
||||
process after the ``fork()`` and before ``execv()``, so the inheritable
|
||||
flag of file descriptors is unchanged in the parent process.
|
||||
|
||||
* ``os.dup2(fd, fd2)`` makes *fd2* inheritable if *fd2* is ``0``
|
||||
|
|
@ -342,23 +341,24 @@ Backward Compatibility
|
|||
|
||||
This PEP break applications relying on inheritance of file descriptors.
|
||||
Developers are encouraged to reuse the high-level Python module
|
||||
``subprocess`` which handle the inheritance of file descriptors in a
|
||||
``subprocess`` which handles the inheritance of file descriptors in a
|
||||
portable way.
|
||||
|
||||
Applications using the ``subprocess`` module with the *pass_fds*
|
||||
parameter or using ``os.dup2()`` to redirect standard streams should not
|
||||
be affected.
|
||||
|
||||
Python does no more conform to POSIX, since file descriptors are made
|
||||
non-inheritable by default. Python was not designed to conform to POSIX,
|
||||
Python is designed to develop portable applications.
|
||||
Python does no more conform to POSIX, since file descriptors are now
|
||||
made non-inheritable by default. Python was not designed to conform to
|
||||
POSIX, but was designed to develop portable applications.
|
||||
|
||||
|
||||
Previous Work
|
||||
=============
|
||||
Related Work
|
||||
============
|
||||
|
||||
The programming languages Go, Perl and Ruby make newly created file
|
||||
descriptors non-inheritable: since Go 1.0, Perl 1.0 and Ruby 2.0.
|
||||
descriptors non-inheritable by default: since Go 1.0 (2009), Perl 1.0
|
||||
(1987) and Ruby 2.0 (2013).
|
||||
|
||||
The SCons project overrides builtin functions ``file()`` and ``open()``
|
||||
to make files non-inheritable on Windows:
|
||||
|
|
@ -382,18 +382,17 @@ No special case for standard streams
|
|||
Functions handling file descriptors should not handle standard streams
|
||||
(file descriptors ``0``, ``1``, ``2``) differently.
|
||||
|
||||
This option does not work on Windows. On Windows,
|
||||
``os.set_inheritable(fd, inheritable)`` (calling
|
||||
``SetHandleInformation()`` to set or clear ``HANDLE_FLAG_INHERIT`` flag)
|
||||
on file descriptor ``0`` (stdin), ``1`` (stdout) or ``2`` (stderr) fails
|
||||
with ``OSError(87, 'invalid argument')``. If ``os.dup2(fd, fd2)`` would
|
||||
always make *fd2* non-inheritable, the function would raise an exception
|
||||
when used to redirect standard streams.
|
||||
This option does not work on Windows. On Windows, calling
|
||||
``SetHandleInformation()`` to set or clear ``HANDLE_FLAG_INHERIT`` flag
|
||||
on standard streams (0, 1, 2) fails with the Windows error 87 (invalid
|
||||
argument). If ``os.dup2(fd, fd2)`` would always make *fd2*
|
||||
non-inheritable, the function would raise an exception when used to
|
||||
redirect standard streams.
|
||||
|
||||
Another option is to add a new *inheritable* parameter to ``os.dup2()``.
|
||||
|
||||
This PEP has a special-case for ``os.dup2()`` to not break backward
|
||||
compatibility on applications redirection standard streams before
|
||||
compatibility on applications redirecting standard streams before
|
||||
calling the C function ``execv()``. Examples in the Python standard
|
||||
library: ``CGIHTTPRequestHandler.run_cgi()`` and ``pty.fork()`` use
|
||||
``os.dup2()`` to redict stdin, stdout and stderr.
|
||||
|
|
|
|||
Loading…
Reference in New Issue