Small benign edits to PEP 446.
This commit is contained in:
Guido van Rossum
parent
bca9fd02fc
commit
091eb3e838
146
pep-0446.txt
146
pep-0446.txt
|
|
@ -16,11 +16,11 @@ Abstract
|
|||
Leaking file descriptors in child processes causes various annoying
|
||||
issues and is a known major security vulnerability. Using the
|
||||
``subprocess`` module with the *close_fds* parameter set to ``True`` is
|
||||
not possible in some cases, and has poor performances on some platforms.
|
||||
not possible in all cases, and has poor performances on some platforms.
|
||||
|
||||
This PEP proposes to make all file descriptors created by Python
|
||||
non-inheritable by default to reduce the risk of these issues. This PEP
|
||||
fixes also a race condition in multithreaded applications on operating
|
||||
fixes also a race condition in multi-threaded applications on operating
|
||||
systems supporting atomic flags to create non-inheritable file
|
||||
descriptors.
|
||||
|
||||
|
|
@ -33,13 +33,13 @@ Inheritance of File Descriptors
|
|||
|
||||
Each operating system handles the inheritance of file descriptors
|
||||
differently. Windows creates non-inheritable handles by default, whereas
|
||||
UNIX and the POSIX API of Windows create inheritable file descriptors by
|
||||
default. Python prefers the POSIX API over the native Windows API to
|
||||
UNIX and the POSIX API on Windows create inheritable file descriptors by
|
||||
default. Python prefers the POSIX API over the native Windows API, to
|
||||
have a single code base and to use the same type for file descriptors,
|
||||
and so it creates inheritable file descriptors.
|
||||
|
||||
There is one exception: ``os.pipe()`` creates non-inheritable pipes on
|
||||
Windows, whereas it creates inheritable pipes on UNIX. The reason is an
|
||||
Windows, whereas it creates inheritable pipes on UNIX. The reason is an
|
||||
implementation artifact: ``os.pipe()`` calls ``CreatePipe()`` on Windows
|
||||
(native API), whereas it calls ``pipe()`` on UNIX (POSIX API). The call
|
||||
to ``CreatePipe()`` was added in Python in 1994, before the introduction
|
||||
|
|
@ -51,18 +51,18 @@ Windows to create inheritable pipes.
|
|||
Inheritance of File Descriptors on Windows
|
||||
------------------------------------------
|
||||
|
||||
On Windows, the native type of file objects are handles (C type
|
||||
On Windows, the native type of file objects is handles (C type
|
||||
``HANDLE``). These handles have a ``HANDLE_FLAG_INHERIT`` flag which
|
||||
defines if a handle can be inherited in a child process or not. For the
|
||||
POSIX API, the C runtime (CRT) provides also file descriptors (C type
|
||||
``int``). The handle of a file descriptor can be get using the
|
||||
POSIX API, the C runtime (CRT) also provides file descriptors (C type
|
||||
``int``). The handle of a file descriptor can be retrieve using the
|
||||
function ``_get_osfhandle(fd)``. A file descriptor can be created from a
|
||||
handle using the function ``_open_osfhandle(handle)``.
|
||||
|
||||
Using `CreateProcess()
|
||||
<http://msdn.microsoft.com/en-us/library/windows/desktop/ms682425%28v=vs.85%29.aspx>`_,
|
||||
handles are only inherited if their inheritable flag
|
||||
(``HANDLE_FLAG_INHERIT``) is set and if the ``bInheritHandles``
|
||||
(``HANDLE_FLAG_INHERIT``) is set **and** the ``bInheritHandles``
|
||||
parameter of ``CreateProcess()`` is ``TRUE``; all file descriptors
|
||||
except standard streams (0, 1, 2) are closed in the child process, even
|
||||
if ``bInheritHandles`` is ``TRUE``. Using the ``spawnv()`` function, all
|
||||
|
|
@ -112,37 +112,37 @@ Call <http://support.microsoft.com/kb/315939/en-us>`_ (last review:
|
|||
November 2006). The `Python issue #16500 "Add an atfork module"
|
||||
<http://bugs.python.org/issue16500>`_ proposes to add such lock, it can
|
||||
be used to make handles non-inheritable without the race condition. Such
|
||||
lock only protects against a race condition between Python threads, C
|
||||
lock only protects against a race condition between Python threads; C
|
||||
threads are not protected.
|
||||
|
||||
Another option is to duplicate handles that must be inherited, pass the
|
||||
number of the duplicated handles to the child process, so the child
|
||||
Another option is to duplicate handles that must be inherited, passing the
|
||||
values of the duplicated handles to the child process, so the child
|
||||
process can steal duplicated handles using `DuplicateHandle()
|
||||
<http://msdn.microsoft.com/en-us/library/windows/apps/ms724251%28v=vs.85%29.aspx>`_
|
||||
with ``DUPLICATE_CLOSE_SOURCE``. Handle numbers change between the
|
||||
parent and the child process because the handles are duplicated (twice),
|
||||
the parent and/or the child process may be adapted to handle this
|
||||
with ``DUPLICATE_CLOSE_SOURCE``. Handle values change between the
|
||||
parent and the child process because the handles are duplicated (twice);
|
||||
the parent and/or the child process must be adapted to handle this
|
||||
change. If the child program cannot be modified, an intermediate program
|
||||
can be used to steal handles from the parent process before spawning the
|
||||
final child program. The intermediate has to pass the handle of the
|
||||
final child program. The intermediate program has to pass the handle from the
|
||||
child process to the parent process. The parent may have to close
|
||||
duplicated handles if all handles were not stolen, if the intermediate
|
||||
process failed for example. If the command line is used to pass the
|
||||
handle numbers, the command line must be modified when handle are
|
||||
duplicated, because their number are modified.
|
||||
duplicated handles if all handles were not stolen, for example if the
|
||||
intermediate process fails. If the command line is used to pass the
|
||||
handle values, the command line must be modified when handles are
|
||||
duplicated, because their values are modified.
|
||||
|
||||
This PEP does not include a solution to this problem because there is no
|
||||
perfect solution working on all Windows versions. This point is deferred
|
||||
until use cases relying on handle or file descriptor inheritance on
|
||||
Windows are well known to choose the best solution, and test carefully
|
||||
the implementation.
|
||||
Windows are well known, so we can choose the best solution and carefully
|
||||
test its implementation.
|
||||
|
||||
|
||||
Inheritance of File Descriptors on UNIX
|
||||
---------------------------------------
|
||||
|
||||
POSIX provides a *close-on-exec* flag on file descriptors to close
|
||||
automatically a file descriptor when the C function ``execv()`` is
|
||||
POSIX provides a *close-on-exec* flag on file descriptors to automatically
|
||||
close a file descriptor when the C function ``execv()`` is
|
||||
called. File descriptors with the *close-on-exec* flag cleared are
|
||||
inherited in the child process, file descriptors with the flag set are
|
||||
closed in the child process.
|
||||
|
|
@ -158,24 +158,24 @@ to set new flags) using ``fcntl()``::
|
|||
res = fcntl(fd, F_SETFD, flags);
|
||||
if (res == -1) { /* handle the error */ }
|
||||
|
||||
FreeBSD, Linux, Mac OS X, NetBSD, OpenBSD and QNX support also setting
|
||||
FreeBSD, Linux, Mac OS X, NetBSD, OpenBSD and QNX also support setting
|
||||
the flag in a single syscall using ioctl()::
|
||||
|
||||
int res;
|
||||
res = ioctl(fd, FIOCLEX, 0);
|
||||
if (!res) { /* handle the error */ }
|
||||
|
||||
The *close-on-exec* flag has no effect on ``fork()``: all file
|
||||
NOTE: The *close-on-exec* flag has no effect on ``fork()``: all file
|
||||
descriptors are inherited by the child process. The `Python issue #16500
|
||||
"Add an atfork module" <http://bugs.python.org/issue16500>`_ proposes to
|
||||
add a new ``atfork`` module to execute code at fork, it may be used to
|
||||
close automatically file descriptors.
|
||||
add a new ``atfork`` module to execute code at fork, which may be used to
|
||||
automatically close file descriptors.
|
||||
|
||||
|
||||
Issues with Inheritable File Descriptors
|
||||
----------------------------------------
|
||||
|
||||
Most of the time, inheritable file descriptors "leaked" in child
|
||||
Most of the time, inheritable file descriptors "leaked" to child
|
||||
processes are not noticed, because they don't cause major bugs. It does
|
||||
not mean that these bugs must not be fixed.
|
||||
|
||||
|
|
@ -185,10 +185,10 @@ Two common issues with inherited file descriptors:
|
|||
in the directory are closed. The same issue can be seen with files,
|
||||
except if the file was created with the ``FILE_SHARE_DELETE`` flag
|
||||
(``O_TEMPORARY`` mode for ``open()``).
|
||||
* If a listening socket is leaked in a child process, the socket address
|
||||
* If a listening socket is leaked to a child process, the socket address
|
||||
cannot be reused before the parent and child processes terminated. For
|
||||
example, if a web server spawn a new program to handle a process, and
|
||||
the server restarts while the program is not done: the server cannot
|
||||
example, if a web server spawns a new program to handle a process, and
|
||||
the server restarts while the program is not done, the server cannot
|
||||
start because the TCP port is still in use.
|
||||
|
||||
Example of issues in open source projects:
|
||||
|
|
@ -224,9 +224,9 @@ Security Vulnerability
|
|||
----------------------
|
||||
|
||||
Leaking sensitive file handles and file descriptors can lead to security
|
||||
vulnerabilities. An untrusted child process can read sensitive data like
|
||||
passwords and take control of the parent process though leaked file
|
||||
descriptors. With a leaked listening socket, a child process can accept
|
||||
vulnerabilities. An untrusted child process might read sensitive data like
|
||||
passwords or take control of the parent process though a leaked file
|
||||
descriptor. With a leaked listening socket, a child process can accept
|
||||
new connections to read sensitive data.
|
||||
|
||||
Example of vulnerabilities:
|
||||
|
|
@ -258,31 +258,31 @@ Inherited file descriptors caused 4 issues in the ``subprocess``
|
|||
module:
|
||||
|
||||
* `Issue #2320: Race condition in subprocess using stdin
|
||||
<http://bugs.python.org/issue2320>`_ (created in 2008)
|
||||
<http://bugs.python.org/issue2320>`_ (opened in 2008)
|
||||
* `Issue #3006: subprocess.Popen causes socket to remain open after
|
||||
close <http://bugs.python.org/issue3006>`_ (created in 2008)
|
||||
close <http://bugs.python.org/issue3006>`_ (opened in 2008)
|
||||
* `Issue #7213: subprocess leaks open file descriptors between Popen
|
||||
instances causing hangs <http://bugs.python.org/issue7213>`_
|
||||
(created in 2009)
|
||||
(opened in 2009)
|
||||
* `Issue #12786: subprocess wait() hangs when stdin is closed
|
||||
<http://bugs.python.org/issue12786>`_ (created in 2011)
|
||||
<http://bugs.python.org/issue12786>`_ (opened in 2011)
|
||||
|
||||
These issues were fixed in Python 3.2 by 4 different changes in the
|
||||
``subprocess`` module:
|
||||
|
||||
* Pipes are now non-inheritable ;
|
||||
* Pipes are now non-inheritable;
|
||||
* The default value of the *close_fds* parameter is now ``True``,
|
||||
with one exception on Windows: the default value is ``False`` if
|
||||
at least one standard stream is replaced ;
|
||||
* A new *pass_fds* parameter has been added ;
|
||||
at least one standard stream is replaced;
|
||||
* A new *pass_fds* parameter has been added;
|
||||
* Creation of a ``_posixsubprocess`` module implemented in C.
|
||||
|
||||
|
||||
Atomic Creation of non-inheritable File Descriptors
|
||||
---------------------------------------------------
|
||||
|
||||
In a multithreaded application, a inheritable file descriptor can be
|
||||
created just before a new program is spawn, before the file descriptor
|
||||
In a multi-threaded application, an inheritable file descriptor may be
|
||||
created just before a new program is spawned, before the file descriptor
|
||||
is made non-inheritable. In this case, the file descriptor is leaked to
|
||||
the child process. This race condition could be avoided if the file
|
||||
descriptor is created directly non-inheritable.
|
||||
|
|
@ -301,7 +301,7 @@ On UNIX, new flags were added for files and sockets:
|
|||
|
||||
* ``O_CLOEXEC``: available on Linux (2.6.23), FreeBSD (8.3),
|
||||
Mac OS 10.8, OpenBSD 5.0, Solaris 11, QNX, BeOS, next NetBSD release
|
||||
(6.1?). This flag is part of POSIX.1-2008.
|
||||
(6.1?). This flag is part of POSIX.1-2008.
|
||||
* ``SOCK_CLOEXEC`` flag for ``socket()`` and ``socketpair()``,
|
||||
available on Linux 2.6.27, OpenBSD 5.2, NetBSD 6.0.
|
||||
* ``fcntl()``: ``F_DUPFD_CLOEXEC`` flag, available on Linux 2.6.24,
|
||||
|
|
@ -330,17 +330,17 @@ On Linux older than 2.6.28, ``accept4()`` fails with ``errno`` set to
|
|||
|
||||
Summary:
|
||||
|
||||
=========================== =============== ====================================
|
||||
Operating System Atomic File Atomic Socket
|
||||
=========================== =============== ====================================
|
||||
FreeBSD 8.3 (2012) X
|
||||
Linux 2.6.23 (2007) 2.6.27 (2008)
|
||||
Mac OS X 10.8 (2012) X
|
||||
NetBSD 6.1 (?) 6.0 (2012)
|
||||
OpenBSD 5.0 (2011) 5.2 (2012)
|
||||
Solaris 11 (2011) X
|
||||
Windows XP (2001) Seven SP1 (2011), 2008 R2 SP1 (2011)
|
||||
=========================== =============== ====================================
|
||||
======================== =============== ====================================
|
||||
Operating System Atomic File Atomic Socket
|
||||
======================== =============== ====================================
|
||||
FreeBSD 8.3 (2012) X
|
||||
Linux 2.6.23 (2007) 2.6.27 (2008)
|
||||
Mac OS X 10.8 (2012) X
|
||||
NetBSD 6.1 (?) 6.0 (2012)
|
||||
OpenBSD 5.0 (2011) 5.2 (2012)
|
||||
Solaris 11 (2011) X
|
||||
Windows XP (2001) Seven SP1 (2011), 2008 R2 SP1 (2011)
|
||||
======================== =============== ====================================
|
||||
|
||||
Legend:
|
||||
|
||||
|
|
@ -398,15 +398,15 @@ file descriptors except standard streams are closed.
|
|||
|
||||
Summary:
|
||||
|
||||
=========================== ================ ================== =============
|
||||
Module FD on UNIX Handles on Windows FD on Windows
|
||||
=========================== ================ ================== =============
|
||||
subprocess, default STD, pass_fds none STD
|
||||
subprocess, replace stdout STD, pass_fds all STD
|
||||
subprocess, close_fds=False all all STD
|
||||
multiprocessing (not applicable) all STD
|
||||
os.execv(), os.spawn() all all all
|
||||
=========================== ================ ================== =============
|
||||
=========================== =============== ================== =============
|
||||
Module FD on UNIX Handles on Windows FD on Windows
|
||||
=========================== =============== ================== =============
|
||||
subprocess, default STD, pass_fds none STD
|
||||
subprocess, replace stdout STD, pass_fds all STD
|
||||
subprocess, close_fds=False all all STD
|
||||
multiprocessing not applicable all STD
|
||||
os.execv(), os.spawn() all all all
|
||||
=========================== =============== ================== =============
|
||||
|
||||
Legend:
|
||||
|
||||
|
|
@ -417,15 +417,15 @@ Legend:
|
|||
inherited in the child process
|
||||
* "pass_fds": file descriptors of the *pass_fds* parameter of the
|
||||
subprocess are inherited
|
||||
* "(not applicable)": on UNIX, the multiprocessing uses ``fork()``,
|
||||
so this case is not concerned by this PEP.
|
||||
* "not applicable": on UNIX, the multiprocessing uses ``fork()``,
|
||||
so this case is not affected by this PEP.
|
||||
|
||||
|
||||
Closing All Open File Descriptors
|
||||
---------------------------------
|
||||
|
||||
On UNIX, the ``subprocess`` module closes almost all file descriptors in
|
||||
the child process. This operation require MAXFD system calls, where
|
||||
the child process. This operation requires MAXFD system calls, where
|
||||
MAXFD is the maximum number of file descriptors, even if there are only
|
||||
few open file descriptors. This maximum can be read using:
|
||||
``os.sysconf("SC_OPEN_MAX")``.
|
||||
|
|
@ -547,10 +547,10 @@ Developers are encouraged to reuse the high-level Python module
|
|||
portable way.
|
||||
|
||||
Applications using the ``subprocess`` module with the *pass_fds*
|
||||
parameter or using ``os.dup2()`` to redirect standard streams should not
|
||||
be affected.
|
||||
parameter or using only ``os.dup2()`` to redirect standard streams should
|
||||
not be affected.
|
||||
|
||||
Python does no more conform to POSIX, since file descriptors are now
|
||||
Python no longer conform to POSIX, since file descriptors are now
|
||||
made non-inheritable by default. Python was not designed to conform to
|
||||
POSIX, but was designed to develop portable applications.
|
||||
|
||||
|
|
@ -587,8 +587,8 @@ Read the mail thread: `[Python-Dev] Proposal for a new function
|
|||
PEP 433
|
||||
-------
|
||||
|
||||
The PEP 433 entitled "Easier suppression of file descriptor inheritance"
|
||||
is a previous attempt proposing various other alternatives, but no
|
||||
PEP 433, "Easier suppression of file descriptor inheritance",
|
||||
was a previous attempt proposing various other alternatives, but no
|
||||
consensus could be reached.
|
||||
|
||||
|
||||
|
|
|
|||
Loading…
Reference in New Issue