iSharkFly-Open
/
python-peps
mirror of https://github.com/python/peps.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Small benign edits to PEP 446.

This commit is contained in:
Guido van Rossum 2013-08-26 14:21:07 -07:00
parent bca9fd02fc
commit 091eb3e838
1 changed files with 73 additions and 73 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

146
pep-0446.txt
View File

@ -16,11 +16,11 @@ Abstract
Leaking file descriptors in child processes causes various annoying Leaking file descriptors in child processes causes various annoying
issues and is a known major security vulnerability. Using the issues and is a known major security vulnerability. Using the
``subprocess`` module with the *close_fds* parameter set to ``True`` is ``subprocess`` module with the *close_fds* parameter set to ``True`` is
not possible in some cases, and has poor performances on some platforms. not possible in all cases, and has poor performances on some platforms.
This PEP proposes to make all file descriptors created by Python This PEP proposes to make all file descriptors created by Python
non-inheritable by default to reduce the risk of these issues. This PEP non-inheritable by default to reduce the risk of these issues. This PEP
fixes also a race condition in multithreaded applications on operating fixes also a race condition in multi-threaded applications on operating
systems supporting atomic flags to create non-inheritable file systems supporting atomic flags to create non-inheritable file
descriptors. descriptors.
@ -33,13 +33,13 @@ Inheritance of File Descriptors
Each operating system handles the inheritance of file descriptors Each operating system handles the inheritance of file descriptors
differently. Windows creates non-inheritable handles by default, whereas differently. Windows creates non-inheritable handles by default, whereas
UNIX and the POSIX API of Windows create inheritable file descriptors by UNIX and the POSIX API on Windows create inheritable file descriptors by
default. Python prefers the POSIX API over the native Windows API to default. Python prefers the POSIX API over the native Windows API, to
have a single code base and to use the same type for file descriptors, have a single code base and to use the same type for file descriptors,
and so it creates inheritable file descriptors. and so it creates inheritable file descriptors.
There is one exception: ``os.pipe()`` creates non-inheritable pipes on There is one exception: ``os.pipe()`` creates non-inheritable pipes on
Windows, whereas it creates inheritable pipes on UNIX. The reason is an Windows, whereas it creates inheritable pipes on UNIX. The reason is an
implementation artifact: ``os.pipe()`` calls ``CreatePipe()`` on Windows implementation artifact: ``os.pipe()`` calls ``CreatePipe()`` on Windows
(native API), whereas it calls ``pipe()`` on UNIX (POSIX API). The call (native API), whereas it calls ``pipe()`` on UNIX (POSIX API). The call
to ``CreatePipe()`` was added in Python in 1994, before the introduction to ``CreatePipe()`` was added in Python in 1994, before the introduction
@ -51,18 +51,18 @@ Windows to create inheritable pipes.
Inheritance of File Descriptors on Windows Inheritance of File Descriptors on Windows
------------------------------------------ ------------------------------------------
On Windows, the native type of file objects are handles (C type On Windows, the native type of file objects is handles (C type
``HANDLE``). These handles have a ``HANDLE_FLAG_INHERIT`` flag which ``HANDLE``). These handles have a ``HANDLE_FLAG_INHERIT`` flag which
defines if a handle can be inherited in a child process or not. For the defines if a handle can be inherited in a child process or not. For the
POSIX API, the C runtime (CRT) provides also file descriptors (C type POSIX API, the C runtime (CRT) also provides file descriptors (C type
``int``). The handle of a file descriptor can be get using the ``int``). The handle of a file descriptor can be retrieve using the
function ``_get_osfhandle(fd)``. A file descriptor can be created from a function ``_get_osfhandle(fd)``. A file descriptor can be created from a
handle using the function ``_open_osfhandle(handle)``. handle using the function ``_open_osfhandle(handle)``.
Using `CreateProcess() Using `CreateProcess()
<http://msdn.microsoft.com/en-us/library/windows/desktop/ms682425%28v=vs.85%29.aspx>`_, <http://msdn.microsoft.com/en-us/library/windows/desktop/ms682425%28v=vs.85%29.aspx>`_,
handles are only inherited if their inheritable flag handles are only inherited if their inheritable flag
(``HANDLE_FLAG_INHERIT``) is set and if the ``bInheritHandles`` (``HANDLE_FLAG_INHERIT``) is set **and** the ``bInheritHandles``
parameter of ``CreateProcess()`` is ``TRUE``; all file descriptors parameter of ``CreateProcess()`` is ``TRUE``; all file descriptors
except standard streams (0, 1, 2) are closed in the child process, even except standard streams (0, 1, 2) are closed in the child process, even
if ``bInheritHandles`` is ``TRUE``. Using the ``spawnv()`` function, all if ``bInheritHandles`` is ``TRUE``. Using the ``spawnv()`` function, all
@ -112,37 +112,37 @@ Call <http://support.microsoft.com/kb/315939/en-us>`_ (last review:
November 2006). The `Python issue #16500 "Add an atfork module" November 2006). The `Python issue #16500 "Add an atfork module"
<http://bugs.python.org/issue16500>`_ proposes to add such lock, it can <http://bugs.python.org/issue16500>`_ proposes to add such lock, it can
be used to make handles non-inheritable without the race condition. Such be used to make handles non-inheritable without the race condition. Such
lock only protects against a race condition between Python threads, C lock only protects against a race condition between Python threads; C
threads are not protected. threads are not protected.
Another option is to duplicate handles that must be inherited, pass the Another option is to duplicate handles that must be inherited, passing the
number of the duplicated handles to the child process, so the child values of the duplicated handles to the child process, so the child
process can steal duplicated handles using `DuplicateHandle() process can steal duplicated handles using `DuplicateHandle()
<http://msdn.microsoft.com/en-us/library/windows/apps/ms724251%28v=vs.85%29.aspx>`_ <http://msdn.microsoft.com/en-us/library/windows/apps/ms724251%28v=vs.85%29.aspx>`_
with ``DUPLICATE_CLOSE_SOURCE``. Handle numbers change between the with ``DUPLICATE_CLOSE_SOURCE``. Handle values change between the
parent and the child process because the handles are duplicated (twice), parent and the child process because the handles are duplicated (twice);
the parent and/or the child process may be adapted to handle this the parent and/or the child process must be adapted to handle this
change. If the child program cannot be modified, an intermediate program change. If the child program cannot be modified, an intermediate program
can be used to steal handles from the parent process before spawning the can be used to steal handles from the parent process before spawning the
final child program. The intermediate has to pass the handle of the final child program. The intermediate program has to pass the handle from the
child process to the parent process. The parent may have to close child process to the parent process. The parent may have to close
duplicated handles if all handles were not stolen, if the intermediate duplicated handles if all handles were not stolen, for example if the
process failed for example. If the command line is used to pass the intermediate process fails. If the command line is used to pass the
handle numbers, the command line must be modified when handle are handle values, the command line must be modified when handles are
duplicated, because their number are modified. duplicated, because their values are modified.
This PEP does not include a solution to this problem because there is no This PEP does not include a solution to this problem because there is no
perfect solution working on all Windows versions. This point is deferred perfect solution working on all Windows versions. This point is deferred
until use cases relying on handle or file descriptor inheritance on until use cases relying on handle or file descriptor inheritance on
Windows are well known to choose the best solution, and test carefully Windows are well known, so we can choose the best solution and carefully
the implementation. test its implementation.
Inheritance of File Descriptors on UNIX Inheritance of File Descriptors on UNIX
--------------------------------------- ---------------------------------------
POSIX provides a *close-on-exec* flag on file descriptors to close POSIX provides a *close-on-exec* flag on file descriptors to automatically
automatically a file descriptor when the C function ``execv()`` is close a file descriptor when the C function ``execv()`` is
called. File descriptors with the *close-on-exec* flag cleared are called. File descriptors with the *close-on-exec* flag cleared are
inherited in the child process, file descriptors with the flag set are inherited in the child process, file descriptors with the flag set are
closed in the child process. closed in the child process.
@ -158,24 +158,24 @@ to set new flags) using ``fcntl()``::
res = fcntl(fd, F_SETFD, flags); res = fcntl(fd, F_SETFD, flags);
if (res == -1) { /* handle the error */ } if (res == -1) { /* handle the error */ }
FreeBSD, Linux, Mac OS X, NetBSD, OpenBSD and QNX support also setting FreeBSD, Linux, Mac OS X, NetBSD, OpenBSD and QNX also support setting
the flag in a single syscall using ioctl():: the flag in a single syscall using ioctl()::
int res; int res;
res = ioctl(fd, FIOCLEX, 0); res = ioctl(fd, FIOCLEX, 0);
if (!res) { /* handle the error */ } if (!res) { /* handle the error */ }
The *close-on-exec* flag has no effect on ``fork()``: all file NOTE: The *close-on-exec* flag has no effect on ``fork()``: all file
descriptors are inherited by the child process. The `Python issue #16500 descriptors are inherited by the child process. The `Python issue #16500
"Add an atfork module" <http://bugs.python.org/issue16500>`_ proposes to "Add an atfork module" <http://bugs.python.org/issue16500>`_ proposes to
add a new ``atfork`` module to execute code at fork, it may be used to add a new ``atfork`` module to execute code at fork, which may be used to
close automatically file descriptors. automatically close file descriptors.
Issues with Inheritable File Descriptors Issues with Inheritable File Descriptors
---------------------------------------- ----------------------------------------
Most of the time, inheritable file descriptors "leaked" in child Most of the time, inheritable file descriptors "leaked" to child
processes are not noticed, because they don't cause major bugs. It does processes are not noticed, because they don't cause major bugs. It does
not mean that these bugs must not be fixed. not mean that these bugs must not be fixed.
@ -185,10 +185,10 @@ Two common issues with inherited file descriptors:
in the directory are closed. The same issue can be seen with files, in the directory are closed. The same issue can be seen with files,
except if the file was created with the ``FILE_SHARE_DELETE`` flag except if the file was created with the ``FILE_SHARE_DELETE`` flag
(``O_TEMPORARY`` mode for ``open()``). (``O_TEMPORARY`` mode for ``open()``).
* If a listening socket is leaked in a child process, the socket address * If a listening socket is leaked to a child process, the socket address
cannot be reused before the parent and child processes terminated. For cannot be reused before the parent and child processes terminated. For
example, if a web server spawn a new program to handle a process, and example, if a web server spawns a new program to handle a process, and
the server restarts while the program is not done: the server cannot the server restarts while the program is not done, the server cannot
start because the TCP port is still in use. start because the TCP port is still in use.
Example of issues in open source projects: Example of issues in open source projects:
@ -224,9 +224,9 @@ Security Vulnerability
---------------------- ----------------------
Leaking sensitive file handles and file descriptors can lead to security Leaking sensitive file handles and file descriptors can lead to security
vulnerabilities. An untrusted child process can read sensitive data like vulnerabilities. An untrusted child process might read sensitive data like
passwords and take control of the parent process though leaked file passwords or take control of the parent process though a leaked file
descriptors. With a leaked listening socket, a child process can accept descriptor. With a leaked listening socket, a child process can accept
new connections to read sensitive data. new connections to read sensitive data.
Example of vulnerabilities: Example of vulnerabilities:
@ -258,31 +258,31 @@ Inherited file descriptors caused 4 issues in the ``subprocess``
module: module:
* `Issue #2320: Race condition in subprocess using stdin * `Issue #2320: Race condition in subprocess using stdin
<http://bugs.python.org/issue2320>`_ (created in 2008) <http://bugs.python.org/issue2320>`_ (opened in 2008)
* `Issue #3006: subprocess.Popen causes socket to remain open after * `Issue #3006: subprocess.Popen causes socket to remain open after
close <http://bugs.python.org/issue3006>`_ (created in 2008) close <http://bugs.python.org/issue3006>`_ (opened in 2008)
* `Issue #7213: subprocess leaks open file descriptors between Popen * `Issue #7213: subprocess leaks open file descriptors between Popen
instances causing hangs <http://bugs.python.org/issue7213>`_ instances causing hangs <http://bugs.python.org/issue7213>`_
(created in 2009) (opened in 2009)
* `Issue #12786: subprocess wait() hangs when stdin is closed * `Issue #12786: subprocess wait() hangs when stdin is closed
<http://bugs.python.org/issue12786>`_ (created in 2011) <http://bugs.python.org/issue12786>`_ (opened in 2011)
These issues were fixed in Python 3.2 by 4 different changes in the These issues were fixed in Python 3.2 by 4 different changes in the
``subprocess`` module: ``subprocess`` module:
* Pipes are now non-inheritable ; * Pipes are now non-inheritable;
* The default value of the *close_fds* parameter is now ``True``, * The default value of the *close_fds* parameter is now ``True``,
with one exception on Windows: the default value is ``False`` if with one exception on Windows: the default value is ``False`` if
at least one standard stream is replaced ; at least one standard stream is replaced;
* A new *pass_fds* parameter has been added ; * A new *pass_fds* parameter has been added;
* Creation of a ``_posixsubprocess`` module implemented in C. * Creation of a ``_posixsubprocess`` module implemented in C.
Atomic Creation of non-inheritable File Descriptors Atomic Creation of non-inheritable File Descriptors
--------------------------------------------------- ---------------------------------------------------
In a multithreaded application, a inheritable file descriptor can be In a multi-threaded application, an inheritable file descriptor may be
created just before a new program is spawn, before the file descriptor created just before a new program is spawned, before the file descriptor
is made non-inheritable. In this case, the file descriptor is leaked to is made non-inheritable. In this case, the file descriptor is leaked to
the child process. This race condition could be avoided if the file the child process. This race condition could be avoided if the file
descriptor is created directly non-inheritable. descriptor is created directly non-inheritable.
@ -301,7 +301,7 @@ On UNIX, new flags were added for files and sockets:
* ``O_CLOEXEC``: available on Linux (2.6.23), FreeBSD (8.3), * ``O_CLOEXEC``: available on Linux (2.6.23), FreeBSD (8.3),
Mac OS 10.8, OpenBSD 5.0, Solaris 11, QNX, BeOS, next NetBSD release Mac OS 10.8, OpenBSD 5.0, Solaris 11, QNX, BeOS, next NetBSD release
(6.1?). This flag is part of POSIX.1-2008. (6.1?). This flag is part of POSIX.1-2008.
* ``SOCK_CLOEXEC`` flag for ``socket()`` and ``socketpair()``, * ``SOCK_CLOEXEC`` flag for ``socket()`` and ``socketpair()``,
available on Linux 2.6.27, OpenBSD 5.2, NetBSD 6.0. available on Linux 2.6.27, OpenBSD 5.2, NetBSD 6.0.
* ``fcntl()``: ``F_DUPFD_CLOEXEC`` flag, available on Linux 2.6.24, * ``fcntl()``: ``F_DUPFD_CLOEXEC`` flag, available on Linux 2.6.24,
@ -330,17 +330,17 @@ On Linux older than 2.6.28, ``accept4()`` fails with ``errno`` set to
Summary: Summary:
=========================== =============== ==================================== ======================== =============== ====================================
Operating System Atomic File Atomic Socket Operating System Atomic File Atomic Socket
=========================== =============== ==================================== ======================== =============== ====================================
FreeBSD 8.3 (2012) X FreeBSD 8.3 (2012) X
Linux 2.6.23 (2007) 2.6.27 (2008) Linux 2.6.23 (2007) 2.6.27 (2008)
Mac OS X 10.8 (2012) X Mac OS X 10.8 (2012) X
NetBSD 6.1 (?) 6.0 (2012) NetBSD 6.1 (?) 6.0 (2012)
OpenBSD 5.0 (2011) 5.2 (2012) OpenBSD 5.0 (2011) 5.2 (2012)
Solaris 11 (2011) X Solaris 11 (2011) X
Windows XP (2001) Seven SP1 (2011), 2008 R2 SP1 (2011) Windows XP (2001) Seven SP1 (2011), 2008 R2 SP1 (2011)
=========================== =============== ==================================== ======================== =============== ====================================
Legend: Legend:
@ -398,15 +398,15 @@ file descriptors except standard streams are closed.
Summary: Summary:
=========================== ================ ================== ============= =========================== =============== ================== =============
Module FD on UNIX Handles on Windows FD on Windows Module FD on UNIX Handles on Windows FD on Windows
=========================== ================ ================== ============= =========================== =============== ================== =============
subprocess, default STD, pass_fds none STD subprocess, default STD, pass_fds none STD
subprocess, replace stdout STD, pass_fds all STD subprocess, replace stdout STD, pass_fds all STD
subprocess, close_fds=False all all STD subprocess, close_fds=False all all STD
multiprocessing (not applicable) all STD multiprocessing not applicable all STD
os.execv(), os.spawn() all all all os.execv(), os.spawn() all all all
=========================== ================ ================== ============= =========================== =============== ================== =============
Legend: Legend:
@ -417,15 +417,15 @@ Legend:
inherited in the child process inherited in the child process
* "pass_fds": file descriptors of the *pass_fds* parameter of the * "pass_fds": file descriptors of the *pass_fds* parameter of the
subprocess are inherited subprocess are inherited
* "(not applicable)": on UNIX, the multiprocessing uses ``fork()``, * "not applicable": on UNIX, the multiprocessing uses ``fork()``,
so this case is not concerned by this PEP. so this case is not affected by this PEP.
Closing All Open File Descriptors Closing All Open File Descriptors
--------------------------------- ---------------------------------
On UNIX, the ``subprocess`` module closes almost all file descriptors in On UNIX, the ``subprocess`` module closes almost all file descriptors in
the child process. This operation require MAXFD system calls, where the child process. This operation requires MAXFD system calls, where
MAXFD is the maximum number of file descriptors, even if there are only MAXFD is the maximum number of file descriptors, even if there are only
few open file descriptors. This maximum can be read using: few open file descriptors. This maximum can be read using:
``os.sysconf("SC_OPEN_MAX")``. ``os.sysconf("SC_OPEN_MAX")``.
@ -547,10 +547,10 @@ Developers are encouraged to reuse the high-level Python module
portable way. portable way.
Applications using the ``subprocess`` module with the *pass_fds* Applications using the ``subprocess`` module with the *pass_fds*
parameter or using ``os.dup2()`` to redirect standard streams should not parameter or using only ``os.dup2()`` to redirect standard streams should
be affected. not be affected.
Python does no more conform to POSIX, since file descriptors are now Python no longer conform to POSIX, since file descriptors are now
made non-inheritable by default. Python was not designed to conform to made non-inheritable by default. Python was not designed to conform to
POSIX, but was designed to develop portable applications. POSIX, but was designed to develop portable applications.
@ -587,8 +587,8 @@ Read the mail thread: `[Python-Dev] Proposal for a new function
PEP 433 PEP 433
------- -------
The PEP 433 entitled "Easier suppression of file descriptor inheritance" PEP 433, "Easier suppression of file descriptor inheritance",
is a previous attempt proposing various other alternatives, but no was a previous attempt proposing various other alternatives, but no
consensus could be reached. consensus could be reached.