2013-01-12 15:15:11 -05:00
|
|
|
PEP: 433
|
2013-01-18 06:45:30 -05:00
|
|
|
Title: Easier suppression of file descriptor inheritance
|
2013-01-12 15:15:11 -05:00
|
|
|
Version: $Revision$
|
|
|
|
Last-Modified: $Date$
|
|
|
|
Author: Victor Stinner <victor.stinner@gmail.com>
|
|
|
|
Status: Draft
|
|
|
|
Type: Standards Track
|
|
|
|
Content-Type: text/x-rst
|
|
|
|
Created: 10-January-2013
|
|
|
|
Python-Version: 3.4
|
|
|
|
|
|
|
|
|
|
|
|
Abstract
|
|
|
|
========
|
|
|
|
|
2013-01-13 08:53:59 -05:00
|
|
|
This PEP proposes to add a new optional parameter ``cloexec`` on
|
2013-01-12 19:30:02 -05:00
|
|
|
functions creating file descriptors in the Python standard library. If
|
2013-01-13 08:53:59 -05:00
|
|
|
the parameter is ``True``, the close-on-exec flag will be set on the
|
2013-01-12 19:30:02 -05:00
|
|
|
new file descriptor.
|
2013-01-12 15:15:11 -05:00
|
|
|
|
|
|
|
|
|
|
|
Rationale
|
|
|
|
=========
|
|
|
|
|
2013-01-24 18:33:38 -05:00
|
|
|
A file descriptor has a close-on-exec flag which indicates if the file
|
|
|
|
descriptor will be inherited or not.
|
2013-01-13 08:53:59 -05:00
|
|
|
|
2013-01-24 18:33:38 -05:00
|
|
|
On UNIX, the file descriptor will be closed on the execution of child processes
|
|
|
|
if the close-on-exec flag is set, the file descriptor is inherited by child
|
|
|
|
processes if the flag is cleared.
|
|
|
|
|
|
|
|
On Windows, the file descriptor is not inherited if the close-on-exec flag is
|
|
|
|
set, the file descriptor is inherited by child processes if the flag is cleared
|
|
|
|
and if ``CreateProcess()`` is called with the *bInheritHandles* parameter set
|
|
|
|
to ``TRUE`` (when ``subprocess.Popen`` is created with ``close_fds=False`` for
|
|
|
|
example).
|
|
|
|
|
|
|
|
|
|
|
|
Status in Python 3.3
|
|
|
|
--------------------
|
|
|
|
|
|
|
|
On UNIX, the subprocess module closes file descriptors greater than 2 by
|
|
|
|
default since Python 3.2 [#subprocess_close]_. All file descriptors created by
|
2013-01-13 08:53:59 -05:00
|
|
|
the parent process are automatically closed in the child process.
|
|
|
|
|
2013-01-12 19:30:02 -05:00
|
|
|
``xmlrpc.server.SimpleXMLRPCServer`` sets the close-on-exec flag of
|
|
|
|
the listening socket, the parent class ``socketserver.BaseServer``
|
|
|
|
does not set this flag.
|
2013-01-12 15:15:11 -05:00
|
|
|
|
2013-01-12 19:30:02 -05:00
|
|
|
There are other cases creating a subprocess or executing a new program
|
|
|
|
where file descriptors are not closed: functions of the os.spawn*()
|
|
|
|
family and third party modules calling ``exec()`` or ``fork()`` +
|
|
|
|
``exec()``. In this case, file descriptors are shared between the
|
|
|
|
parent and the child processes which is usually unexpected and causes
|
|
|
|
various issues.
|
2013-01-12 15:15:11 -05:00
|
|
|
|
2013-01-12 18:04:29 -05:00
|
|
|
This PEP proposes to continue the work started with the change in the
|
2013-01-12 19:30:02 -05:00
|
|
|
subprocess, to fix the issue in any code, and not just code using
|
|
|
|
subprocess.
|
2013-01-12 18:04:29 -05:00
|
|
|
|
2013-01-12 15:15:11 -05:00
|
|
|
|
|
|
|
Inherited file descriptors issues
|
|
|
|
---------------------------------
|
|
|
|
|
2013-01-12 19:30:02 -05:00
|
|
|
Closing the file descriptor in the parent process does not close the
|
|
|
|
related resource (file, socket, ...) because it is still open in the
|
|
|
|
child process.
|
2013-01-12 15:15:11 -05:00
|
|
|
|
2013-01-12 19:30:02 -05:00
|
|
|
The listening socket of TCPServer is not closed on ``exec()``: the
|
|
|
|
child process is able to get connection from new clients; if the
|
|
|
|
parent closes the listening socket and create a new listening socket
|
|
|
|
on the same address, it would get an "address already is used" error.
|
2013-01-12 15:15:11 -05:00
|
|
|
|
2013-01-12 19:30:02 -05:00
|
|
|
Not closing file descriptors can lead to resource exhaustion: even if
|
|
|
|
the parent closes all files, creating a new file descriptor may fail
|
|
|
|
with "too many files" because files are still open in the child
|
|
|
|
process.
|
2013-01-12 15:15:11 -05:00
|
|
|
|
2013-01-13 08:53:59 -05:00
|
|
|
See also the following issues:
|
|
|
|
|
|
|
|
* `Issue #2320: Race condition in subprocess using stdin
|
|
|
|
<http://bugs.python.org/issue2320>`_ (2008)
|
|
|
|
* `Issue #3006: subprocess.Popen causes socket to remain open after
|
|
|
|
close <http://bugs.python.org/issue3006>`_ (2008)
|
|
|
|
* `Issue #7213: subprocess leaks open file descriptors between Popen
|
|
|
|
instances causing hangs <http://bugs.python.org/issue7213>`_ (2009)
|
|
|
|
* `Issue #12786: subprocess wait() hangs when stdin is closed
|
|
|
|
<http://bugs.python.org/issue12786>`_ (2011)
|
|
|
|
|
2013-01-12 15:15:11 -05:00
|
|
|
|
|
|
|
Security
|
|
|
|
--------
|
|
|
|
|
2013-01-12 19:30:02 -05:00
|
|
|
Leaking file descriptors is a major security vulnerability. An
|
|
|
|
untrusted child process can read sensitive data like passwords and
|
|
|
|
take control of the parent process though leaked file descriptors. It
|
|
|
|
is for example a known vulnerability to escape from a chroot.
|
2013-01-12 15:15:11 -05:00
|
|
|
|
2013-01-13 08:53:59 -05:00
|
|
|
See also the CERT recommandation:
|
|
|
|
`FIO42-C. Ensure files are properly closed when they are no longer needed
|
|
|
|
<https://www.securecoding.cert.org/confluence/display/seccode/FIO42-C.+Ensure+files+are+properly+closed+when+they+are+no+longer+needed>`_.
|
|
|
|
|
|
|
|
|
|
|
|
Example of vulnerabilities:
|
|
|
|
|
|
|
|
|
|
|
|
* `OpenSSH Security Advisory: portable-keysign-rand-helper.adv
|
|
|
|
<http://www.openssh.com/txt/portable-keysign-rand-helper.adv>`_
|
|
|
|
(April 2011)
|
|
|
|
* `CWE-403: Exposure of File Descriptor to Unintended Control Sphere
|
|
|
|
<http://cwe.mitre.org/data/definitions/403.html>`_ (2008)
|
|
|
|
* `Hijacking Apache https by mod_php
|
|
|
|
<http://www.securityfocus.com/archive/1/348368>`_ (Dec 2003)
|
|
|
|
|
2013-01-12 15:15:11 -05:00
|
|
|
|
|
|
|
Atomicity
|
|
|
|
---------
|
|
|
|
|
2013-01-12 19:30:02 -05:00
|
|
|
Using ``fcntl()`` to set the close-on-exec flag is not safe in a
|
|
|
|
multithreaded application. If a thread calls ``fork()`` and ``exec()``
|
|
|
|
between the creation of the file descriptor and the call to
|
|
|
|
``fcntl(fd, F_SETFD, new_flags)``: the file descriptor will be
|
|
|
|
inherited by the child process. Modern operating systems offer
|
|
|
|
functions to set the flag during the creation of the file descriptor,
|
|
|
|
which avoids the race condition.
|
2013-01-12 15:15:11 -05:00
|
|
|
|
|
|
|
|
|
|
|
Portability
|
|
|
|
-----------
|
|
|
|
|
|
|
|
Python 3.2 added ``socket.SOCK_CLOEXEC`` flag, Python 3.3 added
|
2013-01-12 19:30:02 -05:00
|
|
|
``os.O_CLOEXEC`` flag and ``os.pipe2()`` function. It is already
|
|
|
|
possible to set atomically close-on-exec flag in Python 3.3 when
|
|
|
|
opening a file and creating a pipe or socket.
|
2013-01-12 15:15:11 -05:00
|
|
|
|
2013-01-12 19:30:02 -05:00
|
|
|
The problem is that these flags and functions are not portable: only
|
|
|
|
recent versions of operating systems support them. ``O_CLOEXEC`` and
|
|
|
|
``SOCK_CLOEXEC`` flags are ignored by old Linux versions and so
|
|
|
|
``FD_CLOEXEC`` flag must be checked using ``fcntl(fd, F_GETFD)``. If
|
|
|
|
the kernel ignores ``O_CLOEXEC`` or ``SOCK_CLOEXEC`` flag, a call to
|
|
|
|
``fcntl(fd, F_SETFD, flags)`` is required to set close-on-exec flag.
|
2013-01-12 15:15:11 -05:00
|
|
|
|
2013-01-12 19:30:02 -05:00
|
|
|
.. note::
|
|
|
|
OpenBSD older 5.2 does not close the file descriptor with
|
|
|
|
close-on-exec flag set if ``fork()`` is used before ``exec()``, but
|
|
|
|
it works correctly if ``exec()`` is called without ``fork()``.
|
2013-01-12 15:15:11 -05:00
|
|
|
|
2013-01-13 08:53:59 -05:00
|
|
|
XXX recheck this OpenBSD bug using a C program. XXX
|
|
|
|
|
2013-01-12 15:15:11 -05:00
|
|
|
|
|
|
|
Scope
|
|
|
|
-----
|
|
|
|
|
|
|
|
Applications still have to close explicitly file descriptors after a
|
|
|
|
``fork()``. The close-on-exec flag only closes file descriptors after
|
|
|
|
``exec()``, and so after ``fork()`` + ``exec()``.
|
|
|
|
|
2013-01-12 19:30:02 -05:00
|
|
|
This PEP only change the close-on-exec flag of file descriptors
|
|
|
|
created by the Python standard library, or by modules using the
|
|
|
|
standard library. Third party modules not using the standard library
|
|
|
|
should be modified to conform to this PEP. The new
|
|
|
|
``os.set_cloexec()`` function can be used for example.
|
2013-01-12 15:15:11 -05:00
|
|
|
|
|
|
|
Impacted functions:
|
|
|
|
|
|
|
|
* ``os.forkpty()``
|
|
|
|
* ``http.server.CGIHTTPRequestHandler.run_cgi()``
|
|
|
|
|
|
|
|
Impacted modules:
|
|
|
|
|
|
|
|
* ``multiprocessing``
|
|
|
|
* ``socketserver``
|
|
|
|
* ``subprocess``
|
|
|
|
* ``tempfile``
|
|
|
|
* ``xmlrpc.server``
|
|
|
|
* Maybe: ``signal``, ``threading``
|
|
|
|
|
2013-01-13 08:53:59 -05:00
|
|
|
XXX Should ``subprocess.Popen`` clear the close-on-exec flag on file
|
|
|
|
XXX descriptors of the constructor the ``pass_fds`` parameter?
|
2013-01-12 15:15:11 -05:00
|
|
|
|
2013-01-12 18:04:29 -05:00
|
|
|
.. note::
|
2013-01-12 19:34:36 -05:00
|
|
|
See `Close file descriptors after fork`_ for a possible solution
|
|
|
|
for ``fork()`` without ``exec()``.
|
2013-01-12 15:15:11 -05:00
|
|
|
|
2013-01-12 18:04:29 -05:00
|
|
|
|
|
|
|
Proposal
|
|
|
|
========
|
2013-01-12 15:15:11 -05:00
|
|
|
|
2013-01-13 08:53:59 -05:00
|
|
|
This PEP proposes to add a new optional parameter ``cloexec`` on
|
2013-01-12 19:34:36 -05:00
|
|
|
functions creating file descriptors in the Python standard library. If
|
2013-01-13 08:53:59 -05:00
|
|
|
the parameter is ``True``, the close-on-exec flag will be set on the
|
2013-01-12 19:34:36 -05:00
|
|
|
new file descriptor.
|
2013-01-12 15:15:11 -05:00
|
|
|
|
2013-01-18 06:45:30 -05:00
|
|
|
Add a new functions:
|
2013-01-12 15:15:11 -05:00
|
|
|
|
2013-01-18 06:45:30 -05:00
|
|
|
* ``os.get_cloexec(fd:int) -> bool``: get the
|
|
|
|
close-on-exec flag of a file descriptor. Not available on all platforms.
|
|
|
|
* ``os.set_cloexec(fd:int, cloexec:bool=True)``: set or clear the
|
|
|
|
close-on-exec flag on a file descriptor. Not available on all platforms.
|
2013-01-12 15:15:11 -05:00
|
|
|
|
2013-01-13 08:53:59 -05:00
|
|
|
Add a new optional ``cloexec`` parameter to:
|
2013-01-12 15:15:11 -05:00
|
|
|
|
|
|
|
* ``open()``: ``os.fdopen()`` is indirectly modified
|
|
|
|
* ``os.dup()``, ``os.dup2()``
|
|
|
|
* ``os.pipe()``
|
2013-01-12 19:30:02 -05:00
|
|
|
* ``socket.socket()``, ``socket.socketpair()``,
|
|
|
|
``socket.socket.accept()``
|
2013-01-12 15:15:11 -05:00
|
|
|
* Maybe also: ``os.open()``, ``os.openpty()``
|
|
|
|
* TODO:
|
|
|
|
|
|
|
|
* ``select.devpoll()``
|
|
|
|
* ``select.poll()``
|
|
|
|
* ``select.epoll()``
|
|
|
|
* ``select.kqueue()``
|
2013-01-12 19:30:02 -05:00
|
|
|
* ``socket.socket.recvmsg()``: use ``MSG_CMSG_CLOEXEC``,
|
|
|
|
or ``os.set_cloexec()``
|
2013-01-12 15:15:11 -05:00
|
|
|
|
2013-01-13 08:53:59 -05:00
|
|
|
The default value of the ``cloexec`` parameter is ``False`` to keep the
|
2013-01-12 19:30:02 -05:00
|
|
|
backward compatibility.
|
2013-01-12 15:15:11 -05:00
|
|
|
|
2013-01-12 19:30:02 -05:00
|
|
|
The close-on-exec flag will not be set on file descriptors 0 (stdin),
|
|
|
|
1 (stdout) and 2 (stderr), because these files are expected to be
|
|
|
|
inherited. It would still be possible to set close-on-exec flag
|
|
|
|
explicitly using ``os.set_cloexec()``.
|
2013-01-12 15:15:11 -05:00
|
|
|
|
2013-01-12 18:04:29 -05:00
|
|
|
Drawbacks:
|
2013-01-12 15:15:11 -05:00
|
|
|
|
2013-01-12 19:30:02 -05:00
|
|
|
* Many functions of the Python standard library creating file
|
2013-01-13 08:53:59 -05:00
|
|
|
descriptors cannot be changed by this proposal, because adding
|
|
|
|
a ``cloexec`` optional parameter would be surprising and too many
|
2013-01-12 19:30:02 -05:00
|
|
|
functions would need it. For example, ``os.urandom()`` uses a
|
|
|
|
temporary file on UNIX, but it calls a function of Windows API on
|
2013-01-13 08:53:59 -05:00
|
|
|
Windows. Adding a ``cloexec`` parameter to ``os.urandom()`` would
|
2013-01-25 12:32:01 -05:00
|
|
|
not make sense. See `Enable file descriptor inheritance by default`_
|
|
|
|
for an incomplete list of functions creating file descriptors.
|
2013-01-12 19:30:02 -05:00
|
|
|
* Checking if a module creates file descriptors is difficult. For
|
|
|
|
example, ``os.urandom()`` creates a file descriptor on UNIX to read
|
|
|
|
``/dev/urandom`` (and closes it at exit), whereas it is implemented
|
|
|
|
using a function call on Windows. It is not possible to control
|
|
|
|
close-on-exec flag of the file descriptor used by ``os.urandom()``,
|
|
|
|
because ``os.urandom()`` API does not allow it.
|
2013-01-12 15:15:11 -05:00
|
|
|
|
|
|
|
|
2013-01-12 18:04:29 -05:00
|
|
|
Alternatives
|
|
|
|
============
|
2013-01-12 15:15:11 -05:00
|
|
|
|
2013-01-18 06:45:30 -05:00
|
|
|
Bikeshedding on the name of the new parameter
|
|
|
|
---------------------------------------------
|
|
|
|
|
|
|
|
* ``inherit``, ``inherited``: closer to Windows definition
|
|
|
|
* ``sensitive``
|
|
|
|
* ``sterile``: "Does not produce offspring."
|
|
|
|
|
|
|
|
|
|
|
|
Enable file descriptor inheritance by default
|
|
|
|
---------------------------------------------
|
2013-01-12 15:15:11 -05:00
|
|
|
|
2013-01-13 08:53:59 -05:00
|
|
|
Set the close-on-exec flag by default on new file descriptors created
|
|
|
|
by Python. This alternative just changes the default value of the new
|
|
|
|
``cloexec`` parameter.
|
2013-01-12 15:15:11 -05:00
|
|
|
|
2013-01-12 19:30:02 -05:00
|
|
|
If a file must be inherited by child processes, ``cloexec=False``
|
2013-01-13 08:53:59 -05:00
|
|
|
parameter can be used.
|
2013-01-12 15:15:11 -05:00
|
|
|
|
2013-01-13 08:53:59 -05:00
|
|
|
``subprocess.Popen`` constructor has an ``pass_fds`` parameter to
|
2013-01-12 19:30:02 -05:00
|
|
|
specify which file descriptors must be inherited. The close-on-exec
|
|
|
|
flag of these file descriptors must be changed with
|
|
|
|
``os.set_cloexec()``.
|
2013-01-12 18:04:29 -05:00
|
|
|
|
2013-01-12 19:30:02 -05:00
|
|
|
Example of functions creating file descriptors which will be modified
|
|
|
|
to set close-on-exec flag:
|
2013-01-12 18:04:29 -05:00
|
|
|
|
|
|
|
* ``os.urandom()`` (on UNIX)
|
|
|
|
* ``curses.window.getwin()``, ``curses.window.putwin()``
|
|
|
|
* ``mmap.mmap()`` (if ``MAP_ANONYMOUS`` is not defined)
|
|
|
|
* ``oss.open()``
|
|
|
|
* ``Modules/main.c``: ``RunStartupFile()``
|
|
|
|
* ``Python/pythonrun.c``: ``PyRun_SimpleFileExFlags()``
|
|
|
|
* ``Modules/getpath.c``: ``search_for_exec_prefix()``
|
|
|
|
* ``Modules/zipimport.c``: ``read_directory()``
|
|
|
|
* ``Modules/_ssl.c``: ``load_dh_params()``
|
|
|
|
* ``PC/getpathp.c``: ``calculate_path()``
|
|
|
|
* ``Python/errors.c``: ``PyErr_ProgramText()``
|
|
|
|
* ``Python/import.c``: ``imp_load_dynamic()``
|
|
|
|
* TODO: ``PC/_msi.c``
|
|
|
|
|
|
|
|
Many functions are impacted indirectly by this alternative. Examples:
|
|
|
|
|
|
|
|
* ``logging.FileHandler``
|
|
|
|
|
|
|
|
Advantages of setting close-on-exec flag by default:
|
|
|
|
|
|
|
|
* There are far more programs that are bitten by FD inheritance upon
|
2013-01-12 19:30:02 -05:00
|
|
|
exec (see `Inherited file descriptors issues`_ and `Security`_)
|
|
|
|
than programs relying on it (see `Applications using inherance of
|
|
|
|
file descriptors`_).
|
2013-01-12 18:04:29 -05:00
|
|
|
|
|
|
|
Drawbacks of setting close-on-exec flag by default:
|
|
|
|
|
2013-01-12 19:30:02 -05:00
|
|
|
* The os module is written as a thin wrapper to system calls (to
|
|
|
|
functions of the C standard library). If atomic flags to set
|
|
|
|
close-on-exec flag are not supported (see `Appendix: Operating
|
|
|
|
system support`_), a single Python function call may call 2 or 3
|
|
|
|
system calls (see `Performances`_ section).
|
|
|
|
* Extra system calls, if any, may slow down Python: see
|
|
|
|
`Performances`_.
|
|
|
|
* It violates the principle of least surprise. Developers using the
|
|
|
|
os module may expect that Python respects the POSIX standard and so
|
|
|
|
that close-on-exec flag is not set by default.
|
2013-01-12 18:04:29 -05:00
|
|
|
|
|
|
|
Backward compatibility: only a few programs rely on inherance of file
|
2013-01-12 19:30:02 -05:00
|
|
|
descriptors, and they only pass a few file descriptors, usually just
|
|
|
|
one. These programs will fail immediatly with ``EBADF`` error, and it
|
2013-01-13 08:53:59 -05:00
|
|
|
will be simple to fix them: add ``cloexec=False`` parameter or use
|
2013-01-12 18:04:29 -05:00
|
|
|
``os.set_cloexec(fd, False)``.
|
|
|
|
|
2013-01-13 08:53:59 -05:00
|
|
|
The ``subprocess`` module will be changed anyway to clear
|
2013-01-12 19:30:02 -05:00
|
|
|
close-on-exec flag on file descriptors listed in the ``pass_fds``
|
2013-01-13 08:53:59 -05:00
|
|
|
parameter of Popen constructor. So it possible that these programs will
|
2013-01-12 19:30:02 -05:00
|
|
|
not need any fix if they use the ``subprocess`` module.
|
2013-01-12 18:04:29 -05:00
|
|
|
|
|
|
|
|
|
|
|
Add a function to set close-on-exec flag by default
|
|
|
|
---------------------------------------------------
|
|
|
|
|
2013-01-12 19:30:02 -05:00
|
|
|
An alternative is to add also a function to change globally the
|
|
|
|
default behaviour. It would be possible to set close-on-exec flag for
|
|
|
|
the whole application including all modules and the Python standard
|
|
|
|
library. This alternative is based on the `Proposal`_ and adds extra
|
|
|
|
changes.
|
2013-01-12 18:04:29 -05:00
|
|
|
|
2013-01-25 17:07:44 -05:00
|
|
|
New functions, command line argument and environment variable:
|
2013-01-12 18:04:29 -05:00
|
|
|
|
|
|
|
* ``sys.getdefaultcloexec() -> bool``: get the default value of the
|
2013-01-25 17:07:44 -05:00
|
|
|
*cloexec* parameter
|
|
|
|
* ``sys.setdefaultcloexec()``, ``-e`` command line option, ``PYTHONCLOEXEC``
|
|
|
|
environment variable (if set): set the default value of the *cloexec*
|
|
|
|
parameter to ``True``
|
2013-01-12 18:04:29 -05:00
|
|
|
|
2013-01-13 08:53:59 -05:00
|
|
|
The major change is that the default value of the ``cloexec`` parameter
|
2013-01-12 19:30:02 -05:00
|
|
|
is ``sys.getdefaultcloexec()``, instead of ``False``.
|
2013-01-12 18:04:29 -05:00
|
|
|
|
2013-01-25 17:07:44 -05:00
|
|
|
When ``sys.setdefaultcloexec()`` is called to set close-on-exec by default, we
|
|
|
|
have the same drawbacks as the `Enable file descriptor inheritance by default`_
|
|
|
|
alternative.
|
2013-01-12 18:04:29 -05:00
|
|
|
|
|
|
|
There are additionnal drawbacks of having two behaviours depending on
|
|
|
|
``sys.getdefaultcloexec()`` value:
|
|
|
|
|
2013-01-12 19:30:02 -05:00
|
|
|
* It is not more possible to know if the close-on-exec flag will be
|
|
|
|
set or not just by reading the source code.
|
2013-01-12 18:04:29 -05:00
|
|
|
|
|
|
|
|
|
|
|
Close file descriptors after fork
|
|
|
|
---------------------------------
|
|
|
|
|
2013-01-12 19:30:02 -05:00
|
|
|
This PEP does not fix issues with applications using ``fork()``
|
|
|
|
without ``exec()``. Python needs a generic process to register
|
2013-01-13 08:53:59 -05:00
|
|
|
callbacks which would be called after a fork, see `Add an 'atfork'
|
2013-01-12 19:30:02 -05:00
|
|
|
module`_. Such registry could be used to close file descriptors just
|
|
|
|
after a ``fork()``.
|
2013-01-12 18:04:29 -05:00
|
|
|
|
|
|
|
Drawbacks:
|
|
|
|
|
2013-01-12 19:30:02 -05:00
|
|
|
* This alternative does not solve the problem for programs using
|
|
|
|
``exec()`` without ``fork()``.
|
|
|
|
* A third party module may call directly the C function ``fork()``
|
|
|
|
which will not call "atfork" callbacks.
|
|
|
|
* All functions creating file descriptors must be changed to register
|
|
|
|
a callback and then unregister their callback when the file is
|
|
|
|
closed. Or a list of *all* open file descriptors must be
|
|
|
|
maintained.
|
|
|
|
* The operating system is a better place than Python to close
|
|
|
|
automatically file descriptors. For example, it is not easy to
|
|
|
|
avoid a race condition between closing the file and unregistering
|
|
|
|
the callback closing the file.
|
2013-01-12 18:04:29 -05:00
|
|
|
|
|
|
|
|
|
|
|
open(): add "e" flag to mode
|
|
|
|
----------------------------
|
|
|
|
|
|
|
|
A new "e" mode would set close-on-exec flag (best-effort).
|
2013-01-12 15:15:11 -05:00
|
|
|
|
2013-01-12 19:30:02 -05:00
|
|
|
This alternative only solves the problem for ``open()``.
|
2013-01-13 08:53:59 -05:00
|
|
|
socket.socket() and os.pipe() do not have a ``mode`` parameter for
|
2013-01-12 19:30:02 -05:00
|
|
|
example.
|
2013-01-12 15:15:11 -05:00
|
|
|
|
2013-01-12 19:30:02 -05:00
|
|
|
Since its version 2.7, the GNU libc supports ``"e"`` flag for
|
|
|
|
``fopen()``. It uses ``O_CLOEXEC`` if available, or use ``fcntl(fd,
|
|
|
|
F_SETFD, FD_CLOEXEC)``. With Visual Studio, fopen() accepts a "N"
|
|
|
|
flag which uses ``O_NOINHERIT``.
|
2013-01-12 15:15:11 -05:00
|
|
|
|
|
|
|
|
2013-01-12 18:04:29 -05:00
|
|
|
Applications using inherance of file descriptors
|
|
|
|
================================================
|
|
|
|
|
2013-01-12 19:30:02 -05:00
|
|
|
Most developers don't know that file descriptors are inherited by
|
|
|
|
default. Most programs do not rely on inherance of file descriptors.
|
|
|
|
For example, ``subprocess.Popen`` was changed in Python 3.2 to close
|
|
|
|
all file descriptors greater than 2 in the child process by default.
|
2013-01-13 08:53:59 -05:00
|
|
|
No user complained about this behavior change yet.
|
2013-01-12 19:30:02 -05:00
|
|
|
|
|
|
|
Network servers using fork may want to pass the client socket to the
|
|
|
|
child process. For example, on UNIX a CGI server pass the socket
|
|
|
|
client through file descriptors 0 (stdin) and 1 (stdout) using
|
|
|
|
``dup2()``. This specific case is not impacted by this PEP because the
|
|
|
|
close-on-exec flag is never set on file descriptors smaller than 3.
|
|
|
|
|
|
|
|
To access a restricted resource like creating a socket listening on a
|
|
|
|
TCP port lower than 1024 or reading a file containing sensitive data
|
|
|
|
like passwords, a common practice is: start as the root user, create a
|
|
|
|
file descriptor, create a child process, pass the file descriptor to
|
|
|
|
the child process and exit. Security is very important in such use
|
|
|
|
case: leaking another file descriptor would be a critical security
|
|
|
|
vulnerability (see `Security`_). The root process may not exit but
|
|
|
|
monitors the child process instead, and restarts a new child process
|
|
|
|
and pass the same file descriptor if the previous child process
|
2013-01-12 18:04:29 -05:00
|
|
|
crashed.
|
|
|
|
|
2013-01-12 19:30:02 -05:00
|
|
|
Example of programs taking file descriptors from the parent process
|
|
|
|
using a command line option:
|
2013-01-12 18:04:29 -05:00
|
|
|
|
|
|
|
* gpg: ``--status-fd <fd>``, ``--logger-fd <fd>``, etc.
|
|
|
|
* openssl: ``-pass fd:<fd>``
|
|
|
|
* qemu: ``-add-fd <fd>``
|
|
|
|
* valgrind: ``--log-fd=<fd>``, ``--input-fd=<fd>``, etc.
|
|
|
|
* xterm: ``-S <fd>``
|
|
|
|
|
2013-01-12 19:30:02 -05:00
|
|
|
On Linux, it is possible to use ``"/dev/fd/<fd>"`` filename to pass a
|
|
|
|
file descriptor to a program expecting a filename.
|
2013-01-12 15:15:11 -05:00
|
|
|
|
|
|
|
|
|
|
|
Performances
|
|
|
|
============
|
|
|
|
|
2013-01-12 19:30:02 -05:00
|
|
|
Setting close-on-exec flag may require additional system calls for
|
|
|
|
each creation of new file descriptors. The number of additional system
|
|
|
|
calls depends on the method used to set the flag:
|
2013-01-12 15:15:11 -05:00
|
|
|
|
|
|
|
* ``O_NOINHERIT``: no additionnal system call
|
2013-01-12 19:30:02 -05:00
|
|
|
* ``O_CLOEXEC``: one addition system call, but only at the creation
|
|
|
|
of the first file descriptor, to check if the flag is supported. If
|
|
|
|
no, Python has to fallback to the next method.
|
|
|
|
* ``ioctl(fd, FIOCLEX)``: one addition system call per file
|
|
|
|
descriptor
|
2013-01-12 15:15:11 -05:00
|
|
|
* ``fcntl(fd, F_SETFD, flags)``: two addition system calls per file
|
|
|
|
descriptor, one to get old flags and one to set new flags
|
|
|
|
|
|
|
|
XXX Benchmark the overhead for these 4 methods. XXX
|
|
|
|
|
|
|
|
|
|
|
|
Implementation
|
|
|
|
==============
|
|
|
|
|
2013-01-18 06:45:30 -05:00
|
|
|
os.get_cloexec(fd)
|
|
|
|
------------------
|
2013-01-12 15:15:11 -05:00
|
|
|
|
2013-01-18 06:45:30 -05:00
|
|
|
Get the close-on-exec flag of a file descriptor.
|
|
|
|
|
|
|
|
Pseudo-code::
|
|
|
|
|
|
|
|
if os.name == 'nt':
|
|
|
|
def get_cloexec(fd):
|
|
|
|
handle = _winapi._get_osfhandle(fd);
|
|
|
|
flags = _winapi.GetHandleInformation(handle)
|
|
|
|
return not(flags & _winapi.HANDLE_FLAG_INHERIT)
|
|
|
|
else:
|
|
|
|
try:
|
|
|
|
import fcntl
|
|
|
|
except ImportError:
|
|
|
|
pass
|
|
|
|
else:
|
|
|
|
def get_cloexec(fd):
|
|
|
|
flags = fcntl.fcntl(fd, fcntl.F_GETFD)
|
|
|
|
return bool(flags & fcntl.FD_CLOEXEC)
|
|
|
|
|
|
|
|
|
|
|
|
os.set_cloexec(fd, cloexec=True)
|
|
|
|
--------------------------------
|
|
|
|
|
|
|
|
Set or clear the close-on-exec flag on a file descriptor. The flag
|
|
|
|
is set after the creation of the file descriptor and so it is not
|
|
|
|
atomic.
|
|
|
|
|
|
|
|
Pseudo-code::
|
2013-01-12 15:15:11 -05:00
|
|
|
|
|
|
|
if os.name == 'nt':
|
|
|
|
def set_cloexec(fd, cloexec=True):
|
2013-01-18 06:45:30 -05:00
|
|
|
handle = _winapi._get_osfhandle(fd);
|
|
|
|
mask = _winapi.HANDLE_FLAG_INHERIT
|
|
|
|
if cloexec:
|
|
|
|
flags = 0
|
|
|
|
else:
|
|
|
|
flags = mask
|
|
|
|
_winapi.SetHandleInformation(handle, mask, flags)
|
2013-01-12 15:15:11 -05:00
|
|
|
else:
|
|
|
|
fnctl = None
|
|
|
|
ioctl = None
|
|
|
|
try:
|
|
|
|
import ioctl
|
|
|
|
except ImportError:
|
|
|
|
try:
|
|
|
|
import fcntl
|
|
|
|
except ImportError:
|
|
|
|
pass
|
|
|
|
if ioctl is not None and hasattr('FIOCLEX', ioctl):
|
|
|
|
def set_cloexec(fd, cloexec=True):
|
|
|
|
if cloexec:
|
|
|
|
ioctl.ioctl(fd, ioctl.FIOCLEX)
|
|
|
|
else:
|
|
|
|
ioctl.ioctl(fd, ioctl.FIONCLEX)
|
|
|
|
elif fnctl is not None:
|
|
|
|
def set_cloexec(fd, cloexec=True):
|
|
|
|
flags = fcntl.fcntl(fd, fcntl.F_GETFD)
|
|
|
|
if cloexec:
|
|
|
|
flags |= FD_CLOEXEC
|
|
|
|
else:
|
|
|
|
flags &= ~FD_CLOEXEC
|
|
|
|
fcntl.fcntl(fd, fcntl.F_SETFD, flags)
|
|
|
|
|
2013-01-12 19:30:02 -05:00
|
|
|
ioctl is preferred over fcntl because it requires only one syscall,
|
|
|
|
instead of two syscalls for fcntl.
|
2013-01-12 15:15:11 -05:00
|
|
|
|
2013-01-12 19:30:02 -05:00
|
|
|
.. note::
|
|
|
|
``fcntl(fd, F_SETFD, flags)`` only supports one flag
|
|
|
|
(``FD_CLOEXEC``), so it would be possible to avoid ``fcntl(fd,
|
|
|
|
F_GETFD)``. But it may drop other flags in the future, and so it is
|
|
|
|
safer to keep the two functions calls.
|
|
|
|
|
|
|
|
.. note::
|
|
|
|
``fopen()`` function of the GNU libc ignores the error if
|
|
|
|
``fcntl(fd, F_SETFD, flags)`` failed.
|
2013-01-12 15:15:11 -05:00
|
|
|
|
|
|
|
open()
|
|
|
|
------
|
|
|
|
|
|
|
|
* Windows: ``open()`` with ``O_NOINHERIT`` flag [atomic]
|
|
|
|
* ``open()`` with ``O_CLOEXEC flag`` [atomic]
|
|
|
|
* ``open()`` + ``os.set_cloexec(fd, True)`` [best-effort]
|
|
|
|
|
|
|
|
os.dup()
|
|
|
|
--------
|
|
|
|
|
|
|
|
* ``fcntl(fd, F_DUPFD_CLOEXEC)`` [atomic]
|
|
|
|
* ``dup()`` + ``os.set_cloexec(fd, True)`` [best-effort]
|
|
|
|
|
|
|
|
os.dup2()
|
|
|
|
---------
|
|
|
|
|
|
|
|
* ``dup3()`` with ``O_CLOEXEC`` flag [atomic]
|
|
|
|
* ``dup2()`` + ``os.set_cloexec(fd, True)`` [best-effort]
|
|
|
|
|
|
|
|
os.pipe()
|
|
|
|
---------
|
|
|
|
|
2013-01-18 06:45:30 -05:00
|
|
|
* Windows: ``CreatePipe()`` with ``SECURITY_ATTRIBUTES.bInheritHandle=TRUE``,
|
|
|
|
or ``_pipe()`` with ``O_NOINHERIT`` flag [atomic]
|
2013-01-12 15:15:11 -05:00
|
|
|
* ``pipe2()`` with ``O_CLOEXEC`` flag [atomic]
|
|
|
|
* ``pipe()`` + ``os.set_cloexec(fd, True)`` [best-effort]
|
|
|
|
|
|
|
|
socket.socket()
|
|
|
|
---------------
|
|
|
|
|
2013-01-18 06:45:30 -05:00
|
|
|
* Windows: ``WSASocket()`` with ``WSA_FLAG_NO_HANDLE_INHERIT`` flag
|
2013-01-12 15:15:11 -05:00
|
|
|
* ``socket()`` with ``SOCK_CLOEXEC`` flag [atomic]
|
|
|
|
* ``socket()`` + ``os.set_cloexec(fd, True)`` [best-effort]
|
|
|
|
|
|
|
|
socket.socketpair()
|
|
|
|
-------------------
|
|
|
|
|
|
|
|
* ``socketpair()`` with ``SOCK_CLOEXEC`` flag [atomic]
|
|
|
|
* ``socketpair()`` + ``os.set_cloexec(fd, True)`` [best-effort]
|
|
|
|
|
|
|
|
socket.socket.accept()
|
|
|
|
----------------------
|
|
|
|
|
|
|
|
* ``accept4()`` with ``SOCK_CLOEXEC`` flag [atomic]
|
|
|
|
* ``accept()`` + ``os.set_cloexec(fd, True)`` [best-effort]
|
|
|
|
|
|
|
|
|
|
|
|
Backward compatibility
|
|
|
|
======================
|
|
|
|
|
2013-01-12 19:30:02 -05:00
|
|
|
There is no backward incompatible change. The default behaviour is
|
|
|
|
unchanged: the close-on-exec flag is not set by default.
|
2013-01-12 15:15:11 -05:00
|
|
|
|
|
|
|
|
|
|
|
Appendix: Operating system support
|
|
|
|
==================================
|
|
|
|
|
|
|
|
Windows
|
|
|
|
-------
|
|
|
|
|
2013-01-12 19:30:02 -05:00
|
|
|
Windows has an ``O_NOINHERIT`` flag: "Do not inherit in child
|
|
|
|
processes".
|
2013-01-12 15:15:11 -05:00
|
|
|
|
|
|
|
For example, it is supported by ``open()`` and ``_pipe()``.
|
|
|
|
|
|
|
|
The value of the flag can be modified using:
|
|
|
|
``SetHandleInformation(fd, HANDLE_FLAG_INHERIT, 1)``.
|
|
|
|
|
2013-01-13 08:53:59 -05:00
|
|
|
``CreateProcess()`` has an ``bInheritHandles`` parameter: if it is
|
2013-01-12 19:30:02 -05:00
|
|
|
FALSE, the handles are not inherited. It is used by
|
|
|
|
``subprocess.Popen`` with ``close_fds`` option.
|
2013-01-12 15:15:11 -05:00
|
|
|
|
|
|
|
fcntl
|
|
|
|
-----
|
|
|
|
|
|
|
|
Functions:
|
|
|
|
|
|
|
|
* ``fcntl(fd, F_GETFD)``
|
|
|
|
* ``fcntl(fd, F_SETFD, flags | FD_CLOEXEC)``
|
|
|
|
|
2013-01-12 19:30:02 -05:00
|
|
|
Availability: AIX, Digital UNIX, FreeBSD, HP-UX, IRIX, Linux, Mac OS
|
|
|
|
X, OpenBSD, Solaris, SunOS, Unicos.
|
2013-01-12 15:15:11 -05:00
|
|
|
|
|
|
|
ioctl
|
|
|
|
-----
|
|
|
|
|
|
|
|
Functions:
|
|
|
|
|
|
|
|
* ``ioctl(fd, FIOCLEX, 0)`` sets close-on-exec flag
|
2013-01-13 08:53:59 -05:00
|
|
|
* ``ioctl(fd, FIONCLEX, 0)`` clears close-on-exec flag
|
2013-01-12 15:15:11 -05:00
|
|
|
|
|
|
|
Availability: Linux, Mac OS X, QNX, NetBSD, OpenBSD, FreeBSD.
|
|
|
|
|
|
|
|
|
|
|
|
Atomic flags
|
|
|
|
------------
|
|
|
|
|
|
|
|
New flags:
|
|
|
|
|
2013-01-12 19:30:02 -05:00
|
|
|
* ``O_CLOEXEC``: available on Linux (2.6.23+), FreeBSD (8.3+),
|
|
|
|
OpenBSD 5.0, QNX, BeOS, next NetBSD release (6.1?). This flag is
|
|
|
|
part of POSIX.1-2008.
|
2013-01-18 06:45:30 -05:00
|
|
|
* ``SOCK_CLOEXEC`` flag for ``socket()`` and ``socketpair()``,
|
|
|
|
available on Linux 2.6.27+, OpenBSD 5.2, NetBSD 6.0.
|
|
|
|
* ``WSA_FLAG_NO_HANDLE_INHERIT`` flag for ``WSASocket()``: supported on
|
|
|
|
Windows 7 with SP1, Windows Server 2008 R2 with SP1, and later
|
2013-01-12 15:15:11 -05:00
|
|
|
* ``fcntl()``: ``F_DUPFD_CLOEXEC`` flag, available on Linux 2.6.24+,
|
2013-01-12 19:30:02 -05:00
|
|
|
OpenBSD 5.0, FreeBSD 9.1, NetBSD 6.0. This flag is part of
|
|
|
|
POSIX.1-2008.
|
|
|
|
* ``recvmsg()``: ``MSG_CMSG_CLOEXEC``, available on Linux 2.6.23+,
|
|
|
|
NetBSD 6.0.
|
2013-01-12 15:15:11 -05:00
|
|
|
|
2013-01-12 19:30:02 -05:00
|
|
|
On Linux older than 2.6.23, ``O_CLOEXEC`` flag is simply ignored. So
|
|
|
|
we have to check that the flag is supported by calling ``fcntl()``. If
|
|
|
|
it does not work, we have to set the flag using ``fcntl()``.
|
2013-01-12 15:15:11 -05:00
|
|
|
|
2013-01-12 19:30:02 -05:00
|
|
|
XXX what is the behaviour on Linux older than 2.6.27
|
|
|
|
XXX with SOCK_CLOEXEC? XXX
|
2013-01-12 15:15:11 -05:00
|
|
|
|
|
|
|
New functions:
|
|
|
|
|
|
|
|
* ``dup3()``: available on Linux 2.6.27+ (and glibc 2.9)
|
|
|
|
* ``pipe2()``: available on Linux 2.6.27+ (and glibc 2.9)
|
|
|
|
* ``accept4()``: available on Linux 2.6.28+ (and glibc 2.10)
|
|
|
|
|
2013-01-12 19:30:02 -05:00
|
|
|
If ``accept4()`` is called on Linux older than 2.6.28, ``accept4()``
|
|
|
|
returns ``-1`` (fail) and errno is set to ``ENOSYS``.
|
2013-01-12 15:15:11 -05:00
|
|
|
|
|
|
|
|
|
|
|
Links
|
|
|
|
=====
|
|
|
|
|
|
|
|
Links:
|
|
|
|
|
|
|
|
* `Secure File Descriptor Handling
|
2013-01-12 19:30:02 -05:00
|
|
|
<http://udrepper.livejournal.com/20407.html>`_ (Ulrich Drepper,
|
|
|
|
2008)
|
2013-01-12 15:15:11 -05:00
|
|
|
* `win32_support.py of the Tornado project
|
|
|
|
<https://bitbucket.org/pvl/gaeseries-tornado/src/c2671cea1842/tornado/win32_support.py>`_:
|
|
|
|
emulate fcntl(fd, F_SETFD, FD_CLOEXEC) using
|
|
|
|
``SetHandleInformation(fd, HANDLE_FLAG_INHERIT, 1)``
|
|
|
|
|
|
|
|
Python issues:
|
|
|
|
|
|
|
|
* `open() does not able to set flags, such as O_CLOEXEC
|
|
|
|
<http://bugs.python.org/issue12105>`_
|
|
|
|
* `Add "e" mode to open(): close-and-exec (O_CLOEXEC) / O_NOINHERIT
|
|
|
|
<http://bugs.python.org/issue16850>`_
|
|
|
|
* `TCP listening sockets created without FD_CLOEXEC flag
|
|
|
|
<http://bugs.python.org/issue12107>`_
|
|
|
|
* `Use O_CLOEXEC in the tempfile module
|
|
|
|
<http://bugs.python.org/issue16860>`_
|
|
|
|
* `Support accept4() for atomic setting of flags at socket creation
|
|
|
|
<http://bugs.python.org/issue10115>`_
|
2013-01-13 08:53:59 -05:00
|
|
|
* `Add an 'atfork' module
|
2013-01-12 15:15:11 -05:00
|
|
|
<http://bugs.python.org/issue16500>`_
|
|
|
|
|
|
|
|
Ruby:
|
|
|
|
|
|
|
|
* `Set FD_CLOEXEC for all fds (except 0, 1, 2)
|
|
|
|
<http://bugs.ruby-lang.org/issues/5041>`_
|
|
|
|
* `O_CLOEXEC flag missing for Kernel::open
|
|
|
|
<http://bugs.ruby-lang.org/issues/1291>`_:
|
|
|
|
`commit reverted
|
2013-01-12 19:30:02 -05:00
|
|
|
<http://bugs.ruby-lang.org/projects/ruby-trunk/repository/revisions/31643>`_
|
|
|
|
later
|
2013-01-12 15:15:11 -05:00
|
|
|
|
|
|
|
Footnotes
|
|
|
|
=========
|
|
|
|
|
2013-01-12 19:30:02 -05:00
|
|
|
.. [#subprocess_close] On UNIX since Python 3.2, subprocess.Popen()
|
|
|
|
closes all file descriptors by default: ``close_fds=True``. It
|
|
|
|
closes file descriptors in range 3 inclusive to ``local_max_fd``
|
|
|
|
exclusive, where ``local_max_fd`` is ``fcntl(0, F_MAXFD)`` on
|
|
|
|
NetBSD, or ``sysconf(_SC_OPEN_MAX)`` otherwise. If the error pipe
|
|
|
|
has a descriptor smaller than 3, ``ValueError`` is raised.
|
|
|
|
|
2013-01-12 15:15:11 -05:00
|
|
|
|