750 lines
25 KiB
ReStructuredText
750 lines
25 KiB
ReStructuredText
PEP: 433
|
||
Title: Easier suppression of file descriptor inheritance
|
||
Version: $Revision$
|
||
Last-Modified: $Date$
|
||
Author: Victor Stinner <vstinner@python.org>
|
||
Status: Superseded
|
||
Type: Standards Track
|
||
Content-Type: text/x-rst
|
||
Created: 10-Jan-2013
|
||
Python-Version: 3.4
|
||
Superseded-By: 446
|
||
|
||
|
||
Abstract
|
||
========
|
||
|
||
Add a new optional *cloexec* parameter on functions creating file
|
||
descriptors, add different ways to change default values of this
|
||
parameter, and add four new functions:
|
||
|
||
* ``os.get_cloexec(fd)``
|
||
* ``os.set_cloexec(fd, cloexec=True)``
|
||
* ``sys.getdefaultcloexec()``
|
||
* ``sys.setdefaultcloexec(cloexec)``
|
||
|
||
|
||
Rationale
|
||
=========
|
||
|
||
A file descriptor has a close-on-exec flag which indicates if the file
|
||
descriptor will be inherited or not.
|
||
|
||
On UNIX, if the close-on-exec flag is set, the file descriptor is not
|
||
inherited: it will be closed at the execution of child processes;
|
||
otherwise the file descriptor is inherited by child processes.
|
||
|
||
On Windows, if the close-on-exec flag is set, the file descriptor is not
|
||
inherited; the file descriptor is inherited by child processes if the
|
||
close-on-exec flag is cleared and if ``CreateProcess()`` is called with
|
||
the *bInheritHandles* parameter set to ``TRUE`` (when
|
||
``subprocess.Popen`` is created with ``close_fds=False`` for example).
|
||
Windows does not have "close-on-exec" flag but an inheritance flag which
|
||
is just the opposite value. For example, setting close-on-exec flag
|
||
means clearing the ``HANDLE_FLAG_INHERIT`` flag of a handle.
|
||
|
||
|
||
Status in Python 3.3
|
||
--------------------
|
||
|
||
On UNIX, the subprocess module closes file descriptors greater than 2 by
|
||
default since Python 3.2 [#subprocess_close]_. All file descriptors
|
||
created by the parent process are automatically closed in the child
|
||
process.
|
||
|
||
``xmlrpc.server.SimpleXMLRPCServer`` sets the close-on-exec flag of
|
||
the listening socket, the parent class ``socketserver.TCPServer``
|
||
does not set this flag.
|
||
|
||
There are other cases creating a subprocess or executing a new program
|
||
where file descriptors are not closed: functions of the ``os.spawn*()``
|
||
and the ``os.exec*()`` families and third party modules calling
|
||
``exec()`` or ``fork()`` + ``exec()``. In this case, file descriptors
|
||
are shared between the parent and the child processes which is usually
|
||
unexpected and causes various issues.
|
||
|
||
This PEP proposes to continue the work started with the change in the
|
||
subprocess in Python 3.2, to fix the issue in any code, and not just
|
||
code using subprocess.
|
||
|
||
|
||
Inherited file descriptors issues
|
||
---------------------------------
|
||
|
||
Closing the file descriptor in the parent process does not close the
|
||
related resource (file, socket, ...) because it is still open in the
|
||
child process.
|
||
|
||
The listening socket of TCPServer is not closed on ``exec()``: the child
|
||
process is able to get connection from new clients; if the parent closes
|
||
the listening socket and create a new listening socket on the same
|
||
address, it would get an "address already is used" error.
|
||
|
||
Not closing file descriptors can lead to resource exhaustion: even if
|
||
the parent closes all files, creating a new file descriptor may fail
|
||
with "too many files" because files are still open in the child process.
|
||
|
||
See also the following issues:
|
||
|
||
* `Issue #2320: Race condition in subprocess using stdin
|
||
<http://bugs.python.org/issue2320>`_ (2008)
|
||
* `Issue #3006: subprocess.Popen causes socket to remain open after
|
||
close <http://bugs.python.org/issue3006>`_ (2008)
|
||
* `Issue #7213: subprocess leaks open file descriptors between Popen
|
||
instances causing hangs <http://bugs.python.org/issue7213>`_ (2009)
|
||
* `Issue #12786: subprocess wait() hangs when stdin is closed
|
||
<http://bugs.python.org/issue12786>`_ (2011)
|
||
|
||
|
||
Security
|
||
--------
|
||
|
||
Leaking file descriptors is a major security vulnerability. An
|
||
untrusted child process can read sensitive data like passwords and
|
||
take control of the parent process though leaked file descriptors. It
|
||
is for example a known vulnerability to escape from a chroot.
|
||
|
||
See also the CERT recommendation:
|
||
`FIO42-C. Ensure files are properly closed when they are no longer needed
|
||
<https://www.securecoding.cert.org/confluence/display/seccode/FIO42-C.+Ensure+files+are+properly+closed+when+they+are+no+longer+needed>`_.
|
||
|
||
|
||
Example of vulnerabilities:
|
||
|
||
|
||
* `OpenSSH Security Advisory: portable-keysign-rand-helper.adv
|
||
<http://www.openssh.com/txt/portable-keysign-rand-helper.adv>`_
|
||
(April 2011)
|
||
* `CWE-403: Exposure of File Descriptor to Unintended Control Sphere
|
||
<http://cwe.mitre.org/data/definitions/403.html>`_ (2008)
|
||
* `Hijacking Apache https by mod_php
|
||
<http://www.securityfocus.com/archive/1/348368>`_ (Dec 2003)
|
||
|
||
* Apache: `Apr should set FD_CLOEXEC if APR_FOPEN_NOCLEANUP is not set
|
||
<https://issues.apache.org/bugzilla/show_bug.cgi?id=46425>`_
|
||
(fixed in 2009)
|
||
* PHP: `system() (and similar) don't cleanup opened handles of Apache
|
||
<https://bugs.php.net/bug.php?id=38915>`_ (not fixed in January
|
||
2013)
|
||
|
||
|
||
Atomicity
|
||
---------
|
||
|
||
Using ``fcntl()`` to set the close-on-exec flag is not safe in a
|
||
multithreaded application. If a thread calls ``fork()`` and ``exec()``
|
||
between the creation of the file descriptor and the call to
|
||
``fcntl(fd, F_SETFD, new_flags)``: the file descriptor will be
|
||
inherited by the child process. Modern operating systems offer
|
||
functions to set the flag during the creation of the file descriptor,
|
||
which avoids the race condition.
|
||
|
||
|
||
Portability
|
||
-----------
|
||
|
||
Python 3.2 added ``socket.SOCK_CLOEXEC`` flag, Python 3.3 added
|
||
``os.O_CLOEXEC`` flag and ``os.pipe2()`` function. It is already
|
||
possible to set atomically close-on-exec flag in Python 3.3 when
|
||
opening a file and creating a pipe or socket.
|
||
|
||
The problem is that these flags and functions are not portable: only
|
||
recent versions of operating systems support them. ``O_CLOEXEC`` and
|
||
``SOCK_CLOEXEC`` flags are ignored by old Linux versions and so
|
||
``FD_CLOEXEC`` flag must be checked using ``fcntl(fd, F_GETFD)``. If
|
||
the kernel ignores ``O_CLOEXEC`` or ``SOCK_CLOEXEC`` flag, a call to
|
||
``fcntl(fd, F_SETFD, flags)`` is required to set close-on-exec flag.
|
||
|
||
.. note::
|
||
OpenBSD older 5.2 does not close the file descriptor with
|
||
close-on-exec flag set if ``fork()`` is used before ``exec()``, but
|
||
it works correctly if ``exec()`` is called without ``fork()``. Try
|
||
`openbsd_bug.py <http://hg.python.org/peps/file/tip/pep-0433/openbsd_bug.py>`_.
|
||
|
||
|
||
Scope
|
||
-----
|
||
|
||
Applications still have to close explicitly file descriptors after a
|
||
``fork()``. The close-on-exec flag only closes file descriptors after
|
||
``exec()``, and so after ``fork()`` + ``exec()``.
|
||
|
||
This PEP only change the close-on-exec flag of file descriptors
|
||
created by the Python standard library, or by modules using the
|
||
standard library. Third party modules not using the standard library
|
||
should be modified to conform to this PEP. The new
|
||
``os.set_cloexec()`` function can be used for example.
|
||
|
||
.. note::
|
||
See `Close file descriptors after fork`_ for a possible solution
|
||
for ``fork()`` without ``exec()``.
|
||
|
||
|
||
Proposal
|
||
========
|
||
|
||
Add a new optional *cloexec* parameter on functions creating file
|
||
descriptors and different ways to change default value of this
|
||
parameter.
|
||
|
||
Add new functions:
|
||
|
||
* ``os.get_cloexec(fd:int) -> bool``: get the
|
||
close-on-exec flag of a file descriptor. Not available on all
|
||
platforms.
|
||
* ``os.set_cloexec(fd:int, cloexec:bool=True)``: set or clear the
|
||
close-on-exec flag on a file descriptor. Not available on all
|
||
platforms.
|
||
* ``sys.getdefaultcloexec() -> bool``: get the current default value
|
||
of the *cloexec* parameter
|
||
* ``sys.setdefaultcloexec(cloexec: bool)``: set the default value
|
||
of the *cloexec* parameter
|
||
|
||
Add a new optional *cloexec* parameter to:
|
||
|
||
* ``asyncore.dispatcher.create_socket()``
|
||
* ``io.FileIO``
|
||
* ``io.open()``
|
||
* ``open()``
|
||
* ``os.dup()``
|
||
* ``os.dup2()``
|
||
* ``os.fdopen()``
|
||
* ``os.open()``
|
||
* ``os.openpty()``
|
||
* ``os.pipe()``
|
||
* ``select.devpoll()``
|
||
* ``select.epoll()``
|
||
* ``select.kqueue()``
|
||
* ``socket.socket()``
|
||
* ``socket.socket.accept()``
|
||
* ``socket.socket.dup()``
|
||
* ``socket.socket.fromfd``
|
||
* ``socket.socketpair()``
|
||
|
||
The default value of the *cloexec* parameter is
|
||
``sys.getdefaultcloexec()``.
|
||
|
||
Add a new command line option ``-e`` and an environment variable
|
||
``PYTHONCLOEXEC`` to the set close-on-exec flag by default.
|
||
|
||
``subprocess`` clears the close-on-exec flag of file descriptors of the
|
||
``pass_fds`` parameter.
|
||
|
||
All functions creating file descriptors in the standard library must
|
||
respect the default value of the *cloexec* parameter:
|
||
``sys.getdefaultcloexec()``.
|
||
|
||
File descriptors 0 (stdin), 1 (stdout) and 2 (stderr) are expected to be
|
||
inherited, but Python does not handle them differently. When
|
||
``os.dup2()`` is used to replace standard streams, ``cloexec=False``
|
||
must be specified explicitly.
|
||
|
||
Drawbacks of the proposal:
|
||
|
||
* It is not more possible to know if the close-on-exec flag will be
|
||
set or not on a newly created file descriptor just by reading the
|
||
source code.
|
||
* If the inheritance of a file descriptor matters, the *cloexec*
|
||
parameter must now be specified explicitly, or the library or the
|
||
application will not work depending on the default value of the
|
||
*cloexec* parameter.
|
||
|
||
|
||
Alternatives
|
||
============
|
||
|
||
Inheritance enabled by default, default no configurable
|
||
-------------------------------------------------------
|
||
|
||
Add a new optional parameter *cloexec* on functions creating file
|
||
descriptors. The default value of the *cloexec* parameter is ``False``,
|
||
and this default cannot be changed. File descriptor inheritance enabled by
|
||
default is also the default on POSIX and on Windows. This alternative is
|
||
the most conservative option.
|
||
|
||
This option does not solve issues listed in the `Rationale`_
|
||
section, it only provides a helper to fix them. All functions creating
|
||
file descriptors have to be modified to set *cloexec=True* in each
|
||
module used by an application to fix all these issues.
|
||
|
||
|
||
Inheritance enabled by default, default can only be set to True
|
||
---------------------------------------------------------------
|
||
|
||
This alternative is based on the proposal: the only difference is that
|
||
``sys.setdefaultcloexec()`` does not take any argument, it can only be
|
||
used to set the default value of the *cloexec* parameter to ``True``.
|
||
|
||
|
||
Disable inheritance by default
|
||
------------------------------
|
||
|
||
This alternative is based on the proposal: the only difference is that
|
||
the default value of the *cloexec* parameter is ``True`` (instead of
|
||
``False``).
|
||
|
||
If a file must be inherited by child processes, ``cloexec=False``
|
||
parameter can be used.
|
||
|
||
Advantages of setting close-on-exec flag by default:
|
||
|
||
* There are far more programs that are bitten by FD inheritance upon
|
||
exec (see `Inherited file descriptors issues`_ and `Security`_)
|
||
than programs relying on it (see `Applications using inheritance of
|
||
file descriptors`_).
|
||
|
||
Drawbacks of setting close-on-exec flag by default:
|
||
|
||
* It violates the principle of least surprise. Developers using the
|
||
os module may expect that Python respects the POSIX standard and so
|
||
that close-on-exec flag is not set by default.
|
||
* The os module is written as a thin wrapper to system calls (to
|
||
functions of the C standard library). If atomic flags to set
|
||
close-on-exec flag are not supported (see `Appendix: Operating
|
||
system support`_), a single Python function call may call 2 or 3
|
||
system calls (see `Performances`_ section).
|
||
* Extra system calls, if any, may slow down Python: see
|
||
`Performances`_.
|
||
|
||
Backward compatibility: only a few programs rely on inheritance of file
|
||
descriptors, and they only pass a few file descriptors, usually just
|
||
one. These programs will fail immediately with ``EBADF`` error, and it
|
||
will be simple to fix them: add ``cloexec=False`` parameter or use
|
||
``os.set_cloexec(fd, False)``.
|
||
|
||
The ``subprocess`` module will be changed anyway to clear
|
||
close-on-exec flag on file descriptors listed in the ``pass_fds``
|
||
parameter of Popen constructor. So it possible that these programs will
|
||
not need any fix if they use the ``subprocess`` module.
|
||
|
||
|
||
Close file descriptors after fork
|
||
---------------------------------
|
||
|
||
This PEP does not fix issues with applications using ``fork()``
|
||
without ``exec()``. Python needs a generic process to register
|
||
callbacks which would be called after a fork, see `#16500:
|
||
Add an atfork module`_. Such registry could be used to close file
|
||
descriptors just after a ``fork()``.
|
||
|
||
Drawbacks:
|
||
|
||
* It does not solve the problem on Windows: ``fork()`` does not exist
|
||
on Windows
|
||
* This alternative does not solve the problem for programs using
|
||
``exec()`` without ``fork()``.
|
||
* A third party module may call directly the C function ``fork()``
|
||
which will not call "atfork" callbacks.
|
||
* All functions creating file descriptors must be changed to register
|
||
a callback and then unregister their callback when the file is
|
||
closed. Or a list of *all* open file descriptors must be
|
||
maintained.
|
||
* The operating system is a better place than Python to close
|
||
automatically file descriptors. For example, it is not easy to
|
||
avoid a race condition between closing the file and unregistering
|
||
the callback closing the file.
|
||
|
||
|
||
open(): add "e" flag to mode
|
||
----------------------------
|
||
|
||
A new "e" mode would set close-on-exec flag (best-effort).
|
||
|
||
This alternative only solves the problem for ``open()``.
|
||
socket.socket() and os.pipe() do not have a ``mode`` parameter for
|
||
example.
|
||
|
||
Since its version 2.7, the GNU libc supports ``"e"`` flag for
|
||
``fopen()``. It uses ``O_CLOEXEC`` if available, or use ``fcntl(fd,
|
||
F_SETFD, FD_CLOEXEC)``. With Visual Studio, fopen() accepts a "N"
|
||
flag which uses ``O_NOINHERIT``.
|
||
|
||
|
||
Bikeshedding on the name of the new parameter
|
||
---------------------------------------------
|
||
|
||
* ``inherit``, ``inherited``: closer to Windows definition
|
||
* ``sensitive``
|
||
* ``sterile``: "Does not produce offspring."
|
||
|
||
|
||
|
||
|
||
Applications using inheritance of file descriptors
|
||
==================================================
|
||
|
||
Most developers don't know that file descriptors are inherited by
|
||
default. Most programs do not rely on inheritance of file descriptors.
|
||
For example, ``subprocess.Popen`` was changed in Python 3.2 to close
|
||
all file descriptors greater than 2 in the child process by default.
|
||
No user complained about this behavior change yet.
|
||
|
||
Network servers using fork may want to pass the client socket to the
|
||
child process. For example, on UNIX a CGI server pass the socket
|
||
client through file descriptors 0 (stdin) and 1 (stdout) using
|
||
``dup2()``.
|
||
|
||
To access a restricted resource like creating a socket listening on a
|
||
TCP port lower than 1024 or reading a file containing sensitive data
|
||
like passwords, a common practice is: start as the root user, create a
|
||
file descriptor, create a child process, drop privileges (ex: change the
|
||
current user), pass the file descriptor to the child process and exit
|
||
the parent process.
|
||
|
||
Security is very important in such use case: leaking another file
|
||
descriptor would be a critical security vulnerability (see `Security`_).
|
||
The root process may not exit but monitors the child process instead,
|
||
and restarts a new child process and pass the same file descriptor if
|
||
the previous child process crashed.
|
||
|
||
Example of programs taking file descriptors from the parent process
|
||
using a command line option:
|
||
|
||
* gpg: ``--status-fd <fd>``, ``--logger-fd <fd>``, etc.
|
||
* openssl: ``-pass fd:<fd>``
|
||
* qemu: ``-add-fd <fd>``
|
||
* valgrind: ``--log-fd=<fd>``, ``--input-fd=<fd>``, etc.
|
||
* xterm: ``-S <fd>``
|
||
|
||
On Linux, it is possible to use ``"/dev/fd/<fd>"`` filename to pass a
|
||
file descriptor to a program expecting a filename.
|
||
|
||
|
||
Performances
|
||
============
|
||
|
||
Setting close-on-exec flag may require additional system calls for
|
||
each creation of new file descriptors. The number of additional system
|
||
calls depends on the method used to set the flag:
|
||
|
||
* ``O_NOINHERIT``: no additional system call
|
||
* ``O_CLOEXEC``: one additional system call, but only at the creation
|
||
of the first file descriptor, to check if the flag is supported. If
|
||
the flag is not supported, Python has to fallback to the next method.
|
||
* ``ioctl(fd, FIOCLEX)``: one additional system call per file
|
||
descriptor
|
||
* ``fcntl(fd, F_SETFD, flags)``: two additional system calls per file
|
||
descriptor, one to get old flags and one to set new flags
|
||
|
||
On Linux, setting the close-on-flag has a low overhead on performances.
|
||
Results of
|
||
`bench_cloexec.py <http://hg.python.org/peps/file/tip/pep-0433/bench_cloexec.py>`_
|
||
on Linux 3.6:
|
||
|
||
* close-on-flag not set: 7.8 us
|
||
* ``O_CLOEXEC``: 1% slower (7.9 us)
|
||
* ``ioctl()``: 3% slower (8.0 us)
|
||
* ``fcntl()``: 3% slower (8.0 us)
|
||
|
||
|
||
Implementation
|
||
==============
|
||
|
||
os.get_cloexec(fd)
|
||
------------------
|
||
|
||
Get the close-on-exec flag of a file descriptor.
|
||
|
||
Pseudo-code::
|
||
|
||
if os.name == 'nt':
|
||
def get_cloexec(fd):
|
||
handle = _winapi._get_osfhandle(fd);
|
||
flags = _winapi.GetHandleInformation(handle)
|
||
return not(flags & _winapi.HANDLE_FLAG_INHERIT)
|
||
else:
|
||
try:
|
||
import fcntl
|
||
except ImportError:
|
||
pass
|
||
else:
|
||
def get_cloexec(fd):
|
||
flags = fcntl.fcntl(fd, fcntl.F_GETFD)
|
||
return bool(flags & fcntl.FD_CLOEXEC)
|
||
|
||
|
||
os.set_cloexec(fd, cloexec=True)
|
||
--------------------------------
|
||
|
||
Set or clear the close-on-exec flag on a file descriptor. The flag
|
||
is set after the creation of the file descriptor and so it is not
|
||
atomic.
|
||
|
||
Pseudo-code::
|
||
|
||
if os.name == 'nt':
|
||
def set_cloexec(fd, cloexec=True):
|
||
handle = _winapi._get_osfhandle(fd);
|
||
mask = _winapi.HANDLE_FLAG_INHERIT
|
||
if cloexec:
|
||
flags = 0
|
||
else:
|
||
flags = mask
|
||
_winapi.SetHandleInformation(handle, mask, flags)
|
||
else:
|
||
fnctl = None
|
||
ioctl = None
|
||
try:
|
||
import ioctl
|
||
except ImportError:
|
||
try:
|
||
import fcntl
|
||
except ImportError:
|
||
pass
|
||
if ioctl is not None and hasattr('FIOCLEX', ioctl):
|
||
def set_cloexec(fd, cloexec=True):
|
||
if cloexec:
|
||
ioctl.ioctl(fd, ioctl.FIOCLEX)
|
||
else:
|
||
ioctl.ioctl(fd, ioctl.FIONCLEX)
|
||
elif fnctl is not None:
|
||
def set_cloexec(fd, cloexec=True):
|
||
flags = fcntl.fcntl(fd, fcntl.F_GETFD)
|
||
if cloexec:
|
||
flags |= FD_CLOEXEC
|
||
else:
|
||
flags &= ~FD_CLOEXEC
|
||
fcntl.fcntl(fd, fcntl.F_SETFD, flags)
|
||
|
||
ioctl is preferred over fcntl because it requires only one syscall,
|
||
instead of two syscalls for fcntl.
|
||
|
||
.. note::
|
||
``fcntl(fd, F_SETFD, flags)`` only supports one flag
|
||
(``FD_CLOEXEC``), so it would be possible to avoid ``fcntl(fd,
|
||
F_GETFD)``. But it may drop other flags in the future, and so it is
|
||
safer to keep the two functions calls.
|
||
|
||
.. note::
|
||
``fopen()`` function of the GNU libc ignores the error if
|
||
``fcntl(fd, F_SETFD, flags)`` failed.
|
||
|
||
open()
|
||
------
|
||
|
||
* Windows: ``open()`` with ``O_NOINHERIT`` flag [atomic]
|
||
* ``open()`` with ``O_CLOEXEC flag`` [atomic]
|
||
* ``open()`` + ``os.set_cloexec(fd, True)`` [best-effort]
|
||
|
||
os.dup()
|
||
--------
|
||
|
||
* Windows: ``DuplicateHandle()`` [atomic]
|
||
* ``fcntl(fd, F_DUPFD_CLOEXEC)`` [atomic]
|
||
* ``dup()`` + ``os.set_cloexec(fd, True)`` [best-effort]
|
||
|
||
os.dup2()
|
||
---------
|
||
|
||
* ``fcntl(fd, F_DUP2FD_CLOEXEC, fd2)`` [atomic]
|
||
* ``dup3()`` with ``O_CLOEXEC`` flag [atomic]
|
||
* ``dup2()`` + ``os.set_cloexec(fd, True)`` [best-effort]
|
||
|
||
os.pipe()
|
||
---------
|
||
|
||
* Windows: ``CreatePipe()`` with
|
||
``SECURITY_ATTRIBUTES.bInheritHandle=TRUE``, or ``_pipe()`` with
|
||
``O_NOINHERIT`` flag [atomic]
|
||
* ``pipe2()`` with ``O_CLOEXEC`` flag [atomic]
|
||
* ``pipe()`` + ``os.set_cloexec(fd, True)`` [best-effort]
|
||
|
||
socket.socket()
|
||
---------------
|
||
|
||
* Windows: ``WSASocket()`` with ``WSA_FLAG_NO_HANDLE_INHERIT`` flag
|
||
[atomic]
|
||
* ``socket()`` with ``SOCK_CLOEXEC`` flag [atomic]
|
||
* ``socket()`` + ``os.set_cloexec(fd, True)`` [best-effort]
|
||
|
||
socket.socketpair()
|
||
-------------------
|
||
|
||
* ``socketpair()`` with ``SOCK_CLOEXEC`` flag [atomic]
|
||
* ``socketpair()`` + ``os.set_cloexec(fd, True)`` [best-effort]
|
||
|
||
socket.socket.accept()
|
||
----------------------
|
||
|
||
* ``accept4()`` with ``SOCK_CLOEXEC`` flag [atomic]
|
||
* ``accept()`` + ``os.set_cloexec(fd, True)`` [best-effort]
|
||
|
||
|
||
Backward compatibility
|
||
======================
|
||
|
||
There is no backward incompatible change. The default behaviour is
|
||
unchanged: the close-on-exec flag is not set by default.
|
||
|
||
|
||
Appendix: Operating system support
|
||
==================================
|
||
|
||
Windows
|
||
-------
|
||
|
||
Windows has an ``O_NOINHERIT`` flag: "Do not inherit in child
|
||
processes".
|
||
|
||
For example, it is supported by ``open()`` and ``_pipe()``.
|
||
|
||
The flag can be cleared using
|
||
``SetHandleInformation(fd, HANDLE_FLAG_INHERIT, 0)``.
|
||
|
||
``CreateProcess()`` has an ``bInheritHandles`` parameter: if it is
|
||
``FALSE``, the handles are not inherited. If it is ``TRUE``, handles
|
||
with ``HANDLE_FLAG_INHERIT`` flag set are inherited.
|
||
``subprocess.Popen`` uses ``close_fds`` option to define
|
||
``bInheritHandles``.
|
||
|
||
|
||
ioctl
|
||
-----
|
||
|
||
Functions:
|
||
|
||
* ``ioctl(fd, FIOCLEX, 0)``: set the close-on-exec flag
|
||
* ``ioctl(fd, FIONCLEX, 0)``: clear the close-on-exec flag
|
||
|
||
Availability: Linux, Mac OS X, QNX, NetBSD, OpenBSD, FreeBSD.
|
||
|
||
|
||
fcntl
|
||
-----
|
||
|
||
Functions:
|
||
|
||
* ``flags = fcntl(fd, F_GETFD); fcntl(fd, F_SETFD, flags | FD_CLOEXEC)``:
|
||
set the close-on-exec flag
|
||
* ``flags = fcntl(fd, F_GETFD); fcntl(fd, F_SETFD, flags & ~FD_CLOEXEC)``:
|
||
clear the close-on-exec flag
|
||
|
||
Availability: AIX, Digital UNIX, FreeBSD, HP-UX, IRIX, Linux, Mac OS
|
||
X, OpenBSD, Solaris, SunOS, Unicos.
|
||
|
||
|
||
Atomic flags
|
||
------------
|
||
|
||
New flags:
|
||
|
||
* ``O_CLOEXEC``: available on Linux (2.6.23), FreeBSD (8.3),
|
||
OpenBSD 5.0, Solaris 11, QNX, BeOS, next NetBSD release (6.1?).
|
||
This flag is part of POSIX.1-2008.
|
||
* ``SOCK_CLOEXEC`` flag for ``socket()`` and ``socketpair()``,
|
||
available on Linux 2.6.27, OpenBSD 5.2, NetBSD 6.0.
|
||
* ``WSA_FLAG_NO_HANDLE_INHERIT`` flag for ``WSASocket()``: supported
|
||
on Windows 7 with SP1, Windows Server 2008 R2 with SP1, and later
|
||
* ``fcntl()``: ``F_DUPFD_CLOEXEC`` flag, available on Linux 2.6.24,
|
||
OpenBSD 5.0, FreeBSD 9.1, NetBSD 6.0, Solaris 11. This flag is part
|
||
of POSIX.1-2008.
|
||
* ``fcntl()``: ``F_DUP2FD_CLOEXEC`` flag, available on FreeBSD 9.1
|
||
and Solaris 11.
|
||
* ``recvmsg()``: ``MSG_CMSG_CLOEXEC``, available on Linux 2.6.23,
|
||
NetBSD 6.0.
|
||
|
||
On Linux older than 2.6.23, ``O_CLOEXEC`` flag is simply ignored. So
|
||
we have to check that the flag is supported by calling ``fcntl()``. If
|
||
it does not work, we have to set the flag using ``ioctl()`` or
|
||
``fcntl()``.
|
||
|
||
On Linux older than 2.6.27, if the ``SOCK_CLOEXEC`` flag is set in the
|
||
socket type, ``socket()`` or ``socketpair()`` fail and ``errno`` is set
|
||
to ``EINVAL``.
|
||
|
||
On Windows XPS3, ``WSASocket()`` with ``WSAEPROTOTYPE`` when
|
||
``WSA_FLAG_NO_HANDLE_INHERIT`` flag is used.
|
||
|
||
New functions:
|
||
|
||
* ``dup3()``: available on Linux 2.6.27 (and glibc 2.9)
|
||
* ``pipe2()``: available on Linux 2.6.27 (and glibc 2.9)
|
||
* ``accept4()``: available on Linux 2.6.28 (and glibc 2.10)
|
||
|
||
If ``accept4()`` is called on Linux older than 2.6.28, ``accept4()``
|
||
returns ``-1`` (fail) and ``errno`` is set to ``ENOSYS``.
|
||
|
||
|
||
Links
|
||
=====
|
||
|
||
Links:
|
||
|
||
* `Secure File Descriptor Handling
|
||
<http://udrepper.livejournal.com/20407.html>`_ (Ulrich Drepper,
|
||
2008)
|
||
* `win32_support.py of the Tornado project
|
||
<https://bitbucket.org/pvl/gaeseries-tornado/src/c2671cea1842/tornado/win32_support.py>`_:
|
||
emulate fcntl(fd, F_SETFD, FD_CLOEXEC) using
|
||
``SetHandleInformation(fd, HANDLE_FLAG_INHERIT, 1)``
|
||
* `LKML: [PATCH] nextfd(2)
|
||
<https://lkml.org/lkml/2012/4/1/71>`_
|
||
|
||
Python issues:
|
||
|
||
* `#10115: Support accept4() for atomic setting of flags at socket
|
||
creation <http://bugs.python.org/issue10115>`_
|
||
* `#12105: open() does not able to set flags, such as O_CLOEXEC
|
||
<http://bugs.python.org/issue12105>`_
|
||
* `#12107: TCP listening sockets created without FD_CLOEXEC flag
|
||
<http://bugs.python.org/issue12107>`_
|
||
* `#16500: Add an atfork module
|
||
<http://bugs.python.org/issue16500>`_
|
||
* `#16850: Add "e" mode to open(): close-and-exec
|
||
(O_CLOEXEC) / O_NOINHERIT <http://bugs.python.org/issue16850>`_
|
||
* `#16860: Use O_CLOEXEC in the tempfile module
|
||
<http://bugs.python.org/issue16860>`_
|
||
* `#17036: Implementation of the PEP 433
|
||
<http://bugs.python.org/issue17036>`_
|
||
* `#16946: subprocess: _close_open_fd_range_safe() does not set
|
||
close-on-exec flag on Linux < 2.6.23 if O_CLOEXEC is defined
|
||
<http://bugs.python.org/issue16946>`_
|
||
* `#17070: PEP 433: Use the new cloexec to improve security and avoid
|
||
bugs <http://bugs.python.org/issue17070>`_
|
||
|
||
Other languages:
|
||
|
||
* Perl sets the close-on-exec flag on newly created file descriptor if
|
||
their number is greater than ``$SYSTEM_FD_MAX`` (``$^F``).
|
||
See `$SYSTEM_FD_MAX documentation
|
||
<http://perldoc.perl.org/perlvar.html#%24SYSTEM_FD_MAX>`_. Perl does
|
||
this since the creation of Perl (it was already present in Perl 1).
|
||
* Ruby: `Set FD_CLOEXEC for all fds (except 0, 1, 2)
|
||
<http://bugs.ruby-lang.org/issues/5041>`_
|
||
* Ruby: `O_CLOEXEC flag missing for Kernel::open
|
||
<http://bugs.ruby-lang.org/issues/1291>`_: the
|
||
`commit was reverted later
|
||
<http://bugs.ruby-lang.org/projects/ruby-trunk/repository/revisions/31643>`_
|
||
* OCaml: `PR#5256: Processes opened using Unix.open_process* inherit
|
||
all opened file descriptors (including sockets)
|
||
<http://caml.inria.fr/mantis/view.php?id=5256>`_. OCaml has a
|
||
``Unix.set_close_on_exec`` function.
|
||
|
||
|
||
Footnotes
|
||
=========
|
||
|
||
.. [#subprocess_close] On UNIX since Python 3.2, subprocess.Popen()
|
||
closes all file descriptors by default: ``close_fds=True``. It
|
||
closes file descriptors in range 3 inclusive to ``local_max_fd``
|
||
exclusive, where ``local_max_fd`` is ``fcntl(0, F_MAXFD)`` on
|
||
NetBSD, or ``sysconf(_SC_OPEN_MAX)`` otherwise. If the error pipe
|
||
has a descriptor smaller than 3, ``ValueError`` is raised.
|
||
|
||
|
||
Copyright
|
||
=========
|
||
|
||
This document has been placed in the public domain.
|
||
|
||
|
||
|
||
..
|
||
Local Variables:
|
||
mode: indented-text
|
||
indent-tabs-mode: nil
|
||
sentence-end-double-space: t
|
||
fill-column: 70
|
||
coding: utf-8
|
||
End:
|