249 lines
8.5 KiB
Plaintext
249 lines
8.5 KiB
Plaintext
PEP: 446
|
|
Title: Add new parameters to configure the inheritance of files and for non-blocking sockets
|
|
Version: $Revision$
|
|
Last-Modified: $Date$
|
|
Author: Victor Stinner <victor.stinner@gmail.com>
|
|
Status: Draft
|
|
Type: Standards Track
|
|
Content-Type: text/x-rst
|
|
Created: 3-July-2013
|
|
Python-Version: 3.4
|
|
|
|
|
|
Abstract
|
|
========
|
|
|
|
This PEP proposes new portable parameters and functions to configure the
|
|
inheritance of file descriptors and the non-blocking flag of sockets.
|
|
|
|
|
|
Rationale
|
|
=========
|
|
|
|
Inheritance of file descriptors
|
|
-------------------------------
|
|
|
|
The inheritance of file descriptors in child processes can be configured
|
|
on each file descriptor using a *close-on-exec* flag. By default, the
|
|
close-on-exec flag is not set.
|
|
|
|
On Windows, the close-on-exec flag is the inverse of ``HANDLE_FLAG_INHERIT``. File
|
|
descriptors are not inherited if the ``bInheritHandles`` parameter of
|
|
the ``CreateProcess()`` function is ``FALSE``, even if the
|
|
``HANDLE_FLAG_INHERIT`` flag is set. If ``bInheritHandles`` is ``TRUE``,
|
|
only file descriptors with ``HANDLE_FLAG_INHERIT`` flag set are
|
|
inherited, others are not.
|
|
|
|
On UNIX, the close-on-exec flag is ``O_CLOEXEC``. File descriptors with
|
|
the ``O_CLOEXEC`` flag set are closed at the execution of a new program
|
|
(ex: when calling ``execv()``).
|
|
|
|
The ``O_CLOEXEC`` flag has no effect on ``fork()``, all file descriptors
|
|
are inherited by the child process. Futhermore, most properties file
|
|
descriptors are shared between the parent and the child processes,
|
|
except file attributes which are duplicated (``O_CLOEXEC`` is the only
|
|
file attribute). Setting ``O_CLOEXEC`` flag of a file descriptor in the
|
|
child process does not change the ``O_CLOEXEC`` flag of the file
|
|
descriptor in the parent process.
|
|
|
|
|
|
Issues of the inheritance of file descriptors
|
|
---------------------------------------------
|
|
|
|
Inheritance of file descriptors causes issues. For example, closing a
|
|
file descriptor in the parent process does not release the resource
|
|
(file, socket, ...), because the file descriptor is still open in the
|
|
child process.
|
|
|
|
Leaking file descriptors is also a major security vulnerability. An
|
|
untrusted child process can read sensitive data like passwords and take
|
|
control of the parent process though leaked file descriptors. It is for
|
|
example a known vulnerability to escape from a chroot.
|
|
|
|
|
|
Non-blocking sockets
|
|
--------------------
|
|
|
|
To handle multiple network clients in a single thread, a multiplexing
|
|
function like ``select()`` can be used. For best performances, sockets
|
|
must be configured as non-blocking. Operations like ``send()`` and
|
|
``recv()`` return an ``EAGAIN`` or ``EWOULDBLOCK`` error if the
|
|
operation would block.
|
|
|
|
By default, newly created sockets are blocking. Setting the non-blocking
|
|
mode requires additional system calls.
|
|
|
|
On UNIX, the blocking flag is ``O_NONBLOCK``: a pipe and a socket are
|
|
non-blocking if the ``O_NONBLOCK`` flag is set.
|
|
|
|
|
|
Setting flags at the creation of the file descriptor
|
|
----------------------------------------------------
|
|
|
|
Windows and recent versions of other operating systems like Linux
|
|
support setting the close-on-exec flag directly at the creation of file
|
|
descriptors, and close-on-exec and blocking flags at the creation of
|
|
sockets.
|
|
|
|
Setting these flags at the creation is atomic and avoids additional
|
|
system calls.
|
|
|
|
|
|
Proposal
|
|
========
|
|
|
|
New cloexec And blocking Parameters
|
|
-----------------------------------
|
|
|
|
Add a new optional *cloexec* on functions creating file descriptors:
|
|
|
|
* ``io.FileIO``
|
|
* ``io.open()``
|
|
* ``open()``
|
|
* ``os.dup()``
|
|
* ``os.dup2()``
|
|
* ``os.fdopen()``
|
|
* ``os.open()``
|
|
* ``os.openpty()``
|
|
* ``os.pipe()``
|
|
* ``select.devpoll()``
|
|
* ``select.epoll()``
|
|
* ``select.kqueue()``
|
|
|
|
Add new optional *cloexec* and *blocking* parameters to functions
|
|
creating sockets:
|
|
|
|
* ``asyncore.dispatcher.create_socket()``
|
|
* ``socket.socket()``
|
|
* ``socket.socket.accept()``
|
|
* ``socket.socket.dup()``
|
|
* ``socket.socket.fromfd``
|
|
* ``socket.socketpair()``
|
|
|
|
The default value of *cloexec* is ``False`` and the default value of
|
|
*blocking* is ``True``.
|
|
|
|
The atomicity is not guaranteed. If the platform does not support
|
|
setting close-on-exec and blocking flags at the creation of the file
|
|
descriptor or socket, the flags are set using additional system calls.
|
|
|
|
|
|
New Functions
|
|
-------------
|
|
|
|
Add new functions the get and set the close-on-exec flag of a file
|
|
descriptor, available on all platforms:
|
|
|
|
* ``os.get_cloexec(fd:int) -> bool``
|
|
* ``os.set_cloexec(fd:int, cloexec: bool)``
|
|
|
|
Add new functions the get and set the blocking flag of a file
|
|
descriptor, only available on UNIX:
|
|
|
|
* ``os.get_blocking(fd:int) -> bool``
|
|
* ``os.set_blocking(fd:int, blocking: bool)``
|
|
|
|
|
|
Other Changes
|
|
-------------
|
|
|
|
The ``subprocess.Popen`` class must clear the close-on-exec flag of file
|
|
descriptors of the ``pass_fds`` parameter. The flag is cleared in the
|
|
child process before executing the program; the change does not change
|
|
the flag in the parent process.
|
|
|
|
The close-on-exec flag must also be set on private file descriptors and
|
|
sockets in the Python standard library. For example, on UNIX,
|
|
os.urandom() opens ``/dev/urandom`` to read some random bytes and the
|
|
file descriptor is closed at function exit. The file descriptor is not
|
|
expected to be inherited by child processes.
|
|
|
|
|
|
Rejected Alternatives
|
|
=====================
|
|
|
|
PEP 433
|
|
-------
|
|
|
|
The PEP 433 entitled "Easier suppression of file descriptor inheritance"
|
|
is a previous attempt proposing various other alternatives, but no
|
|
consensus could be reached.
|
|
|
|
This PEP has a well defined behaviour (the default value of the new
|
|
*cloexec* parameter is not configurable), is more conservative (no
|
|
backward compatibility issue), and is much simpler.
|
|
|
|
|
|
Add blocking parameter for file descriptors and use Windows overlapped I/O
|
|
--------------------------------------------------------------------------
|
|
|
|
Windows supports non-blocking operations on files using an extension of
|
|
the Windows API called "Overlapped I/O". Using this extension requires
|
|
to modify the Python standard library and applications to pass a
|
|
``OVERLAPPED`` structure and an event loop to wait for the completion of
|
|
operations.
|
|
|
|
This PEP only tries to expose portable flags on file descriptors and
|
|
sockets. Supporting overlapped I/O requires an abstraction providing a
|
|
high-level and portable API for asynchronous operations on files and
|
|
sockets. Overlapped I/O are out of the scope of this PEP.
|
|
|
|
UNIX supports non-blocking files, moreover recent versions of operating
|
|
systems support setting the non-blocking flag at the creation of a file
|
|
descriptor. It would be possible to add a new optional *blocking*
|
|
parameter to Python functions creating file descriptors. On Windows,
|
|
creating a file descriptor with ``blocking=False`` would raise a
|
|
``NotImplementedError``. This behaviour is not acceptable for the ``os``
|
|
module which is designed as a thin wrapper on the C functions of the
|
|
operating system. If a platform does not support a function, the
|
|
function should not be available on the platform. For example,
|
|
the ``os.fork()`` function is not available on Windows.
|
|
|
|
UNIX has more flag on file descriptors: ``O_DSYNC``, ``O_SYNC``,
|
|
``O_DIRECT``, etc. Adding all these flags complicates the signature and
|
|
the implementation of functions creating file descriptor like open().
|
|
Moreover, these flags do not work on any file type, and are not
|
|
portable.
|
|
|
|
For all these reasons, this alternative was rejected. The PEP 3156
|
|
proposes an abstraction for asynchronous I/O supporting non-blocking
|
|
files on Windows.
|
|
|
|
|
|
Links
|
|
=====
|
|
|
|
Python issues:
|
|
|
|
* `#10115: Support accept4() for atomic setting of flags at socket
|
|
creation <http://bugs.python.org/issue10115>`_
|
|
* `#12105: open() does not able to set flags, such as O_CLOEXEC
|
|
<http://bugs.python.org/issue12105>`_
|
|
* `#12107: TCP listening sockets created without FD_CLOEXEC flag
|
|
<http://bugs.python.org/issue12107>`_
|
|
* `#16850: Add "e" mode to open(): close-and-exec
|
|
(O_CLOEXEC) / O_NOINHERIT <http://bugs.python.org/issue16850>`_
|
|
* `#16860: Use O_CLOEXEC in the tempfile module
|
|
<http://bugs.python.org/issue16860>`_
|
|
* `#16946: subprocess: _close_open_fd_range_safe() does not set
|
|
close-on-exec flag on Linux < 2.6.23 if O_CLOEXEC is defined
|
|
<http://bugs.python.org/issue16946>`_
|
|
* `#17070: Use the new cloexec to improve security and avoid bugs
|
|
<http://bugs.python.org/issue17070>`_
|
|
|
|
Other links:
|
|
|
|
* `Secure File Descriptor Handling
|
|
<http://udrepper.livejournal.com/20407.html>`_ (Ulrich Drepper,
|
|
2008)
|
|
* `Ghosts of Unix past, part 2: Conflated designs
|
|
<http://lwn.net/Articles/412131/>`_ (Neil Brown, 2010) explains the
|
|
history of ``O_CLOEXEC`` and ``O_NONBLOCK`` flags
|
|
|
|
|
|
Copyright
|
|
=========
|
|
|
|
This document has been placed into the public domain.
|
|
|