python-peps/pep-0446.txt

PEP: 446
Title: Add new parameters to configure the inheritance of files and for non-blocking sockets
Version: $Revision$
Last-Modified: $Date$
Author: Victor Stinner <victor.stinner@gmail.com>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 3-July-2013
Python-Version: 3.4


Abstract
========

This PEP proposes new portable parameters and functions to configure the
inheritance of file descriptors and the non-blocking flag of sockets.


Rationale
=========

Inheritance of file descriptors
-------------------------------

The inheritance of file descriptors in child processes can be configured
on each file descriptor using a *close-on-exec* flag. By default, the
close-on-exec flag is not set.

On Windows, the close-on-exec flag is the inverse of ``HANDLE_FLAG_INHERIT``. File
descriptors are not inherited if the ``bInheritHandles`` parameter of
the ``CreateProcess()`` function is ``FALSE``, even if the
``HANDLE_FLAG_INHERIT`` flag is set. If ``bInheritHandles`` is ``TRUE``,
only file descriptors with ``HANDLE_FLAG_INHERIT`` flag set are
inherited, others are not.

On UNIX, the close-on-exec flag is ``O_CLOEXEC``. File descriptors with
the ``O_CLOEXEC`` flag set are closed at the execution of a new program
(ex: when calling ``execv()``).

The ``O_CLOEXEC`` flag has no effect on ``fork()``, all file descriptors
are inherited by the child process. Futhermore, most properties file
descriptors are shared between the parent and the child processes,
except file attributes which are duplicated (``O_CLOEXEC`` is the only
file attribute).  Setting ``O_CLOEXEC`` flag of a file descriptor in the
child process does not change the ``O_CLOEXEC`` flag of the file
descriptor in the parent process.


Issues of the inheritance of file descriptors
---------------------------------------------

Inheritance of file descriptors causes issues. For example, closing a
file descriptor in the parent process does not release the resource
(file, socket, ...), because the file descriptor is still open in the
child process.

Leaking file descriptors is also a major security vulnerability. An
untrusted child process can read sensitive data like passwords and take
control of the parent process though leaked file descriptors. It is for
example a known vulnerability to escape from a chroot.


Non-blocking sockets
--------------------

To handle multiple network clients in a single thread, a multiplexing
function like ``select()`` can be used. For best performances, sockets
must be configured as non-blocking. Operations like ``send()`` and
``recv()`` return an ``EAGAIN`` or ``EWOULDBLOCK`` error if the
operation would block.

By default, newly created sockets are blocking. Setting the non-blocking
mode requires additional system calls.

On UNIX, the blocking flag is ``O_NONBLOCK``: a pipe and a socket are
non-blocking if the ``O_NONBLOCK`` flag is set.


Setting flags at the creation of the file descriptor
----------------------------------------------------

Windows and recent versions of other operating systems like Linux
support setting the close-on-exec flag directly at the creation of file
descriptors, and close-on-exec and blocking flags at the creation of
sockets.

Setting these flags at the creation is atomic and avoids additional
system calls.


Proposal
========

New cloexec And blocking Parameters
-----------------------------------

Add a new optional *cloexec* on functions creating file descriptors:

* ``io.FileIO``
* ``io.open()``
* ``open()``
* ``os.dup()``
* ``os.dup2()``
* ``os.fdopen()``
* ``os.open()``
* ``os.openpty()``
* ``os.pipe()``
* ``select.devpoll()``
* ``select.epoll()``
* ``select.kqueue()``

Add new optional *cloexec* and *blocking* parameters to functions
creating sockets:

* ``asyncore.dispatcher.create_socket()``
* ``socket.socket()``
* ``socket.socket.accept()``
* ``socket.socket.dup()``
* ``socket.socket.fromfd``
* ``socket.socketpair()``

The default value of *cloexec* is ``False`` and the default value of
*blocking* is ``True``.

The atomicity is not guaranteed. If the platform does not support
setting close-on-exec and blocking flags at the creation of the file
descriptor or socket, the flags are set using additional system calls.


New Functions
-------------

Add new functions the get and set the close-on-exec flag of a file
descriptor, available on all platforms:

* ``os.get_cloexec(fd:int) -> bool``
* ``os.set_cloexec(fd:int, cloexec: bool)``

Add new functions the get and set the blocking flag of a file
descriptor, only available on UNIX:

* ``os.get_blocking(fd:int) -> bool``
* ``os.set_blocking(fd:int, blocking: bool)``


Other Changes
-------------

The ``subprocess.Popen`` class must clear the close-on-exec flag of file
descriptors of the ``pass_fds`` parameter. The flag is cleared in the
child process before executing the program; the change does not change
the flag in the parent process.

The close-on-exec flag must also be set on private file descriptors and
sockets in the Python standard library. For example, on UNIX,
os.urandom() opens ``/dev/urandom`` to read some random bytes and the
file descriptor is closed at function exit. The file descriptor is not
expected to be inherited by child processes.


Rejected Alternatives
=====================

PEP 433
-------

The PEP 433 entitled "Easier suppression of file descriptor inheritance"
is a previous attempt proposing various other alternatives, but no
consensus could be reached.

This PEP has a well defined behaviour (the default value of the new
*cloexec* parameter is not configurable), is more conservative (no
backward compatibility issue), and is much simpler.


Add blocking parameter for file descriptors and use Windows overlapped I/O
--------------------------------------------------------------------------

Windows supports non-blocking operations on files using an extension of
the Windows API called "Overlapped I/O". Using this extension requires
to modify the Python standard library and applications to pass a
``OVERLAPPED`` structure and an event loop to wait for the completion of
operations.

This PEP only tries to expose portable flags on file descriptors and
sockets. Supporting overlapped I/O requires an abstraction providing a
high-level and portable API for asynchronous operations on files and
sockets. Overlapped I/O are out of the scope of this PEP.

UNIX supports non-blocking files, moreover recent versions of operating
systems support setting the non-blocking flag at the creation of a file
descriptor. It would be possible to add a new optional *blocking*
parameter to Python functions creating file descriptors. On Windows,
creating a file descriptor with ``blocking=False``  would raise a
``NotImplementedError``. This behaviour is not acceptable for the ``os``
module which is designed as a thin wrapper on the C functions of the
operating system. If a platform does not support a function, the
function should not be available on the platform. For example,
the ``os.fork()`` function is not available on Windows.

UNIX has more flag on file descriptors: ``O_DSYNC``, ``O_SYNC``,
``O_DIRECT``, etc.  Adding all these flags complicates the signature and
the implementation of functions creating file descriptor like open().
Moreover, these flags do not work on any file type, and are not
portable.

For all these reasons, this alternative was rejected. The PEP 3156
proposes an abstraction for asynchronous I/O supporting non-blocking
files on Windows.


Links
=====

Python issues:

* `#10115: Support accept4() for atomic setting of flags at socket
  creation <http://bugs.python.org/issue10115>`_
* `#12105: open() does not able to set flags, such as O_CLOEXEC
  <http://bugs.python.org/issue12105>`_
* `#12107: TCP listening sockets created without FD_CLOEXEC flag
  <http://bugs.python.org/issue12107>`_
* `#16850: Add "e" mode to open(): close-and-exec
  (O_CLOEXEC) / O_NOINHERIT <http://bugs.python.org/issue16850>`_
* `#16860: Use O_CLOEXEC in the tempfile module
  <http://bugs.python.org/issue16860>`_
* `#16946: subprocess: _close_open_fd_range_safe() does not set
  close-on-exec flag on Linux < 2.6.23 if O_CLOEXEC is defined
  <http://bugs.python.org/issue16946>`_
* `#17070: Use the new cloexec to improve security and avoid bugs
  <http://bugs.python.org/issue17070>`_

Other links:

* `Secure File Descriptor Handling
  <http://udrepper.livejournal.com/20407.html>`_ (Ulrich Drepper,
  2008)
* `Ghosts of Unix past, part 2: Conflated designs
  <http://lwn.net/Articles/412131/>`_ (Neil Brown, 2010) explains the
  history of ``O_CLOEXEC`` and ``O_NONBLOCK`` flags


Copyright
=========

This document has been placed into the public domain.