python-peps/pep-0554.rst

PEP: 554
Title: Multiple Interpreters in the Stdlib
Author: Eric Snow <ericsnowcurrently@gmail.com>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 2017-09-05
Python-Version: 3.7
Post-History:


Abstract
========

This proposal introduces the stdlib ``interpreters`` module.  It exposes
the basic functionality of subinterpreters that already exists in the
C-API.  Each subinterpreter runs with its own state (see
``Interpreter Isolation`` below).


Rationale
=========

Running code in multiple interpreters provides a useful level of
isolation within the same process.  This can be leveraged in number
of ways.  Furthermore, subinterpreters provide a well-defined framework
in which such isolation may extended.

CPython has supported subinterpreters, with increasing levels of
support, since version 1.5.  While the feature has the potential
to be a powerful tool, subinterpreters have suffered from neglect
because they are not available directly from Python.  Exposing the
existing functionality in the stdlib will help reverse the situation.

This proposal is focused on enabling the fundamental capability of
multiple isolated interpreters in the same Python process.  This is a
new area for Python so there is relative uncertainly about the best
tools to provide as companions to subinterpreters.  Thus we minimize
the functionality we add in the proposal as much as possible.


Proposal
========

The ``interpreters`` module will be added to the stdlib.  It will
provide a high-level interface to subinterpreters and wrap the low-level
``_interpreters`` module.  The proposed API is inspired by the
``threading`` module.

The module provides the following functions:

``enumerate()``::

   Return a list of all existing interpreters.

``get_current()``::

   Return the currently running interpreter.

``get_main()``::

   Return the main interpreter.

``create()``::

   Initialize a new Python interpreter and return it.  The
   interpreter will be created in the current thread and will remain
   idle until something is run in it.

The module also provides the following class:

``Interpreter(id)``::

   ``id``::

      The interpreter's ID (read-only).

   ``is_running()``::

      Return whether or not the interpreter is currently running.

   ``destroy()``::

      Finalize and destroy the interpreter.

   ``run(code)``::

      Run the provided Python code in the interpreter, in the current
      OS thread.  Supported code: source text.


Deferred Functionality
======================

In the interest of keeping this proposal minimal, the following
functionality has been left out for future consideration.  Note that
this is not a judgement against any of said capability, but rather a
deferment.  That said, each is arguably valid.

Queues (Channels)
-----------------

Subinterpreters are inherently isolated, in contrast to threads.  This
enables a different concurrency model than currently exists in Python.
CSP (Communicating Sequential Processes), upon which Go's concurrency
is based, is one example of this model.

A key component of this approach to concurrency is message passing.  So
providing a message/object passing mechanism alongside ``Interpreter``
is entirely sensible and even advisable.  However, it isn't strictly
necessary to expose the existing functionality from the C-API.

The key challenge here is that sharing objects between interpreters
faces complexity due to CPython's memory model.  This is substantially
more challenging under a possible future where interpreters do not share
the GIL.

In the spirit of minimalism explained above and given the complexity
involved with sharing objects between interpreters, this proposal leaves
the addition of queues to future consideration.

Interpreter.call()
------------------

It would be convenient to run existing functions in subinterpreters
directly.  ``Interpreter.run()`` could be adjusted to support this or
a ``call()`` method could be added::

   Interpreter.call(f, *args, **kwargs)

This suffers from the same problem as sharing objects between
interpreters via queues.  The minimal solution (running a source string)
is sufficient for us to get the feature out where it can be explored.


Interpreter Isolation
=====================

CPython's interpreters are intended to be strictly isolated from each
other.  Each interpreter has its own copy of all modules, classes,
functions, and variables.  The same applies to state in C, including in
extension modules.  The CPython C-API docs explain more. [c-api]_

However, there are ways in which interpreters share some state.  First
of all, some process-global state remains shared, like file descriptors.
There are no plans to change this.

Second, some isolation is faulty due to bugs or implementations that did
not take subinterpreters into account.  This includes things like
at-exit handlers and extension modules that rely on C globals.  In these
cases bugs should be opened (some are already).

Finally, some potential isolation is missing due to the current design
of CPython.  This includes the GIL and memory management.  Improvements
are currently going on to address gaps in this area.


Open Questions
==============

* Add queues to the proposal anyway?


References
==========

.. [c-api]
   https://docs.python.org/3/c-api/init.html#bugs-and-caveats


Copyright
=========

This document has been placed in the public domain.



..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   coding: utf-8
   End: