157 lines
4.4 KiB
ReStructuredText
157 lines
4.4 KiB
ReStructuredText
PEP: 554
|
||
Title: Multiple Interpreters in the Stdlib
|
||
Author: Eric Snow <ericsnowcurrently@gmail.com>
|
||
Status: Draft
|
||
Type: Standards Track
|
||
Content-Type: text/x-rst
|
||
Created: 2017-09-05
|
||
Python-Version: 3.7
|
||
Post-History:
|
||
|
||
|
||
Abstract
|
||
========
|
||
|
||
This proposal introduces the stdlib ``interpreters`` module. It exposes
|
||
the basic functionality of subinterpreters that already exists in the
|
||
C-API. Each subinterpreter runs with its own state, including its own
|
||
copy of all modules, classes, functions, and variables.
|
||
|
||
|
||
Rationale
|
||
=========
|
||
|
||
Running code in multiple interpreters provides a useful level of
|
||
isolation within the same process. This can be leveraged in number
|
||
of ways. Furthermore, subinterpreters provide a well-defined framework
|
||
in which such isolation may extended.
|
||
|
||
CPython has supported subinterpreters, with increasing levels of
|
||
support, since version 1.5. While the feature has the potential
|
||
to be a powerful tool, subinterpreters have suffered from neglect
|
||
because they are not available directly from Python. Exposing the
|
||
existing functionality in the stdlib will help reverse the situation.
|
||
|
||
This proposal is focused on enabling the fundamental capability of
|
||
multiple isolated interpreters in the same Python process. This is a
|
||
new area for Python so there is relative uncertainly about the best
|
||
tools to provide as companions to subinterpreters. Thus we minimize
|
||
the functionality we add in the proposal as much as possible.
|
||
|
||
|
||
Proposal
|
||
========
|
||
|
||
The ``interpreters`` module will be added to the stdlib. It will
|
||
provide a high-level interface to subinterpreters and wrap the low-level
|
||
``_interpreters`` module. The proposed API is inspired by the
|
||
``threading`` module.
|
||
|
||
The module provides the following functions:
|
||
|
||
``enumerate()``::
|
||
|
||
Return a list of all existing interpreters.
|
||
|
||
``get_current()``::
|
||
|
||
Return the currently running interpreter.
|
||
|
||
``get_main()``::
|
||
|
||
Return the main interpreter.
|
||
|
||
``create()``::
|
||
|
||
Initialize a new Python interpreter and return it. The
|
||
interpreter will be created in the current thread and will remain
|
||
idle until something is run in it.
|
||
|
||
The module also provides the following class:
|
||
|
||
``Interpreter(id)``::
|
||
|
||
``id``::
|
||
|
||
The interpreter's ID (read-only).
|
||
|
||
``is_running()``::
|
||
|
||
Return whether or not the interpreter is currently running.
|
||
|
||
``destroy()``::
|
||
|
||
Finalize and destroy the interpreter.
|
||
|
||
``run(code)``::
|
||
|
||
Run the provided Python code in the interpreter, in the current
|
||
OS thread. Supported code: source text.
|
||
|
||
|
||
Deferred Functionality
|
||
======================
|
||
|
||
In the interest of keeping this proposal minimal, the following
|
||
functionality has been left out for future consideration. Note that
|
||
this is not a judgement against any of said capability, but rather a
|
||
deferment. That said, each is arguably valid.
|
||
|
||
Queues (Channels)
|
||
-----------------
|
||
|
||
Subinterpreters are inherently isolated, in contrast to threads. This
|
||
enables a different concurrency model than currently exists in Python.
|
||
CSP (Communicating Sequential Processes), upon which Go's concurrency
|
||
is based, is one example of this model.
|
||
|
||
A key component of this approach to concurrency is message passing. So
|
||
providing a message/object passing mechanism alongside ``Interpreter``
|
||
is entirely sensible and even advisable. However, it isn't strictly
|
||
necessary to expose the existing functionality from the C-API.
|
||
|
||
The key challenge here is that sharing objects between interpreters
|
||
faces complexity due to CPython's memory model. This is substantially
|
||
more challenging under a possible future where interpreters do not share
|
||
the GIL.
|
||
|
||
In the spirit of minimalism explained above and given the complexity
|
||
involved with sharing objects between interpreters, this proposal leaves
|
||
the addition of queues to future consideration.
|
||
|
||
Interpreter.call()
|
||
------------------
|
||
|
||
It would be convenient to run existing functions in subinterpreters
|
||
directly. ``Interpreter.run()`` could be adjusted to support this or
|
||
a ``call()`` method could be added::
|
||
|
||
Interpreter.call(f, *args, **kwargs)
|
||
|
||
This suffers from the same problem as sharing objects between
|
||
interpreters via queues. The minimal solution (running a source string)
|
||
is sufficient for us to get the feature out where it can be explored.
|
||
|
||
|
||
Open Questions
|
||
==============
|
||
|
||
* Add queues to the proposal anyway?
|
||
|
||
|
||
Copyright
|
||
=========
|
||
|
||
This document has been placed in the public domain.
|
||
|
||
|
||
|
||
..
|
||
Local Variables:
|
||
mode: indented-text
|
||
indent-tabs-mode: nil
|
||
sentence-end-double-space: t
|
||
fill-column: 70
|
||
coding: utf-8
|
||
End:
|