1402 lines
54 KiB
ReStructuredText
1402 lines
54 KiB
ReStructuredText
PEP: 554
|
||
Title: Multiple Interpreters in the Stdlib
|
||
Author: Eric Snow <ericsnowcurrently@gmail.com>
|
||
Status: Draft
|
||
Type: Standards Track
|
||
Content-Type: text/x-rst
|
||
Created: 2017-09-05
|
||
Python-Version: 3.8
|
||
Post-History: 07-Sep-2017, 08-Sep-2017, 13-Sep-2017, 05-Dec-2017,
|
||
09-May-2018
|
||
|
||
|
||
Abstract
|
||
========
|
||
|
||
CPython has supported multiple interpreters in the same process (AKA
|
||
"subinterpreters") since version 1.5 (1997). The feature has been
|
||
available via the C-API. [c-api]_ Subinterpreters operate in
|
||
`relative isolation from one another <Interpreter Isolation_>`_, which
|
||
provides the basis for an
|
||
`alternative concurrency model <Concurrency_>`_.
|
||
|
||
This proposal introduces the stdlib ``interpreters`` module. The module
|
||
will be `provisional <Provisional Status_>`_. It exposes the basic
|
||
functionality of subinterpreters already provided by the C-API, along
|
||
with new functionality for sharing data between interpreters.
|
||
|
||
|
||
Proposal
|
||
========
|
||
|
||
The ``interpreters`` module will be added to the stdlib. It will
|
||
provide a high-level interface to subinterpreters and wrap a new
|
||
low-level ``_interpreters`` (in the same was as the ``threading``
|
||
module). See the `Examples`_ section for concrete usage and use cases.
|
||
|
||
Along with exposing the existing (in CPython) subinterpreter support,
|
||
the module will also provide a mechanism for sharing data between
|
||
interpreters. This mechanism centers around "channels", which are
|
||
similar to queues and pipes.
|
||
|
||
Note that *objects* are not shared between interpreters since they are
|
||
tied to the interpreter in which they were created. Instead, the
|
||
objects' *data* is passed between interpreters. See the `Shared data`_
|
||
section for more details about sharing between interpreters.
|
||
|
||
At first only the following types will be supported for sharing:
|
||
|
||
* None
|
||
* bytes
|
||
* str
|
||
* int
|
||
* PEP 3118 buffer objects (via ``send_buffer()``)
|
||
|
||
Support for other basic types (e.g. int, Ellipsis) will be added later.
|
||
|
||
API summary for interpreters module
|
||
-----------------------------------
|
||
|
||
Here is a summary of the API for the ``interpreters`` module. For a
|
||
more in-depth explanation of the proposed classes and functions, see
|
||
the `"interpreters" Module API`_ section below.
|
||
|
||
For creating and using interpreters:
|
||
|
||
+------------------------------+----------------------------------------------+
|
||
| signature | description |
|
||
+============================+=+==============================================+
|
||
| list_all() -> [Intepreter] | Get all existing interpreters. |
|
||
+------------------------------+----------------------------------------------+
|
||
| get_current() -> Interpreter | Get the currently running interpreter. |
|
||
+------------------------------+----------------------------------------------+
|
||
| create() -> Interpreter | Initialize a new (idle) Python interpreter. |
|
||
+------------------------------+----------------------------------------------+
|
||
|
||
|
|
||
|
||
+-----------------------+-----------------------------------------------------+
|
||
| signature | description |
|
||
+=======================+=====================================================+
|
||
| class Interpreter(id) | A single interpreter. |
|
||
+-----------------------+-----------------------------------------------------+
|
||
| .id | The interpreter's ID (read-only). |
|
||
+-----------------------+-----------------------------------------------------+
|
||
| .is_running() -> Bool | Is the interpreter currently executing code? |
|
||
+-----------------------+-----------------------------------------------------+
|
||
| .destroy() | Finalize and destroy the interpreter. |
|
||
+-----------------------+-----------------------------------------------------+
|
||
| .run(src_str, /, \*, | | Run the given source code in the interpreter. |
|
||
| channels=None) | | (This blocks the current thread until done.) |
|
||
+-----------------------+-----------------------------------------------------+
|
||
|
||
|
|
||
|
||
+----------------+--------------+------------------------------------------------------+
|
||
| exception | base | description |
|
||
+================+==============+======================================================+
|
||
| RunFailedError | RuntimeError | Interpreter.run() resulted in an uncaught exception. |
|
||
+----------------+--------------+------------------------------------------------------+
|
||
|
||
For sharing data between interpreters:
|
||
|
||
+--------------------------------+--------------------------------------------+
|
||
| signature | description |
|
||
+================================+============================================+
|
||
| is_shareable(obj) -> Bool | | Can the object's data be shared |
|
||
| | | between interpreters? |
|
||
+--------------------------------+--------------------------------------------+
|
||
| create_channel() -> | | Create a new channel for passing |
|
||
| (RecvChannel, SendChannel) | | data between interpreters. |
|
||
+--------------------------------+--------------------------------------------+
|
||
| list_all_channels() -> | Get all open channels. |
|
||
| [(RecvChannel, SendChannel)] | |
|
||
+--------------------------------+--------------------------------------------+
|
||
|
||
|
|
||
|
||
+-------------------------------+-----------------------------------------------+
|
||
| signature | description |
|
||
+===============================+===============================================+
|
||
| class RecvChannel(id) | The receiving end of a channel. |
|
||
+-------------------------------+-----------------------------------------------+
|
||
| .id | The channel's unique ID. |
|
||
+-------------------------------+-----------------------------------------------+
|
||
| .interpreters | The list of associated interpreters. |
|
||
+-------------------------------+-----------------------------------------------+
|
||
| .recv() -> object | | Get the next object from the channel, |
|
||
| | | and wait if none have been sent. |
|
||
| | | Associate the interpreter with the channel. |
|
||
+-------------------------------+-----------------------------------------------+
|
||
| .recv_nowait(default=None) -> | | Like recv(), but return the default |
|
||
| object | | instead of waiting. |
|
||
+-------------------------------+-----------------------------------------------+
|
||
| .release() | | No longer associate the current interpreter |
|
||
| | | with the channel (on the receiving end). |
|
||
+-------------------------------+-----------------------------------------------+
|
||
| .close(force=False) | | Close the channel in all interpreters. |
|
||
+-------------------------------+-----------------------------------------------+
|
||
|
||
|
|
||
|
||
+---------------------------+-------------------------------------------------+
|
||
| signature | description |
|
||
+===========================+=================================================+
|
||
| class SendChannel(id) | The sending end of a channel. |
|
||
+---------------------------+-------------------------------------------------+
|
||
| .id | The channel's unique ID. |
|
||
+---------------------------+-------------------------------------------------+
|
||
| .interpreters | The list of associated interpreters. |
|
||
+---------------------------+-------------------------------------------------+
|
||
| .send(obj) | | Send the object (i.e. its data) to the |
|
||
| | | receiving end of the channel and wait. |
|
||
| | | Associate the interpreter with the channel. |
|
||
+---------------------------+-------------------------------------------------+
|
||
| .send_nowait(obj) | | Like send(), but Fail if not received. |
|
||
+---------------------------+-------------------------------------------------+
|
||
| .send_buffer(obj) | | Send the object's (PEP 3118) buffer to the |
|
||
| | | receiving end of the channel and wait. |
|
||
| | | Associate the interpreter with the channel. |
|
||
+---------------------------+-------------------------------------------------+
|
||
| .send_buffer_nowait(obj) | | Like send_buffer(), but fail if not received. |
|
||
+---------------------------+-------------------------------------------------+
|
||
| .release() | | No longer associate the current interpreter |
|
||
| | | with the channel (on the sending end). |
|
||
+---------------------------+-------------------------------------------------+
|
||
| .close(force=False) | | Close the channel in all interpreters. |
|
||
+---------------------------+-------------------------------------------------+
|
||
|
||
|
|
||
|
||
+----------------------+--------------------+------------------------------------------------+
|
||
| exception | base | description |
|
||
+======================+====================+================================================+
|
||
| ChannelError | Exception | The base class for channel-related exceptions. |
|
||
+----------------------+--------------------+------------------------------------------------+
|
||
| ChannelNotFoundError | ChannelError | The identified channel was not found. |
|
||
+----------------------+--------------------+------------------------------------------------+
|
||
| ChannelEmptyError | ChannelError | The channel was unexpectedly empty. |
|
||
+----------------------+--------------------+------------------------------------------------+
|
||
| ChannelNotEmptyError | ChannelError | The channel was unexpectedly not empty. |
|
||
+----------------------+--------------------+------------------------------------------------+
|
||
| NotReceivedError | ChannelError | Nothing was waiting to receive a sent object. |
|
||
+----------------------+--------------------+------------------------------------------------+
|
||
| ChannelClosedError | ChannelError | The channel is closed. |
|
||
+----------------------+--------------------+------------------------------------------------+
|
||
| ChannelReleasedError | ChannelClosedError | The channel is released (but not yet closed). |
|
||
+----------------------+--------------------+------------------------------------------------+
|
||
|
||
|
||
Examples
|
||
========
|
||
|
||
Run isolated code
|
||
-----------------
|
||
|
||
::
|
||
|
||
interp = interpreters.create()
|
||
print('before')
|
||
interp.run('print("during")')
|
||
print('after')
|
||
|
||
Run in a thread
|
||
---------------
|
||
|
||
::
|
||
|
||
interp = interpreters.create()
|
||
def run():
|
||
interp.run('print("during")')
|
||
t = threading.Thread(target=run)
|
||
print('before')
|
||
t.start()
|
||
print('after')
|
||
|
||
Pre-populate an interpreter
|
||
---------------------------
|
||
|
||
::
|
||
|
||
interp = interpreters.create()
|
||
interp.run(tw.dedent("""
|
||
import some_lib
|
||
import an_expensive_module
|
||
some_lib.set_up()
|
||
"""))
|
||
wait_for_request()
|
||
interp.run(tw.dedent("""
|
||
some_lib.handle_request()
|
||
"""))
|
||
|
||
Handling an exception
|
||
---------------------
|
||
|
||
::
|
||
|
||
interp = interpreters.create()
|
||
try:
|
||
interp.run(tw.dedent("""
|
||
raise KeyError
|
||
"""))
|
||
except KeyError:
|
||
print("got the error from the subinterpreter")
|
||
|
||
Synchronize using a channel
|
||
---------------------------
|
||
|
||
::
|
||
|
||
interp = interpreters.create()
|
||
r, s = interpreters.create_channel()
|
||
def run():
|
||
interp.run(tw.dedent("""
|
||
reader.recv()
|
||
print("during")
|
||
reader.release()
|
||
"""),
|
||
shared=dict(
|
||
reader=r,
|
||
),
|
||
)
|
||
t = threading.Thread(target=run)
|
||
print('before')
|
||
t.start()
|
||
print('after')
|
||
s.send(b'')
|
||
s.release()
|
||
|
||
Sharing a file descriptor
|
||
-------------------------
|
||
|
||
::
|
||
|
||
interp = interpreters.create()
|
||
r1, s1 = interpreters.create_channel()
|
||
r2, s2 = interpreters.create_channel()
|
||
def run():
|
||
interp.run(tw.dedent("""
|
||
fd = int.from_bytes(
|
||
reader.recv(), 'big')
|
||
for line in os.fdopen(fd):
|
||
print(line)
|
||
writer.send(b'')
|
||
"""),
|
||
shared=dict(
|
||
reader=r,
|
||
writer=s2,
|
||
),
|
||
)
|
||
t = threading.Thread(target=run)
|
||
t.start()
|
||
with open('spamspamspam') as infile:
|
||
fd = infile.fileno().to_bytes(1, 'big')
|
||
s.send(fd)
|
||
r.recv()
|
||
|
||
Passing objects via marshal
|
||
---------------------------
|
||
|
||
::
|
||
|
||
interp = interpreters.create()
|
||
r, s = interpreters.create_fifo()
|
||
interp.run(tw.dedent("""
|
||
import marshal
|
||
"""),
|
||
shared=dict(
|
||
reader=r,
|
||
),
|
||
)
|
||
def run():
|
||
interp.run(tw.dedent("""
|
||
data = reader.recv()
|
||
while data:
|
||
obj = marshal.loads(data)
|
||
do_something(obj)
|
||
data = reader.recv()
|
||
reader.release()
|
||
"""))
|
||
t = threading.Thread(target=run)
|
||
t.start()
|
||
for obj in input:
|
||
data = marshal.dumps(obj)
|
||
s.send(data)
|
||
s.send(None)
|
||
|
||
Passing objects via pickle
|
||
--------------------------
|
||
|
||
::
|
||
|
||
interp = interpreters.create()
|
||
r, s = interpreters.create_channel()
|
||
interp.run(tw.dedent("""
|
||
import pickle
|
||
"""),
|
||
shared=dict(
|
||
reader=r,
|
||
),
|
||
)
|
||
def run():
|
||
interp.run(tw.dedent("""
|
||
data = reader.recv()
|
||
while data:
|
||
obj = pickle.loads(data)
|
||
do_something(obj)
|
||
data = reader.recv()
|
||
reader.release()
|
||
"""))
|
||
t = threading.Thread(target=run)
|
||
t.start()
|
||
for obj in input:
|
||
data = pickle.dumps(obj)
|
||
s.send(data)
|
||
s.send(None)
|
||
|
||
Running a module
|
||
----------------
|
||
|
||
::
|
||
|
||
interp = interpreters.create()
|
||
main_module = mod_name
|
||
interp.run(f'import runpy; runpy.run_module({main_module!r})')
|
||
|
||
Running as script (including zip archives & directories)
|
||
--------------------------------------------------------
|
||
|
||
::
|
||
|
||
interp = interpreters.create()
|
||
main_script = path_name
|
||
interp.run(f"import runpy; runpy.run_path({main_script!r})")
|
||
|
||
Running in a thread pool executor
|
||
---------------------------------
|
||
|
||
::
|
||
|
||
interps = [interpreters.create() for i in range(5)]
|
||
with concurrent.futures.ThreadPoolExecutor(max_workers=len(interps)) as pool:
|
||
print('before')
|
||
for interp in interps:
|
||
pool.submit(interp.run, 'print("starting"); print("stopping")'
|
||
print('after')
|
||
|
||
|
||
Rationale
|
||
=========
|
||
|
||
Running code in multiple interpreters provides a useful level of
|
||
isolation within the same process. This can be leveraged in a number
|
||
of ways. Furthermore, subinterpreters provide a well-defined framework
|
||
in which such isolation may extended.
|
||
|
||
Nick Coghlan explained some of the benefits through a comparison with
|
||
multi-processing [benefits]_::
|
||
|
||
[I] expect that communicating between subinterpreters is going
|
||
to end up looking an awful lot like communicating between
|
||
subprocesses via shared memory.
|
||
|
||
The trade-off between the two models will then be that one still
|
||
just looks like a single process from the point of view of the
|
||
outside world, and hence doesn't place any extra demands on the
|
||
underlying OS beyond those required to run CPython with a single
|
||
interpreter, while the other gives much stricter isolation
|
||
(including isolating C globals in extension modules), but also
|
||
demands much more from the OS when it comes to its IPC
|
||
capabilities.
|
||
|
||
The security risk profiles of the two approaches will also be quite
|
||
different, since using subinterpreters won't require deliberately
|
||
poking holes in the process isolation that operating systems give
|
||
you by default.
|
||
|
||
CPython has supported subinterpreters, with increasing levels of
|
||
support, since version 1.5. While the feature has the potential
|
||
to be a powerful tool, subinterpreters have suffered from neglect
|
||
because they are not available directly from Python. Exposing the
|
||
existing functionality in the stdlib will help reverse the situation.
|
||
|
||
This proposal is focused on enabling the fundamental capability of
|
||
multiple isolated interpreters in the same Python process. This is a
|
||
new area for Python so there is relative uncertainly about the best
|
||
tools to provide as companions to subinterpreters. Thus we minimize
|
||
the functionality we add in the proposal as much as possible.
|
||
|
||
Concerns
|
||
--------
|
||
|
||
* "subinterpreters are not worth the trouble"
|
||
|
||
Some have argued that subinterpreters do not add sufficient benefit
|
||
to justify making them an official part of Python. Adding features
|
||
to the language (or stdlib) has a cost in increasing the size of
|
||
the language. So an addition must pay for itself. In this case,
|
||
subinterpreters provide a novel concurrency model focused on isolated
|
||
threads of execution. Furthermore, they provide an opportunity for
|
||
changes in CPython that will allow simulateous use of multiple CPU
|
||
cores (currently prevented by the GIL).
|
||
|
||
Alternatives to subinterpreters include threading, async, and
|
||
multiprocessing. Threading is limited by the GIL and async isn't
|
||
the right solution for every problem (nor for every person).
|
||
Multiprocessing is likewise valuable in some but not all situations.
|
||
Direct IPC (rather than via the multiprocessing module) provides
|
||
similar benefits but with the same caveat.
|
||
|
||
Notably, subinterpreters are not intended as a replacement for any of
|
||
the above. Certainly they overlap in some areas, but the benefits of
|
||
subinterpreters include isolation and (potentially) performance. In
|
||
particular, subinterpreters provide a direct route to an alternate
|
||
concurrency model (e.g. CSP) which has found success elsewhere and
|
||
will appeal to some Python users. That is the core value that the
|
||
``interpreters`` module will provide.
|
||
|
||
* "stdlib support for subinterpreters adds extra burden
|
||
on C extension authors"
|
||
|
||
In the `Interpreter Isolation`_ section below we identify ways in
|
||
which isolation in CPython's subinterpreters is incomplete. Most
|
||
notable is extension modules that use C globals to store internal
|
||
state. PEP 3121 and PEP 489 provide a solution for most of the
|
||
problem, but one still remains. [petr-c-ext]_ Until that is resolved,
|
||
C extension authors will face extra difficulty to support
|
||
subinterpreters.
|
||
|
||
Consequently, projects that publish extension modules may face an
|
||
increased maintenance burden as their users start using subinterpreters,
|
||
where their modules may break. This situation is limited to modules
|
||
that use C globals (or use libraries that use C globals) to store
|
||
internal state. For numpy, the reported-bug rate is one every 6
|
||
months. [bug-rate]_
|
||
|
||
Ultimately this comes down to a question of how often it will be a
|
||
problem in practice: how many projects would be affected, how often
|
||
their users will be affected, what the additional maintenance burden
|
||
will be for projects, and what the overall benefit of subinterpreters
|
||
is to offset those costs. The position of this PEP is that the actual
|
||
extra maintenance burden will be small and well below the threshold at
|
||
which subinterpreters are worth it.
|
||
|
||
|
||
About Subinterpreters
|
||
=====================
|
||
|
||
Concurrency
|
||
-----------
|
||
|
||
Concurrency is a challenging area of software development. Decades of
|
||
research and practice have led to a wide variety of concurrency models,
|
||
each with different goals. Most center on correctness and usability.
|
||
|
||
One class of concurrency models focuses on isolated threads of
|
||
execution that interoperate through some message passing scheme. A
|
||
notable example is `Communicating Sequential Processes`_ (CSP), upon
|
||
which Go's concurrency is based. The isolation inherent to
|
||
subinterpreters makes them well-suited to this approach.
|
||
|
||
Shared data
|
||
-----------
|
||
|
||
Subinterpreters are inherently isolated (with caveats explained below),
|
||
in contrast to threads. So the same communicate-via-shared-memory
|
||
approach doesn't work. Without an alternative, effective use of
|
||
concurrency via subinterpreters is significantly limited.
|
||
|
||
The key challenge here is that sharing objects between interpreters
|
||
faces complexity due to various constraints on object ownership,
|
||
visibility, and mutability. At a conceptual level it's easier to
|
||
reason about concurrency when objects only exist in one interpreter
|
||
at a time. At a technical level, CPython's current memory model
|
||
limits how Python *objects* may be shared safely between interpreters;
|
||
effectively objects are bound to the interpreter in which they were
|
||
created. Furthermore the complexity of *object* sharing increases as
|
||
subinterpreters become more isolated, e.g. after GIL removal.
|
||
|
||
Consequently,the mechanism for sharing needs to be carefully considered.
|
||
There are a number of valid solutions, several of which may be
|
||
appropriate to support in Python. This proposal provides a single basic
|
||
solution: "channels". Ultimately, any other solution will look similar
|
||
to the proposed one, which will set the precedent. Note that the
|
||
implementation of ``Interpreter.run()`` can be done in a way that allows
|
||
for multiple solutions to coexist, but doing so is not technically
|
||
a part of the proposal here.
|
||
|
||
Regarding the proposed solution, "channels", it is a basic, opt-in data
|
||
sharing mechanism that draws inspiration from pipes, queues, and CSP's
|
||
channels. [fifo]_
|
||
|
||
As simply described earlier by the API summary,
|
||
channels have two operations: send and receive. A key characteristic
|
||
of those operations is that channels transmit data derived from Python
|
||
objects rather than the objects themselves. When objects are sent,
|
||
their data is extracted. When the "object" is received in the other
|
||
interpreter, the data is converted back into an object.
|
||
|
||
To make this work, the mutable shared state will be managed by the
|
||
Python runtime, not by any of the interpreters. Initially we will
|
||
support only one type of objects for shared state: the channels provided
|
||
by ``create_channel()``. Channels, in turn, will carefully manage
|
||
passing objects between interpreters.
|
||
|
||
This approach, including keeping the API minimal, helps us avoid further
|
||
exposing any underlying complexity to Python users. Along those same
|
||
lines, we will initially restrict the types that may be passed through
|
||
channels to the following:
|
||
|
||
* None
|
||
* bytes
|
||
* str
|
||
* int
|
||
* PEP 3118 buffer objects (via ``send_buffer()``)
|
||
|
||
Limiting the initial shareable types is a practical matter, reducing
|
||
the potential complexity of the initial implementation. There are a
|
||
number of strategies we may pursue in the future to expand supported
|
||
objects and object sharing strategies.
|
||
|
||
Interpreter Isolation
|
||
---------------------
|
||
|
||
CPython's interpreters are intended to be strictly isolated from each
|
||
other. Each interpreter has its own copy of all modules, classes,
|
||
functions, and variables. The same applies to state in C, including in
|
||
extension modules. The CPython C-API docs explain more. [caveats]_
|
||
|
||
However, there are ways in which interpreters share some state. First
|
||
of all, some process-global state remains shared:
|
||
|
||
* file descriptors
|
||
* builtin types (e.g. dict, bytes)
|
||
* singletons (e.g. None)
|
||
* underlying static module data (e.g. functions) for
|
||
builtin/extension/frozen modules
|
||
|
||
There are no plans to change this.
|
||
|
||
Second, some isolation is faulty due to bugs or implementations that did
|
||
not take subinterpreters into account. This includes things like
|
||
extension modules that rely on C globals. [cryptography]_ In these
|
||
cases bugs should be opened (some are already):
|
||
|
||
* readline module hook functions (http://bugs.python.org/issue4202)
|
||
* memory leaks on re-init (http://bugs.python.org/issue21387)
|
||
|
||
Finally, some potential isolation is missing due to the current design
|
||
of CPython. Improvements are currently going on to address gaps in this
|
||
area:
|
||
|
||
* interpreters share the GIL
|
||
* interpreters share memory management (e.g. allocators, gc)
|
||
* GC is not run per-interpreter [global-gc]_
|
||
* at-exit handlers are not run per-interpreter [global-atexit]_
|
||
* extensions using the ``PyGILState_*`` API are incompatible [gilstate]_
|
||
|
||
Existing Usage
|
||
--------------
|
||
|
||
Subinterpreters are not a widely used feature. In fact, the only
|
||
documented cases of wide-spread usage are
|
||
`mod_wsgi <https://github.com/GrahamDumpleton/mod_wsgi>`_ and
|
||
`JEP <https://github.com/ninia/jep>`_. On the one hand, this case
|
||
provides confidence that existing subinterpreter support is relatively
|
||
stable. On the other hand, there isn't much of a sample size from which
|
||
to judge the utility of the feature.
|
||
|
||
|
||
Provisional Status
|
||
==================
|
||
|
||
The new ``interpreters`` module will be added with "provisional" status
|
||
(see PEP 411). This allows Python users to experiment with the feature
|
||
and provide feedback while still allowing us to adjust to that feedback.
|
||
The module will be provisional in Python 3.7 and we will make a decision
|
||
before the 3.8 release whether to keep it provisional, graduate it, or
|
||
remove it.
|
||
|
||
|
||
Alternate Python Implementations
|
||
================================
|
||
|
||
I'll be soliciting feedback from the different Python implementors about
|
||
subinterpreter support.
|
||
|
||
Multiple-interpter support in the major Python implementations:
|
||
|
||
TBD
|
||
|
||
* jython: yes [jython]_
|
||
* ironpython: yes?
|
||
* pypy: maybe not? [pypy]_
|
||
* micropython: ???
|
||
|
||
|
||
"interpreters" Module API
|
||
=========================
|
||
|
||
The module provides the following functions:
|
||
|
||
``list_all()``::
|
||
|
||
Return a list of all existing interpreters.
|
||
|
||
``get_current()``::
|
||
|
||
Return the currently running interpreter.
|
||
|
||
``create()``::
|
||
|
||
Initialize a new Python interpreter and return it. The
|
||
interpreter will be created in the current thread and will remain
|
||
idle until something is run in it. The interpreter may be used
|
||
in any thread and will run in whichever thread calls
|
||
``interp.run()``.
|
||
|
||
|
||
The module also provides the following class:
|
||
|
||
``Interpreter(id)``::
|
||
|
||
id:
|
||
|
||
The interpreter's ID (read-only).
|
||
|
||
is_running():
|
||
|
||
Return whether or not the interpreter is currently executing code.
|
||
Calling this on the current interpreter will always return True.
|
||
|
||
destroy():
|
||
|
||
Finalize and destroy the interpreter.
|
||
|
||
This may not be called on an already running interpreter. Doing
|
||
so results in a RuntimeError.
|
||
|
||
run(source_str, /, *, channels=None):
|
||
|
||
Run the provided Python source code in the interpreter. If the
|
||
"channels" keyword argument is provided (and is a mapping of
|
||
attribute names to channels) then it is added to the interpreter's
|
||
execution namespace (the interpreter's "__main__" module). If any
|
||
of the values are not are not RecvChannel or SendChannel instances
|
||
then ValueError gets raised.
|
||
|
||
This may not be called on an already running interpreter. Doing
|
||
so results in a RuntimeError.
|
||
|
||
A "run()" call is similar to a function call. Once it completes,
|
||
the code that called "run()" continues executing (in the original
|
||
interpreter). Likewise, if there is any uncaught exception then
|
||
it effectively (see below) propagates into the code where
|
||
``run()`` was called. However, unlike function calls (but like
|
||
threads), there is no return value. If any value is needed, pass
|
||
it out via a channel.
|
||
|
||
The big difference is that "run()" executes the code in an
|
||
entirely different interpreter, with entirely separate state.
|
||
The state of the current interpreter in the current OS thread
|
||
is swapped out with the state of the target interpreter (the one
|
||
that will execute the code). When the target finishes executing,
|
||
the original interpreter gets swapped back in and its execution
|
||
resumes.
|
||
|
||
So calling "run()" will effectively cause the current Python
|
||
thread to pause. Sometimes you won't want that pause, in which
|
||
case you should make the "run()" call in another thread. To do
|
||
so, add a function that calls "run()" and then run that function
|
||
in a normal "threading.Thread".
|
||
|
||
Note that the interpreter's state is never reset, neither before
|
||
"run()" executes the code nor after. Thus the interpreter
|
||
state is preserved between calls to "run()". This includes
|
||
"sys.modules", the "builtins" module, and the internal state
|
||
of C extension modules.
|
||
|
||
Also note that "run()" executes in the namespace of the "__main__"
|
||
module, just like scripts, the REPL, "-m", and "-c". Just as
|
||
the interpreter's state is not ever reset, the "__main__" module
|
||
is never reset. You can imagine concatenating the code from each
|
||
"run()" call into one long script. This is the same as how the
|
||
REPL operates.
|
||
|
||
Supported code: source text.
|
||
|
||
|
||
Uncaught Exceptions
|
||
-------------------
|
||
|
||
Regarding uncaught exceptions in ``Interpreter.run()``, we noted that
|
||
they are "effectively" propagated into the code where ``run()`` was
|
||
called. To prevent leaking exceptions (and tracebacks) between
|
||
interpreters, we create a surrogate of the exception and its traceback
|
||
(see ``traceback.TracebackException``), set it to ``__cause__`` on a
|
||
new ``RunFailedError``, and raise that.
|
||
|
||
Raising (a proxy of) the exception is problematic since it's harder to
|
||
distinguish between an error in the ``run()`` call and an uncaught
|
||
exception from the subinterpreter.
|
||
|
||
|
||
API for sharing data
|
||
--------------------
|
||
|
||
Subinterpreters are less useful without a mechanism for sharing data
|
||
between them. Sharing actual Python objects between interpreters,
|
||
however, has enough potential problems that we are avoiding support
|
||
for that here. Instead, only mimimum set of types will be supported.
|
||
Initially this will include ``None``, ``bytes``, ``str``, ``int``,
|
||
and channels. Further types may be supported later.
|
||
|
||
The ``interpreters`` module provides a way for users to determine
|
||
whether an object is shareable or not:
|
||
|
||
``is_shareable(obj)``::
|
||
|
||
Return True if the object may be shared between interpreters. This
|
||
does not necessarily mean that the actual objects will be shared.
|
||
Insead, it means that the objects' underlying data will be shared in
|
||
a cross-interpreter way, whether via a proxy, a copy, or some other
|
||
means.
|
||
|
||
This proposal provides two ways to do share such objects between
|
||
interpreters.
|
||
|
||
First, shareable objects may be passed to ``run()`` as keyword arguments,
|
||
where they are effectively injected into the target interpreter's
|
||
``__main__`` module. This is mainly intended for sharing meta-objects
|
||
(e.g. channels) between interpreters, as it is less useful to pass other
|
||
objects (like ``bytes``) to ``run``.
|
||
|
||
Second, the main mechanism for sharing objects (i.e. their data) between
|
||
interpreters is through channels. A channel is a simplex FIFO similar
|
||
to a pipe. The main difference is that channels can be associated with
|
||
zero or more interpreters on either end. Unlike queues, which are also
|
||
many-to-many, channels have no buffer.
|
||
|
||
``create_channel()``::
|
||
|
||
Create a new channel and return (recv, send), the RecvChannel and
|
||
SendChannel corresponding to the ends of the channel. The channel
|
||
is not closed and destroyed (i.e. garbage-collected) until the number
|
||
of associated interpreters returns to 0 (including when the channel
|
||
is explicitly closed).
|
||
|
||
An interpreter gets associated with a channel by calling its "send()"
|
||
or "recv()" method. That association gets dropped by calling
|
||
"release()" on the channel.
|
||
|
||
Both ends of the channel are supported "shared" objects (i.e. may be
|
||
safely shared by different interpreters. Thus they may be passed as
|
||
keyword arguments to "Interpreter.run()".
|
||
|
||
``list_all_channels()``::
|
||
|
||
Return a list of all open (RecvChannel, SendChannel) pairs.
|
||
|
||
|
||
``RecvChannel(id)``::
|
||
|
||
The receiving end of a channel. An interpreter may use this to
|
||
receive objects from another interpreter. At first only bytes will
|
||
be supported.
|
||
|
||
id:
|
||
|
||
The channel's unique ID.
|
||
|
||
interpreters:
|
||
|
||
The list of associated interpreters: those that have called
|
||
the "recv()" or "__next__()" methods and haven't called
|
||
"release()" (and the channel hasn't been explicitly closed).
|
||
|
||
recv():
|
||
|
||
Return the next object (i.e. the data from the sent object) from
|
||
the channel. If none have been sent then wait until the next
|
||
send. This associates the current interpreter with the channel.
|
||
|
||
If the channel is already closed then raise ChannelClosedError.
|
||
If the channel isn't closed but the current interpreter already
|
||
called the "release()" method (which drops its association with
|
||
the channel) then raise ChannelReleasedError (which is a subclass
|
||
of ChannelClosedError).
|
||
|
||
recv_nowait(default=None):
|
||
|
||
Return the next object from the channel. If none have been sent
|
||
then return the default. Otherwise, this is the same as the
|
||
"recv()" method.
|
||
|
||
release():
|
||
|
||
No longer associate the current interpreter with the channel (on
|
||
the receiving end) and block future association (via the "recv()"
|
||
method). If the interpreter was never associated with the channel
|
||
then still block future association. Once an interpreter is no
|
||
longer associated with the channel, subsequent (or current) send()
|
||
and recv() calls from that interpreter will raise
|
||
ChannelReleasedError (or ChannelClosedError if the channel
|
||
is actually marked as closed).
|
||
|
||
Once the number of associated interpreters on both ends drops
|
||
to 0, the channel is actually marked as closed. The Python
|
||
runtime will garbage collect all closed channels, though it may
|
||
not be immediately. Note that "release()" is automatically called
|
||
in behalf of the current interpreter when the channel is no longer
|
||
used (i.e. has no references) in that interpreter.
|
||
|
||
This operation is idempotent. Return True if "release()" has not
|
||
been called before by the current interpreter.
|
||
|
||
close(force=False):
|
||
|
||
Close both ends of the channel (in all interpreters). This means
|
||
that any further use of the channel raises ChannelClosedError. If
|
||
the channel is not empty then raise ChannelNotEmptyError (if
|
||
"force" is False) or discard the remaining objects (if "force"
|
||
is True) and close it.
|
||
|
||
|
||
``SendChannel(id)``::
|
||
|
||
The sending end of a channel. An interpreter may use this to send
|
||
objects to another interpreter. At first only bytes will be
|
||
supported.
|
||
|
||
id:
|
||
|
||
The channel's unique ID.
|
||
|
||
interpreters:
|
||
|
||
The list of associated interpreters (those that have called
|
||
the "send()" method).
|
||
|
||
send(obj):
|
||
|
||
Send the object (i.e. its data) to the receiving end of the
|
||
channel. Wait until the object is received. If the the
|
||
object is not shareable then ValueError is raised. Currently
|
||
only bytes are supported.
|
||
|
||
If the channel is already closed then raise ChannelClosedError.
|
||
If the channel isn't closed but the current interpreter already
|
||
called the "release()" method (which drops its association with
|
||
the channel) then raise ChannelReleasedError.
|
||
|
||
send_nowait(obj):
|
||
|
||
Send the object to the receiving end of the channel. If the other
|
||
end is not currently receiving then raise NotReceivedError.
|
||
Otherwise this is the same as "send()".
|
||
|
||
send_buffer(obj):
|
||
|
||
Send a MemoryView of the object rather than the object. Otherwise
|
||
this is the same as send(). Note that the object must implement
|
||
the PEP 3118 buffer protocol.
|
||
|
||
send_buffer_nowait(obj):
|
||
|
||
Send a MemoryView of the object rather than the object. If the
|
||
other end is not currently receiving then raise NotReceivedError.
|
||
Otherwise this is the same as "send_buffer()".
|
||
|
||
release():
|
||
|
||
This is the same as "RecvChannel.release(), but applied to the
|
||
sending end of the channel.
|
||
|
||
close(force=False):
|
||
|
||
Close both ends of the channel (in all interpreters). No matter
|
||
what the "send" end of the channel is immediately closed. If the
|
||
channel is empty then close the "recv" end immediately too.
|
||
Otherwise wait until the channel is empty before closing it (if
|
||
"force" is False) or discard the remaining items and close
|
||
immediately (if "force" is True).
|
||
|
||
Note that ``send_buffer()`` is similar to how
|
||
``multiprocessing.Connection`` works. [mp-conn]_
|
||
|
||
|
||
Open Questions
|
||
==============
|
||
|
||
* "force" argument to ``ch.release()``?
|
||
* add a "tp_share" type slot instead of using a global registry
|
||
for shareable types?
|
||
|
||
|
||
Open Implementation Questions
|
||
=============================
|
||
|
||
Does every interpreter think that their thread is the "main" thread?
|
||
--------------------------------------------------------------------
|
||
|
||
(This is more of an implementation detail that an issue for the PEP.)
|
||
|
||
CPython's interpreter implementation identifies the OS thread in which
|
||
it was started as the "main" thread. The interpreter the has slightly
|
||
different behavior depending on if the current thread is the main one
|
||
or not. This presents a problem in cases where "main thread" is meant
|
||
to imply "main thread in the main interpreter" [main-thread]_, where
|
||
the main interpreter is the initial one.
|
||
|
||
Disallow subinterpreters in the main thread?
|
||
--------------------------------------------
|
||
|
||
(This is more of an implementation detail that an issue for the PEP.)
|
||
|
||
This is a specific case of the above issue. Currently in CPython,
|
||
"we need a main \*thread\* in order to sensibly manage the way signal
|
||
handling works across different platforms". [main-thread]_
|
||
|
||
Since signal handlers are part of the interpreter state, running a
|
||
subinterpreter in the main thread means that the main interpreter
|
||
can no longer properly handle signals (since it's effectively paused).
|
||
|
||
Furthermore, running a subinterpreter in the main thread would
|
||
conceivably allow setting signal handlers on that interpreter, which
|
||
would likewise impact signal handling when that interpreter isn't
|
||
running or is running in a different thread.
|
||
|
||
Ultimately, running subinterpreters in the main OS thread introduces
|
||
complications to the signal handling implementation. So it may make
|
||
the most sense to disallow running subinterpreters in the main thread.
|
||
Support for it could be considered later. The downside is that folks
|
||
wanting to try out subinterpreters would be required to take the extra
|
||
step of using threads. This could slow adoption and experimentation,
|
||
whereas without the restriction there's less of an obstacle.
|
||
|
||
|
||
Deferred Functionality
|
||
======================
|
||
|
||
In the interest of keeping this proposal minimal, the following
|
||
functionality has been left out for future consideration. Note that
|
||
this is not a judgement against any of said capability, but rather a
|
||
deferment. That said, each is arguably valid.
|
||
|
||
Interpreter.call()
|
||
------------------
|
||
|
||
It would be convenient to run existing functions in subinterpreters
|
||
directly. ``Interpreter.run()`` could be adjusted to support this or
|
||
a ``call()`` method could be added::
|
||
|
||
Interpreter.call(f, *args, **kwargs)
|
||
|
||
This suffers from the same problem as sharing objects between
|
||
interpreters via queues. The minimal solution (running a source string)
|
||
is sufficient for us to get the feature out where it can be explored.
|
||
|
||
timeout arg to recv() and send()
|
||
--------------------------------
|
||
|
||
Typically functions that have a ``block`` argument also have a
|
||
``timeout`` argument. It sometimes makes sense to do likewise for
|
||
functions that otherwise block, like the channel ``recv()`` and
|
||
``send()`` methods. We can add it later if needed.
|
||
|
||
get_main()
|
||
----------
|
||
|
||
CPython has a concept of a "main" interpreter. This is the initial
|
||
interpreter created during CPython's runtime initialization. It may
|
||
be useful to identify the main interpreter. For instance, the main
|
||
interpreter should not be destroyed. However, for the basic
|
||
functionality of a high-level API a ``get_main()`` function is not
|
||
necessary. Furthermore, there is no requirement that a Python
|
||
implementation have a concept of a main interpreter. So until there's
|
||
a clear need we'll leave ``get_main()`` out.
|
||
|
||
Interpreter.run_in_thread()
|
||
---------------------------
|
||
|
||
This method would make a ``run()`` call for you in a thread. Doing this
|
||
using only ``threading.Thread`` and ``run()`` is relatively trivial so
|
||
we've left it out.
|
||
|
||
Synchronization Primitives
|
||
--------------------------
|
||
|
||
The ``threading`` module provides a number of synchronization primitives
|
||
for coordinating concurrent operations. This is especially necessary
|
||
due to the shared-state nature of threading. In contrast,
|
||
subinterpreters do not share state. Data sharing is restricted to
|
||
channels, which do away with the need for explicit synchronization. If
|
||
any sort of opt-in shared state support is added to subinterpreters in
|
||
the future, that same effort can introduce synchronization primitives
|
||
to meet that need.
|
||
|
||
CSP Library
|
||
-----------
|
||
|
||
A ``csp`` module would not be a large step away from the functionality
|
||
provided by this PEP. However, adding such a module is outside the
|
||
minimalist goals of this proposal.
|
||
|
||
Syntactic Support
|
||
-----------------
|
||
|
||
The ``Go`` language provides a concurrency model based on CSP, so
|
||
it's similar to the concurrency model that subinterpreters support.
|
||
``Go`` provides syntactic support, as well several builtin concurrency
|
||
primitives, to make concurrency a first-class feature. Conceivably,
|
||
similar syntactic (and builtin) support could be added to Python using
|
||
subinterpreters. However, that is *way* outside the scope of this PEP!
|
||
|
||
Multiprocessing
|
||
---------------
|
||
|
||
The ``multiprocessing`` module could support subinterpreters in the same
|
||
way it supports threads and processes. In fact, the module's
|
||
maintainer, Davin Potts, has indicated this is a reasonable feature
|
||
request. However, it is outside the narrow scope of this PEP.
|
||
|
||
C-extension opt-in/opt-out
|
||
--------------------------
|
||
|
||
By using the ``PyModuleDef_Slot`` introduced by PEP 489, we could easily
|
||
add a mechanism by which C-extension modules could opt out of support
|
||
for subinterpreters. Then the import machinery, when operating in
|
||
a subinterpreter, would need to check the module for support. It would
|
||
raise an ImportError if unsupported.
|
||
|
||
Alternately we could support opting in to subinterpreter support.
|
||
However, that would probably exclude many more modules (unnecessarily)
|
||
than the opt-out approach.
|
||
|
||
The scope of adding the ModuleDef slot and fixing up the import
|
||
machinery is non-trivial, but could be worth it. It all depends on
|
||
how many extension modules break under subinterpreters. Given the
|
||
relatively few cases we know of through mod_wsgi, we can leave this
|
||
for later.
|
||
|
||
Poisoning channels
|
||
------------------
|
||
|
||
CSP has the concept of poisoning a channel. Once a channel has been
|
||
poisoned, and ``send()`` or ``recv()`` call on it will raise a special
|
||
exception, effectively ending execution in the interpreter that tried
|
||
to use the poisoned channel.
|
||
|
||
This could be accomplished by adding a ``poison()`` method to both ends
|
||
of the channel. The ``close()`` method can be used in this way
|
||
(mostly), but these semantics are relatively specialized and can wait.
|
||
|
||
Sending channels over channels
|
||
------------------------------
|
||
|
||
Some advanced usage of subinterpreters could take advantage of the
|
||
ability to send channels over channels, in addition to bytes. Given
|
||
that channels will already be multi-interpreter safe, supporting then
|
||
in ``RecvChannel.recv()`` wouldn't be a big change. However, this can
|
||
wait until the basic functionality has been ironed out.
|
||
|
||
Reseting __main__
|
||
-----------------
|
||
|
||
As proposed, every call to ``Interpreter.run()`` will execute in the
|
||
namespace of the interpreter's existing ``__main__`` module. This means
|
||
that data persists there between ``run()`` calls. Sometimes this isn't
|
||
desireable and you want to execute in a fresh ``__main__``. Also,
|
||
you don't necessarily want to leak objects there that you aren't using
|
||
any more.
|
||
|
||
Note that the following won't work right because it will clear too much
|
||
(e.g. ``__name__`` and the other "__dunder__" attributes::
|
||
|
||
interp.run('globals().clear()')
|
||
|
||
Possible solutions include:
|
||
|
||
* a ``create()`` arg to indicate resetting ``__main__`` after each
|
||
``run`` call
|
||
* an ``Interpreter.reset_main`` flag to support opting in or out
|
||
after the fact
|
||
* an ``Interpreter.reset_main()`` method to opt in when desired
|
||
* ``importlib.util.reset_globals()`` [reset_globals]_
|
||
|
||
Also note that reseting ``__main__`` does nothing about state stored
|
||
in other modules. So any solution would have to be clear about the
|
||
scope of what is being reset. Conceivably we could invent a mechanism
|
||
by which any (or every) module could be reset, unlike ``reload()``
|
||
which does not clear the module before loading into it. Regardless,
|
||
since ``__main__`` is the execution namespace of the interpreter,
|
||
resetting it has a much more direct correlation to interpreters and
|
||
their dynamic state than does resetting other modules. So a more
|
||
generic module reset mechanism may prove unnecessary.
|
||
|
||
This isn't a critical feature initially. It can wait until later
|
||
if desirable.
|
||
|
||
File descriptors and sockets in channels
|
||
----------------------------------------
|
||
|
||
Given that file descriptors and sockets are process-global resources,
|
||
support for passing them through channels is a reasonable idea. They
|
||
would be a good candidate for the first effort at expanding the types
|
||
that channels support. They aren't strictly necessary for the initial
|
||
API.
|
||
|
||
Integration with async
|
||
----------------------
|
||
|
||
Per Antoine Pitrou [async]_::
|
||
|
||
Has any thought been given to how FIFOs could integrate with async
|
||
code driven by an event loop (e.g. asyncio)? I think the model of
|
||
executing several asyncio (or Tornado) applications each in their
|
||
own subinterpreter may prove quite interesting to reconcile multi-
|
||
core concurrency with ease of programming. That would require the
|
||
FIFOs to be able to synchronize on something an event loop can wait
|
||
on (probably a file descriptor?).
|
||
|
||
A possible solution is to provide async implementations of the blocking
|
||
channel methods (``__next__()``, ``recv()``, and ``send()``). However,
|
||
the basic functionality of subinterpreters does not depend on async and
|
||
can be added later.
|
||
|
||
Support for iteration
|
||
---------------------
|
||
|
||
Supporting iteration on ``RecvChannel`` (via ``__iter__()`` or
|
||
``_next__()``) may be useful. A trivial implementation would use the
|
||
``recv()`` method, similar to how files do iteration. Since this isn't
|
||
a fundamental capability and has a simple analog, adding iteration
|
||
support can wait until later.
|
||
|
||
Channel context managers
|
||
------------------------
|
||
|
||
Context manager support on ``RecvChannel`` and ``SendChannel`` may be
|
||
helpful. The implementation would be simple, wrapping a call to
|
||
``close()`` (or maybe ``release()``) like files do. As with iteration,
|
||
this can wait.
|
||
|
||
Pipes and Queues
|
||
----------------
|
||
|
||
With the proposed object passing machanism of "channels", other similar
|
||
basic types aren't required to achieve the minimal useful functionality
|
||
of subinterpreters. Such types include pipes (like channels, but
|
||
one-to-one) and queues (like channels, but buffered). See below in
|
||
`Rejected Ideas` for more information.
|
||
|
||
Even though these types aren't part of this proposal, they may still
|
||
be useful in the context of concurrency. Adding them later is entirely
|
||
reasonable. The could be trivially implemented as wrappers around
|
||
channels. Alternatively they could be implemented for efficiency at the
|
||
same low level as channels.
|
||
|
||
Buffering
|
||
---------
|
||
|
||
The proposed channels are unbuffered. This simplifies the API and
|
||
implementation. If buffering is desireable we can add it later.
|
||
|
||
Return a lock from send()
|
||
-------------------------
|
||
|
||
When sending an object through a channel, you don't have a way of knowing
|
||
when the object gets received on the other end. One way to work around
|
||
this is to return a locked ``threading.Lock`` from ``SendChannel.send()``
|
||
that unlocks once the object is received.
|
||
|
||
This matters for buffered channels (i.e. queues). For unbuffered
|
||
channels it is a non-issue. So this can be dealt with once channels
|
||
support buffering.
|
||
|
||
Add a "reraise" method to RunFailedError
|
||
----------------------------------------
|
||
|
||
While having ``__cause__`` set on ``RunFailedError`` helps produce a
|
||
more useful traceback, it's less helpful when handling the original
|
||
error. To help facilitate this, we could add
|
||
``RunFailedError.reraise()``. This method would enable the following
|
||
pattern::
|
||
|
||
try:
|
||
interp.run(script)
|
||
except RunFailedError as exc:
|
||
try:
|
||
exc.reraise()
|
||
except MyException:
|
||
...
|
||
|
||
This would be made even simpler if there existed a ``__reraise__``
|
||
protocol.
|
||
|
||
|
||
Rejected Ideas
|
||
==============
|
||
|
||
Explicit channel association
|
||
----------------------------
|
||
|
||
Interpreters are implicitly associated with channels upon ``recv()`` and
|
||
``send()`` calls. They are de-associated with ``release()`` calls. The
|
||
alternative would be explicit methods. It would be either
|
||
``add_channel()`` and ``remove_channel()`` methods on ``Interpreter``
|
||
objects or something similar on channel objects.
|
||
|
||
In practice, this level of management shouldn't be necessary for users.
|
||
So adding more explicit support would only add clutter to the API.
|
||
|
||
Use pipes instead of channels
|
||
-----------------------------
|
||
|
||
A pipe would be a simplex FIFO between exactly two interpreters. For
|
||
most use cases this would be sufficient. It could potentially simplify
|
||
the implementation as well. However, it isn't a big step to supporting
|
||
a many-to-many simplex FIFO via channels. Also, with pipes the API
|
||
ends up being slightly more complicated, requiring naming the pipes.
|
||
|
||
Use queues instead of channels
|
||
------------------------------
|
||
|
||
The main difference between queues and channels is that queues support
|
||
buffering. This would complicate the blocking semantics of ``recv()``
|
||
and ``send()``. Also, queues can be built on top of channels.
|
||
|
||
"enumerate"
|
||
-----------
|
||
|
||
The ``list_all()`` function provides the list of all interpreters.
|
||
In the threading module, which partly inspired the proposed API, the
|
||
function is called ``enumerate()``. The name is different here to
|
||
avoid confusing Python users that are not already familiar with the
|
||
threading API. For them "enumerate" is rather unclear, whereas
|
||
"list_all" is clear.
|
||
|
||
Alternate solutions to prevent leaking exceptions across interpreters
|
||
---------------------------------------------------------------------
|
||
|
||
In function calls, uncaught exceptions propagate to the calling frame.
|
||
The same approach could be taken with ``run()``. However, this would
|
||
mean that exception objects would leak across the inter-interpreter
|
||
boundary. Likewise, the frames in the traceback would potentially leak.
|
||
|
||
While that might not be a problem currently, it would be a problem once
|
||
interpreters get better isolation relative to memory management (which
|
||
is necessary to stop sharing the GIL between interpreters). We've
|
||
resolved the semantics of how the exceptions propagate by raising a
|
||
``RunFailedError`` instead, for which ``__cause__`` wraps a safe proxy
|
||
for the original exception and traceback.
|
||
|
||
Rejected possible solutions:
|
||
|
||
* reproduce the exception and traceback in the original interpreter
|
||
and raise that.
|
||
* raise a subclass of RunFailedError that proxies the original
|
||
exception and traceback.
|
||
* raise RuntimeError instead of RunFailedError
|
||
* convert at the boundary (a la ``subprocess.CalledProcessError``)
|
||
(requires a cross-interpreter representation)
|
||
* support customization via ``Interpreter.excepthook``
|
||
(requires a cross-interpreter representation)
|
||
* wrap in a proxy at the boundary (including with support for
|
||
something like ``err.raise()`` to propagate the traceback).
|
||
* return the exception (or its proxy) from ``run()`` instead of
|
||
raising it
|
||
* return a result object (like ``subprocess`` does) [result-object]_
|
||
(unecessary complexity?)
|
||
* throw the exception away and expect users to deal with unhandled
|
||
exceptions explicitly in the script they pass to ``run()``
|
||
(they can pass error info out via channels); with threads you have
|
||
to do something similar
|
||
|
||
|
||
References
|
||
==========
|
||
|
||
.. [c-api]
|
||
https://docs.python.org/3/c-api/init.html#sub-interpreter-support
|
||
|
||
.. _Communicating Sequential Processes:
|
||
|
||
.. [CSP]
|
||
https://en.wikipedia.org/wiki/Communicating_sequential_processes
|
||
https://github.com/futurecore/python-csp
|
||
|
||
.. [fifo]
|
||
https://docs.python.org/3/library/multiprocessing.html#multiprocessing.Pipe
|
||
https://docs.python.org/3/library/multiprocessing.html#multiprocessing.Queue
|
||
https://docs.python.org/3/library/queue.html#module-queue
|
||
http://stackless.readthedocs.io/en/2.7-slp/library/stackless/channels.html
|
||
https://golang.org/doc/effective_go.html#sharing
|
||
http://www.jtolds.com/writing/2016/03/go-channels-are-bad-and-you-should-feel-bad/
|
||
|
||
.. [caveats]
|
||
https://docs.python.org/3/c-api/init.html#bugs-and-caveats
|
||
|
||
.. [petr-c-ext]
|
||
https://mail.python.org/pipermail/import-sig/2016-June/001062.html
|
||
https://mail.python.org/pipermail/python-ideas/2016-April/039748.html
|
||
|
||
.. [cryptography]
|
||
https://github.com/pyca/cryptography/issues/2299
|
||
|
||
.. [global-gc]
|
||
http://bugs.python.org/issue24554
|
||
|
||
.. [gilstate]
|
||
https://bugs.python.org/issue10915
|
||
http://bugs.python.org/issue15751
|
||
|
||
.. [global-atexit]
|
||
https://bugs.python.org/issue6531
|
||
|
||
.. [mp-conn]
|
||
https://docs.python.org/3/library/multiprocessing.html#multiprocessing.Connection
|
||
|
||
.. [bug-rate]
|
||
https://mail.python.org/pipermail/python-ideas/2017-September/047094.html
|
||
|
||
.. [benefits]
|
||
https://mail.python.org/pipermail/python-ideas/2017-September/047122.html
|
||
|
||
.. [main-thread]
|
||
https://mail.python.org/pipermail/python-ideas/2017-September/047144.html
|
||
https://mail.python.org/pipermail/python-dev/2017-September/149566.html
|
||
|
||
.. [reset_globals]
|
||
https://mail.python.org/pipermail/python-dev/2017-September/149545.html
|
||
|
||
.. [async]
|
||
https://mail.python.org/pipermail/python-dev/2017-September/149420.html
|
||
https://mail.python.org/pipermail/python-dev/2017-September/149585.html
|
||
|
||
.. [result-object]
|
||
https://mail.python.org/pipermail/python-dev/2017-September/149562.html
|
||
|
||
.. [jython]
|
||
https://mail.python.org/pipermail/python-ideas/2017-May/045771.html
|
||
|
||
.. [pypy]
|
||
https://mail.python.org/pipermail/python-ideas/2017-September/046973.html
|
||
|
||
|
||
Copyright
|
||
=========
|
||
|
||
This document has been placed in the public domain.
|
||
|
||
|
||
|
||
..
|
||
Local Variables:
|
||
mode: indented-text
|
||
indent-tabs-mode: nil
|
||
sentence-end-double-space: t
|
||
fill-column: 70
|
||
coding: utf-8
|
||
End:
|