iSharkFly-Open
/
python-peps
mirror of https://github.com/python/peps.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

PEP 554: Seasonal Updates. (#944)

This commit is contained in:
Eric Snow 2019-03-23 00:12:14 -06:00 committed by GitHub
parent ad0f49e0a2
commit 1ff0e1d362
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 119 additions and 104 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

223
pep-0554.rst
View File

@ -15,7 +15,7 @@ Abstract
CPython has supported multiple interpreters in the same process (AKA CPython has supported multiple interpreters in the same process (AKA
"subinterpreters") since version 1.5 (1997). The feature has been "subinterpreters") since version 1.5 (1997). The feature has been
available via the C-API. [c-api]_ Subinterpreters operate in available via the C-API. [c-api]_ Subinterpreters operate in
`relative isolation from one another <Interpreter Isolation_>`_, which `relative isolation from one another <Interpreter Isolation_>`_, which
provides the basis for an provides the basis for an
`alternative concurrency model <Concurrency_>`_. `alternative concurrency model <Concurrency_>`_.
@ -51,6 +51,7 @@ At first only the following types will be supported for sharing:
* str * str
* int * int
* PEP 3118 buffer objects (via ``send_buffer()``) * PEP 3118 buffer objects (via ``send_buffer()``)
* PEP 554 channels
Support for other basic types (e.g. bool, float, Ellipsis) will be added later. Support for other basic types (e.g. bool, float, Ellipsis) will be added later.
@ -152,7 +153,7 @@ For sharing data between interpreters:
| | | receiving end of the channel and wait. | | | | receiving end of the channel and wait. |
| | | Associate the interpreter with the channel. | | | | Associate the interpreter with the channel. |
+---------------------------+-------------------------------------------------+ +---------------------------+-------------------------------------------------+
| .send_nowait(obj) | | Like send(), but Fail if not received. | | .send_nowait(obj) | | Like send(), but fail if not received. |
+---------------------------+-------------------------------------------------+ +---------------------------+-------------------------------------------------+
| .send_buffer(obj) | | Send the object's (PEP 3118) buffer to the | | .send_buffer(obj) | | Send the object's (PEP 3118) buffer to the |
| | | receiving end of the channel and wait. | | | | receiving end of the channel and wait. |
@ -242,6 +243,24 @@ Handling an exception
except interpreters.RunFailedError as exc: except interpreters.RunFailedError as exc:
print(f"got the error from the subinterpreter: {exc}") print(f"got the error from the subinterpreter: {exc}")
Re-raising an exception
-----------------------
::
interp = interpreters.create()
try:
try:
interp.run(tw.dedent("""
raise KeyError
"""))
except interpreters.RunFailedError as exc:
raise exc.__cause__
except KeyError:
print("got a KeyError from the subinterpreter")
Note that this pattern is a candidate for later improvement.
Synchronize using a channel Synchronize using a channel
--------------------------- ---------------------------
@ -494,8 +513,8 @@ each with different goals. Most center on correctness and usability.
One class of concurrency models focuses on isolated threads of One class of concurrency models focuses on isolated threads of
execution that interoperate through some message passing scheme. A execution that interoperate through some message passing scheme. A
notable example is `Communicating Sequential Processes`_ (CSP), upon notable example is `Communicating Sequential Processes`_ (CSP) (upon
which Go's concurrency is based. The isolation inherent to which Go's concurrency is roughly based). The isolation inherent to
subinterpreters makes them well-suited to this approach. subinterpreters makes them well-suited to this approach.
Shared data Shared data
@ -521,9 +540,9 @@ There are a number of valid solutions, several of which may be
appropriate to support in Python. This proposal provides a single basic appropriate to support in Python. This proposal provides a single basic
solution: "channels". Ultimately, any other solution will look similar solution: "channels". Ultimately, any other solution will look similar
to the proposed one, which will set the precedent. Note that the to the proposed one, which will set the precedent. Note that the
implementation of ``Interpreter.run()`` can be done in a way that allows implementation of ``Interpreter.run()`` will be done in a way that
for multiple solutions to coexist, but doing so is not technically allows for multiple solutions to coexist, but doing so is not
a part of the proposal here. technically a part of the proposal here.
Regarding the proposed solution, "channels", it is a basic, opt-in data Regarding the proposed solution, "channels", it is a basic, opt-in data
sharing mechanism that draws inspiration from pipes, queues, and CSP's sharing mechanism that draws inspiration from pipes, queues, and CSP's
@ -534,7 +553,8 @@ channels have two operations: send and receive. A key characteristic
of those operations is that channels transmit data derived from Python of those operations is that channels transmit data derived from Python
objects rather than the objects themselves. When objects are sent, objects rather than the objects themselves. When objects are sent,
their data is extracted. When the "object" is received in the other their data is extracted. When the "object" is received in the other
interpreter, the data is converted back into an object. interpreter, the data is converted back into an object owned by that
interpreter.
To make this work, the mutable shared state will be managed by the To make this work, the mutable shared state will be managed by the
Python runtime, not by any of the interpreters. Initially we will Python runtime, not by any of the interpreters. Initially we will
@ -552,6 +572,7 @@ channels to the following:
* str * str
* int * int
* PEP 3118 buffer objects (via ``send_buffer()``) * PEP 3118 buffer objects (via ``send_buffer()``)
* channels
Limiting the initial shareable types is a practical matter, reducing Limiting the initial shareable types is a practical matter, reducing
the potential complexity of the initial implementation. There are a the potential complexity of the initial implementation. There are a
@ -589,11 +610,11 @@ Finally, some potential isolation is missing due to the current design
of CPython. Improvements are currently going on to address gaps in this of CPython. Improvements are currently going on to address gaps in this
area: area:
* interpreters share the GIL
* interpreters share memory management (e.g. allocators, gc)
* GC is not run per-interpreter [global-gc]_ * GC is not run per-interpreter [global-gc]_
* at-exit handlers are not run per-interpreter [global-atexit]_ * at-exit handlers are not run per-interpreter [global-atexit]_
* extensions using the ``PyGILState_*`` API are incompatible [gilstate]_ * extensions using the ``PyGILState_*`` API are incompatible [gilstate]_
* interpreters share memory management (e.g. allocators, gc)
* interpreters share the GIL
Existing Usage Existing Usage
-------------- --------------
@ -683,7 +704,7 @@ The module also provides the following class:
"channels" keyword argument is provided (and is a mapping of "channels" keyword argument is provided (and is a mapping of
attribute names to channels) then it is added to the interpreter's attribute names to channels) then it is added to the interpreter's
execution namespace (the interpreter's "__main__" module). If any execution namespace (the interpreter's "__main__" module). If any
of the values are not are not RecvChannel or SendChannel instances of the values are not RecvChannel or SendChannel instances
then ValueError gets raised. then ValueError gets raised.
This may not be called on an already running interpreter. Doing This may not be called on an already running interpreter. Doing
@ -737,9 +758,9 @@ interpreters, we create a surrogate of the exception and its traceback
(see ``traceback.TracebackException``), set it to ``__cause__`` on a (see ``traceback.TracebackException``), set it to ``__cause__`` on a
new ``RunFailedError``, and raise that. new ``RunFailedError``, and raise that.
Raising (a proxy of) the exception is problematic since it's harder to Raising (a proxy of) the exception directly is problematic since it's
distinguish between an error in the ``run()`` call and an uncaught harder to distinguish between an error in the ``run()`` call and an
exception from the subinterpreter. uncaught exception from the subinterpreter.
API for sharing data API for sharing data
@ -763,14 +784,15 @@ whether an object is shareable or not:
a cross-interpreter way, whether via a proxy, a copy, or some other a cross-interpreter way, whether via a proxy, a copy, or some other
means. means.
This proposal provides two ways to do share such objects between This proposal provides two ways to share such objects between
interpreters. interpreters.
First, shareable objects may be passed to ``run()`` as keyword arguments, First, channels may be passed to ``run()`` via the ``channels``
where they are effectively injected into the target interpreter's keyword argument, where they are effectively injected into the target
``__main__`` module. This is mainly intended for sharing meta-objects interpreter's ``__main__`` module. While passing arbitrary shareable
(e.g. channels) between interpreters, as it is less useful to pass other objects this way is possible, doing so is mainly intended for sharing
objects (like ``bytes``) to ``run``. meta-objects (e.g. channels) between interpreters. It is less useful
to pass other objects (like ``bytes``) to ``run`` directly.
Second, the main mechanism for sharing objects (i.e. their data) between Second, the main mechanism for sharing objects (i.e. their data) between
interpreters is through channels. A channel is a simplex FIFO similar interpreters is through channels. A channel is a simplex FIFO similar
@ -778,6 +800,9 @@ to a pipe. The main difference is that channels can be associated with
zero or more interpreters on either end. Unlike queues, which are also zero or more interpreters on either end. Unlike queues, which are also
many-to-many, channels have no buffer. many-to-many, channels have no buffer.
The ``interpreters`` module provides the following functions and
classes related to channels:
``create_channel()``:: ``create_channel()``::
Create a new channel and return (recv, send), the RecvChannel and Create a new channel and return (recv, send), the RecvChannel and
@ -802,24 +827,25 @@ many-to-many, channels have no buffer.
``RecvChannel(id)``:: ``RecvChannel(id)``::
The receiving end of a channel. An interpreter may use this to The receiving end of a channel. An interpreter may use this to
receive objects from another interpreter. At first only bytes will receive objects from another interpreter. At first only a few of
be supported. the simple, immutable builtin types will be supported.
id: id:
The channel's unique ID. The channel's unique ID. This is shared with the "send" end.
interpreters: interpreters:
The list of associated interpreters: those that have called The list of associated interpreters: those that have called
the "recv()" or "__next__()" methods and haven't called the "recv()" method and haven't called "release()" (and the
"release()" (and the channel hasn't been explicitly closed). channel hasn't been explicitly closed).
recv(): recv():
Return the next object (i.e. the data from the sent object) from Return the next object (i.e. the data from the sent object) from
the channel. If none have been sent then wait until the next the channel. If none have been sent then wait until the next
send. This associates the current interpreter with the channel. send. This associates the current interpreter with the "recv"
end of the channel.
If the channel is already closed then raise ChannelClosedError. If the channel is already closed then raise ChannelClosedError.
If the channel isn't closed but the current interpreter already If the channel isn't closed but the current interpreter already
@ -848,7 +874,7 @@ many-to-many, channels have no buffer.
to 0, the channel is actually marked as closed. The Python to 0, the channel is actually marked as closed. The Python
runtime will garbage collect all closed channels, though it may runtime will garbage collect all closed channels, though it may
not be immediately. Note that "release()" is automatically called not be immediately. Note that "release()" is automatically called
in behalf of the current interpreter when the channel is no longer on behalf of the current interpreter when the channel is no longer
used (i.e. has no references) in that interpreter. used (i.e. has no references) in that interpreter.
This operation is idempotent. Return True if "release()" has not This operation is idempotent. Return True if "release()" has not
@ -857,21 +883,21 @@ many-to-many, channels have no buffer.
close(force=False): close(force=False):
Close both ends of the channel (in all interpreters). This means Close both ends of the channel (in all interpreters). This means
that any further use of the channel raises ChannelClosedError. If that any further use of the channel anywhere raises
the channel is not empty then raise ChannelNotEmptyError (if ChannelClosedError. If the channel is not empty then raise
"force" is False) or discard the remaining objects (if "force" ChannelNotEmptyError (if "force" is False) or discard the
is True) and close it. remaining objects (if "force" is True) and close it.
``SendChannel(id)``:: ``SendChannel(id)``::
The sending end of a channel. An interpreter may use this to send The sending end of a channel. An interpreter may use this to send
objects to another interpreter. At first only bytes will be objects to another interpreter. At first only a few of
supported. the simple, immutable builtin types will be supported.
id: id:
The channel's unique ID. The channel's unique ID. This is shared with the "recv" end.
interpreters: interpreters:
@ -882,8 +908,9 @@ many-to-many, channels have no buffer.
Send the object (i.e. its data) to the receiving end of the Send the object (i.e. its data) to the receiving end of the
channel. Wait until the object is received. If the the channel. Wait until the object is received. If the the
object is not shareable then ValueError is raised. Currently object is not shareable then ValueError is raised. This
only bytes are supported. associates the current interpreter with the "send" end of the
channel.
If the channel is already closed then raise ChannelClosedError. If the channel is already closed then raise ChannelClosedError.
If the channel isn't closed but the current interpreter already If the channel isn't closed but the current interpreter already
@ -892,9 +919,10 @@ many-to-many, channels have no buffer.
send_nowait(obj): send_nowait(obj):
Send the object to the receiving end of the channel. If the other Send the object to the receiving end of the channel. If no
end is not currently receiving then raise NotReceivedError. interpreter is currently receiving (waiting on the other end)
Otherwise this is the same as "send()". then raise NotReceivedError. Otherwise this is the same as
"send()".
send_buffer(obj): send_buffer(obj):
@ -918,9 +946,9 @@ many-to-many, channels have no buffer.
Close both ends of the channel (in all interpreters). No matter Close both ends of the channel (in all interpreters). No matter
what the "send" end of the channel is immediately closed. If the what the "send" end of the channel is immediately closed. If the
channel is empty then close the "recv" end immediately too. channel is empty then close the "recv" end immediately too.
Otherwise wait until the channel is empty before closing it (if Otherwise, if "force" if False, close the "recv" end (and hence
"force" is False) or discard the remaining items and close the full channel) once the channel becomes empty; or, if "force"
immediately (if "force" is True). is True, discard the remaining items and close immediately.
Note that ``send_buffer()`` is similar to how Note that ``send_buffer()`` is similar to how
``multiprocessing.Connection`` works. [mp-conn]_ ``multiprocessing.Connection`` works. [mp-conn]_
@ -929,53 +957,10 @@ Note that ``send_buffer()`` is similar to how
Open Questions Open Questions
============== ==============
* "force" argument to ``ch.release()``?
* add a "tp_share" type slot instead of using a global registry * add a "tp_share" type slot instead of using a global registry
for shareable types? for shareable types?
Open Implementation Questions
=============================
Does every interpreter think that their thread is the "main" thread?
--------------------------------------------------------------------
(This is more of an implementation detail that an issue for the PEP.)
CPython's interpreter implementation identifies the OS thread in which
it was started as the "main" thread. The interpreter the has slightly
different behavior depending on if the current thread is the main one
or not. This presents a problem in cases where "main thread" is meant
to imply "main thread in the main interpreter" [main-thread]_, where
the main interpreter is the initial one.
Disallow subinterpreters in the main thread?
--------------------------------------------
(This is more of an implementation detail that an issue for the PEP.)
This is a specific case of the above issue. Currently in CPython,
"we need a main \*thread\* in order to sensibly manage the way signal
handling works across different platforms". [main-thread]_
Since signal handlers are part of the interpreter state, running a
subinterpreter in the main thread means that the main interpreter
can no longer properly handle signals (since it's effectively paused).
Furthermore, running a subinterpreter in the main thread would
conceivably allow setting signal handlers on that interpreter, which
would likewise impact signal handling when that interpreter isn't
running or is running in a different thread.
Ultimately, running subinterpreters in the main OS thread introduces
complications to the signal handling implementation. So it may make
the most sense to disallow running subinterpreters in the main thread.
Support for it could be considered later. The downside is that folks
wanting to try out subinterpreters would be required to take the extra
step of using threads. This could slow adoption and experimentation,
whereas without the restriction there's less of an obstacle.
Deferred Functionality Deferred Functionality
====================== ======================
@ -1048,10 +1033,11 @@ Syntactic Support
The ``Go`` language provides a concurrency model based on CSP, so The ``Go`` language provides a concurrency model based on CSP, so
it's similar to the concurrency model that subinterpreters support. it's similar to the concurrency model that subinterpreters support.
``Go`` provides syntactic support, as well several builtin concurrency However, ``Go`` also provides syntactic support, as well several builtin
primitives, to make concurrency a first-class feature. Conceivably, concurrency primitives, to make concurrency a first-class feature.
similar syntactic (and builtin) support could be added to Python using Conceivably, similar syntactic (and builtin) support could be added to
subinterpreters. However, that is *way* outside the scope of this PEP! Python using subinterpreters. However, that is *way* outside the scope
of this PEP!
Multiprocessing Multiprocessing
--------------- ---------------
@ -1072,19 +1058,21 @@ raise an ImportError if unsupported.
Alternately we could support opting in to subinterpreter support. Alternately we could support opting in to subinterpreter support.
However, that would probably exclude many more modules (unnecessarily) However, that would probably exclude many more modules (unnecessarily)
than the opt-out approach. than the opt-out approach. Also, note that PEP 489 defined that an
extension's use of the PEP's machinery implies support for
subinterpreters.
The scope of adding the ModuleDef slot and fixing up the import The scope of adding the ModuleDef slot and fixing up the import
machinery is non-trivial, but could be worth it. It all depends on machinery is non-trivial, but could be worth it. It all depends on
how many extension modules break under subinterpreters. Given the how many extension modules break under subinterpreters. Given that
relatively few cases we know of through mod_wsgi, we can leave this there are relatively few cases we know of through mod_wsgi, we can
for later. leave this for later.
Poisoning channels Poisoning channels
------------------ ------------------
CSP has the concept of poisoning a channel. Once a channel has been CSP has the concept of poisoning a channel. Once a channel has been
poisoned, and ``send()`` or ``recv()`` call on it will raise a special poisoned, any ``send()`` or ``recv()`` call on it would raise a special
exception, effectively ending execution in the interpreter that tried exception, effectively ending execution in the interpreter that tried
to use the poisoned channel. to use the poisoned channel.
@ -1092,15 +1080,6 @@ This could be accomplished by adding a ``poison()`` method to both ends
of the channel. The ``close()`` method can be used in this way of the channel. The ``close()`` method can be used in this way
(mostly), but these semantics are relatively specialized and can wait. (mostly), but these semantics are relatively specialized and can wait.
Sending channels over channels
------------------------------
Some advanced usage of subinterpreters could take advantage of the
ability to send channels over channels, in addition to bytes. Given
that channels will already be multi-interpreter safe, supporting then
in ``RecvChannel.recv()`` wouldn't be a big change. However, this can
wait until the basic functionality has been ironed out.
Reseting __main__ Reseting __main__
----------------- -----------------
@ -1161,7 +1140,7 @@ Per Antoine Pitrou [async]_::
on (probably a file descriptor?). on (probably a file descriptor?).
A possible solution is to provide async implementations of the blocking A possible solution is to provide async implementations of the blocking
channel methods (``__next__()``, ``recv()``, and ``send()``). However, channel methods (``recv()``, and ``send()``). However,
the basic functionality of subinterpreters does not depend on async and the basic functionality of subinterpreters does not depend on async and
can be added later. can be added later.
@ -1320,6 +1299,39 @@ Rejected possible solutions:
to do something similar to do something similar
Implementation
==============
The implementation of the PEP has 4 parts:
* the high-level module described in this PEP (mostly a light wrapper
around a low-level C extension
* the low-level C extension module
* additions to the ("private") C=API needed by the low-level module
* secondary fixes/changes in the CPython runtime that facilitate
the low-level module (among other benefits)
These are at various levels of completion, with more done the lower
you go:
* the high-level module has been, at best, roughly implemented.
However, fully implementing it will be almost trivial.
* the low-level module is mostly complete. The bulk of the
implementation was merged into master in December 2018 as the
"_xxsubinterpreters" module (for the sake of testing subinterpreter
functionality). Only 3 parts of the implementation remain:
"send_wait()", "send_buffer()", and exception propagation. All three
have been mostly finished, but were blocked by work related to ceval.
That blocker is basically resolved now and finishing the low-level
will not require extensive work.
* all necessary C-API work has been finished
* all anticipated work in the runtime has been finished
The implementation effort for PEP 554 is being tracked as part of
a larger project aimed at improving multi-core support in CPython.
[multi-core-project]_
References References
========== ==========
@ -1389,6 +1401,9 @@ References
.. [pypy] .. [pypy]
https://mail.python.org/pipermail/python-ideas/2017-September/046973.html https://mail.python.org/pipermail/python-ideas/2017-September/046973.html
.. [multi-core-project]
https://github.com/ericsnowcurrently/multi-core-python
Copyright Copyright
========= =========