iSharkFly-Open
/
python-peps
mirror of https://github.com/python/peps.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Add an object passing mechanism to PEP 554. (#401) 

This change also enumerates some of the concerns discussed on the mailing list.
This commit is contained in:
Eric Snow 2017-09-08 14:04:39 -07:00 committed by GitHub
parent e2eb97526c
commit 77ed1dcc30
1 changed files with 170 additions and 31 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

201
pep-0554.rst
View File

@ -15,7 +15,8 @@ Abstract
This proposal introduces the stdlib ``interpreters`` module. It exposes
the basic functionality of subinterpreters that already exists in the
C-API. Each subinterpreter runs with its own state (see
``Interpreter Isolation`` below).
``Interpreter Isolation`` below). The module will be "provisional", as
described by PEP 411.
Rationale
@ -38,6 +39,60 @@ new area for Python so there is relative uncertainly about the best
tools to provide as companions to subinterpreters. Thus we minimize
the functionality we add in the proposal as much as possible.
Concerns
--------
* "subinterpreters are not worth the trouble"
Some have argued that subinterpreters do not add sufficient benefit
to justify making them an official part of Python. Adding features
to the language (or stdlib) has a cost in increasing the size of
the language. So it must pay for itself. In this case, subinterpreters
provide a novel concurrency model focused on isolated threads of
execution. Furthermore, they present an opportunity for changes in
CPython that will allow simulateous use of multiple CPU cores (currently
prevented by the GIL).
Alternatives to subinterpreters include threading, async, and
multiprocessing. Threading is limited by the GIL and async isn't
the right solution for every problem (nor for every person).
Multiprocessing is likewise valuable in some but not all situations.
Direct IPC (rather than via the multiprocessing module) provides
similar benefits but with the same caveat.
Notably, subinterpreters are not intended as a replacement for any of
the above. Certainly they overlap in some areas, but the benefits of
subinterpreters include isolation and (potentially) performance. In
particular, subinterpreters provide a direct route to an alternate
concurrency model (e.g. CSP) which has found success elsewhere and
will appeal to some Python users. That is the core value that the
``interpreters`` module will provide.
* stdlib support for subinterpreters adds extra burden
on C extension authors
In the ``Interpreter Isolation`` section below we identify ways in
which isolation in CPython's subinterpreters is incomplete. Most
notable is extension modules that use C globals to store internal
state. PEP 3121 and PEP 489 provide a solution for most of the
problem, but one still remains. [petr-c-ext]_ Until that is resolved,
C extension authors will face extra difficulty to support
subinterpreters.
Consequently, projects that publish extension modules may face an
increased maintenance burden as their users start using subinterpreters,
where their modules may break. This situation is limited to modules
that use C globals (or use libraries that use C globals) to store
internal state.
Ultimately this comes down to a question of how often it will be a
problem in practice: how many projects would be affected, how often
their users will be affected, what the additional maintenance burden
will be for projects, and what the overall benefit of subinterpreters
is to offset those costs. The position of this PEP is that the actual
extra maintenance burden will be small and well below the threshold at
which subinterpreters are worth it.
Proposal
========
@ -67,27 +122,124 @@ The module provides the following functions:
interpreter will be created in the current thread and will remain
idle until something is run in it.
The module also provides the following class:
The module also provides the following classes:
``Interpreter(id)``::
``id``::
id:
The interpreter's ID (read-only).
``is_running()``::
is_running():
Return whether or not the interpreter is currently running.
``destroy()``::
destroy():
Finalize and destroy the interpreter.
``run(code)``::
run(code):
Run the provided Python code in the interpreter, in the current
OS thread. Supported code: source text.
get_fifo(name):
Return the FIFO object with the given name that is associated
with this interpreter. If no such FIFO exists then raise
KeyError. The FIFO will be either a "FIFOReader" or a
"FIFOWriter", depending on how "add_fifo()" was called.
list_fifos():
Return a list of all fifos associated with the interpreter.
add_fifo(name=None, *, recv=True):
Create a new FIFO associated with this interpreter and return
the opposite end of the FIFO. For example, if "recv" is True
then a "FIFOReader" is associated with this interpreter and a
"FIFOWriter" is returned. The returned FIFO is also associated
with the interpreter in which "add_fifo()" was called.
The FIFO's name is set to the provided value. If no name is
provided then a dynamically generated one is used. If a FIFO
with the given name is already associated with this interpreter
or with the one in which "add_fifo()" was called then raise
KeyError.
remove_fifo(name):
Drop the association between the named FIFO and this interpreter.
If the named FIFO is not found then raise KeyError.
``FIFOReader(name)``::
The receiving end of a FIFO. An interpreter may use this to receive
objects from another interpreter. At first only bytes and None will
be supported.
name:
The FIFO's name.
__next__():
Return the next bytes object from the pipe. If none have been
pushed on then block.
pop(*, block=True):
Return the next bytes object from the pipe. If none have been
pushed on and "block" is True (the default) then block.
Otherwise return None.
``FIFOWriter(name)``::
The sending end of a FIFO. An interpreter may use this to send
objects to another interpreter. At first only bytes and None will
be supported.
name:
The FIFO's name.
push(object, *, block=True):
Add the object to the FIFO. If "block" is true then block
until the object is popped off. If the FIFO does not support
the object's type then TypeError is raised.
About FIFOs
-----------
Subinterpreters are inherently isolated (with caveats explained below),
in contrast to threads. This enables a different concurrency model than
currently exists in Python. CSP (Communicating Sequential Processes),
upon which Go's concurrency is based, is one example of this model.
A key component of this approach to concurrency is message passing. So
providing a message/object passing mechanism alongside ``Interpreter``
is a fundamental requirement. This proposal includes a basic mechanism
upon which more complex machinery may be built. That basic mechanism
draws inspiration from pipes, queues, and CSP's channels.
The key challenge here is that sharing objects between interpreters
faces complexity due in part to CPython's current memory model.
Furthermore, in this class of concurrency, the ideal is that objects
only exist in one interpreter at a time. However, this is not practical
for Python so we initially constrain supported objects to ``bytes`` and
``None``. There are a number of strategies we may pursue in the future
to expand supported objects and object sharing strategies.
Note that the complexity of object sharing increases as subinterpreters
become more isolated, e.g. after GIL removal. So the mechanism for
message passing needs to be carefully considered. Keeping the API
minimal and initially restricting the supported types helps us avoid
further exposing any underlying complexity to Python users.
Deferred Functionality
======================
@ -97,28 +249,6 @@ functionality has been left out for future consideration. Note that
this is not a judgement against any of said capability, but rather a
deferment. That said, each is arguably valid.
Queues (Channels)
-----------------
Subinterpreters are inherently isolated, in contrast to threads. This
enables a different concurrency model than currently exists in Python.
CSP (Communicating Sequential Processes), upon which Go's concurrency
is based, is one example of this model.
A key component of this approach to concurrency is message passing. So
providing a message/object passing mechanism alongside ``Interpreter``
is entirely sensible and even advisable. However, it isn't strictly
necessary to expose the existing functionality from the C-API.
The key challenge here is that sharing objects between interpreters
faces complexity due to CPython's memory model. This is substantially
more challenging under a possible future where interpreters do not share
the GIL.
In the spirit of minimalism explained above and given the complexity
involved with sharing objects between interpreters, this proposal leaves
the addition of queues to future consideration.
Interpreter.call()
------------------
@ -155,10 +285,15 @@ of CPython. This includes the GIL and memory management. Improvements
are currently going on to address gaps in this area.
Open Questions
==============
Provisional Status
==================
* Add queues to the proposal anyway?
The new ``interpreters`` module will be added with "provisional" status
(see PEP 411). This allows Python users to experiment with the feature
and provide feedback while still allowing us to adjust to that feedback.
The module will be provisional in Python 3.7 and we will make a decision
before the 3.8 release whether to keep it provisional, graduate it, or
remove it.
References
@ -167,6 +302,10 @@ References
.. [c-api]
https://docs.python.org/3/c-api/init.html#bugs-and-caveats
.. [petr-c-ext]
https://mail.python.org/pipermail/import-sig/2016-June/001062.html
https://mail.python.org/pipermail/python-ideas/2016-April/039748.html
Copyright
=========