From 502a13a43b71bcf5620937312dbc2a219e4b1a06 Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Wed, 1 Nov 2023 17:06:47 -0600 Subject: [PATCH] PEP 554: Clean Up and Add Channels Back In (#3498) The bulk of the new text comes from what I removed in gh-2812 (a48b585). One notable addition: Interpreter.set_main_attrs() (and Interpreter.get_main_attr()). --- peps/pep-0554.rst | 973 +++++++++++++++++++++++++++++++++++----------- 1 file changed, 737 insertions(+), 236 deletions(-) diff --git a/peps/pep-0554.rst b/peps/pep-0554.rst index a5c500c46..cf731669d 100644 --- a/peps/pep-0554.rst +++ b/peps/pep-0554.rst @@ -27,22 +27,9 @@ facilitates novel alternative approaches to This proposal introduces the stdlib ``interpreters`` module. It exposes the basic functionality of multiple interpreters already provided by the -C-API, along with describing a *very* basic way to communicate -(i.e. pass data between interpreters). - - -A Disclaimer about the GIL -========================== - -To avoid any confusion up front: This PEP is meant to be independent -of any efforts to stop sharing the GIL between interpreters (:pep:`684`). -At most this proposal will allow users to take advantage of any -GIL-related work. - -The author's position here is that exposing multiple interpreters -to Python code is worth doing, even if they still share the GIL. -Conversations with past steering councils indicates they do not -necessarily agree. +C-API, along with basic support for communicating between interpreters. +This module is especially relevant since :pep:`684` introduced a +per-interpreter GIL in Python 3.12. Proposal @@ -51,6 +38,7 @@ Proposal Summary: * add a new stdlib module: "interpreters" +* add concurrent.futures.InterpreterPoolExecutor * help for extension module maintainers @@ -63,16 +51,17 @@ to the multiple interpreter functionality, and wrap a new low-level See the `Examples`_ section for concrete usage and use cases. Along with exposing the existing (in CPython) multiple interpreter -support, the module will also support a very basic mechanism for -passing data between interpreters. That involves setting simple objects -in the ``__main__`` module of a target subinterpreter. If one end of -an ``os.pipe()`` is passed this way then that pipe can be used to send -bytes between the two interpreters. +support, the module will also support a basic mechanism for +passing data between interpreters. That involves setting "shareable" +objects in the ``__main__`` module of a target subinterpreter. Some +such objects, like ``os.pipe()``, may be used to communicate further. +The module will also provide a minimal implementation of "channels" +as a demonstration of cross-interpreter communication. Note that *objects* are not shared between interpreters since they are tied to the interpreter in which they were created. Instead, the objects' *data* is passed between interpreters. See the `Shared Data`_ -and `API For Sharing Data`_ sections for more details about +and `API For Communication`_ sections for more details about sharing/communicating between interpreters. API summary for interpreters module @@ -109,22 +98,39 @@ For creating and using interpreters: +----------------------------------+---------------------------------------------------+ | ``.close()`` | Finalize and destroy the interpreter. | +----------------------------------+---------------------------------------------------+ -| ``.run(src_str, /)`` | | Run the given source code in the interpreter | +| ``.set_main_attrs(**kwargs)`` | Bind "shareable" objects in ``__main__``. | ++----------------------------------+---------------------------------------------------+ +| ``.get_main_attr(name)`` | Get a "shareable" object from ``__main__``. | ++----------------------------------+---------------------------------------------------+ +| ``.exec(src_str, /)`` | | Run the given source code in the interpreter | | | | (in the current thread). | +----------------------------------+---------------------------------------------------+ -.. XXX Support blocking interp.run() until the interpreter +.. XXX Support blocking interp.exec() until the interpreter finishes its current work. -| +For communicating between interpreters: -+--------------------+------------------+------------------------------------------------------+ -| exception | base | description | -+====================+==================+======================================================+ -| ``RunFailedError`` | ``RuntimeError`` | Interpreter.run() resulted in an uncaught exception. | -+--------------------+------------------+------------------------------------------------------+ ++---------------------------------------------------------+--------------------------------------------+ +| signature | description | ++=========================================================+============================================+ +| ``is_shareable(obj) -> Bool`` | | Can the object's data be passed | +| | | between interpreters? | ++---------------------------------------------------------+--------------------------------------------+ +| ``create_channel() -> (RecvChannel, SendChannel)`` | | Create a new channel for passing | +| | | data between interpreters. | ++---------------------------------------------------------+--------------------------------------------+ -.. XXX Add "InterpreterAlreadyRunningError"? +concurrent.futures.InterpreterPoolExecutor +------------------------------------------ + +An executor will be added that extends ``ThreadPoolExecutor`` to run +per-thread tasks in subinterpreters. Initially, the only supported +tasks will be whatever ``Interpreter.exec()`` takes (e.g. a ``str`` +script). However, we may also support some functions, as well as +eventually a separate method for pickling the task and arguments, +to reduce friction (at the expense of performance +for short-running tasks). Help for Extension Module Maintainers ------------------------------------- @@ -154,24 +160,24 @@ following: Examples ======== -Run isolated code ------------------ +Run isolated code in current OS thread +-------------------------------------- :: interp = interpreters.create() print('before') - interp.run('print("during")') + interp.exec('print("during")') print('after') -Run in a thread ---------------- +Run in a different thread +------------------------- :: interp = interpreters.create() def run(): - interp.run('print("during")') + interp.exec('print("during")') t = threading.Thread(target=run) print('before') t.start() @@ -184,13 +190,13 @@ Pre-populate an interpreter :: interp = interpreters.create() - interp.run(tw.dedent(""" + interp.exec(tw.dedent(""" import some_lib import an_expensive_module some_lib.set_up() """)) wait_for_request() - interp.run(tw.dedent(""" + interp.exec(tw.dedent(""" some_lib.handle_request() """)) @@ -201,7 +207,7 @@ Handling an exception interp = interpreters.create() try: - interp.run(tw.dedent(""" + interp.exec(tw.dedent(""" raise KeyError """)) except interpreters.RunFailedError as exc: @@ -215,7 +221,7 @@ Re-raising an exception interp = interpreters.create() try: try: - interp.run(tw.dedent(""" + interp.exec(tw.dedent(""" raise KeyError """)) except interpreters.RunFailedError as exc: @@ -225,21 +231,43 @@ Re-raising an exception Note that this pattern is a candidate for later improvement. +Interact with the __main__ namespace +------------------------------------ + +:: + + interp = interpreters.create() + interp.set_main_attrs(a=1, b=2) + interp.exec(tw.dedent(""" + res = do_something(a, b) + """)) + res = interp.get_main_attr('res') + Synchronize using an OS pipe ---------------------------- :: interp = interpreters.create() - r, s = os.pipe() - print('before') - interp.run(tw.dedent(f""" + r1, s1 = os.pipe() + r2, s2 = os.pipe() + + def task(): + interp.exec(tw.dedent(f""" import os - os.read({r}, 1) - print("during") + os.read({r1}, 1) + print('during B') + os.write({s2}, '') """)) + + t = threading.thread(target=task) + t.start() + print('before') + os.write(s1, '') + print('during A') + os.read(r2, 1) print('after') - os.write(s, '') + t.join() Sharing a file descriptor ------------------------- @@ -247,20 +275,13 @@ Sharing a file descriptor :: interp = interpreters.create() - r1, s1 = os.pipe() - r2, s2 = os.pipe() - interp.run(tw.dedent(f""" + with open('spamspamspam') as infile: + interp.set_main_attrs(fd=infile.fileno()) + interp.exec(tw.dedent(f""" import os - fd = int.from_bytes( - os.read({r1}, 10), 'big') for line in os.fdopen(fd): print(line) - os.write({s2}, b'') """)) - with open('spamspamspam') as infile: - fd = infile.fileno().to_bytes(1, 'big') - os.write(s1, fd) - os.read(r2, 1) Passing objects via pickle -------------------------- @@ -269,12 +290,12 @@ Passing objects via pickle interp = interpreters.create() r, s = os.pipe() - interp.run(tw.dedent(f""" + interp.exec(tw.dedent(f""" import os import pickle reader = {r} """)) - interp.run(tw.dedent(""" + interp.exec(tw.dedent(""" data = b'' c = os.read(reader, 1) while c != b'\x00': @@ -299,11 +320,22 @@ Capturing an interpreter's stdout interp = interpreters.create() stdout = io.StringIO() with contextlib.redirect_stdout(stdout): - interp.run(tw.dedent(""" - print('spam!') - """)) + interp.exec(tw.dedent(""" + print('spam!') + """)) assert(stdout.getvalue() == 'spam!') + # alternately: + interp.exec(tw.dedent(""" + import contextlib, io + stdout = io.StringIO() + with contextlib.redirect_stdout(stdout): + print('spam!') + captured = stdout.getvalue() + """)) + captured = interp.get_main_attr('captured') + assert(captured == 'spam!') + A pipe (``os.pipe()``) could be used similarly. Running a module @@ -313,7 +345,7 @@ Running a module interp = interpreters.create() main_module = mod_name - interp.run(f'import runpy; runpy.run_module({main_module!r})') + interp.exec(f'import runpy; runpy.run_module({main_module!r})') Running as script (including zip archives & directories) -------------------------------------------------------- @@ -322,7 +354,92 @@ Running as script (including zip archives & directories) interp = interpreters.create() main_script = path_name - interp.run(f"import runpy; runpy.run_path({main_script!r})") + interp.exec(f"import runpy; runpy.run_path({main_script!r})") + +Using a channel to communicate +------------------------------ + +:: + + tasks_recv, tasks = interpreters.create_channel() + results, results_send = interpreters.create_channel() + + def worker(): + interp = interpreters.create() + interp.set_main_attrs(tasks=tasks_recv, results=results_send) + interp.exec(tw.dedent(""" + def handle_request(req): + ... + + def capture_exception(exc): + ... + + while True: + try: + req = tasks.recv() + except Exception: + # channel closed + break + try: + res = handle_request(req) + except Exception as exc: + res = capture_exception(exc) + results.send_nowait(res) + """)) + threads = [threading.Thread(target=worker) for _ in range(20)] + for t in threads: + t.start() + + requests = ... + for req in requests: + tasks.send(req) + tasks.close() + + for t in threads: + t.join() + +Sharing a memoryview (imagine map-reduce) +----------------------------------------- + +:: + + data, chunksize = read_large_data_set() + buf = memoryview(data) + numchunks = (len(buf) + 1) / chunksize + results = memoryview(b'\0' * numchunks) + + tasks_recv, tasks = interpreters.create_channel() + + def worker(): + interp = interpreters.create() + interp.set_main_attrs(data=buf, results=results, tasks=tasks_recv) + interp.exec(tw.dedent(""" + while True: + try: + req = tasks.recv() + except Exception: + # channel closed + break + resindex, start, end = req + chunk = data[start: end] + res = reduce_chunk(chunk) + results[resindex] = res + """)) + t = threading.Thread(target=worker) + t.start() + + for i in range(numchunks): + if not workers_running(): + raise ... + start = i * chunksize + end = start + chunksize + if end > len(buf): + end = len(buf) + tasks.send((start, end, i)) + tasks.close() + t.join() + + use_results(results) Rationale @@ -434,7 +551,7 @@ PEP proposes and likely deserves significant time on PyPI to mature. (See `Nathaniel's post `_ on python-dev.) However, this PEP does not propose any new concurrency API. -At most it exposes minimal tools (e.g. subinterpreters) +At most it exposes minimal tools (e.g. subinterpreters, channels) which may be used to write code that follows patterns associated with (relatively) new-to-Python `concurrency models `_. Those tools could also be used as the basis for APIs for such @@ -504,24 +621,34 @@ is mitigated somewhat for some "immortal" objects (see :pep:`683`). Consequently, the mechanism for sharing needs to be carefully considered. There are a number of valid solutions, several of which may be -appropriate to support in Python. Earlier versions of this proposal -included a basic capability ("channels"), though most of the options -were quite similar. +appropriate to support in Python's stdlib and C-API. Any such solution +is likely to share many characteristics with the others. -Note that the implementation of ``Interpreter.run()`` will be done -in a way that allows for may of these solutions to be implemented -independently and to coexist, but doing so is not technically -a part of the proposal here. +In the meantime, we propose here a minimal solution +(``Interpreter.set_main_attrs()``), which sets some precedent for how +objects are shared. More importantly, it facilitates the introduction +of more advanced approaches later and allows them to coexist and cooperate. +In part to demonstrate that, we will provide a basic implementation of +"channels", as a somewhat more advanced sharing solution. + +Separate proposals may cover: + +* the addition of a public C-API based on the implementation + ``Interpreter.set_main_attrs()`` +* the addition of other sharing approaches to the "interpreters" module The fundamental enabling feature for communication is that most objects can be converted to some encoding of underlying raw data, which is safe to be passed between interpreters. For example, an ``int`` object can -be turned into a C ``long`` value, send to another interpreter, and -turned back into an ``int`` object there. +be turned into a C ``long`` value, sent to another interpreter, and +turned back into an ``int`` object there. As another example, +``None`` may be passed as-is. -Regardless, the effort to determine the best way forward here is outside -the scope of this PEP. In the meantime, this proposal provides a basic -interim solution, described in `API For Sharing Data`_ below. +Regardless, the effort to determine the best way forward here is mostly +outside the scope of this PEP. In the meantime, this proposal describes +a basic interim solution using pipes (``os.pipe()``), as well as +providing a dedicated capability ("channels"). +See `API For Communication`_ below. Interpreter Isolation --------------------- @@ -531,10 +658,12 @@ other. Each interpreter has its own copy of all modules, classes, functions, and variables. The same applies to state in C, including in extension modules. The CPython C-API docs explain more. [caveats]_ -However, there are ways in which interpreters share some state. First -of all, some process-global state remains shared: +However, there are ways in which interpreters do share some state. +First of all, some process-global state remains shared: * file descriptors +* low-level env vars +* process memory (though allocators *are* isolated) * builtin types (e.g. dict, bytes) * singletons (e.g. None) * underlying static module data (e.g. functions) for @@ -554,17 +683,14 @@ Finally, some potential isolation is missing due to the current design of CPython. Improvements are currently going on to address gaps in this area: -* GC is not run per-interpreter [global-gc]_ -* at-exit handlers are not run per-interpreter [global-atexit]_ -* extensions using the ``PyGILState_*`` API are incompatible [gilstate]_ -* interpreters share memory management (e.g. allocators, gc) -* interpreters share the GIL +* extensions using the ``PyGILState_*`` API are somewhat incompatible [gilstate]_ Existing Usage -------------- -Multiple interpreter support is not a widely used feature. In fact, -the only documented cases of widespread usage are +Multiple interpreter support has not been a widely used feature. +In fact, there have been only a handful of documented cases of +widespread usage, including `mod_wsgi `_, `OpenStack Ceph `_, and `JEP `_. On the one hand, these cases @@ -591,6 +717,7 @@ without a lot of trouble. Here are the projects I contacted: .. _interpreters-get-current: .. _interpreters-create: .. _interpreters-Interpreter: +.. _interpreters-is-shareable: "interpreters" Module API ========================= @@ -616,8 +743,16 @@ The module provides the following functions:: It will remain idle until something is run in it and always run in its own thread. + is_shareable(obj) -> bool: -The module also provides the following classes:: + Return True if the object may be "shared" between interpreters. + This does not necessarily mean that the actual objects will be + shared. Insead, it means that the objects' underlying data will + be shared in a cross-interpreter way, whether via a proxy, a + copy, or some other means. + + +The module also provides the following class:: class Interpreter(id): @@ -627,7 +762,7 @@ The module also provides the following classes:: is_running() -> bool: - Return whether or not the interpreter's "run()" is currently + Return whether or not the interpreter's "exec()" is currently executing code. Code running in subthreads is ignored. Calling this on the current interpreter will always return True. @@ -638,7 +773,31 @@ The module also provides the following classes:: This may not be called on an already running interpreter. Doing so results in a RuntimeError. - run(source_str, /): + set_main_attrs(iterable_or_mapping, /): + set_main_attrs(**kwargs): + + Set attributes in the interpreter's __main__ module + corresponding to the given name-value pairs. Each value + must be a "shareable" object and will be converted to a new + object (e.g. copy, proxy) in whatever way that object's type + defines. If an attribute with the same name is already set, + it will be overwritten. + + This method is helpful for setting up an interpreter before + calling exec(). + + get_main_attr(name, default=None, /): + + Return the value of the corresponding attribute of the + interpreter's __main__ module. If the attribute isn't set + then the default is returned. If it is set, but the value + isn't "shareable" then a ValueError is raised. + + This may be used to introspect the __main__ module, as well + as a very basic mechanism for "returning" one or more results + from Interpreter.exec(). + + exec(source_str, /): Run the provided Python source code in the interpreter, in its __main__ module. @@ -646,24 +805,26 @@ The module also provides the following classes:: This may not be called on an already running interpreter. Doing so results in a RuntimeError. - A "run()" call is similar to an exec() call (or calling - a function that returns None). Once "run()" completes, - the code that called "run()" continues executing (in the - original interpreter). Likewise, if there is any uncaught - exception then it effectively (see below) propagates into - the code where ``run()`` was called. Like exec() (and threads), - but unlike function calls, there is no return value. If any - "return" value from the code is needed, send the data out - via a pipe (os.pipe()). + An "interp.exec()" call is similar to a builtin exec() call + (or to calling a function that returns None). Once + "interp.exec()" completes, the code that called "exec()" + continues executing (in the original interpreter). Likewise, + if there is any uncaught exception then it effectively + (see below) propagates into the code where ``interp.exec()`` + was called. Like exec() (and threads), but unlike function + calls, there is no return value. If any "return" value from + the code is needed, send the data out via a pipe (os.pipe()) + or channel or other cross-interpreter communication mechanism. - The big difference from exec() or functions is that "run()" - executes the code in an entirely different interpreter, - with entirely separate state. The interpreters are completely - isolated from each other, so the state of the original interpreter - (including the code it was executing in the current OS thread) - does not affect the state of the target interpreter - (the one that will execute the code). Likewise, the target - does not affect the original, nor any of its other threads. + The big difference from exec() or functions is that + "interp.exec()" executes the code in an entirely different + interpreter, with entirely separate state. The interpreters + are completely isolated from each other, so the state of the + original interpreter (including the code it was executing in + the current OS thread) does not affect the state of the target + interpreter (the one that will execute the code). Likewise, + the target does not affect the original, nor any of its other + threads. Instead, the state of the original interpreter (for this thread) is frozen, and the code it's executing code completely blocks. @@ -671,76 +832,47 @@ The module also provides the following classes:: OS thread. Then, when it finishes executing, the original interpreter gets control back and continues executing. - So calling "run()" will effectively cause the current Python - thread to completely pause. Sometimes you won't want that pause, - in which case you should make the "run()" call in another thread. - To do so, add a function that calls "run()" and then run that - function in a normal "threading.Thread". + So calling "interp.exec()" will effectively cause the current + Python thread to completely pause. Sometimes you won't want + that pause, in which case you should make the "exec()" call in + another thread. To do so, add a function that calls + "interp.exec()" and then run that function in a normal + "threading.Thread". Note that the interpreter's state is never reset, neither - before "run()" executes the code nor after. Thus the - interpreter state is preserved between calls to "run()". - This includes "sys.modules", the "builtins" module, and the - internal state of C extension modules. + before "interp.exec()" executes the code nor after. Thus the + interpreter state is preserved between calls to + "interp.exec()". This includes "sys.modules", the "builtins" + module, and the internal state of C extension modules. - Also note that "run()" executes in the namespace of the + Also note that "interp.exec()" executes in the namespace of the "__main__" module, just like scripts, the REPL, "-m", and "-c". Just as the interpreter's state is not ever reset, the "__main__" module is never reset. You can imagine - concatenating the code from each "run()" call into one long - script. This is the same as how the REPL operates. + concatenating the code from each "interp.exec()" call into one + long script. This is the same as how the REPL operates. Supported code: source text. +.. XXX Add "InterpreterAlreadyRunningError"? + +In addition to the functionality of ``Interpreter.set_main_attrs()``, +the module provides a related way to pass data between interpreters: +channels. See `Channels`_ below. + Uncaught Exceptions ------------------- -Regarding uncaught exceptions in ``Interpreter.run()``, we noted that -they are "effectively" propagated into the code where ``run()`` was -called. To prevent leaking exceptions (and tracebacks) between +Regarding uncaught exceptions in ``Interpreter.exec()``, we noted that +they are "effectively" propagated into the code where ``interp.exec()`` +was called. To prevent leaking exceptions (and tracebacks) between interpreters, we create a surrogate of the exception and its traceback (see :class:`traceback.TracebackException`), set it to ``__cause__`` -on a new ``RunFailedError``, and raise that. +on a new ``interpreters.RunFailedError``, and raise that. Directly raising (a proxy of) the exception is problematic since it's -harder to distinguish between an error in the ``run()`` call and an -uncaught exception from the subinterpreter. - -API For Sharing Data --------------------- - -As discussed in `Shared Data`_ above, multiple interpreter support -is less useful without a mechanism for sharing data (communicating) -between them. Sharing actual Python objects between interpreters, -however, has enough potential problems that we are avoiding support -for that in this proposal. Nor, as mentioned earlier, are we adding -anything more than the most minimal mechanism for communication. - -That very basic mechanism, using pipes (see ``os.pipe()``), will allow -users to send data (bytes) from one interpreter to another. We'll -take a closer look in a moment. Fundamentally, it's a simple -application of the underlying sharing capability proposed here. - -The various aspects of the approach, including keeping the API minimal, -helps us avoid further exposing any underlying complexity -to Python users. - -Communicating Through OS Pipes -'''''''''''''''''''''''''''''' - -As noted, this proposal enables a very basic mechanism for -communicating between interpreters, which makes use of -``Interpreter.run()``: - -1. interpreter A calls ``os.pipe()`` to get a read/write pair - of file descriptors (both ``int`` objects) -2. interpreter A calls ``run()`` on interpreter B, including - the read FD via string formatting -3. interpreter A writes some bytes to the write FD -4. interpreter B reads those bytes - -Several of the earlier examples demonstrate this, such as -`Synchronize using an OS pipe`_. +harder to distinguish between an error in the ``interp.exec()`` call +and an uncaught exception from the subinterpreter. Interpreter Restrictions @@ -761,7 +893,7 @@ Note that interpreters created with the existing C-API do not have these restrictions. The same is true for the "main" interpreter, so existing use of Python will not change. -.. Mention the similar restrictions in PEP 684? +.. XXX Mention the similar restrictions in PEP 684? We may choose to later loosen some of the above restrictions or provide a way to enable/disable granular restrictions individually. Regardless, @@ -769,6 +901,204 @@ requiring multi-phase init from extension modules will always be a default restriction. +API For Communication +===================== + +As discussed in `Shared Data`_ above, multiple interpreter support +is less useful without a mechanism for sharing data (communicating) +between them. Sharing actual Python objects between interpreters, +however, has enough potential problems that we are avoiding support +for that in this proposal. Nor, as mentioned earlier, are we adding +anything more than a basic mechanism for communication. + +That mechanism is the ``Interpreter.set_main_attrs()`` method. +It may be used to set up global variables before ``Interpreter.exec()`` +is called. The name-value pairs passed to ``set_main_attrs()`` are +bound as attributes of the interpreter's ``__main__`` module. +The values must be "shareable". See `Shareable Types`_ below. + +Additional approaches to communicating and sharing objects are enabled +through ``Interpreter.set_main_attrs()``. A shareable object could be +implemented which works like a queue, but with cross-interpreter safety. +In fact, this PEP does include an example of such an approach: channels. + +Shareable Types +--------------- + +An object is "shareable" if its type supports shareable instances. +The type must implement a new internal protocol, which is used to +convert an object to interpreter-independent data and then coverted +back to an object on the other side. Also see +`is_shareable() `_ above. + +A minimal set of simple, immutable builtin types will be supported +initially, including: + +* ``None`` +* ``bool`` +* ``bytes`` +* ``str`` +* ``int`` +* ``float`` + +We will also support a small number of complex types initially: + +* ``memoryview``, to allow sharing :pep:`3118` buffers +* `channels `_ + +Further builtin types may be supported later, complex or not. +Limiting the initial shareable types is a practical matter, reducing +the potential complexity of the initial implementation. There are a +number of strategies we may pursue in the future to expand supported +objects, once we have more experience with interpreter isolation. + +In the meantime, a separate proposal will discuss making the internal +protocol (and C-API) used by ``Interpreter.set_main_attrs()`` public. +With that protocol, support for other types could be added +by extension modules. + +Communicating Through OS Pipes +'''''''''''''''''''''''''''''' + +Even without a dedicated object for communication, users may already +use existing tools. For example, one basic approach for sending data +between interpreters is to use a pipe (see ``os.pipe()``): + +1. interpreter A calls ``os.pipe()`` to get a read/write pair + of file descriptors (both ``int`` objects) +2. interpreter A calls ``interp.set_main_attrs()``, binding the read FD + (or embeds it using string formatting) +3. interpreter A calls ``interp.exec()`` on interpreter B +4. interpreter A writes some bytes to the write FD +5. interpreter B reads those bytes + +Several of the earlier examples demonstrate this, such as +`Synchronize using an OS pipe`_. + +.. _interpreters-create-channel: +.. _interpreters-RecvChannel: +.. _interpreters-SendChannel: + +Channels +-------- + +The ``interpreters`` module will include a dedicated solution for +passing object data between interpreters: channels. They are included +in the module in part to provide an easier mechanism than using +``os.pipe()`` and in part to demonstrate how libraries may take +advantage of ``Interpreter.set_main_attrs()`` +and the protocol it uses. + +A channel is a simplex FIFO. It is a basic, opt-in data sharing +mechanism that draws inspiration from pipes, queues, and CSP's +channels. [fifo]_ The main difference from pipes is that channels can +be associated with zero or more interpreters on either end. Like +queues, which are also many-to-many, channels are buffered (though +they also offer methods with unbuffered semantics). + +Channels have two operations: send and receive. A key characteristic +of those operations is that channels transmit data derived from Python +objects rather than the objects themselves. When objects are sent, +their data is extracted. When the "object" is received in the other +interpreter, the data is converted back into an object owned by that +interpreter. + +To make this work, the mutable shared state will be managed by the +Python runtime, not by any of the interpreters. Initially we will +support only one type of objects for shared state: the channels provided +by ``interpreters.create_channel()``. Channels, in turn, will carefully +manage passing objects between interpreters. + +This approach, including keeping the API minimal, helps us avoid further +exposing any underlying complexity to Python users. + +The ``interpreters`` module provides the following function related +to channels:: + + create_channel() -> (RecvChannel, SendChannel): + + Create a new channel and return (recv, send), the RecvChannel + and SendChannel corresponding to the ends of the channel. + + Both ends of the channel are supported "shared" objects (i.e. + may be safely shared by different interpreters. Thus they + may be set using "Interpreter.set_main_attrs()". + +The module also provides the following channel-related classes:: + + class RecvChannel(id): + + The receiving end of a channel. An interpreter may use this to + receive objects from another interpreter. Any type supported by + Interpreter.set_main_attrs() will be supported here, though at + first only a few of the simple, immutable builtin types + will be supported. + + id -> int: + + The channel's unique ID. The "send" end has the same one. + + recv(*, timeout=None): + + Return the next object from the channel. If none have been + sent then wait until the next send (or until the timeout is hit). + + At the least, the object will be equivalent to the sent object. + That will almost always mean the same type with the same data, + though it could also be a compatible proxy. Regardless, it may + use a copy of that data or actually share the data. That's up + to the object's type. + + recv_nowait(default=None): + + Return the next object from the channel. If none have been + sent then return the default. Otherwise, this is the same + as the "recv()" method. + + + class SendChannel(id): + + The sending end of a channel. An interpreter may use this to + send objects to another interpreter. Any type supported by + Interpreter.set_main_attrs() will be supported here, though + at first only a few of the simple, immutable builtin types + will be supported. + + id -> int: + + The channel's unique ID. The "recv" end has the same one. + + send(obj, *, timeout=None): + + Send the object (i.e. its data) to the "recv" end of the + channel. Wait until the object is received. If the object + is not shareable then ValueError is raised. + + The builtin memoryview is supported, so sending a buffer + across involves first wrapping the object in a memoryview + and then sending that. + + send_nowait(obj): + + Send the object to the "recv" end of the channel. This + behaves the same as "send()", except for the waiting part. + If no interpreter is currently receiving (waiting on the + other end) then queue the object and return False. Otherwise + return True. + +Caveats For Shared Objects +-------------------------- + +Again, Python objects are not shared between interpreters. +However, in some cases data those objects wrap is actually shared +and not just copied. One example might be :pep:`3118` buffers. + +In those cases the object in the original interpreter is kept alive +until the shared data in the other interpreter is no longer used. +Then object destruction can happen like normal in the original +interpreter, along with the previously shared data. + + Documentation ============= @@ -798,8 +1128,8 @@ Note that the documentation will play a large part in mitigating any negative impact that the new ``interpreters`` module might have on extension module maintainers. -Also, the ``ImportError`` for incompatible extension modules will have -a message that clearly says it is due to missing multiple interpreters +Also, the ``ImportError`` for incompatible extension modules will be +updated to clearly say it is due to missing multiple interpreters compatibility and that extensions are not required to provide it. This will help set user expectations properly. @@ -818,6 +1148,18 @@ to ``concurrent.futures``. There are several reasons why that wouldn't work: Similar reasoning applies for support in the ``multiprocessing`` module. +Open Questions +============== + +* will is be too confusing that ``interp.exec()`` runs in the current thread? +* should we add pickling fallbacks right now for ``interp.exec()``, and/or + ``Interpreter.set_main_attrs()`` and ``Interpreter.get_main_attr()``? +* should we support (limited) functions in ``interp.exec()`` right now? +* rename ``Interpreter.close()`` to ``Interpreter.destroy()``? +* drop ``Interpreter.get_main_attr()``, since we have channels? +* should channels be its own PEP? + + Deferred Functionality ====================== @@ -826,19 +1168,70 @@ functionality has been left out for future consideration. Note that this is not a judgement against any of said capability, but rather a deferment. That said, each is arguably valid. -Shareable Objects ------------------ +Add convenience API +------------------- -Earlier versions of this proposal included a mechanism by which the -data underlying a given object could be passed to another interpreter -or even shared, even if the object can't be. Without channels there -isn't enough benefit to keep the concept of shareable objects around. +There are a number of things I can imagine would smooth out +*hypothetical* rough edges with the new module: + +* add something like ``Interpreter.run()`` or ``Interpreter.call()`` + that calls ``interp.exec()`` and falls back to pickle +* fall back to pickle in ``Interpreter.set_main_attrs()`` + and ``Interpreter.get_main_attr()`` + +These would be easy to do if this proves to be a pain point. + +Avoid possible confusion about interpreters running in the current thread +------------------------------------------------------------------------- + +One regular point of confusion has been that ``Interpreter.exec()`` +executes in the current OS thread, temporarily blocking the current +Python thread. It may be worth doing something to avoid that confusion. + +Some possible solutions for this hypothetical problem: + +* by default, run in a new thread? +* add ``Interpreter.exec_in_thread()``? +* add ``Interpreter.exec_in_current_thread()``? + +In earlier versions of this PEP the method was ``interp.run()``. +The simple change to ``interp.exec()`` alone will probably reduce +confusion sufficiently, when coupled with educating users via +the docs. It it turns out to be a real problem, we can pursue +one of the alternatives at that point. + +Clarify "running" vs. "has threads" +----------------------------------- + +``Interpreter.is_running()`` refers specifically to whether or not +``Interpreter.exec()`` (or similar) is running somewhere. It does not +say anything about if the interpreter has any subthreads running. That +information might be helpful. + +Some things we could do: + +* rename ``Interpreter.is_running()`` to ``Interpreter.is_running_main()`` +* add ``Interpreter.has_threads()``, to complement ``Interpreter.is_running()`` +* expand to ``Interpreter.is_running(main=True, threads=False)`` + +None of these are urgent and any could be done later, if desired. + +A Dunder Method For Sharing +--------------------------- + +We could add a special method, like ``__xid__`` to correspond to ``tp_xid``. +At the very least, it would allow Python types to convert their instances +to some other type that implements ``tp_xid``. + +The problem is that exposing this capability to Python code presents +a degree of complixity that hasn't been explored yet, nor is there +a compelling case to investigate that complexity. Interpreter.call() ------------------ It would be convenient to run existing functions in subinterpreters -directly. ``Interpreter.run()`` could be adjusted to support this or +directly. ``Interpreter.exec()`` could be adjusted to support this or a ``call()`` method could be added:: Interpreter.call(f, *args, **kwargs) @@ -850,9 +1243,9 @@ is sufficient for us to get the feature out where it can be explored. Interpreter.run_in_thread() --------------------------- -This method would make a ``run()`` call for you in a thread. Doing this -using only ``threading.Thread`` and ``run()`` is relatively trivial so -we've left it out. +This method would make a ``interp.exec()`` call for you in a thread. +Doing this using only ``threading.Thread`` and ``interp.exec()`` is +relatively trivial so we've left it out. Synchronization Primitives -------------------------- @@ -913,25 +1306,37 @@ how many extension modules break under subinterpreters. Given that there are relatively few cases we know of through mod_wsgi, we can leave this for later. +Poisoning channels +------------------ + +CSP has the concept of poisoning a channel. Once a channel has been +poisoned, any ``send()`` or ``recv()`` call on it would raise a special +exception, effectively ending execution in the interpreter that tried +to use the poisoned channel. + +This could be accomplished by adding a ``poison()`` method to both ends +of the channel. The ``close()`` method can be used in this way +(mostly), but these semantics are relatively specialized and can wait. + Resetting __main__ ------------------ -As proposed, every call to ``Interpreter.run()`` will execute in the +As proposed, every call to ``Interpreter.exec()`` will execute in the namespace of the interpreter's existing ``__main__`` module. This means -that data persists there between ``run()`` calls. Sometimes this isn't -desirable and you want to execute in a fresh ``__main__``. Also, -you don't necessarily want to leak objects there that you aren't using -any more. +that data persists there between ``interp.exec()`` calls. Sometimes +this isn't desirable and you want to execute in a fresh ``__main__``. +Also, you don't necessarily want to leak objects there that you aren't +using any more. Note that the following won't work right because it will clear too much (e.g. ``__name__`` and the other "__dunder__" attributes:: - interp.run('globals().clear()') + interp.exec('globals().clear()') Possible solutions include: * a ``create()`` arg to indicate resetting ``__main__`` after each - ``run`` call + ``interp.exec()`` call * an ``Interpreter.reset_main`` flag to support opting in or out after the fact * an ``Interpreter.reset_main()`` method to opt in when desired @@ -1006,13 +1411,34 @@ Per Antoine Pitrou [async]_:: The basic functionality of multiple interpreters support does not depend on async and can be added later. -channels --------- +A possible solution is to provide async implementations of the blocking +channel methods (``recv()``, and ``send()``). -We could introduce some relatively efficient, native data types for -passing data between interpreters, to use instead of OS pipes. Earlier -versions of this PEP introduced one such mechanism, called "channels". -This can be pursued later. +Alternately, "readiness callbacks" could be used to simplify use in +async scenarios. This would mean adding an optional ``callback`` +(kw-only) parameter to the ``recv_nowait()`` and ``send_nowait()`` +channel methods. The callback would be called once the object was sent +or received (respectively). + +(Note that making channels buffered makes readiness callbacks less +important.) + +Support for iteration +--------------------- + +Supporting iteration on ``RecvChannel`` (via ``__iter__()`` or +``_next__()``) may be useful. A trivial implementation would use the +``recv()`` method, similar to how files do iteration. Since this isn't +a fundamental capability and has a simple analog, adding iteration +support can wait until later. + +Channel context managers +------------------------ + +Context manager support on ``RecvChannel`` and ``SendChannel`` may be +helpful. The implementation would be simple, wrapping a call to +``close()`` (or maybe ``release()``) like files do. As with iteration, +this can wait. Pipes and Queues ---------------- @@ -1029,6 +1455,26 @@ reasonable. The could be trivially implemented as wrappers around channels. Alternatively they could be implemented for efficiency at the same low level as channels. +Return a lock from send() +------------------------- + +When sending an object through a channel, you don't have a way of knowing +when the object gets received on the other end. One way to work around +this is to return a locked ``threading.Lock`` from ``SendChannel.send()`` +that unlocks once the object is received. + +Alternately, the proposed ``SendChannel.send()`` (blocking) and +``SendChannel.send_nowait()`` provide an explicit distinction that is +less likely to confuse users. + +Note that returning a lock would matter for buffered channels +(i.e. queues). For unbuffered channels it is a non-issue. + +Support prioritization in channels +---------------------------------- + +A simple example is ``queue.PriorityQueue`` in the stdlib. + Support inheriting settings (and more?) --------------------------------------- @@ -1052,11 +1498,20 @@ leap to make them shareable. However, as noted elsewhere, it isn't essential or (particularly common) so we can wait on doing that. +Make everything shareable through serialization +----------------------------------------------- + +We could use pickle (or marshal) to serialize everything and thus +make them shareable. Doing this is potentially inefficient, +but it may be a matter of convenience in the end. +We can add it later, but trying to remove it later +would be significantly more painful. + Make RunFailedError.__cause__ lazy ---------------------------------- -An uncaught exception in a subinterpreter (from ``run()``) is copied -to the calling interpreter and set as ``__cause__`` on a +An uncaught exception in a subinterpreter (from ``interp.exec()``) is +copied to the calling interpreter and set as ``__cause__`` on a ``RunFailedError`` which is then raised. That copying part involves some sort of deserialization in the calling interpreter, which can be expensive (e.g. due to imports) yet is not always necessary. @@ -1071,24 +1526,15 @@ It may also make sense to have ``RunFailedError.__cause__`` be a descriptor that does the lazy deserialization (and set ``__cause__``) on the ``RunFailedError`` instance. -Make everything shareable through serialization ------------------------------------------------ +Return a value from ``interp.exec()`` +------------------------------------- -We could use pickle (or marshal) to serialize everything and thus -make them shareable. Doing this is potentially inefficient, -but it may be a matter of convenience in the end. -We can add it later, but trying to remove it later -would be significantly more painful. - -Return a value from ``run()`` ------------------------------ - -Currently ``run()`` always returns None. One idea is to return the -return value from whatever the subinterpreter ran. However, for now +Currently ``interp.exec()`` always returns None. One idea is to return +the return value from whatever the subinterpreter ran. However, for now it doesn't make sense. The only thing folks can run is a string of code (i.e. a script). This is equivalent to ``PyRun_StringFlags()``, ``exec()``, or a module body. None of those "return" anything. We can -revisit this once ``run()`` supports functions, etc. +revisit this once ``interp.exec()`` supports functions, etc. Add a shareable synchronization primitive ----------------------------------------- @@ -1107,7 +1553,7 @@ The exception types that inherit from ``BaseException`` (aside from ``Exception``) are usually treated specially. These types are: ``KeyboardInterrupt``, ``SystemExit``, and ``GeneratorExit``. It may make sense to treat them specially when it comes to propagation from -``run()``. Here are some options:: +``interp.exec()``. Here are some options:: * propagate like normal via RunFailedError * do not propagate (handle them somehow in the subinterpreter) @@ -1117,16 +1563,64 @@ make sense to treat them specially when it comes to propagation from We aren't going to worry about handling them differently. Threads already ignore ``SystemExit``, so for now we will follow that pattern. +Add an explicit release() and close() to channel end classes +------------------------------------------------------------ + +It can be convenient to have an explicit way to close a channel against +further global use. Likewise it could be useful to have an explicit +way to release one of the channel ends relative to the current +interpreter. Among other reasons, such a mechanism is useful for +communicating overall state between interpreters without the extra +boilerplate that passing objects through a channel directly would +require. + +The challenge is getting automatic release/close right without making +it hard to understand. This is especially true when dealing with a +non-empty channel. We should be able to get by without release/close +for now. + +Add SendChannel.send_buffer() +----------------------------- + +This method would allow no-copy sending of an object through a channel +if it supports the :pep:`3118` buffer protocol (e.g. memoryview). + +Support for this is not fundamental to channels and can be added on +later without much disruption. + +Auto-run in a thread +-------------------- + +The PEP proposes a hard separation between subinterpreters and threads: +if you want to run in a thread you must create the thread yourself and +call ``interp.exec()`` in it. However, it might be convenient if +``interp.exec()`` could do that for you, meaning there would be less +boilerplate. + +Furthermore, we anticipate that users will want to run in a thread much +more often than not. So it would make sense to make this the default +behavior. We would add a kw-only param "threaded" (default ``True``) +to ``interp.exec()`` to allow the run-in-the-current-thread operation. + Rejected Ideas ============== +Explicit channel association +---------------------------- + +Interpreters are implicitly associated with channels upon ``recv()`` and +``send()`` calls. They are de-associated with ``release()`` calls. The +alternative would be explicit methods. It would be either +``add_channel()`` and ``remove_channel()`` methods on ``Interpreter`` +objects or something similar on channel objects. + +In practice, this level of management shouldn't be necessary for users. +So adding more explicit support would only add clutter to the API. + Add an API based on pipes ------------------------- -(Earlier versions of this PEP proposed "channels" for communicating -between interpreters. This idea is written relative to that.) - A pipe would be a simplex FIFO between exactly two interpreters. For most use cases this would be sufficient. It could potentially simplify the implementation as well. However, it isn't a big step to supporting @@ -1136,9 +1630,6 @@ ends up being slightly more complicated, requiring naming the pipes. Add an API based on queues -------------------------- -(Earlier versions of this PEP proposed "channels" for communicating -between interpreters. This idea is written relative to that.) - Queues and buffered channels are almost the same thing. The main difference is that channels have a stronger relationship with context (i.e. the associated interpreter). @@ -1160,8 +1651,8 @@ Alternate solutions to prevent leaking exceptions across interpreters --------------------------------------------------------------------- In function calls, uncaught exceptions propagate to the calling frame. -The same approach could be taken with ``run()``. However, this would -mean that exception objects would leak across the inter-interpreter +The same approach could be taken with ``interp.exec()``. However, this +would mean that exception objects would leak across the inter-interpreter boundary. Likewise, the frames in the traceback would potentially leak. While that might not be a problem currently, it would be a problem once @@ -1184,13 +1675,13 @@ Rejected possible solutions: (requires a cross-interpreter representation) * wrap in a proxy at the boundary (including with support for something like ``err.raise()`` to propagate the traceback). -* return the exception (or its proxy) from ``run()`` instead of +* return the exception (or its proxy) from ``interp.exec()`` instead of raising it * return a result object (like ``subprocess`` does) [result-object]_ (unnecessary complexity?) * throw the exception away and expect users to deal with unhandled - exceptions explicitly in the script they pass to ``run()`` - (they can pass error info out via ``os.pipe()``); + exceptions explicitly in the script they pass to ``interp.exec()`` + (they can pass error info out via channels); with threads you have to do something similar Always associate each new interpreter with its own thread @@ -1204,16 +1695,16 @@ same thread may run more than one interpreter (though obviously not at the same time). The proposed module maintains this behavior. Interpreters are not -tied to threads. Only calls to ``Interpreter.run()`` are. However, +tied to threads. Only calls to ``Interpreter.exec()`` are. However, one of the key objectives of this PEP is to provide a more human-centric concurrency model. With that in mind, from a conceptual standpoint the module *might* be easier to understand if each interpreter were associated with its own thread. That would mean ``interpreters.create()`` would create a new thread -and ``Interpreter.run()`` would only execute in that thread (and +and ``Interpreter.exec()`` would only execute in that thread (and nothing else would). The benefit is that users would not have to -wrap ``Interpreter.run()`` calls in a new ``threading.Thread``. Nor +wrap ``Interpreter.exec()`` calls in a new ``threading.Thread``. Nor would they be in a position to accidentally pause the current interpreter (in the current thread) while their interpreter executes. @@ -1228,17 +1719,27 @@ require extra runtime modifications. It would also make the module's implementation overly complicated. Finally, it might not even make the module easier to understand. -Allow multiple simultaneous calls to Interpreter.run() ------------------------------------------------------- +Only associate interpreters upon use +------------------------------------ -This would make sense especially if ``Interpreter.run()`` were to +Associate interpreters with channel ends only once ``recv()``, +``send()``, etc. are called. + +Doing this is potentially confusing and also can lead to unexpected +races where a channel is auto-closed before it can be used in the +original (creating) interpreter. + +Allow multiple simultaneous calls to Interpreter.exec() +------------------------------------------------------- + +This would make sense especially if ``Interpreter.exec()`` were to manage new threads for you (which we've rejected). Essentially, each call would run independently, which would be mostly fine from a narrow technical standpoint, since each interpreter can have multiple threads. The problem is that the interpreter has only one ``__main__`` module -and simultaneous ``Interpreter.run()`` calls would have to sort out +and simultaneous ``Interpreter.exec()`` calls would have to sort out sharing ``__main__`` or we'd have to invent a new mechanism. Neither would be simple enough to be worth doing. @@ -1253,7 +1754,7 @@ pattern:: try: try: - interp.run(script) + interp.exec(script) except RunFailedError as exc: exc.reraise() except MyException: @@ -1267,7 +1768,7 @@ is good enough:: try: try: - interp.run(script) + interp.exec(script) except RunFailedError as exc: raise exc.__cause__ except MyException: @@ -1277,7 +1778,7 @@ Note that in extreme cases it may require a little extra boilerplate:: try: try: - interp.run(script) + interp.exec(script) except RunFailedError as exc: if exc.__cause__ is not None: raise exc.__cause__ @@ -1294,7 +1795,7 @@ The implementation of the PEP has 4 parts: * the high-level module described in this PEP (mostly a light wrapper around a low-level C extension * the low-level C extension module -* additions to the ("private") C=API needed by the low-level module +* additions to the internal C-API needed by the low-level module * secondary fixes/changes in the CPython runtime that facilitate the low-level module (among other benefits) @@ -1306,11 +1807,9 @@ you go: * the low-level module is mostly complete. The bulk of the implementation was merged into master in December 2018 as the "_xxsubinterpreters" module (for the sake of testing multiple - interpreters functionality). Only 3 parts of the implementation - remain: "send_wait()", "send_buffer()", and exception propagation. - All three have been mostly finished, but were blocked by work - related to ceval. That blocker is basically resolved now and - finishing the low-level will not require extensive work. + interpreters functionality). Only the exception propagation + implementation remains to be finished, which will not require + extensive work. * all necessary C-API work has been finished * all anticipated work in the runtime has been finished @@ -1329,22 +1828,24 @@ References https://en.wikipedia.org/wiki/Communicating_sequential_processes https://github.com/futurecore/python-csp +.. [fifo] + https://docs.python.org/3/library/multiprocessing.html#multiprocessing.Pipe + https://docs.python.org/3/library/multiprocessing.html#multiprocessing.Queue + https://docs.python.org/3/library/queue.html#module-queue + http://stackless.readthedocs.io/en/2.7-slp/library/stackless/channels.html + https://golang.org/doc/effective_go.html#sharing + http://www.jtolds.com/writing/2016/03/go-channels-are-bad-and-you-should-feel-bad/ + .. [caveats] https://docs.python.org/3/c-api/init.html#bugs-and-caveats .. [cryptography] https://github.com/pyca/cryptography/issues/2299 -.. [global-gc] - http://bugs.python.org/issue24554 - .. [gilstate] https://bugs.python.org/issue10915 http://bugs.python.org/issue15751 -.. [global-atexit] - https://bugs.python.org/issue6531 - .. [bug-rate] https://mail.python.org/pipermail/python-ideas/2017-September/047094.html