PEP 554: Minor Updates (#3067)

This includes minor fixes and other edits based on some feedback (https://discuss.python.org/t/pep-554-multiple-interpreters-in-the-stdlib/24855/7).

pep-0554.rst

@@ -291,6 +291,21 @@ Passing objects via pickle
        os.write(s, b'\x00')
    os.write(s, b'\x00')
 
+Capturing an interpreter's stdout
+---------------------------------
+
+::
+
+   interp = interpreters.create()
+   stdout = io.StringIO()
+   with contextlib.redirect_stdout(stdout):
+       interp.run(tw.dedent("""
+               print('spam!')
+               """))
+   assert(stdout.getvalue() == 'spam!')
+
+A pipe (``os.pipe()``) could be used similarly.
+
 Running a module
 ----------------
 
@@ -390,10 +405,8 @@ will appeal to some Python users.  That is the core value that the
 In the `Interpreter Isolation`_ section below we identify ways in
 which isolation in CPython's subinterpreters is incomplete.  Most
 notable is extension modules that use C globals to store internal
-state.  :pep:`3121` and :pep:`489` provide a solution for most of the
-problem, but one still remains. [petr-c-ext]_  Until that is resolved
-(see :pep:`573`), C extension authors will face extra difficulty
-to support subinterpreters.
+state.  (:pep:`3121` and :pep:`489` provide a solution to that problem,
+followed by some extra APIs that improve efficiency, e.g. :pep:`573`).
 
 Consequently, projects that publish extension modules may face an
 increased maintenance burden as their users start using subinterpreters,
@@ -416,12 +429,12 @@ which subinterpreters are worth it.
 
 Introducing an API for a new concurrency model, like happened with
 asyncio, is an extremely large project that requires a lot of careful
-consideration.  It is not something that can be done a simply as this
+consideration.  It is not something that can be done as simply as this
 PEP proposes and likely deserves significant time on PyPI to mature.
 (See `Nathaniel's post <nathaniel-asyncio_>`_ on python-dev.)
 
 However, this PEP does not propose any new concurrency API.
-At most it exposes minimal tools (e.g. subinterpreters, simple "sharing")
+At most it exposes minimal tools (e.g. subinterpreters)
 which may be used to write code that follows patterns associated with
 (relatively) new-to-Python `concurrency models <Concurrency_>`_.
 Those tools could also be used as the basis for APIs for such
@@ -686,8 +699,8 @@ Regarding uncaught exceptions in ``Interpreter.run()``, we noted that
 they are "effectively" propagated into the code where ``run()`` was
 called.  To prevent leaking exceptions (and tracebacks) between
 interpreters, we create a surrogate of the exception and its traceback
-(see ``traceback.TracebackException``), set it to ``__cause__`` on a
-new ``RunFailedError``, and raise that.
+(see :class:`traceback.TracebackException`), set it to ``__cause__``
+on a new ``RunFailedError``, and raise that.
 
 Directly raising (a proxy of) the exception is problematic since it's
 harder to distinguish between an error in the ``run()`` call and an
@@ -928,11 +941,12 @@ Also note that resetting ``__main__`` does nothing about state stored
 in other modules.  So any solution would have to be clear about the
 scope of what is being reset.  Conceivably we could invent a mechanism
 by which any (or every) module could be reset, unlike ``reload()``
-which does not clear the module before loading into it.  Regardless,
-since ``__main__`` is the execution namespace of the interpreter,
-resetting it has a much more direct correlation to interpreters and
-their dynamic state than does resetting other modules.  So a more
-generic module reset mechanism may prove unnecessary.
+which does not clear the module before loading into it.
+
+Regardless, since ``__main__`` is the execution namespace of the
+interpreter, resetting it has a much more direct correlation to
+interpreters and their dynamic state than does resetting other modules.
+So a more generic module reset mechanism may prove unnecessary.
 
 This isn't a critical feature initially.  It can wait until later
 if desirable.
@@ -958,6 +972,16 @@ otherwise available from Python code, it isn't a fundamental
 functionality.  So in the spirit of minimalism here, this can wait.
 Regardless, I doubt it would be controversial to add it post-PEP.
 
+Copy an existing interpreter's state
+------------------------------------
+
+Relatedly, it may be useful to support creating a new interpreter
+based on an existing one, e.g. ``Interpreter.copy()``.  This ties
+into the idea that a snapshot could be made of an interpreter's memory,
+which would make starting up CPython, or creating new interpreters,
+faster in general.  The same mechanism could be used for a
+hypothetical ``Interpreter.reset()``, as described previously.
+
 Shareable file descriptors and sockets
 --------------------------------------
 
@@ -1093,25 +1117,15 @@ make sense to treat them specially when it comes to propagation from
 We aren't going to worry about handling them differently.  Threads
 already ignore ``SystemExit``, so for now we will follow that pattern.
 
-Auto-run in a thread
---------------------
-
-The PEP proposes a hard separation between interpreters and threads:
-if you want to run in a thread you must create the thread yourself and
-call ``run()`` in it.  However, it might be convenient if ``run()``
-could do that for you, meaning there would be less boilerplate.
-
-Furthermore, we anticipate that users will want to run in a thread much
-more often than not.  So it would make sense to make this the default
-behavior.  We would add a kw-only param "threaded" (default ``True``)
-to ``run()`` to allow the run-in-the-current-thread operation.
-
 
 Rejected Ideas
 ==============
 
-Use pipes instead of channels
------------------------------
+Add an API based on pipes
+-------------------------
+
+(Earlier versions of this PEP proposed "channels" for communicating
+between interpreters.  This idea is written relative to that.)
 
 A pipe would be a simplex FIFO between exactly two interpreters.  For
 most use cases this would be sufficient.  It could potentially simplify
@@ -1119,8 +1133,11 @@ the implementation as well.  However, it isn't a big step to supporting
 a many-to-many simplex FIFO via channels.  Also, with pipes the API
 ends up being slightly more complicated, requiring naming the pipes.
 
-Use queues instead of channels
-------------------------------
+Add an API based on queues
+--------------------------
+
+(Earlier versions of this PEP proposed "channels" for communicating
+between interpreters.  This idea is written relative to that.)
 
 Queues and buffered channels are almost the same thing.  The main
 difference is that channels have a stronger relationship with context
@@ -1211,6 +1228,20 @@ require extra runtime modifications.  It would also make the module's
 implementation overly complicated.  Finally, it might not even make
 the module easier to understand.
 
+Allow multiple simultaneous calls to Interpreter.run()
+------------------------------------------------------
+
+This would make sense especially if ``Interpreter.run()`` were to
+manage new threads for you (which we've rejected).  Essentially,
+each call would run independently, which would be mostly fine
+from a narrow technical standpoint, since each interpreter
+can have multiple threads.
+
+The problem is that the interpreter has only one ``__main__`` module
+and simultaneous ``Interpreter.run()`` calls would have to sort out
+sharing ``__main__`` or we'd have to invent a new mechanism.  Neither
+would be simple enough to be worth doing.
+
 Add a "reraise" method to RunFailedError
 ----------------------------------------