iSharkFly-Open
/
python-peps
mirror of https://github.com/python/peps.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

minor style conformance changes

This commit is contained in:
Barry Warsaw 2001-03-21 17:34:33 +00:00
parent 414241bf39
commit 741090d1c4
1 changed files with 134 additions and 116 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

250
pep-0219.txt
View File

@ -4,166 +4,184 @@ Version: $Revision$
Author: gmcm@hypernet.com (Gordon McMillan) Author: gmcm@hypernet.com (Gordon McMillan)
Status: Draft Status: Draft
Type: Standards Track Type: Standards Track
Python-Version: 2.1 Created: 14-Aug-2000 Created: 14-Aug-2000
Python-Version: 2.1
Post-History: Post-History:
Introduction Introduction
This PEP discusses changes required to core Python in order to This PEP discusses changes required to core Python in order to
efficiently support generators, microthreads and coroutines. It efficiently support generators, microthreads and coroutines. It is
is related to PEP 220, which describes how Python should be extended related to PEP 220, which describes how Python should be extended
to support these facilities. The focus of this PEP is strictly on to support these facilities. The focus of this PEP is strictly on
the changes required to allow these extensions to work. the changes required to allow these extensions to work.
While these PEPs are based on Christian Tismer's Stackless[1] While these PEPs are based on Christian Tismer's Stackless[1]
implementation, they do not regard Stackless as a reference implementation, they do not regard Stackless as a reference
implementation. Stackless (with an extension module) implements implementation. Stackless (with an extension module) implements
continuations, and from continuations one can implement coroutines, continuations, and from continuations one can implement
microthreads (as has been done by Will Ware[2]) and generators. But coroutines, microthreads (as has been done by Will Ware[2]) and
in more that a year, no one has found any other productive use of generators. But in more that a year, no one has found any other
continuations, so there seems to be no demand for their support. productive use of continuations, so there seems to be no demand
for their support.
However, Stackless support for continuations is a relatively minor However, Stackless support for continuations is a relatively minor
piece of the implementation, so one might regard it as "a" reference piece of the implementation, so one might regard it as "a"
implementation (rather than "the" reference implementation). reference implementation (rather than "the" reference
implementation).
Background Background
Generators and coroutines have been implmented in a number of languages in Generators and coroutines have been implemented in a number of
a number of ways. Indeed, Tim Peters has done pure Python implementations languages in a number of ways. Indeed, Tim Peters has done pure
of generators[3] and coroutines[4] using threads (and a thread-based Python implementations of generators[3] and coroutines[4] using
coroutine implementation exists for Java). However, the horrendous threads (and a thread-based coroutine implementation exists for
overhead of a thread-based implementation severely limits the usefulness Java). However, the horrendous overhead of a thread-based
of this approach. implementation severely limits the usefulness of this approach.
Microthreads (a.k.a "green" or "user" threads) and coroutines involve Microthreads (a.k.a "green" or "user" threads) and coroutines
transfers of control that are difficult to accomodate in a language involve transfers of control that are difficult to accommodate in
implementation based on a single stack. (Generators can be done on a a language implementation based on a single stack. (Generators can
single stack, but they can also be regarded as a very simple case of be done on a single stack, but they can also be regarded as a very
coroutines.) simple case of coroutines.)
Real threads allocate a full-sized stack for each thread of control, and Real threads allocate a full-sized stack for each thread of
this is the major source of overhead. However, coroutines and microthreads control, and this is the major source of overhead. However,
can be implemented in Python in a way that involves almost no overhead. coroutines and microthreads can be implemented in Python in a way
This PEP, therefor, offers a way for making Python able to realistically that involves almost no overhead. This PEP, therefor, offers a
manage thousands of separate "threads" of activity (vs. todays limit of way for making Python able to realistically manage thousands of
perhaps dozens of separate threads of activity). separate "threads" of activity (vs. todays limit of perhaps dozens
of separate threads of activity).
Another justification for this PEP (explored in PEP 220) is that
coroutines and generators often allow a more direct expression of
an algorithm than is possible in today's Python.
Another justification for this PEP (explored in PEP 220) is that
coroutines and generators often allow a more direct expression of an
algorithm than is possible in today's Python.
Discussion Discussion
The first thing to note is that Python, while it mingles interpreter data The first thing to note is that Python, while it mingles
(normal C stack usage) with Python data (the state of the interpreted interpreter data (normal C stack usage) with Python data (the
program) on the stack, the two are logically separate. They just happen to state of the interpreted program) on the stack, the two are
use the same stack. logically separate. They just happen to use the same stack.
A real thread gets something approaching a process-sized stack because the A real thread gets something approaching a process-sized stack
implementation has no way of knowing how much stack space the thread will because the implementation has no way of knowing how much stack
require. The stack space required for an individual frame is likely to be space the thread will require. The stack space required for an
reasonable, but stack switching is an arcane and non-portable process, individual frame is likely to be reasonable, but stack switching
not supported by C. is an arcane and non-portable process, not supported by C.
Once Python stops putting Python data on the C stack, however, stack Once Python stops putting Python data on the C stack, however,
switching becomes easy. stack switching becomes easy.
The fundamental approach of the PEP is based on these two ideas. First, The fundamental approach of the PEP is based on these two
separate C's stack usage from Python's stack usage. Secondly, associate ideas. First, separate C's stack usage from Python's stack
with each frame enough stack space to handle that frame's execution. usage. Secondly, associate with each frame enough stack space to
handle that frame's execution.
In the normal usage, Stackless Python has a normal stack
structure, except that it is broken into chunks. But in the
presence of a coroutine / microthread extension, this same
mechanism supports a stack with a tree structure. That is, an
extension can support transfers of control between frames outside
the normal "call / return" path.
In the normal usage, Stackless Python has a normal stack structure,
except that it is broken into chunks. But in the presence of a
coroutine / microthread extension, this same mechanism supports a stack
with a tree structure. That is, an extension can support transfers of
control between frames outside the normal "call / return" path.
Problems Problems
The major difficulty with this approach is C calling Python. The problem The major difficulty with this approach is C calling Python. The
is that the C stack now holds a nested execution of the byte-code problem is that the C stack now holds a nested execution of the
interpreter. In that situation, a coroutine / microthread extension cannot byte-code interpreter. In that situation, a coroutine /
be permitted to transfer control to a frame in a different invocation of the microthread extension cannot be permitted to transfer control to a
byte-code interpreter. If a frame were to complete and exit back to C from frame in a different invocation of the byte-code interpreter. If a
the wrong interpreter, the C stack could be trashed. frame were to complete and exit back to C from the wrong
interpreter, the C stack could be trashed.
The ideal solution is to create a mechanism where nested executions of the The ideal solution is to create a mechanism where nested
byte code interpreter are never needed. The easy solution is for the executions of the byte code interpreter are never needed. The easy
coroutine / microthread extension(s) to recognize the situation and refuse solution is for the coroutine / microthread extension(s) to
to allow transfers outside the current invocation. recognize the situation and refuse to allow transfers outside the
current invocation.
We can categorize code that involves C calling Python into two camps: We can categorize code that involves C calling Python into two
Python's implementation, and C extensions. And hopefully we can offer a camps: Python's implementation, and C extensions. And hopefully we
compromise: Python's internal usage (and C extension writers who want to can offer a compromise: Python's internal usage (and C extension
go to the effort) will no longer use a nested invocation of the writers who want to go to the effort) will no longer use a nested
interpreter. Extensions which do not go to the effort will still be invocation of the interpreter. Extensions which do not go to the
safe, but will not play well with coroutines / microthreads. effort will still be safe, but will not play well with coroutines
/ microthreads.
Generally, when a recursive call is transformed into a loop, a bit of Generally, when a recursive call is transformed into a loop, a bit
extra bookkeeping is required. The loop will need to keep it's own of extra bookkeeping is required. The loop will need to keep it's
"stack" of arguments and results since the real stack can now only hold own "stack" of arguments and results since the real stack can now
the most recent. The code will be more verbose, because it's not quite only hold the most recent. The code will be more verbose, because
as obvious when we're done. While Stackless is not implemented this way, it's not quite as obvious when we're done. While Stackless is not
it has to deal with the same issues. implemented this way, it has to deal with the same issues.
In normal Python, PyEval_EvalCode is used to build a frame and execute In normal Python, PyEval_EvalCode is used to build a frame and
it. Stackless Python introduces the concept of a FrameDispatcher. Like execute it. Stackless Python introduces the concept of a
PyEval_EvalCode, it executes one frame. But the interpreter may signal FrameDispatcher. Like PyEval_EvalCode, it executes one frame. But
the FrameDispatcher that a new frame has been swapped in, and the new the interpreter may signal the FrameDispatcher that a new frame
frame should be executed. When a frame completes, the FrameDispatcher has been swapped in, and the new frame should be executed. When a
follows the back pointer to resume the "calling" frame. frame completes, the FrameDispatcher follows the back pointer to
resume the "calling" frame.
So Stackless transforms recursions into a loop, but it is not the So Stackless transforms recursions into a loop, but it is not the
FrameDispatcher that manages the frames. This is done by the interpreter FrameDispatcher that manages the frames. This is done by the
(or an extension that knows what it's doing). interpreter (or an extension that knows what it's doing).
The general idea is that where C code needs to execute Python code, it The general idea is that where C code needs to execute Python
creates a frame for the Python code, setting its back pointer to the code, it creates a frame for the Python code, setting its back
current frame. Then it swaps in the frame, signals the FrameDispatcher pointer to the current frame. Then it swaps in the frame, signals
and gets out of the way. The C stack is now clean - the Python code can the FrameDispatcher and gets out of the way. The C stack is now
transfer control to any other frame (if an extension gives it the means clean - the Python code can transfer control to any other frame
to do so). (if an extension gives it the means to do so).
In the vanilla case, this magic can be hidden from the programmer (even, In the vanilla case, this magic can be hidden from the programmer
in most cases, from the Python-internals programmer). Many situations (even, in most cases, from the Python-internals programmer). Many
present another level of difficulty, however. situations present another level of difficulty, however.
The map builtin function involves two obstacles to this approach. It The map builtin function involves two obstacles to this
cannot simply construct a frame and get out of the way, not just because approach. It cannot simply construct a frame and get out of the
there's a loop involved, but each pass through the loop requires some way, not just because there's a loop involved, but each pass
"post" processing. In order to play well with others, Stackless through the loop requires some "post" processing. In order to play
constructs a frame object for map itself. well with others, Stackless constructs a frame object for map
itself.
Most recursions of the interpreter are not this complex, but fairly Most recursions of the interpreter are not this complex, but
frequently, some "post" operations are required. Stackless does not fairly frequently, some "post" operations are required. Stackless
fix these situations because of amount of code changes required. Instead, does not fix these situations because of amount of code changes
Stackless prohibits transfers out of a nested interpreter. While not required. Instead, Stackless prohibits transfers out of a nested
ideal (and sometimes puzzling), this limitation is hardly crippling. interpreter. While not ideal (and sometimes puzzling), this
limitation is hardly crippling.
Advantages Advantages
For normal Python, the advantage to this approach is that C stack usage For normal Python, the advantage to this approach is that C stack
becomes much smaller and more predictable. Unbounded recursion in Python usage becomes much smaller and more predictable. Unbounded
code becomes a memory error, instead of a stack error (and thus, in recursion in Python code becomes a memory error, instead of a
non-Cupertino operating systems, something that can be recovered from). stack error (and thus, in non-Cupertino operating systems,
The price, of course, is the added complexity that comes from transforming something that can be recovered from). The price, of course, is
recursions of the byte-code interpreter loop into a higher order loop the added complexity that comes from transforming recursions of
(and the attendant bookkeeping involved). the byte-code interpreter loop into a higher order loop (and the
attendant bookkeeping involved).
The big advantage comes from realizing that the Python stack is
really a tree, and the frame dispatcher can transfer control
freely between leaf nodes of the tree, thus allowing things like
microthreads and coroutines.
The big advantage comes from realizing that the Python stack is really
a tree, and the frame dispatcher can transfer control freely between
leaf nodes of the tree, thus allowing things like microthreads and
coroutines.
References References
[1] www.stackless.com [1] www.stackless.com
[2] http://world.std.com/~wware/uthread.html [2] http://world.std.com/~wware/uthread.html
[3] Demo/threads/Generator.py in the source distribution [3] Demo/threads/Generator.py in the source distribution
[4] http://www.stackless.com/coroutines.tim.peters.html [4] http://www.stackless.com/coroutines.tim.peters.html
Local Variables: Local Variables: