Add PEP 380: Syntax for Delegating to a Subgenerator.
Also updated Greg's email address in other PEPs.
This commit is contained in:
Brett Cannon
parent
0543929f96
commit
b26070deec
|
|
@ -2,8 +2,8 @@ PEP: 284
|
|||
Title: Integer for-loops
|
||||
Version: $Revision$
|
||||
Last-Modified: $Date$
|
||||
Author: eppstein@ics.uci.edu (David Eppstein),
|
||||
greg@cosc.canterbury.ac.nz (Gregory Ewing)
|
||||
Author: David Eppstein <eppstein@ics.uci.edu>,
|
||||
Greg Ewing <greg.ewing@canterbury.ac.nz>
|
||||
Status: Rejected
|
||||
Type: Standards Track
|
||||
Created: 1-Mar-2002
|
||||
|
|
|
|||
|
|
@ -2,7 +2,7 @@ PEP: 335
|
|||
Title: Overloadable Boolean Operators
|
||||
Version: $Revision$
|
||||
Last-Modified: $Date$
|
||||
Author: Gregory Ewing <greg@cosc.canterbury.ac.nz>
|
||||
Author: Greg Ewing <greg.ewing@canterbury.ac.nz>
|
||||
Status: Draft
|
||||
Type: Standards Track
|
||||
Content-Type: text/x-rst
|
||||
|
|
|
|||
|
|
@ -0,0 +1,347 @@
|
|||
PEP: 380
|
||||
Title: Syntax for Delegating to a Subgenerator
|
||||
Version: $Revision$
|
||||
Last-Modified: $Date$
|
||||
Author: Gregory Ewing <greg.ewing@canterbury.ac.nz>
|
||||
Status: Draft
|
||||
Type: Standards Track
|
||||
Content-Type: text/x-rst
|
||||
Created: 13-Feb-2009
|
||||
Python-Version: 2.7
|
||||
Post-History:
|
||||
|
||||
|
||||
Abstract
|
||||
========
|
||||
|
||||
A syntax is proposed for a generator to delegate part of its
|
||||
operations to another generator. This allows a section of code
|
||||
containing 'yield' to be factored out and placed in another
|
||||
generator. Additionally, the subgenerator is allowed to return with a
|
||||
value, and the value is made available to the delegating generator.
|
||||
|
||||
The new syntax also opens up some opportunities for optimisation when
|
||||
one generator re-yields values produced by another.
|
||||
|
||||
|
||||
Proposal
|
||||
========
|
||||
|
||||
The following new expression syntax will be allowed in the body of a
|
||||
generator:
|
||||
|
||||
::
|
||||
|
||||
yield from <expr>
|
||||
|
||||
where <expr> is an expression evaluating to an iterable, from which an
|
||||
iterator is extracted. The iterator is run to exhaustion, during which
|
||||
time it yields and receives values directly to or from the caller of
|
||||
the generator containing the ``yield from`` expression (the
|
||||
"delegating generator").
|
||||
|
||||
When the iterator is another generator, the effect is the same as if
|
||||
the body of the subgenerator were inlined at the point of the ``yield
|
||||
from`` expression. Furthermore, the subgenerator is allowed to execute
|
||||
a ``return`` statement with a value, and that value becomes the value of
|
||||
the ``yield from`` expression.
|
||||
|
||||
In general, the semantics can be understood in terms of the iterator
|
||||
protocol as follows:
|
||||
|
||||
* Any values that the iterator yields are passed directly to the
|
||||
caller.
|
||||
|
||||
* Any values sent to the delegating generator using ``send()``
|
||||
are passed directly to the iterator. If the sent value is None,
|
||||
the iterator's ``next()`` method is called. If the sent value is
|
||||
not None, the iterator's ``send()`` method is called. Any exception
|
||||
resulting from attempting to call ``next`` or ``send`` is raised
|
||||
in the delegating generator.
|
||||
|
||||
* Calls to the ``throw()`` method of the delegating generator are
|
||||
forwarded to the iterator. If the iterator does not have a
|
||||
``throw()`` method, the thrown-in exception is raised in the
|
||||
delegating generator.
|
||||
|
||||
* If the delegating generator's ``close()`` method is called, the
|
||||
``close() method of the iterator is called first if it has one,
|
||||
then the delegating generator is finalised.
|
||||
|
||||
* The value of the ``yield from`` expression is the first argument
|
||||
to the ``StopIteration`` exception raised by the iterator when it
|
||||
terminates.
|
||||
|
||||
* ``return expr`` in a generator causes ``StopIteration(expr)`` to
|
||||
be raised.
|
||||
|
||||
For convenience, the ``StopIteration`` exception will be given a
|
||||
``value`` attribute that holds its first argument, or None if there
|
||||
are no arguments.
|
||||
|
||||
|
||||
Formal Semantics
|
||||
----------------
|
||||
|
||||
1. The statement
|
||||
|
||||
::
|
||||
|
||||
result = yield from expr
|
||||
|
||||
is semantically equivalent to
|
||||
|
||||
::
|
||||
|
||||
_i = iter(expr)
|
||||
try:
|
||||
_u = _i.next()
|
||||
while 1:
|
||||
try:
|
||||
_v = yield _u
|
||||
except Exception, _e:
|
||||
_m = getattr(_i, 'throw', None)
|
||||
if _m is not None:
|
||||
_u = _m(_e)
|
||||
else:
|
||||
raise
|
||||
else:
|
||||
if _v is None:
|
||||
_u = _i.next()
|
||||
else:
|
||||
_u = _i.send(_v)
|
||||
except StopIteration, _e:
|
||||
result = _e.value
|
||||
finally:
|
||||
_m = getattr(_i, 'close', None)
|
||||
if _m is not None:
|
||||
_m()
|
||||
|
||||
except that implementations are free to cache bound methods for the 'next',
|
||||
'send', 'throw' and 'close' methods of the iterator.
|
||||
|
||||
2. In a generator, the statement
|
||||
|
||||
::
|
||||
|
||||
return value
|
||||
|
||||
is semantically equivalent to
|
||||
|
||||
::
|
||||
|
||||
raise StopIteration(value)
|
||||
|
||||
except that, as currently, the exception cannot be caught by ``except``
|
||||
clauses within the returning generator.
|
||||
|
||||
3. The StopIteration exception behaves as though defined thusly:
|
||||
|
||||
::
|
||||
|
||||
class StopIteration(Exception):
|
||||
|
||||
def __init__(self, *args):
|
||||
if len(args) > 0:
|
||||
self.value = args[0]
|
||||
else:
|
||||
self.value = None
|
||||
Exception.__init__(self, *args)
|
||||
|
||||
|
||||
Rationale
|
||||
=========
|
||||
|
||||
A Python generator is a form of coroutine, but has the limitation that
|
||||
it can only yield to its immediate caller. This means that a piece of
|
||||
code containing a ``yield`` cannot be factored out and put into a
|
||||
separate function in the same way as other code. Performing such a
|
||||
factoring causes the called function to itself become a generator, and
|
||||
it is necessary to explicitly iterate over this second generator and
|
||||
re-yield any values that it produces.
|
||||
|
||||
If yielding of values is the only concern, this is not very arduous
|
||||
and can be performed with a loop such as
|
||||
|
||||
::
|
||||
|
||||
for v in g:
|
||||
yield v
|
||||
|
||||
However, if the subgenerator is to interact properly with the caller
|
||||
in the case of calls to ``send()``, ``throw()`` and ``close()``, things
|
||||
become considerably more complicated. As the formal expansion presented
|
||||
above illustrates, the necessary code is very longwinded, and it is tricky
|
||||
to handle all the corner cases correctly. In this situation, the advantages
|
||||
of a specialised syntax should be clear.
|
||||
|
||||
|
||||
Generators as Threads
|
||||
---------------------
|
||||
|
||||
A motivating use case for generators being able to return values
|
||||
concerns the use of generators to implement lightweight threads. When
|
||||
using generators in that way, it is reasonable to want to spread the
|
||||
computation performed by the lightweight thread over many functions.
|
||||
One would like to be able to call a subgenerator as though it were
|
||||
an ordinary function, passing it parameters and receiving a returned
|
||||
value.
|
||||
|
||||
Using the proposed syntax, a statement such as
|
||||
|
||||
::
|
||||
|
||||
y = f(x)
|
||||
|
||||
where f is an ordinary function, can be transformed into a delegation
|
||||
call
|
||||
|
||||
::
|
||||
|
||||
y = yield from g(x)
|
||||
|
||||
where g is a generator. One can reason about the behaviour of the
|
||||
resulting code by thinking of g as an ordinary function that can be
|
||||
suspended using a ``yield`` statement.
|
||||
|
||||
When using generators as threads in this way, typically one is not
|
||||
interested in the values being passed in or out of the yields.
|
||||
However, there are use cases for this as well, where the thread is
|
||||
seen as a producer or consumer of items. The ``yield from``
|
||||
expression allows the logic of the thread to be spread over as
|
||||
many functions as desired, with the production or consumption of
|
||||
items occuring in any subfunction, and the items are automatically
|
||||
routed to or from their ultimate source or destination.
|
||||
|
||||
Concerning ``throw()`` and ``close()``, it is reasonable to expect
|
||||
that if an exception is thrown into the thread from outside, it should
|
||||
first be raised in the innermost generator where the thread is suspended,
|
||||
and propagate outwards from there; and that if the thread is terminated
|
||||
from outside by calling ``close()``, the chain of active generators
|
||||
should be finalised from the innermost outwards.
|
||||
|
||||
|
||||
Syntax
|
||||
------
|
||||
|
||||
The particular syntax proposed has been chosen as suggestive of its
|
||||
meaning, while not introducing any new keywords and clearly standing
|
||||
out as being different from a plain ``yield``.
|
||||
|
||||
|
||||
Optimisations
|
||||
-------------
|
||||
|
||||
Using a specialised syntax opens up possibilities for optimisation
|
||||
when there is a long chain of generators. Such chains can arise, for
|
||||
instance, when recursively traversing a tree structure. The overhead
|
||||
of passing ``next()`` calls and yielded values down and up the chain
|
||||
can cause what ought to be an O(n) operation to become O(n\*\*2).
|
||||
|
||||
A possible strategy is to add a slot to generator objects to hold a
|
||||
generator being delegated to. When a ``next()`` or ``send()`` call is
|
||||
made on the generator, this slot is checked first, and if it is
|
||||
nonempty, the generator that it references is resumed instead. If it
|
||||
raises StopIteration, the slot is cleared and the main generator is
|
||||
resumed.
|
||||
|
||||
This would reduce the delegation overhead to a chain of C function
|
||||
calls involving no Python code execution. A possible enhancement would
|
||||
be to traverse the whole chain of generators in a loop and directly
|
||||
resume the one at the end, although the handling of StopIteration is
|
||||
more complicated then.
|
||||
|
||||
|
||||
Use of StopIteration to return values
|
||||
-------------------------------------
|
||||
|
||||
There are a variety of ways that the return value from the generator
|
||||
could be passed back. Some alternatives include storing it as an
|
||||
attribute of the generator-iterator object, or returning it as the
|
||||
value of the ``close()`` call to the subgenerator. However, the proposed
|
||||
mechanism is attractive for a couple of reasons:
|
||||
|
||||
* Using the StopIteration exception makes it easy for other kinds
|
||||
of iterators to participate in the protocol without having to
|
||||
grow extra attributes or a close() method.
|
||||
|
||||
* It simplifies the implementation, because the point at which the
|
||||
return value from the subgenerator becomes available is the same
|
||||
point at which StopIteration is raised. Delaying until any later
|
||||
time would require storing the return value somewhere.
|
||||
|
||||
|
||||
Criticisms
|
||||
==========
|
||||
|
||||
Under this proposal, the value of a ``yield from`` expression would
|
||||
be derived in a very different way from that of an ordinary ``yield``
|
||||
expression. This suggests that some other syntax not containing the
|
||||
word ``yield`` might be more appropriate, but no acceptable alternative
|
||||
has so far been proposed.
|
||||
|
||||
It has been suggested that some mechanism other than ``return`` in
|
||||
the subgenerator should be used to establish the value returned by
|
||||
the ``yield from`` expression. However, this would interfere with
|
||||
the goal of being able to think of the subgenerator as a suspendable
|
||||
function, since it would not be able to return values in the same way
|
||||
as other functions.
|
||||
|
||||
The use of an argument to StopIteration to pass the return value
|
||||
has been criticised as an "abuse of exceptions", without any
|
||||
concrete justification of this claim. In any case, this is only
|
||||
one suggested implementation; another mechanism could be used
|
||||
without losing any essential features of the proposal.
|
||||
|
||||
It has been suggested that a different exception, such as
|
||||
GeneratorReturn, should be used instead of StopIteration to return a
|
||||
value. However, no convincing practical reason for this has been put
|
||||
forward, and the addition of a ``value`` attribute to StopIteration
|
||||
mitigates any difficulties in extracting a return value from a
|
||||
StopIteration exception that may or may not have one. Also, using a
|
||||
different exception would mean that, unlike ordinary functions,
|
||||
'return' without a value in a generator would not be equivalent to
|
||||
'return None'.
|
||||
|
||||
|
||||
Alternative Proposals
|
||||
=====================
|
||||
|
||||
Proposals along similar lines have been made before, some using the
|
||||
syntax ``yield *`` instead of ``yield from``. While ``yield *`` is
|
||||
more concise, it could be argued that it looks too similar to an
|
||||
ordinary ``yield`` and the difference might be overlooked when reading
|
||||
code.
|
||||
|
||||
To the author's knowledge, previous proposals have focused only on
|
||||
yielding values, and thereby suffered from the criticism that the
|
||||
two-line for-loop they replace is not sufficiently tiresome to write
|
||||
to justify a new syntax. By also dealing with calls to ``send()``,
|
||||
``throw()`` and ``close()``, this proposal provides considerably more
|
||||
benefit.
|
||||
|
||||
|
||||
Implementation
|
||||
==============
|
||||
|
||||
A `prototype implementation`_ is available, based on the first
|
||||
optimisation outlined above.
|
||||
|
||||
.. _prototype implementation: http://www.cosc.canterbury.ac.nz/greg.ewing/python/yield-from/
|
||||
|
||||
|
||||
Copyright
|
||||
=========
|
||||
|
||||
This document has been placed in the public domain.
|
||||
|
||||
|
||||
|
||||
..
|
||||
Local Variables:
|
||||
mode: indented-text
|
||||
indent-tabs-mode: nil
|
||||
sentence-end-double-space: t
|
||||
fill-column: 70
|
||||
coding: utf-8
|
||||
End:
|
||||
Loading…
Reference in New Issue