Make PEP 479 (Change StopIteration) be more specific, improve some wording, etc.

pep-0479.txt

@@ -15,8 +15,12 @@ Abstract
 ========
 
 This PEP proposes a semantic change to ``StopIteration`` when raised
-inside a generator, unifying the behaviour of list comprehensions and
+inside a generator.  This would unify the behaviour of list
-generator expressions somewhat.
+comprehensions and generator expressions, reducing surprises such as
+the one that started this discussion [1]_.  This is also the main
+backwards incompatibility of the proposal -- any generator that
+depends on an implicitly-raised ``StopIteration`` to terminate it will
+have to be rewritten to either catch that exception or use a for-loop.
 
 
 Rationale
@@ -37,7 +41,10 @@ When a generator frame is (re)started as a result of a ``__next__()``
 
 * A yield point is reached, and the yielded value is returned.
 * The frame is returned from; ``StopIteration`` is raised.
-* An exception is thrown, which bubbles out.
+* An exception is raised, which bubbles out.
+
+In the latter two cases the frame is abandoned (and the generator
+object's ``gi_frame`` attribute is set to None).
 
 
 Proposal
@@ -49,33 +56,47 @@ is replaced with ``RuntimeError``, which causes the ``next()`` call
 From then on it's just like any old exception. [3]_
 
 This affects the third outcome listed above, without altering any
-other effects.
+other effects.  Furthermore, it only affects this outcome when the
+exception raised is StopIteration (or a subclass thereof).
+
+Note that the proposed replacement happens at the point where the
+exception is about to bubble out of the frame, i.e. after any
+``except`` or ``finally`` blocks that could affect it have been
+exited.  The ``StopIteration`` raised by returning from the frame is
+not affected (the point being that ``StopIteration`` means that the
+generator terminated "normally", i.e. it did not raise an exception).
 
 
-Consequences to existing code
+Consequences for existing code
-=============================
+==============================
 
 This change will affect existing code that depends on
 ``StopIteration`` bubbling up.  The pure Python reference
-implementation of ``groupby`` [1]_ currently has comments "Exit on
+implementation of ``groupby`` [2]_ currently has comments "Exit on
 ``StopIteration``" where it is expected that the exception will
 propagate and then be handled.  This will be unusual, but not unknown,
-and such constructs will fail.
+and such constructs will fail.  Other examples abound, e.g. [5]_, [6]_.
 
 (Nick Coghlan comments: """If you wanted to factor out a helper
 function that terminated the generator you'd have to do "return
 yield from helper()" rather than just "helper()".""")
 
+There are also examples of generator expressions floating around that
+rely on a StopIteration raised by the expression, the target or the
+predicate (rather than by the __next__() call implied in the ``for``
+loop proper).
+
 As this can break code, it is proposed to utilize the ``__future__``
-mechanism to introduce this, finally making it standard in Python 3.6
+mechanism to introduce this in Python 3.5, finally making it standard
-or 3.7.  Any generator function constructed in the presence of this
+in Python 3.6 or 3.7.  The proposed syntax is::
-directive will have a flag set on its code object, and generators with
+
-the flag set will behave according to this proposal.  Once the feature
+    from __future__ import replace_stopiteration_in_generators
-becomes standard, the flag may be dropped; code should not inspect
+
-generators for it.  (GvR: """And the flag should somehow be
+Any generator function constructed under the influence of this
-transferred to the stack frame when the function is executed, so the
+directive will have the REPLACE_STOPITERATION flag set on its code
-right action can be taken when an exception is about to bubble out of
+object, and generators with the flag set will behave according to this
-that frame.""")
+proposal.  Once the feature becomes standard, the flag may be dropped;
+code should not inspect generators for it.
 
 
 Alternate proposals
@@ -123,19 +144,25 @@ the original ``GeneratorReturn``, and would reference the original
 exception in its ``__cause__``.  If uncaught, this would clearly show
 the chaining of exceptions.
 
-This does *not* affect the discrepancy between generator expressions
+This alternative does *not* affect the discrepancy between generator expressions
 and list comprehensions, but allows generator-aware code (such as the
 contextlib and asyncio modules) to reliably differentiate between the
 second and third outcomes listed above.
 
+However, once code exists that depends on this distinction between
+``GeneratorReturn`` and ``StopIteration``, a generator that invokes
+another generator and relies on the latter's ``StopIteration`` to
+bubble out would still be potentially wrong, depending on the use made
+of the distinction between the two exception types.
+
 
 Criticism
 =========
 
 Unofficial and apocryphal statistics suggest that this is seldom, if
 ever, a problem. [4]_  Code does exist which relies on the current
-behaviour, and there is the concern that this would be unnecessary
+behaviour (e.g. [2]_, [5]_, [6]_), and there is the concern that this
-code churn to achieve little or no gain.
+would be unnecessary code churn to achieve little or no gain.
 
 
 References
@@ -153,6 +180,12 @@ References
 .. [4] Response by Steven D'Aprano
    (https://mail.python.org/pipermail/python-ideas/2014-November/029994.html)
 
+.. [5] Split a sequence or generator using a predicate
+   (http://code.activestate.com/recipes/578416-split-a-sequence-or-generator-using-a-predicate/)
+
+.. [6] wrap unbounded generator to restrict its output
+   (http://code.activestate.com/recipes/66427-wrap-unbounded-generator-to-restrict-its-output/)
+
 
 Copyright
 =========