python-peps/pep-0479.txt

PEP: 479
Title: Change StopIteration handling inside generators
Version: $Revision$
Last-Modified: $Date$
Author: Chris Angelico <rosuav@gmail.com>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 15-Nov-2014
Python-Version: 3.5
Post-History: 15-Nov-2014


Abstract
========

This PEP proposes a semantic change to ``StopIteration`` when raised
inside a generator, unifying the behaviour of list comprehensions and
generator expressions somewhat.


Rationale
=========

The interaction of generators and ``StopIteration`` is currently
somewhat surprising, and can conceal obscure bugs.  An unexpected
exception should not result in subtly altered behaviour, but should
cause a noisy and easily-debugged traceback.  Currently,
``StopIteration`` can be absorbed by the generator construct.


Background information
======================

When a generator frame is (re)started as a result of a ``__next__()``
(or ``send()`` or ``throw()``) call, one of three outcomes can occur:

* A yield point is reached, and the yielded value is returned.
* The frame is returned from; ``StopIteration`` is raised.
* An exception is thrown, which bubbles out.


Proposal
========

If a ``StopIteration`` is about to bubble out of a generator frame, it
is replaced with ``RuntimeError``, which causes the ``next()`` call
(which invoked the generator) to fail, passing that exception out.
From then on it's just like any old exception. [3]_

This affects the third outcome listed above, without altering any
other effects.


Consequences to existing code
=============================

This change will affect existing code that depends on
``StopIteration`` bubbling up.  The pure Python reference
implementation of ``groupby`` [1]_ currently has comments "Exit on
``StopIteration``" where it is expected that the exception will
propagate and then be handled.  This will be unusual, but not unknown,
and such constructs will fail.

(Nick Coghlan comments: """If you wanted to factor out a helper
function that terminated the generator you'd have to do "return
yield from helper()" rather than just "helper()".""")

As this can break code, it is proposed to utilize the ``__future__``
mechanism to introduce this, finally making it standard in Python 3.6
or 3.7.  Any generator function constructed in the presence of this
directive will have a flag set on its code object, and generators with
the flag set will behave according to this proposal.  Once the feature
becomes standard, the flag may be dropped; code should not inspect
generators for it.  (GvR: """And the flag should somehow be
transferred to the stack frame when the function is executed, so the
right action can be taken when an exception is about to bubble out of
that frame.""")


Alternate proposals
===================

Raising something other than RuntimeError
-----------------------------------------

Rather than the generic ``RuntimeError``, it might make sense to raise
a new exception type ``UnexpectedStopIteration``.  This has the
downside of implicitly encouraging that it be caught; the correct
action is to catch the original ``StopIteration``, not the chained
exception.


Supplying a specific exception to raise on return
-------------------------------------------------

Nick Coghlan suggested a means of providing a specific
``StopIteration`` instance to the generator; if any other instance of
``StopIteration`` is raised, it is an error, but if that particular
one is raised, the generator has properly completed.  This subproposal
has been withdrawn in favour of better options, but is retained for
reference.


Making return-triggered StopIterations obvious
----------------------------------------------

For certain situations, a simpler and fully backward-compatible
solution may be sufficient: when a generator returns, instead of
raising ``StopIteration``, it raises a specific subclass of
``StopIteration`` which can then be detected.  If it is not that
subclass, it is an escaping exception rather than a return statement.

Of the three outcomes listed above:

* A yielded value, obviously, would still be returned.
* If the frame is returned from, ``GeneratorReturn`` is raised.
* If an instance of ``GeneratorReturn`` would be raised, instead an
  instance of ``StopIteration`` would be raised.

In the third case, the ``StopIteration`` would have the ``value`` of
the original ``GeneratorReturn``, and would reference the original
exception in its ``__cause__``.  If uncaught, this would clearly show
the chaining of exceptions.

This does *not* affect the discrepancy between generator expressions
and list comprehensions, but allows generator-aware code (such as the
contextlib and asyncio modules) to reliably differentiate between the
second and third outcomes listed above.


Criticism
=========

Unofficial and apocryphal statistics suggest that this is seldom, if
ever, a problem. [4]_  Code does exist which relies on the current
behaviour, and there is the concern that this would be unnecessary
code churn to achieve little or no gain.


References
==========

.. [1] Initial mailing list comment
   (https://mail.python.org/pipermail/python-ideas/2014-November/029906.html)

.. [2] Pure Python implementation of groupby
   (https://docs.python.org/3/library/itertools.html#itertools.groupby)

.. [3] Proposal by GvR
   (https://mail.python.org/pipermail/python-ideas/2014-November/029953.html)

.. [4] Response by Steven D'Aprano
   (https://mail.python.org/pipermail/python-ideas/2014-November/029994.html)


Copyright
=========

This document has been placed in the public domain.



..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   coding: utf-8
   End: