iSharkFly-Open
/
python-peps
mirror of https://github.com/python/peps.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

A couple PEP 654 edits (#1830) 

* Markup edits. Use REsT references, fix a couple of typos.

* Mention TaskGroups in Motivation; add an example of trio.MultiError API
This commit is contained in:
Yury Selivanov 2021-02-22 11:38:55 -08:00 committed by GitHub
parent 4ca905e7ff
commit cef697a03c
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 119 additions and 77 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

196
pep-0654.rst
View File

@ -1,6 +1,8 @@
PEP: 654
Title: Exception Groups and except*
Author: Irit Katriel <iritkatriel@gmail.com>, Yury Selivanov <yury@edgedb.com>, Guido van Rossum <guido@python.org>
Author: Irit Katriel <iritkatriel@gmail.com>,
Yury Selivanov <yury@edgedb.com>,
Guido van Rossum <guido@python.org>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
@ -23,8 +25,7 @@ Motivation
==========
The interpreter is currently able to propagate at most one exception at a
time. The chaining features introduced in
`PEP 3134 <https://www.python.org/dev/peps/pep-3134/>`_ link together
time. The chaining features introduced in :pep:`3134` link together
exceptions that are related to each other as the cause or context, but
there are situations where multiple unrelated exceptions need to be propagated
together as the stack unwinds. Several real world use cases are listed below.
@ -32,40 +33,42 @@ together as the stack unwinds. Several real world use cases are listed below.
* **Concurrent errors**. Libraries for async concurrency provide APIs to invoke
multiple tasks and return their results in aggregate. There isn't currently
a good way for such libraries to handle situations where multiple tasks
raise exceptions. The Python standard library's
`asyncio.gather() <https://docs.python.org/3/library/asyncio-task.html#asyncio.gather>`_
raise exceptions. The Python standard library's ``asyncio.gather()`` [1]_
function provides two options: raise the first exception, or return the
exceptions in the results list. The `Trio <https://trio.readthedocs.io/en/stable/>`_
exceptions in the results list. The Trio [2]_
library has a ``MultiError`` exception type which it raises to report a
collection of errors. Work on this PEP was initially motivated by the
difficulties in handling ``MultiErrors``, which are detailed in a design
document for an improved version, `MultiError2 <https://github.com/python-trio/trio/issues/611>`_.
difficulties in handling ``MultiErrors`` [9]_, which are detailed in a design
document for an improved version, ``MultiError2`` [3]_.
That document demonstrates how difficult it is to create an effective API
for reporting and handling multiple errors without the language changes we
are proposing.
are proposing (see also the `Programming Without 'except \*'`_ section.)
Implementing a better task spawning API in asyncio, inspired by Trio
nurseries, was the main motivation for this PEP. That work is currently
blocked on Python not having native language level support for exception
groups.
* **Multiple failures when retrying an operation.** The Python standard
library's ``socket.create_connection`` function may attempt to connect to
different addresses, and if all attempts fail it needs to report that to the
user. It is an open issue how to aggregate these errors, particularly when
they are different [`Python issue 29980 <https://bugs.python.org/issue29980>`_].
they are different (see issue 29980 [4]_.)
* **Multiple user callbacks fail.** Python's ``atexit.register()`` function
allows users to register functions that are called on system exit. If any of
them raise exceptions, only the last one is reraised, but it would be better
to reraised all of them together
[`atexit documentation <https://docs.python.org/3/library/atexit.html#atexit.register>`_].
to reraised all of them together (see ``atexit`` documentation [5]_.)
Similarly, the pytest library allows users to register finalizers which
are executed at teardown. If more than one of these finalizers raises an
exception, only the first is reported to the user. This can be improved with
``ExceptionGroups``, as explained in this issue by pytest developer Ran Benita
[`Pytest issue 8217 <https://github.com/pytest-dev/pytest/issues/8217>`_].
(see pytest issue 8217 [6]_.)
* **Multiple errors in a complex calculation.** The Hypothesis library performs
automatic bug reduction (simplifying code that demonstrates a bug). In the
process it may find variations that generate different errors, and
(optionally) reports all of them
[`Hypothesis documentation <https://hypothesis.readthedocs.io/en/latest/settings.html#hypothesis.settings.report_multiple_bugs>`_].
(optionally) reports all of them (see the Hypothesis documentation [7]_.)
An ``ExceptionGroup`` mechanism as we are proposing here can resolve some of
the difficulties with debugging that are mentioned in the link above, and
which are due to the loss of context/cause information (communicated
@ -77,7 +80,8 @@ together as the stack unwinds. Several real world use cases are listed below.
effectively masked an exception that the user's code raised inside the context
manager scope. While the user's exception was chained as the context of the
cleanup error, it was not caught by the user's except clause
[`Python issue 40857 <https://bugs.python.org/issue40857>`_].
(see issue 40857 [8]_.)
The issue was resolved by making the cleanup code ignore errors, thus
sidestepping the multiple exception problem. With the features we propose
here, it would be possible for ``__exit__`` to raise an ``ExceptionGroup``
@ -90,12 +94,11 @@ Rationale
Grouping several exceptions together can be done without changes to the
language, simply by creating a container exception type.
`Trio <https://trio.readthedocs.io/en/stable/>`_ is an example of a library
that has made use of this technique in its
`MultiError <https://trio.readthedocs.io/en/stable/reference-core.html#trio.MultiError>`_
type. However, such approaches require calling code to catch the container
exception type, and then inspect it to determine the types of errors that had
occurred, extract the ones it wants to handle and reraise the rest.
Trio [2]_ is an example of a library that has made use of this technique in its
``MultiError`` [9]_ type. However, such an approach requires calling code to catch
the container exception type, and then to inspect it to determine the types of
errors that had occurred, extract the ones it wants to handle, and reraise the
rest.
Changes to the language are required in order to extend support for
``ExceptionGroups`` in the style of existing exception handling mechanisms. At
@ -129,8 +132,8 @@ sequence of the nested exceptions, for example:
``ExceptionGroup('issues', [ValueError('bad value'), TypeError('bad type')])``.
The ``ExceptionGroup`` class exposes these parameters in the fields ``message``
and ``errors``. A nested exception can also be an ``ExceptionGroup`` so the class
represents a tree of exceptions, where the leaves are plain exceptions and
and ``errors``. A nested exception can also be an ``ExceptionGroup`` so the
class represents a tree of exceptions, where the leaves are plain exceptions and
each internal node represent a time at which the program grouped some
unrelated exceptions into a new ``ExceptionGroup`` and raised them together.
The ``ExceptionGroup`` class is final, i.e., it cannot be subclassed.
@ -182,8 +185,8 @@ new copy. Leaf exceptions are not copied, nor are ``ExceptionGroups`` which are
fully contained in the result. When it is necessary to partition an
``ExceptionGroup`` because the condition holds for some, but not all of its
contained exceptions, a new ``ExceptionGroup`` is created but the ``__cause__``,
``__context__`` and ``__traceback__`` fields are copied by reference, so are shared
with the original ``eg``.
``__context__`` and ``__traceback__`` fields are copied by reference, so are
shared with the original ``eg``.
If both the subgroup and its complement are needed, the
``ExceptionGroup.split(condition)`` method can be used:
@ -218,7 +221,12 @@ other side:
.. code-block::
>>> other_errors.split(lambda e: isinstance(e, SyntaxError))
(None, ExceptionGroup('one', [ExceptionGroup('two', [ValueError(3)]), ExceptionGroup('three', [OSError(4)])]))
(None, ExceptionGroup('one', [
ExceptionGroup('two', [
ValueError(3)
]),
ExceptionGroup('three', [
OSError(4)])]))
Since splitting by exception type is a very common use case, ``subgroup`` and
``split`` can take an exception type or tuple of exception types and treat it
@ -814,7 +822,8 @@ It is not possible to use both traditional ``except`` blocks and the new
except ValueError:
pass
except *CancelledError: # <- SyntaxError:
pass # combining ``except`` and ``except*`` is prohibited
pass # combining ``except`` and ``except*``
# is prohibited
It is possible to catch the ``ExceptionGroup`` type with ``except``, but not
with ``except*`` because the latter is ambiguous:
@ -869,13 +878,13 @@ propose in this PEP will not break any existing code:
Once programs begin to use these features, there will be migration issues to
consider:
* An ``except Exception:`` clause will not catch ``ExceptionGroups`` because they
are derived from ``BaseException``. Any such clause will need to be replaced
* An ``except Exception:`` clause will not catch ``ExceptionGroup`` because it
is derived from ``BaseException``. Any such clause will need to be replaced
by ``except (Exception, ExceptionGroup):`` or ``except *Exception:``.
* Similarly, any ``except T:`` clause that wraps code which is now potentially
raising ``ExceptionGroup`` needs to become ``except *T:``, and its body may need
to be updated.
raising ``ExceptionGroup`` needs to become ``except *T:``, and its body may
need to be updated.
* Libraries that need to support older Python versions will not be able to use
``except*`` or raise ``ExceptionGroups``.
@ -885,15 +894,16 @@ How to Teach This
=================
``ExceptionGroups`` and ``except*`` will be documented as part of the language
standard. Libraries that raise ``ExceptionGroups`` such as ``asyncio`` will need to
specify this in their documentation and clarify which API calls need to be
standard. Libraries that raise ``ExceptionGroups`` such as ``asyncio`` will need
to specify this in their documentation and clarify which API calls need to be
wrapped with ``try-except*`` rather than ``try-except``.
Reference Implementation
========================
We developed these concepts (and the examples for this PEP) with
the help of `a reference implementation <https://github.com/iritkatriel/cpython/tree/exceptionGroup-stage5>`_.
the help of the reference implementation [10]_.
It has the builtin ``ExceptionGroup`` along with the changes to the traceback
formatting code, in addition to the grammar, compiler and interpreter changes
@ -1048,72 +1058,104 @@ clauses. We rejected this because it would be less obvious. The fact that we
are handling ``ExceptionGroups`` of ``T`` rather than only naked ``Ts`` should be
specified in the same place where we state ``T``.
Programming Without 'except \*'
===============================
Consider the following simple example of the ``except *`` syntax (pretending
Trio natively supported this proposal):
.. code-block::
try:
async with trio.open_nursery() as nursery:
# Make two concurrent calls to child()
nursery.start_soon(child)
nursery.start_soon(child)
except *ValueError:
pass
Here is how this code would look in Python 3.9:
.. code-block::
def handle_ValueError(exc):
if isinstance(exc, ValueError):
return None
else:
return exc # reraise exc
with MultiError.catch(handle_ValueError):
async with trio.open_nursery() as nursery:
# Make two concurrent calls to child()
nursery.start_soon(child)
nursery.start_soon(child)
This example clearly demonstrates how unintuitive and cumbersome handling
of multiple errors is in current Python. The exception handling logic has
to be in a separate closure and is fairly low level, requiring the writer to
have non-trivial understanding of both Python exceptions mechanics and the
Trio APIs. Instead of using the ``try..except`` block we have to use a
``with`` block. We need to explicitly reraise exceptions we are not handling.
Handling more exception types or implementing more complex
exception handling logic will only further complicate the code to the point
of it being unreadable.
See Also
========
* An analysis of how exception groups will likely be used in asyncio
programs:
https://github.com/python/exceptiongroups/issues/3#issuecomment-716203284
programs: [11]_.
* The issue where the ``except*`` concept was first formalized: [12]_.
* ``MultiError2`` design document: [3]_.
* Reporting Multiple Errors in the Hypothesis library: [7]_.
* The issue where the ``except*`` concept was first formalized:
https://github.com/python/exceptiongroups/issues/4
References
==========
* Reference implementation:
.. [1] https://docs.python.org/3/library/asyncio-task.html#asyncio.gather
Branch: https://github.com/iritkatriel/cpython/tree/exceptionGroup-stage5
.. [2] https://trio.readthedocs.io/en/stable/
PR: https://github.com/iritkatriel/cpython/pull/10
.. [3] https://github.com/python-trio/trio/issues/611
* PEP 3134: Exception Chaining and Embedded Tracebacks
.. [4] https://bugs.python.org/issue29980
https://www.python.org/dev/peps/pep-3134/
.. [5] https://docs.python.org/3/library/atexit.html#atexit.register
* The ``asyncio`` standard library
.. [6] https://github.com/pytest-dev/pytest/issues/8217
https://docs.python.org/3/library/asyncio.html
.. [7] https://hypothesis.readthedocs.io/en/latest/settings.html#hypothesis.settings.report_multiple_bugs
``asyncio.gather()``:
https://docs.python.org/3/library/asyncio-task.html#asyncio.gather
.. [8] https://bugs.python.org/issue40857
* The Trio Library
.. [9] https://trio.readthedocs.io/en/stable/reference-core.html#trio.MultiError
Trio: https://trio.readthedocs.io/en/stable/
.. [10] https://github.com/iritkatriel/cpython/tree/exceptionGroup-stage5
``MultiError``:
https://trio.readthedocs.io/en/stable/reference-core.html#trio.MultiError
.. [11] https://github.com/python/exceptiongroups/issues/3#issuecomment-716203284
``MultiError2`` design document: https://github.com/python-trio/trio/issues/611.
.. [12] https://github.com/python/exceptiongroups/issues/4
* Python issue 29980: OSError: multiple exceptions should preserve the
exception type if it is common
https://bugs.python.org/issue29980
* Python issue 40857: ``tempfile.TemporaryDirectory()```` context manager can fail
to propagate exceptions generated within its context
https://bugs.python.org/issue40857
* ``atexit`` documentation:
https://docs.python.org/3/library/atexit.html#atexit.register
* PyTest issue 8217: Improve reporting when multiple teardowns raise an exception
https://github.com/pytest-dev/pytest/issues/8217
* The Hypothesis Library
https://hypothesis.readthedocs.io/en/latest/index.html
Reporting Multiple Errors:
https://hypothesis.readthedocs.io/en/latest/settings.html#hypothesis.settings.report_multiple_bugs
.. [13] https://trio.readthedocs.io/en/stable/reference-core.html#nurseries-and-spawning
Copyright
=========
This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.
..
Local Variables:
mode: indented-text
indent-tabs-mode: nil
sentence-end-double-space: t
fill-column: 70
coding: utf-8
End: