1516 lines
59 KiB
ReStructuredText
1516 lines
59 KiB
ReStructuredText
PEP: 654
|
|
Title: Exception Groups and except*
|
|
Author: Irit Katriel <iritkatriel@gmail.com>,
|
|
Yury Selivanov <yury@edgedb.com>,
|
|
Guido van Rossum <guido@python.org>
|
|
Status: Accepted
|
|
Type: Standards Track
|
|
Content-Type: text/x-rst
|
|
Created: 22-Feb-2021
|
|
Post-History: 22-Feb-2021, 20-Mar-2021
|
|
|
|
|
|
Abstract
|
|
========
|
|
|
|
This document proposes language extensions that allow programs to raise and handle
|
|
multiple unrelated exceptions simultaneously:
|
|
|
|
* A new standard exception type, the ``ExceptionGroup``, which represents a
|
|
group of unrelated exceptions being propagated together.
|
|
|
|
* A new syntax ``except*`` for handling ``ExceptionGroups``.
|
|
|
|
Motivation
|
|
==========
|
|
|
|
The interpreter is currently able to propagate at most one exception at a
|
|
time. The chaining features introduced in :pep:`3134` link together
|
|
exceptions that are related to each other as the cause or context, but
|
|
there are situations where multiple unrelated exceptions need to be propagated
|
|
together as the stack unwinds. Several real world use cases are listed below.
|
|
|
|
* **Concurrent errors**. Libraries for async concurrency provide APIs to invoke
|
|
multiple tasks and return their results in aggregate. There isn't currently
|
|
a good way for such libraries to handle situations where multiple tasks
|
|
raise exceptions. The Python standard library's ``asyncio.gather()`` [1]_
|
|
function provides two options: raise the first exception, or return the
|
|
exceptions in the results list. The Trio [2]_
|
|
library has a ``MultiError`` exception type which it raises to report a
|
|
collection of errors. Work on this PEP was initially motivated by the
|
|
difficulties in handling ``MultiErrors`` [9]_, which are detailed in a design
|
|
document for an improved version, ``MultiError2`` [3]_.
|
|
That document demonstrates how difficult it is to create an effective API
|
|
for reporting and handling multiple errors without the language changes we
|
|
are proposing (see also the `Programming Without 'except \*'`_ section.)
|
|
|
|
Implementing a better task spawning API in asyncio, inspired by Trio
|
|
nurseries, was the main motivation for this PEP. That work is currently
|
|
blocked on Python not having native language level support for exception
|
|
groups.
|
|
|
|
* **Multiple failures when retrying an operation.** The Python standard
|
|
library's ``socket.create_connection`` function may attempt to connect to
|
|
different addresses, and if all attempts fail it needs to report that to the
|
|
user. It is an open issue how to aggregate these errors, particularly when
|
|
they are different (see issue 29980 [4]_.)
|
|
|
|
* **Multiple user callbacks fail.** Python's ``atexit.register()`` function
|
|
allows users to register functions that are called on system exit. If any of
|
|
them raise exceptions, only the last one is reraised, but it would be better
|
|
to reraise all of them together (see ``atexit`` documentation [5]_.)
|
|
Similarly, the pytest library allows users to register finalizers which
|
|
are executed at teardown. If more than one of these finalizers raises an
|
|
exception, only the first is reported to the user. This can be improved with
|
|
``ExceptionGroups``, as explained in this issue by pytest developer Ran Benita
|
|
(see pytest issue 8217 [6]_.)
|
|
|
|
* **Multiple errors in a complex calculation.** The Hypothesis library performs
|
|
automatic bug reduction (simplifying code that demonstrates a bug). In the
|
|
process it may find variations that generate different errors, and
|
|
(optionally) reports all of them (see the Hypothesis documentation [7]_.)
|
|
An ``ExceptionGroup`` mechanism as we are proposing here can resolve some of
|
|
the difficulties with debugging that are mentioned in the link above, and
|
|
which are due to the loss of context/cause information (communicated
|
|
by Hypothesis Core Developer Zac Hatfield-Dodds).
|
|
|
|
* **Errors in wrapper code.** The Python standard library's
|
|
``tempfile.TemporaryDirectory`` context manager
|
|
had an issue where an exception raised during cleanup in ``__exit__``
|
|
effectively masked an exception that the user's code raised inside the context
|
|
manager scope. While the user's exception was chained as the context of the
|
|
cleanup error, it was not caught by the user's except clause
|
|
(see issue 40857 [8]_.)
|
|
|
|
The issue was resolved by making the cleanup code ignore errors, thus
|
|
sidestepping the multiple exception problem. With the features we propose
|
|
here, it would be possible for ``__exit__`` to raise an ``ExceptionGroup``
|
|
containing its own errors along with the user's errors, and this would allow
|
|
the user to catch their own exceptions by their types.
|
|
|
|
|
|
Rationale
|
|
=========
|
|
|
|
Grouping several exceptions together can be done without changes to the
|
|
language, simply by creating a container exception type.
|
|
Trio [2]_ is an example of a library that has made use of this technique in its
|
|
``MultiError`` [9]_ type. However, such an approach requires calling code to catch
|
|
the container exception type, and then to inspect it to determine the types of
|
|
errors that had occurred, extract the ones it wants to handle, and reraise the
|
|
rest. Furthermore, exceptions in Python have important information attached to
|
|
their ``__traceback__``, ``__cause__`` and ``__context__`` fields, and
|
|
designing a container type that preserves the integrity of this information
|
|
requires care; it is not as simple as collecting exceptions into a set.
|
|
|
|
Changes to the language are required in order to extend support for
|
|
exception groups in the style of existing exception handling mechanisms. At
|
|
the very least we would like to be able to catch an exception group only if
|
|
it contains an exception of a type that we choose to handle. Exceptions of
|
|
other types in the same group need to be automatically reraised,
|
|
otherwise it is too easy for user code to inadvertently swallow exceptions
|
|
that it is not handling.
|
|
|
|
We considered whether it is possible to modify the semantics of ``except``
|
|
for this purpose, in a backwards-compatible manner, and found that it is not.
|
|
See the `Rejected Ideas`_ section for more on this.
|
|
|
|
The purpose of this PEP, then, is to add the ``ExceptionGroup`` builtin type
|
|
and the ``except*`` syntax for handling exception groups in the interpreter.
|
|
The desired semantics of ``except*`` are sufficiently different from the
|
|
current exception handling semantics that we are not proposing to modify the
|
|
behavior of the ``except`` keyword but rather to add the new ``except*``
|
|
syntax.
|
|
|
|
Our premise is that exception groups and ``except*`` will be used
|
|
selectively, only when they are needed. We do not expect them to become
|
|
the default mechanism for exception handling. The decision to raise
|
|
exception groups from a library needs to be considered carefully and
|
|
regarded as an API-breaking change. We expect that this will normally be
|
|
done by introducing a new API rather than modifying an existing one.
|
|
|
|
|
|
Specification
|
|
=============
|
|
|
|
ExceptionGroup and BaseExceptionGroup
|
|
-------------------------------------
|
|
|
|
We propose to add two new builtin exception types:
|
|
``BaseExceptionGroup(BaseException)`` and
|
|
``ExceptionGroup(BaseExceptionGroup, Exception)``. They are assignable to
|
|
``Exception.__cause__`` and ``Exception.__context__``, and they can be
|
|
raised and handled as any exception with ``raise ExceptionGroup(...)`` and
|
|
``try: ... except ExceptionGroup: ...`` or ``raise BaseExceptionGroup(...)``
|
|
and ``try: ... except BaseExceptionGroup: ...``.
|
|
|
|
Both have a constructor that takes two positional-only arguments: a message
|
|
string and a sequence of the nested exceptions, which are exposed in the
|
|
fields ``message`` and ``exceptions``. For example:
|
|
``ExceptionGroup('issues', [ValueError('bad value'), TypeError('bad type')])``.
|
|
The difference between them is that ``ExceptionGroup`` can only wrap
|
|
``Exception`` subclasses while ``BaseExceptionGroup`` can wrap any
|
|
``BaseException`` subclass. The ``BaseExceptionGroup`` constructor
|
|
inspects the nested exceptions and if they are all ``Exception`` subclasses,
|
|
it returns an ``ExceptionGroup`` rather than a ``BaseExceptionGroup``. The
|
|
``ExceptionGroup`` constructor raises a ``TypeError`` if any of the nested
|
|
exceptions is not an ``Exception`` instance. In the rest of the document,
|
|
when we refer to an exception group, we mean either an ``ExceptionGroup``
|
|
or a ``BaseExceptionGroup``. When it is necessary to make the distinction,
|
|
we use the class name. For brevity, we will use ``ExceptionGroup`` in code
|
|
examples that are relevant to both.
|
|
|
|
Since an exception group can be nested, it represents a tree of exceptions,
|
|
where the leaves are plain exceptions and each internal node represents a time
|
|
at which the program grouped some unrelated exceptions into a new group and
|
|
raised them together.
|
|
|
|
The ``BaseExceptionGroup.subgroup(condition)`` method gives us a way to obtain
|
|
an exception group that has the same metadata (message, cause, context,
|
|
traceback) as the original group, and the same nested structure of groups, but
|
|
contains only those exceptions for which the condition is true:
|
|
|
|
.. code-block::
|
|
|
|
>>> eg = ExceptionGroup(
|
|
... "one",
|
|
... [
|
|
... TypeError(1),
|
|
... ExceptionGroup(
|
|
... "two",
|
|
... [TypeError(2), ValueError(3)]
|
|
... ),
|
|
... ExceptionGroup(
|
|
... "three",
|
|
... [OSError(4)]
|
|
... )
|
|
... ]
|
|
... )
|
|
>>> traceback.print_exception(eg)
|
|
| ExceptionGroup: one
|
|
+-+---------------- 1 ----------------
|
|
| TypeError: 1
|
|
+---------------- 2 ----------------
|
|
| ExceptionGroup: two
|
|
+-+---------------- 1 ----------------
|
|
| TypeError: 2
|
|
+---------------- 2 ----------------
|
|
| ValueError: 3
|
|
+------------------------------------
|
|
+---------------- 3 ----------------
|
|
| ExceptionGroup: three
|
|
+-+---------------- 1 ----------------
|
|
| OSError: 4
|
|
+------------------------------------
|
|
|
|
>>> type_errors = eg.subgroup(lambda e: isinstance(e, TypeError))
|
|
>>> traceback.print_exception(type_errors)
|
|
| ExceptionGroup: one
|
|
+-+---------------- 1 ----------------
|
|
| TypeError: 1
|
|
+---------------- 2 ----------------
|
|
| ExceptionGroup: two
|
|
+-+---------------- 1 ----------------
|
|
| TypeError: 2
|
|
+------------------------------------
|
|
>>>
|
|
|
|
|
|
The match condition is also applied to interior nodes (the exception
|
|
groups), and a match causes the whole subtree rooted at this node
|
|
to be included in the result.
|
|
|
|
Empty nested groups are omitted from the result, as in the
|
|
case of ``ExceptionGroup("three")`` in the example above. If none of the
|
|
exceptions match the condition, ``subgroup`` returns ``None`` rather
|
|
than an empty group. The original ``eg``
|
|
is unchanged by ``subgroup``, but the value returned is not necessarily a full
|
|
new copy. Leaf exceptions are not copied, nor are exception groups which are
|
|
fully contained in the result. When it is necessary to partition a
|
|
group because the condition holds for some, but not all of its
|
|
contained exceptions, a new ``ExceptionGroup`` or ``BaseExceptionGroup``
|
|
instance is created, while the ``__cause__``, ``__context__`` and
|
|
``__traceback__`` fields are copied by reference, so they are shared with
|
|
the original ``eg``.
|
|
|
|
If both the subgroup and its complement are needed, the
|
|
``BaseExceptionGroup.split(condition)`` method can be used:
|
|
|
|
.. code-block::
|
|
|
|
>>> type_errors, other_errors = eg.split(lambda e: isinstance(e, TypeError))
|
|
>>> traceback.print_exception(type_errors)
|
|
| ExceptionGroup: one
|
|
+-+---------------- 1 ----------------
|
|
| TypeError: 1
|
|
+---------------- 2 ----------------
|
|
| ExceptionGroup: two
|
|
+-+---------------- 1 ----------------
|
|
| TypeError: 2
|
|
+------------------------------------
|
|
>>> traceback.print_exception(other_errors)
|
|
| ExceptionGroup: one
|
|
+-+---------------- 1 ----------------
|
|
| ExceptionGroup: two
|
|
+-+---------------- 1 ----------------
|
|
| ValueError: 3
|
|
+------------------------------------
|
|
+---------------- 2 ----------------
|
|
| ExceptionGroup: three
|
|
+-+---------------- 1 ----------------
|
|
| OSError: 4
|
|
+------------------------------------
|
|
>>>
|
|
|
|
|
|
If a split is trivial (one side is empty), then None is returned for the
|
|
other side:
|
|
|
|
.. code-block::
|
|
|
|
>>> other_errors.split(lambda e: isinstance(e, SyntaxError))
|
|
(None, ExceptionGroup('one', [
|
|
ExceptionGroup('two', [
|
|
ValueError(3)
|
|
]),
|
|
ExceptionGroup('three', [
|
|
OSError(4)])]))
|
|
|
|
Since splitting by exception type is a very common use case, ``subgroup`` and
|
|
``split`` can take an exception type or tuple of exception types and treat it
|
|
as a shorthand for matching that type: ``eg.split(T)`` divides ``eg`` into the
|
|
subgroup of leaf exceptions that match the type ``T``, and the subgroup of those
|
|
that do not (using the same check as ``except`` for a match).
|
|
|
|
Subclassing Exception Groups
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
It is possible to subclass exception groups, but when doing that it is
|
|
usually necessary to specify how ``subgroup()`` and ``split()`` should
|
|
create new instances for the matching or non-matching part of the partition.
|
|
``BaseExceptionGroup`` exposes an instance method ``derive(self, excs)``
|
|
which is called whenever ``subgroup`` and ``split`` need to create a new
|
|
exception group. The parameter ``excs`` is the sequence of exceptions to
|
|
include in the new group. Since ``derive`` has access to self, it can
|
|
copy data from it to the new object. For example, if we need an exception
|
|
group subclass that has an additional error code field, we can do this:
|
|
|
|
.. code-block::
|
|
|
|
class MyExceptionGroup(ExceptionGroup):
|
|
def __new__(cls, message, excs, errcode):
|
|
obj = super().__new__(cls, message, excs)
|
|
obj.errcode = errcode
|
|
return obj
|
|
|
|
def derive(self, excs):
|
|
return MyExceptionGroup(self.message, excs, self.errcode)
|
|
|
|
|
|
Note that we override ``__new__`` rather than ``__init__``; this is because
|
|
``BaseExceptionGroup.__new__`` needs to inspect the constructor arguments, and
|
|
its signature is different from that of the subclass. Note also that our
|
|
``derive`` function does not copy the ``__context__``, ``__cause__`` and
|
|
``__traceback__`` fields, because ``subgroup`` and ``split`` do that for us.
|
|
|
|
With the class defined above, we have the following:
|
|
|
|
.. code-block::
|
|
|
|
>>> eg = MyExceptionGroup("eg", [TypeError(1), ValueError(2)], 42)
|
|
>>>
|
|
>>> match, rest = eg.split(ValueError)
|
|
>>> print(f'match: {match!r}: {match.errcode}')
|
|
match: MyExceptionGroup('eg', [ValueError(2)], 42): 42
|
|
>>> print(f'rest: {rest!r}: {rest.errcode}')
|
|
rest: MyExceptionGroup('eg', [TypeError(1)], 42): 42
|
|
>>>
|
|
|
|
If we do not override ``derive``, then split calls the one defined
|
|
on ``BaseExceptionGroup``, which returns an instance of ``ExceptionGroup``
|
|
if all contained exceptions are of type ``Exception``, and
|
|
``BaseExceptionGroup`` otherwise. For example:
|
|
|
|
.. code-block::
|
|
|
|
>>> class MyExceptionGroup(BaseExceptionGroup):
|
|
... pass
|
|
...
|
|
>>> eg = MyExceptionGroup("eg", [ValueError(1), KeyboardInterrupt(2)])
|
|
>>> match, rest = eg.split(ValueError)
|
|
>>> print(f'match: {match!r}')
|
|
match: ExceptionGroup('eg', [ValueError(1)])
|
|
>>> print(f'rest: {rest!r}')
|
|
rest: BaseExceptionGroup('eg', [KeyboardInterrupt(2)])
|
|
>>>
|
|
|
|
|
|
The Traceback of an Exception Group
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
For regular exceptions, the traceback represents a simple path of frames,
|
|
from the frame in which the exception was raised to the frame in which it
|
|
was caught or, if it hasn't been caught yet, the frame that the program's
|
|
execution is currently in. The list is constructed by the interpreter, which
|
|
appends any frame from which it exits to the traceback of the 'current
|
|
exception' if one exists. To support efficient appends, the links in a
|
|
traceback's list of frames are from the oldest to the newest frame. Appending
|
|
a new frame is then simply a matter of inserting a new head to the linked
|
|
list referenced from the exception's ``__traceback__`` field. Crucially, the
|
|
traceback's frame list is immutable in the sense that frames only need to be
|
|
added at the head, and never need to be removed.
|
|
|
|
We do not need to make any changes to this data structure. The ``__traceback__``
|
|
field of the exception group instance represents the path that the contained
|
|
exceptions travelled through together after being joined into the
|
|
group, and the same field on each of the nested exceptions
|
|
represents the path through which this exception arrived at the frame of the
|
|
merge.
|
|
|
|
What we do need to change is any code that interprets and displays tracebacks,
|
|
because it now needs to continue into tracebacks of nested exceptions, as
|
|
in the following example:
|
|
|
|
.. code-block::
|
|
|
|
>>> def f(v):
|
|
... try:
|
|
... raise ValueError(v)
|
|
... except ValueError as e:
|
|
... return e
|
|
...
|
|
>>> try:
|
|
... raise ExceptionGroup("one", [f(1)])
|
|
... except ExceptionGroup as e:
|
|
... eg = e
|
|
...
|
|
>>> raise ExceptionGroup("two", [f(2), eg])
|
|
+ Exception Group Traceback (most recent call last):
|
|
| File "<stdin>", line 1, in <module>
|
|
| ExceptionGroup: two
|
|
+-+---------------- 1 ----------------
|
|
| Traceback (most recent call last):
|
|
| File "<stdin>", line 3, in f
|
|
| ValueError: 2
|
|
+---------------- 2 ----------------
|
|
| Exception Group Traceback (most recent call last):
|
|
| File "<stdin>", line 2, in <module>
|
|
| ExceptionGroup: one
|
|
+-+---------------- 1 ----------------
|
|
| Traceback (most recent call last):
|
|
| File "<stdin>", line 3, in f
|
|
| ValueError: 1
|
|
+------------------------------------
|
|
>>>
|
|
|
|
Handling Exception Groups
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
We expect that when programs catch and handle exception groups, they will
|
|
typically either query to check if it has leaf exceptions for which some
|
|
condition holds (using ``subgroup`` or ``split``) or format the exception
|
|
(using the ``traceback`` module's methods).
|
|
|
|
It is less likely to be useful to iterate over the individual leaf exceptions.
|
|
To see why, suppose that an application caught an exception group raised by
|
|
an ``asyncio.gather()`` call. At this stage, the context for each specific
|
|
exception is lost. Any recovery for this exception should have been performed
|
|
before it was grouped with other exceptions [10]_.
|
|
Furthermore, the application is likely to react in the same way to any number
|
|
of instances of a certain exception type, so it is more likely that we will
|
|
want to know whether ``eg.subgroup(T)`` is None or not, than we are to be
|
|
interested in the number of ``Ts`` in ``eg``.
|
|
|
|
However, there are situations where it is necessary to inspect the
|
|
individual leaf exceptions. For example, suppose that we have an
|
|
exception group ``eg`` and that we want to log the ``OSErrors`` that have a
|
|
specific error code and reraise everything else. We can do this by passing
|
|
a function with side effects to ``subgroup``, as follows:
|
|
|
|
.. code-block::
|
|
|
|
def log_and_ignore_ENOENT(err):
|
|
if isinstance(err, OSError) and err.errno == ENOENT:
|
|
log(err)
|
|
return False
|
|
else:
|
|
return True
|
|
|
|
try:
|
|
. . .
|
|
except ExceptionGroup as eg:
|
|
eg = eg.subgroup(log_and_ignore_ENOENT)
|
|
if eg is not None:
|
|
raise eg
|
|
|
|
|
|
In the previous example, when ``log_and_ignore_ENOENT`` is invoked on a leaf
|
|
exception, only part of this exception's traceback is accessible -- the part
|
|
referenced from its ``__traceback__`` field. If we need the full traceback,
|
|
we need to look at the concatenation of the tracebacks of the exceptions on
|
|
the path from the root to this leaf. We can get that with direct iteration,
|
|
recursively, as follows:
|
|
|
|
.. code-block::
|
|
|
|
def leaf_generator(exc, tbs=None):
|
|
if tbs is None:
|
|
tbs = []
|
|
|
|
tbs.append(exc.__traceback__)
|
|
if isinstance(exc, BaseExceptionGroup):
|
|
for e in exc.exceptions:
|
|
yield from leaf_generator(e, tbs)
|
|
else:
|
|
# exc is a leaf exception and its traceback
|
|
# is the concatenation of the traceback
|
|
# segments in tbs.
|
|
|
|
# Note: the list returned (tbs) is reused in each iteration
|
|
# through the generator. Make a copy if your use case holds
|
|
# on to it beyond the current iteration or mutates its contents.
|
|
|
|
yield exc, tbs
|
|
tbs.pop()
|
|
|
|
|
|
We can then process the full tracebacks of the leaf exceptions:
|
|
|
|
.. code-block::
|
|
|
|
>>> import traceback
|
|
>>>
|
|
>>> def g(v):
|
|
... try:
|
|
... raise ValueError(v)
|
|
... except Exception as e:
|
|
... return e
|
|
...
|
|
>>> def f():
|
|
... raise ExceptionGroup("eg", [g(1), g(2)])
|
|
...
|
|
>>> try:
|
|
... f()
|
|
... except BaseException as e:
|
|
... eg = e
|
|
...
|
|
>>> for (i, (exc, tbs)) in enumerate(leaf_generator(eg)):
|
|
... print(f"\n=== Exception #{i+1}:")
|
|
... traceback.print_exception(exc)
|
|
... print(f"The complete traceback for Exception #{i+1}:")
|
|
... for tb in tbs:
|
|
... traceback.print_tb(tb)
|
|
...
|
|
|
|
=== Exception #1:
|
|
Traceback (most recent call last):
|
|
File "<stdin>", line 3, in g
|
|
ValueError: 1
|
|
The complete traceback for Exception #1
|
|
File "<stdin>", line 2, in <module>
|
|
File "<stdin>", line 2, in f
|
|
File "<stdin>", line 3, in g
|
|
|
|
=== Exception #2:
|
|
Traceback (most recent call last):
|
|
File "<stdin>", line 3, in g
|
|
ValueError: 2
|
|
The complete traceback for Exception #2:
|
|
File "<stdin>", line 2, in <module>
|
|
File "<stdin>", line 2, in f
|
|
File "<stdin>", line 3, in g
|
|
>>>
|
|
|
|
except*
|
|
-------
|
|
|
|
We are proposing to introduce a new variant of the ``try..except`` syntax to
|
|
simplify working with exception groups. The ``*`` symbol indicates that multiple
|
|
exceptions can be handled by each ``except*`` clause:
|
|
|
|
.. code-block::
|
|
|
|
try:
|
|
...
|
|
except* SpamError:
|
|
...
|
|
except* FooError as e:
|
|
...
|
|
except* (BarError, BazError) as e:
|
|
...
|
|
|
|
In a traditional ``try-except`` statement there is only one exception to handle,
|
|
so the body of at most one ``except`` clause executes; the first one that matches
|
|
the exception. With the new syntax, an ``except*`` clause can match a subgroup
|
|
of the exception group that was raised, while the remaining part is matched
|
|
by following ``except*`` clauses. In other words, a single exception group can
|
|
cause several ``except*`` clauses to execute, but each such clause executes at
|
|
most once (for all matching exceptions from the group) and each exception is
|
|
either handled by exactly one clause (the first one that matches its type)
|
|
or is reraised at the end. The manner in which each exception is handled by
|
|
a ``try-except*`` block is independent of any other exceptions in the group.
|
|
|
|
For example, suppose that the body of the ``try`` block above raises
|
|
``eg = ExceptionGroup('msg', [FooError(1), FooError(2), BazError()])``.
|
|
The ``except*`` clauses are evaluated in order by calling ``split`` on the
|
|
``unhandled`` exception group, which is initially equal to ``eg`` and then shrinks
|
|
as exceptions are matched and extracted from it. In the first ``except*`` clause,
|
|
``unhandled.split(SpamError)`` returns ``(None, unhandled)`` so the body of this
|
|
block is not executed and ``unhandled`` is unchanged. For the second block,
|
|
``unhandled.split(FooError)`` returns a non-trivial split ``(match, rest)`` with
|
|
``match = ExceptionGroup('msg', [FooError(1), FooError(2)])``
|
|
and ``rest = ExceptionGroup('msg', [BazError()])``. The body of this ``except*``
|
|
block is executed, with the value of ``e`` and ``sys.exc_info()`` set to ``match``.
|
|
Then, ``unhandled`` is set to ``rest``.
|
|
Finally, the third block matches the remaining exception so it is executed
|
|
with ``e`` and ``sys.exc_info()`` set to ``ExceptionGroup('msg', [BazError()])``.
|
|
|
|
|
|
Exceptions are matched using a subclass check. For example:
|
|
|
|
.. code-block::
|
|
|
|
try:
|
|
low_level_os_operation()
|
|
except* OSError as eg:
|
|
for e in eg.exceptions:
|
|
print(type(e).__name__)
|
|
|
|
could output:
|
|
|
|
.. code-block::
|
|
|
|
BlockingIOError
|
|
ConnectionRefusedError
|
|
OSError
|
|
InterruptedError
|
|
BlockingIOError
|
|
|
|
The order of ``except*`` clauses is significant just like with the regular
|
|
``try..except``:
|
|
|
|
.. code-block::
|
|
|
|
>>> try:
|
|
... raise ExceptionGroup("problem", [BlockingIOError()])
|
|
... except* OSError as e: # Would catch the error
|
|
... print(repr(e))
|
|
... except* BlockingIOError: # Would never run
|
|
... print('never')
|
|
...
|
|
ExceptionGroup('problem', [BlockingIOError()])
|
|
|
|
Recursive Matching
|
|
~~~~~~~~~~~~~~~~~~
|
|
|
|
The matching of ``except*`` clauses against an exception group is performed
|
|
recursively, using the ``split()`` method:
|
|
|
|
.. code-block::
|
|
|
|
>>> try:
|
|
... raise ExceptionGroup(
|
|
... "eg",
|
|
... [
|
|
... ValueError('a'),
|
|
... TypeError('b'),
|
|
... ExceptionGroup(
|
|
... "nested",
|
|
... [TypeError('c'), KeyError('d')])
|
|
... ]
|
|
... )
|
|
... except* TypeError as e1:
|
|
... print(f'e1 = {e1!r}')
|
|
... except* Exception as e2:
|
|
... print(f'e2 = {e2!r}')
|
|
...
|
|
e1 = ExceptionGroup('eg', [TypeError('b'), ExceptionGroup('nested', [TypeError('c')])])
|
|
e2 = ExceptionGroup('eg', [ValueError('a'), ExceptionGroup('nested', [KeyError('d')])])
|
|
>>>
|
|
|
|
Unmatched Exceptions
|
|
~~~~~~~~~~~~~~~~~~~~
|
|
|
|
If not all exceptions in an exception group were matched by the ``except*``
|
|
clauses, the remaining part of the group is propagated on:
|
|
|
|
.. code-block::
|
|
|
|
>>> try:
|
|
... try:
|
|
... raise ExceptionGroup(
|
|
... "msg", [
|
|
... ValueError('a'), TypeError('b'),
|
|
... TypeError('c'), KeyError('e')
|
|
... ]
|
|
... )
|
|
... except* ValueError as e:
|
|
... print(f'got some ValueErrors: {e!r}')
|
|
... except* TypeError as e:
|
|
... print(f'got some TypeErrors: {e!r}')
|
|
... except ExceptionGroup as e:
|
|
... print(f'propagated: {e!r}')
|
|
...
|
|
got some ValueErrors: ExceptionGroup('msg', [ValueError('a')])
|
|
got some TypeErrors: ExceptionGroup('msg', [TypeError('b'), TypeError('c')])
|
|
propagated: ExceptionGroup('msg', [KeyError('e')])
|
|
>>>
|
|
|
|
|
|
Naked Exceptions
|
|
~~~~~~~~~~~~~~~~
|
|
|
|
If the exception raised inside the ``try`` body is not of type ``ExceptionGroup``
|
|
or ``BaseExceptionGroup``, we call it a ``naked`` exception. If its type matches
|
|
one of the ``except*`` clauses, it is caught and wrapped by an ``ExceptionGroup``
|
|
(or ``BaseExceptionGroup`` if it is not an ``Exception`` subclass) with an empty
|
|
message string. This is to make the type of ``e`` consistent and statically known:
|
|
|
|
.. code-block::
|
|
|
|
>>> try:
|
|
... raise BlockingIOError
|
|
... except* OSError as e:
|
|
... print(repr(e))
|
|
...
|
|
ExceptionGroup('', [BlockingIOError()])
|
|
|
|
However, if a naked exception is not caught, it propagates in its original
|
|
naked form:
|
|
|
|
.. code-block::
|
|
|
|
>>> try:
|
|
... try:
|
|
... raise ValueError(12)
|
|
... except* TypeError as e:
|
|
... print('never')
|
|
... except ValueError as e:
|
|
... print(f'caught ValueError: {e!r}')
|
|
...
|
|
caught ValueError: ValueError(12)
|
|
>>>
|
|
|
|
Raising exceptions in an ``except*`` block
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
In a traditional ``except`` block, there are two ways to raise exceptions:
|
|
``raise e`` to explicitly raise an exception object ``e``, or naked ``raise`` to
|
|
reraise the 'current exception'. When ``e`` is the current exception, the two
|
|
forms are not equivalent because a reraise does not add the current frame to
|
|
the stack:
|
|
|
|
.. code-block::
|
|
|
|
def foo(): | def foo():
|
|
try: | try:
|
|
1 / 0 | 1 / 0
|
|
except ZeroDivisionError as e: | except ZeroDivisionError:
|
|
raise e | raise
|
|
|
|
|
foo() | foo()
|
|
|
|
|
Traceback (most recent call last): | Traceback (most recent call last):
|
|
File "/Users/guido/a.py", line 7 | File "/Users/guido/b.py", line 7
|
|
foo() | foo()
|
|
File "/Users/guido/a.py", line 5 | File "/Users/guido/b.py", line 3
|
|
raise e | 1/0
|
|
File "/Users/guido/a.py", line 3 | ZeroDivisionError: division by zero
|
|
1/0 |
|
|
ZeroDivisionError: division by zero |
|
|
|
|
|
|
This holds for exception groups as well, but the situation is now more complex
|
|
because there can be exceptions raised and reraised from multiple ``except*``
|
|
clauses, as well as unhandled exceptions that need to propagate.
|
|
The interpreter needs to combine all those exceptions into a result, and
|
|
raise that.
|
|
|
|
The reraised exceptions and the unhandled exceptions are subgroups of the
|
|
original group, and share its metadata (cause, context, traceback).
|
|
On the other hand, each of the explicitly raised exceptions has its own
|
|
metadata - the traceback contains the line from which it was raised, its
|
|
cause is whatever it may have been explicitly chained to, and its context is the
|
|
value of ``sys.exc_info()`` in the ``except*`` clause of the raise.
|
|
|
|
In the aggregated exception group, the reraised and unhandled exceptions have
|
|
the same relative structure as in the original exception, as if they were split
|
|
off together in one ``subgroup`` call. For example, in the snippet below the
|
|
inner ``try-except*`` block raises an ``ExceptionGroup`` that contains all
|
|
``ValueErrors`` and ``TypeErrors`` merged back into the same shape they had in
|
|
the original ``ExceptionGroup``:
|
|
|
|
.. code-block::
|
|
|
|
>>> try:
|
|
... try:
|
|
... raise ExceptionGroup(
|
|
... "eg",
|
|
... [
|
|
... ValueError(1),
|
|
... TypeError(2),
|
|
... OSError(3),
|
|
... ExceptionGroup(
|
|
... "nested",
|
|
... [OSError(4), TypeError(5), ValueError(6)])
|
|
... ]
|
|
... )
|
|
... except* ValueError as e:
|
|
... print(f'*ValueError: {e!r}')
|
|
... raise
|
|
... except* OSError as e:
|
|
... print(f'*OSError: {e!r}')
|
|
... except ExceptionGroup as e:
|
|
... print(repr(e))
|
|
...
|
|
*ValueError: ExceptionGroup('eg', [ValueError(1), ExceptionGroup('nested', [ValueError(6)])])
|
|
*OSError: ExceptionGroup('eg', [OSError(3), ExceptionGroup('nested', [OSError(4)])])
|
|
ExceptionGroup('eg', [ValueError(1), TypeError(2), ExceptionGroup('nested', [TypeError(5), ValueError(6)])])
|
|
>>>
|
|
|
|
|
|
When exceptions are raised explicitly, they are independent of the original
|
|
exception group, and cannot be merged with it (they have their own cause,
|
|
context and traceback). Instead, they are combined into a new ``ExceptionGroup``
|
|
(or ``BaseExceptionGroup``), which also contains the reraised/unhandled
|
|
subgroup described above.
|
|
|
|
In the following example, the ``ValueErrors`` were raised so they are in their
|
|
own ``ExceptionGroup``, while the ``OSErrors`` were reraised so they were
|
|
merged with the unhandled ``TypeErrors``.
|
|
|
|
.. code-block::
|
|
|
|
>>> try:
|
|
... raise ExceptionGroup(
|
|
... "eg",
|
|
... [
|
|
... ValueError(1),
|
|
... TypeError(2),
|
|
... OSError(3),
|
|
... ExceptionGroup(
|
|
... "nested",
|
|
... [OSError(4), TypeError(5), ValueError(6)])
|
|
... ]
|
|
... )
|
|
... except* ValueError as e:
|
|
... print(f'*ValueError: {e!r}')
|
|
... raise e
|
|
... except* OSError as e:
|
|
... print(f'*OSError: {e!r}')
|
|
... raise
|
|
...
|
|
*ValueError: ExceptionGroup('eg', [ValueError(1), ExceptionGroup('nested', [ValueError(6)])])
|
|
*OSError: ExceptionGroup('eg', [OSError(3), ExceptionGroup('nested', [OSError(4)])])
|
|
| ExceptionGroup
|
|
+-+---------------- 1 ----------------
|
|
| Exception Group Traceback (most recent call last):
|
|
| File "<stdin>", line 15, in <module>
|
|
| File "<stdin>", line 2, in <module>
|
|
| ExceptionGroup: eg
|
|
+-+---------------- 1 ----------------
|
|
| ValueError: 1
|
|
+---------------- 2 ----------------
|
|
| ExceptionGroup: nested
|
|
+-+---------------- 1 ----------------
|
|
| ValueError: 6
|
|
+------------------------------------
|
|
+---------------- 2 ----------------
|
|
| Exception Group Traceback (most recent call last):
|
|
| File "<stdin>", line 2, in <module>
|
|
| ExceptionGroup: eg
|
|
+-+---------------- 1 ----------------
|
|
| TypeError: 2
|
|
+---------------- 2 ----------------
|
|
| OSError: 3
|
|
+---------------- 3 ----------------
|
|
| ExceptionGroup: nested
|
|
+-+---------------- 1 ----------------
|
|
| OSError: 4
|
|
+---------------- 2 ----------------
|
|
| TypeError: 5
|
|
+------------------------------------
|
|
>>>
|
|
|
|
|
|
Chaining
|
|
~~~~~~~~
|
|
|
|
Explicitly raised exception groups are chained as with any exceptions. The
|
|
following example shows how part of ``ExceptionGroup`` "one" became the
|
|
context for ``ExceptionGroup`` "two", while the other part was combined with
|
|
it into the new ``ExceptionGroup``.
|
|
|
|
.. code-block::
|
|
|
|
>>> try:
|
|
... raise ExceptionGroup("one", [ValueError('a'), TypeError('b')])
|
|
... except* ValueError:
|
|
... raise ExceptionGroup("two", [KeyError('x'), KeyError('y')])
|
|
...
|
|
| ExceptionGroup
|
|
+-+---------------- 1 ----------------
|
|
| Exception Group Traceback (most recent call last):
|
|
| File "<stdin>", line 2, in <module>
|
|
| ExceptionGroup: one
|
|
+-+---------------- 1 ----------------
|
|
| ValueError: a
|
|
+------------------------------------
|
|
|
|
|
| During handling of the above exception, another exception occurred:
|
|
|
|
|
| Exception Group Traceback (most recent call last):
|
|
| File "<stdin>", line 4, in <module>
|
|
| ExceptionGroup: two
|
|
+-+---------------- 1 ----------------
|
|
| KeyError: 'x'
|
|
+---------------- 2 ----------------
|
|
| KeyError: 'y'
|
|
+------------------------------------
|
|
+---------------- 2 ----------------
|
|
| Exception Group Traceback (most recent call last):
|
|
| File "<stdin>", line 2, in <module>
|
|
| ExceptionGroup: one
|
|
+-+---------------- 1 ----------------
|
|
| TypeError: b
|
|
+------------------------------------
|
|
>>>
|
|
|
|
|
|
Raising New Exceptions
|
|
~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
In the previous examples the explicit raises were of the exceptions that
|
|
were caught, so for completion we show a new exception being raised, with
|
|
chaining:
|
|
|
|
.. code-block::
|
|
|
|
>>> try:
|
|
... raise TypeError('bad type')
|
|
... except* TypeError as e:
|
|
... raise ValueError('bad value') from e
|
|
...
|
|
| ExceptionGroup
|
|
+-+---------------- 1 ----------------
|
|
| ExceptionGroup
|
|
+-+---------------- 1 ----------------
|
|
| Traceback (most recent call last):
|
|
| File "<stdin>", line 2, in <module>
|
|
| TypeError: bad type
|
|
+------------------------------------
|
|
|
|
|
| The above exception was the direct cause of the following exception:
|
|
|
|
|
| Traceback (most recent call last):
|
|
| File "<stdin>", line 4, in <module>
|
|
| ValueError: bad value
|
|
+------------------------------------
|
|
>>>
|
|
|
|
|
|
Note that exceptions raised in one ``except*`` clause are not eligible to match
|
|
other clauses from the same ``try`` statement:
|
|
|
|
.. code-block::
|
|
|
|
>>> try:
|
|
... raise TypeError(1)
|
|
... except* TypeError:
|
|
... raise ValueError(2) from None # <- not caught in the next clause
|
|
... except* ValueError:
|
|
... print('never')
|
|
...
|
|
| ExceptionGroup
|
|
+-+---------------- 1 ----------------
|
|
| Traceback (most recent call last):
|
|
| File "<stdin>", line 4, in <module>
|
|
| ValueError: 2
|
|
+----------------------------------------
|
|
>>>
|
|
|
|
|
|
Raising a new instance of a naked exception does not cause this exception to
|
|
be wrapped by an exception group. Rather, the exception is raised as is, and
|
|
if it needs to be combined with other propagated exceptions, it becomes a
|
|
direct child of the new exception group created for that:
|
|
|
|
|
|
.. code-block::
|
|
|
|
>>> try:
|
|
... raise ExceptionGroup("eg", [ValueError('a')])
|
|
... except* ValueError:
|
|
... raise KeyError('x')
|
|
...
|
|
| ExceptionGroup
|
|
+-+---------------- 1 ----------------
|
|
| Exception Group Traceback (most recent call last):
|
|
| File "<stdin>", line 2, in <module>
|
|
| ExceptionGroup: eg
|
|
+-+---------------- 1 ----------------
|
|
| ValueError: a
|
|
+------------------------------------
|
|
|
|
|
| During handling of the above exception, another exception occurred:
|
|
|
|
|
| Traceback (most recent call last):
|
|
| File "<stdin>", line 4, in <module>
|
|
| KeyError: 'x'
|
|
+------------------------------------
|
|
>>>
|
|
>>> try:
|
|
... raise ExceptionGroup("eg", [ValueError('a'), TypeError('b')])
|
|
... except* ValueError:
|
|
... raise KeyError('x')
|
|
...
|
|
| ExceptionGroup
|
|
+-+---------------- 1 ----------------
|
|
| Exception Group Traceback (most recent call last):
|
|
| File "<stdin>", line 2, in <module>
|
|
| ExceptionGroup: eg
|
|
+-+---------------- 1 ----------------
|
|
| ValueError: a
|
|
+------------------------------------
|
|
|
|
|
| During handling of the above exception, another exception occurred:
|
|
|
|
|
| Traceback (most recent call last):
|
|
| File "<stdin>", line 4, in <module>
|
|
| KeyError: 'x'
|
|
+---------------- 2 ----------------
|
|
| Exception Group Traceback (most recent call last):
|
|
| File "<stdin>", line 2, in <module>
|
|
| ExceptionGroup: eg
|
|
+-+---------------- 1 ----------------
|
|
| TypeError: b
|
|
+------------------------------------
|
|
>>>
|
|
|
|
|
|
Finally, as an example of how the proposed semantics can help us work
|
|
effectively with exception groups, the following code ignores all ``EPIPE``
|
|
OS errors, while letting all other exceptions propagate.
|
|
|
|
.. code-block::
|
|
|
|
try:
|
|
low_level_os_operation()
|
|
except* OSError as errors:
|
|
exc = errors.subgroup(lambda e: e.errno != errno.EPIPE)
|
|
if exc is not None:
|
|
raise exc from None
|
|
|
|
|
|
Caught Exception Objects
|
|
~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
It is important to point out that the exception group bound to ``e`` in an
|
|
``except*`` clause is an ephemeral object. Raising it via ``raise`` or
|
|
``raise e`` will not cause changes to the overall shape of the original
|
|
exception group. Any modifications to ``e`` will likely be lost:
|
|
|
|
.. code-block::
|
|
|
|
>>> eg = ExceptionGroup("eg", [TypeError(12)])
|
|
>>> eg.foo = 'foo'
|
|
>>> try:
|
|
... raise eg
|
|
... except* TypeError as e:
|
|
... e.foo = 'bar'
|
|
... # ^----------- ``e`` is an ephemeral object that might get
|
|
>>> # destroyed after the ``except*`` clause.
|
|
>>> eg.foo
|
|
'foo'
|
|
|
|
|
|
Forbidden Combinations
|
|
~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
It is not possible to use both traditional ``except`` blocks and the new
|
|
``except*`` clauses in the same ``try`` statement. The following is a
|
|
``SyntaxError``:
|
|
|
|
.. code-block::
|
|
|
|
try:
|
|
...
|
|
except ValueError:
|
|
pass
|
|
except* CancelledError: # <- SyntaxError:
|
|
pass # combining ``except`` and ``except*``
|
|
# is prohibited
|
|
|
|
It is possible to catch the ``ExceptionGroup`` and ``BaseExceptionGroup``
|
|
types with ``except``, but not with ``except*`` because the latter is
|
|
ambiguous:
|
|
|
|
.. code-block::
|
|
|
|
try:
|
|
...
|
|
except ExceptionGroup: # <- This works
|
|
pass
|
|
|
|
try:
|
|
...
|
|
except* ExceptionGroup: # <- Runtime error
|
|
pass
|
|
|
|
try:
|
|
...
|
|
except* (TypeError, ExceptionGroup): # <- Runtime error
|
|
pass
|
|
|
|
|
|
An empty "match anything" ``except*`` block is not supported as its meaning may
|
|
be confusing:
|
|
|
|
.. code-block::
|
|
|
|
try:
|
|
...
|
|
except*: # <- SyntaxError
|
|
pass
|
|
|
|
|
|
``continue``, ``break``, and ``return`` are disallowed in ``except*`` clauses,
|
|
causing a ``SyntaxError``. This is because the exceptions in an
|
|
``ExceptionGroup`` are assumed to be independent, and the presence or absence
|
|
of one of them should not impact handling of the others, as could happen if we
|
|
allow an ``except*`` clause to change the way control flows through other
|
|
clauses.
|
|
|
|
|
|
Backwards Compatibility
|
|
=======================
|
|
|
|
Backwards compatibility was a requirement of our design, and the changes we
|
|
propose in this PEP will not break any existing code:
|
|
|
|
* The addition of the new builtin exception types ``ExceptionGroup`` and
|
|
``BaseExceptionGroup`` does not impact existing programs. The way that
|
|
existing exceptions are handled and displayed does not change in any way.
|
|
|
|
* The behaviour of ``except`` is unchanged so existing code will continue to work.
|
|
Programs will only be impacted by the changes proposed in this PEP once they
|
|
begin to use exception groups and ``except*``.
|
|
|
|
* An important concern was that ``except Exception:`` will continue to catch
|
|
almost all exceptions, and by making ``ExceptionGroup`` extend ``Exception``
|
|
we ensured that this will be the case. ``BaseExceptionGroups`` will not be
|
|
caught, which is appropriate because they include exceptions that would not
|
|
have been caught by ``except Exception``.
|
|
|
|
Once programs begin to use these features, there will be migration issues to
|
|
consider:
|
|
|
|
* An ``except T:`` clause that wraps code which is now potentially raising
|
|
an exception group may need to become ``except* T:``, and its body may
|
|
need to be updated. This means that raising an exception group is an
|
|
API-breaking change and will likely be done in new APIs rather than
|
|
added to existing ones.
|
|
|
|
* Libraries that need to support older Python versions will not be able to use
|
|
``except*`` or raise exception groups.
|
|
|
|
|
|
How to Teach This
|
|
=================
|
|
|
|
Exception groups and ``except*`` will be documented as part of the language
|
|
standard. Libraries that raise exception groups such as ``asyncio`` will need
|
|
to specify this in their documentation and clarify which API calls need to be
|
|
wrapped with ``try-except*`` rather than ``try-except``.
|
|
|
|
|
|
Reference Implementation
|
|
========================
|
|
|
|
We developed these concepts (and the examples for this PEP) with
|
|
the help of the reference implementation [11]_.
|
|
|
|
It has the builtin ``ExceptionGroup`` along with the changes to the traceback
|
|
formatting code, in addition to the grammar, compiler and interpreter changes
|
|
required to support ``except*``. ``BaseExceptionGroup`` will be added
|
|
soon.
|
|
|
|
Two opcodes were added: one implements the exception type match check via
|
|
``ExceptionGroup.split()``, and the other is used at the end of a ``try-except``
|
|
construct to merge all unhandled, raised and reraised exceptions (if any).
|
|
The raised/reraised exceptions are collected in a list on the runtime stack.
|
|
For this purpose, the body of each ``except*`` clause is wrapped in a traditional
|
|
``try-except`` which captures any exceptions raised. Both raised and reraised
|
|
exceptions are collected in the same list. When the time comes to merge them
|
|
into a result, the raised and reraised exceptions are distinguished by comparing
|
|
their metadata fields (context, cause, traceback) with those of the originally
|
|
raised exception. As mentioned above, the reraised exceptions have the same
|
|
metadata as the original, while the raised ones do not.
|
|
|
|
Rejected Ideas
|
|
==============
|
|
|
|
Make Exception Groups Iterable
|
|
------------------------------
|
|
|
|
We considered making exception groups iterable, so that ``list(eg)`` would
|
|
produce a flattened list of the leaf exceptions contained in the group.
|
|
We decided that this would not be a sound API, because the metadata
|
|
(cause, context and traceback) of the individual exceptions in a group is
|
|
incomplete and this could create problems.
|
|
|
|
Furthermore, as we explained in the `Handling Exception Groups`_ section, we
|
|
find it unlikely that iteration over leaf exceptions will have many use cases.
|
|
We did, however, provide there the code for a traversal algorithm that
|
|
correctly constructs each leaf exceptions' metadata. If it does turn out to
|
|
be useful in practice, we can in the future add that utility to the standard
|
|
library or even make exception groups iterable.
|
|
|
|
Make ``ExceptionGroup`` Extend ``BaseException``
|
|
------------------------------------------------
|
|
|
|
We considered making ``ExceptionGroup`` subclass only ``BaseException``,
|
|
and not ``Exception``. The rationale of this was that we expect exception
|
|
groups to be used in a deliberate manner where they are needed, and raised
|
|
only by APIs that are specifically designed and documented to do so. In
|
|
this context, an ``ExceptionGroup`` escaping from an API that is not
|
|
intended to raise one is a bug, and we wanted to give it "fatal error"
|
|
status so that ``except Exception`` will not inadvertently swallow it.
|
|
This would have been consistent with the way ``except T:`` does not catch
|
|
exception groups that contain ``T`` for all other types, and would help
|
|
contain ``ExceptionGroups`` to the parts of the program in which they are
|
|
supposed to appear. However, it was clear from the public discussion that
|
|
``T=Exception`` is a special case, and there are developers who feel strongly
|
|
that ``except Exception:`` should catch "almost everything", including
|
|
exception groups. This is why we decided to make ``ExceptionGroup`` a
|
|
subclass of ``Exception``.
|
|
|
|
Make it Impossible to Wrap ``BaseExceptions`` in an Exception Group
|
|
-------------------------------------------------------------------
|
|
|
|
A consequence of the decision to make ``ExceptionGroup`` extend
|
|
``Exception`` is that ``ExceptionGroup`` should not wrap ``BaseExceptions``
|
|
like ``KeyboardInterrupt``, as they are not currently caught by
|
|
``except Exception:``. We considered the option of simply making it
|
|
impossible to wrap ``BaseExceptions``, but eventually decided to make
|
|
it possible through the ``BaseExceptionGroup`` type, which extends
|
|
``BaseException`` rather than ``Exception``. Making this possible
|
|
adds flexibility to the language and leaves it for the programmer to
|
|
weigh the benefit of wrapping ``BaseExceptions`` rather than propagating
|
|
them in their naked form while discarding any other exceptions.
|
|
|
|
Traceback Representation
|
|
------------------------
|
|
|
|
We considered options for adapting the traceback data structure to represent
|
|
trees, but it became apparent that a traceback tree is not meaningful once
|
|
separated from the exceptions it refers to. While a simple-path traceback can
|
|
be attached to any exception by a ``with_traceback()`` call, it is hard to
|
|
imagine a case where it makes sense to assign a traceback tree to an exception
|
|
group. Furthermore, a useful display of the traceback includes information
|
|
about the nested exceptions. For these reasons we decided that it is best to
|
|
leave the traceback mechanism as it is and modify the traceback display code.
|
|
|
|
Extend ``except`` to Handle Exception Groups
|
|
---------------------------------------------
|
|
|
|
We considered extending the semantics of ``except`` to handle
|
|
exception groups, instead of introducing ``except*``. There were two
|
|
backwards compatibility concerns with this. The first is the type of the
|
|
caught exception. Consider this example:
|
|
|
|
.. code-block::
|
|
|
|
try:
|
|
. . .
|
|
except OSError as err:
|
|
if err.errno != ENOENT:
|
|
raise
|
|
|
|
If the value assigned to err is an exception group containing all of
|
|
the ``OSErrors`` that were raised, then the attribute access ``err.errno``
|
|
no longer works. So we would need to execute the body of the ``except``
|
|
clause multiple times, once for each exception in the group. However, this
|
|
too is a potentially breaking change because at the moment we write ``except``
|
|
clauses with the knowledge that they are only executed once. If there is
|
|
a non-idempotent operation there, such as releasing a resource, the
|
|
repetition could be harmful.
|
|
|
|
The idea of making ``except`` iterate over the leaf exceptions of an exception
|
|
group is at the heart of an `alternative proposal to this PEP by Nathaniel J. Smith
|
|
<https://discuss.python.org/t/flat-exception-groups-alternative-to-pep-654/10433>`_,
|
|
and the discussion about that proposal further elaborates on the pitfalls of
|
|
changing ``except`` semantics in a mature language like Python, as well as
|
|
deviating from the semantics that parallel constructs have in other languages.
|
|
|
|
Another option that came up in the public discussion was to add ``except*``,
|
|
but also make ``except`` treat ``ExceptionGroups`` as a special case.
|
|
``except`` would then do something along the lines of extracting one exception
|
|
of matching type from the group in order to handle it (while discarding all
|
|
the other exceptions in the group). The motivation behind
|
|
these suggestions was to make the adoption of exception groups safer, in that
|
|
``except T`` catches ``Ts`` that are wrapped in exception groups. We decided
|
|
that such an approach adds considerable complexity to the semantics of the
|
|
language without making it more powerful. Even if it would make the adoption
|
|
of exception groups slightly easier (which is not at all obvious), these are
|
|
not the semantics we would like to have in the long term.
|
|
|
|
|
|
A New ``except`` Alternative
|
|
----------------------------
|
|
|
|
We considered introducing a new keyword (such as ``catch``) which can be used
|
|
to handle both naked exceptions and exception groups. Its semantics would
|
|
be the same as those of ``except*`` when catching an exception group, but
|
|
it would not wrap a naked exception to create an exception group. This
|
|
would have been part of a long term plan to replace ``except`` by ``catch``,
|
|
but we decided that deprecating ``except`` in favour of an enhanced keyword
|
|
would be too confusing for users at this time, so it is more appropriate
|
|
to introduce the ``except*`` syntax for exception groups while ``except``
|
|
continues to be used for simple exceptions.
|
|
|
|
Applying an ``except*`` Clause on One Exception at a Time
|
|
---------------------------------------------------------
|
|
|
|
We explained above that it is unsafe to execute an ``except`` clause in
|
|
existing code more than once, because the code may not be idempotent.
|
|
We considered doing this in the new ``except*`` clauses,
|
|
where the backwards compatibility considerations do not exist.
|
|
The idea is to always execute an ``except*`` clause on a single exception,
|
|
possibly executing the same clause multiple times when it matches multiple
|
|
exceptions. We decided instead to execute each ``except*`` clause at most
|
|
once, giving it an exception group that contains all matching exceptions. The
|
|
reason for this decision was the observation that when a program needs to know
|
|
the particular context of an exception it is handling, the exception is
|
|
handled before it is grouped and raised together with other exceptions.
|
|
|
|
For example, ``KeyError`` is an exception that typically relates to a certain
|
|
operation. Any recovery code would be local to the place where the error
|
|
occurred, and would use the traditional ``except``:
|
|
|
|
.. code-block::
|
|
|
|
try:
|
|
dct[key]
|
|
except KeyError:
|
|
# handle the exception
|
|
|
|
It is unlikely that asyncio users would want to do something like this:
|
|
|
|
.. code-block::
|
|
|
|
try:
|
|
async with asyncio.TaskGroup() as g:
|
|
g.create_task(task1); g.create_task(task2)
|
|
except* KeyError:
|
|
# handling KeyError here is meaningless, there's
|
|
# no context to do anything with it but to log it.
|
|
|
|
When a program handles a collection of exceptions that were aggregated into
|
|
an exception group, it would not typically attempt to recover from any
|
|
particular failed operation, but will rather use the types of the errors to
|
|
determine how they should impact the program's control flow or what logging
|
|
or cleanup is required. This decision is likely to be the same whether the group
|
|
contains a single or multiple instances of something like a ``KeyboardInterrupt``
|
|
or ``asyncio.CancelledError``. Therefore, it is more convenient to handle all
|
|
exceptions matching an ``except*`` at once. If it does turn out to be necessary,
|
|
the handler can inpect the exception group and process the individual
|
|
exceptions in it.
|
|
|
|
Not Matching Naked Exceptions in ``except*``
|
|
--------------------------------------------
|
|
|
|
We considered the option of making ``except* T`` match only exception groups
|
|
that contain ``Ts``, but not naked ``Ts``. To see why we thought this would
|
|
not be a desirable feature, return to the distinction in the previous paragraph
|
|
between operation errors and control flow exceptions. If we don't know whether
|
|
we should expect naked exceptions or exception groups from the body of a
|
|
``try`` block, then we're not in the position of handling operation errors.
|
|
Rather, we are likely calling a fairly generic function and will be handling
|
|
errors to make control flow decisions. We are likely to do the same thing
|
|
whether we catch a naked exception of type ``T`` or an exception group
|
|
with one or more ``Ts``. Therefore, the burden of having to explicitly handle
|
|
both is not likely to have semantic benefit.
|
|
|
|
If it does turn out to be necessary to make the distinction, it is always
|
|
possible to nest in the ``try-except*`` clause an additional ``try-except``
|
|
clause which intercepts and handles a naked exception before the ``except*``
|
|
clause has a chance to wrap it in an exception group. In this case the
|
|
overhead of specifying both is not additional burden - we really do need to
|
|
write a separate code block to handle each case:
|
|
|
|
.. code-block::
|
|
|
|
try:
|
|
try:
|
|
...
|
|
except SomeError:
|
|
# handle the naked exception
|
|
except* SomeError:
|
|
# handle the exception group
|
|
|
|
|
|
Allow mixing ``except:`` and ``except*:`` in the same ``try``
|
|
-------------------------------------------------------------
|
|
|
|
This option was rejected because it adds complexity without adding useful
|
|
semantics. Presumably the intention would be that an ``except T:`` block handles
|
|
only naked exceptions of type ``T``, while ``except* T:`` handles ``T`` in
|
|
exception groups. We already discussed above why this is unlikely
|
|
to be useful in practice, and if it is needed then the nested ``try-except``
|
|
block can be used instead to achieve the same result.
|
|
|
|
``try*`` instead of ``except*``
|
|
-------------------------------
|
|
|
|
Since either all or none of the clauses of a ``try`` construct are ``except*``,
|
|
we considered changing the syntax of the ``try`` instead of all the ``except*``
|
|
clauses. We rejected this because it would be less obvious. The fact that we
|
|
are handling exception groups of ``T`` rather than only naked ``Ts`` should be
|
|
specified in the same place where we state ``T``.
|
|
|
|
Alternative syntax options
|
|
--------------------------
|
|
|
|
Alternatives to the ``except*`` syntax were evaluated in a `discussion on python-dev
|
|
<http://groups.google.com/g/dev-python/c/G3p9_jovyus>`_, and it was suggested to use
|
|
``except group``. Upon careful evaluation this was rejected because the following
|
|
would be ambiguous, as it is currently valid syntax where ``group`` is interpreted
|
|
as a callable. The same is true for any valid identifier.
|
|
|
|
.. code-block::
|
|
|
|
try:
|
|
...
|
|
except group (T1, T2):
|
|
...
|
|
|
|
|
|
Programming Without 'except \*'
|
|
===============================
|
|
|
|
Consider the following simple example of the ``except*`` syntax (pretending
|
|
Trio natively supported this proposal):
|
|
|
|
.. code-block::
|
|
|
|
try:
|
|
async with trio.open_nursery() as nursery:
|
|
# Make two concurrent calls to child()
|
|
nursery.start_soon(child)
|
|
nursery.start_soon(child)
|
|
except* ValueError:
|
|
pass
|
|
|
|
Here is how this code would look in Python 3.9:
|
|
|
|
.. code-block::
|
|
|
|
def handle_ValueError(exc):
|
|
if isinstance(exc, ValueError):
|
|
return None
|
|
else:
|
|
return exc # reraise exc
|
|
|
|
with MultiError.catch(handle_ValueError):
|
|
async with trio.open_nursery() as nursery:
|
|
# Make two concurrent calls to child()
|
|
nursery.start_soon(child)
|
|
nursery.start_soon(child)
|
|
|
|
This example clearly demonstrates how unintuitive and cumbersome handling
|
|
of multiple errors is in current Python. The exception handling logic has
|
|
to be in a separate closure and is fairly low level, requiring the writer to
|
|
have non-trivial understanding of both Python exceptions mechanics and the
|
|
Trio APIs. Instead of using the ``try..except`` block we have to use a
|
|
``with`` block. We need to explicitly reraise exceptions we are not handling.
|
|
Handling more exception types or implementing more complex
|
|
exception handling logic will only further complicate the code to the point
|
|
of it being unreadable.
|
|
|
|
|
|
See Also
|
|
========
|
|
|
|
* An analysis of how exception groups will likely be used in asyncio
|
|
programs: [10]_.
|
|
|
|
* The issue where the ``except*`` concept was first formalized: [12]_.
|
|
|
|
* ``MultiError2`` design document: [3]_.
|
|
|
|
* Reporting Multiple Errors in the Hypothesis library: [7]_.
|
|
|
|
|
|
Acknowledgements
|
|
================
|
|
|
|
We wish to thank Nathaniel J. Smith and the other Trio developers for their
|
|
work on structured concurrency. We borrowed the idea of constructing an
|
|
exception tree whose nodes are exceptions from MultiError, and the ``split()``
|
|
API from the design document for MultiError V2. The discussions on python-dev
|
|
and elsewhere helped us improve upon the first draft of the PEP in multiple
|
|
ways, both the design and the exposition. For this we appreciate all those who
|
|
contributed ideas and asked good questions: Ammar Askar, Matthew Barnett,
|
|
Ran Benita, Emily Bowman, Brandt Bucher, Joao Bueno, Baptiste Carvello,
|
|
Rob Cliffe, Nick Coghlan, Steven D'Aprano, Caleb Donovick, Steve Dower,
|
|
Greg Ewing, Ethan Furman, Pablo Salgado, Jonathan Goble, Joe Gottman, Thomas Grainger,
|
|
Larry Hastings, Zac Hatfield-Dodds, Chris Jerdonek, Jim Jewett, Sven Kunze,
|
|
Łukasz Langa, Glenn Linderman, Paul Moore, Antoine Pitrou, Ivan Pozdeev,
|
|
Patrick Reader, Terry Reedy, Sascha Schlemmer, Barry Scott, Mark Shannon,
|
|
Damian Shaw, Cameron Simpson, Gregory Smith, Paul Sokolovsky, Calvin Spealman,
|
|
Steve Stagg, Victor Stinner, Marco Sulla, Petr Viktorin and Barry Warsaw.
|
|
|
|
|
|
Acceptance
|
|
==========
|
|
|
|
PEP 654 was `accepted by Thomas Wouters on Sep 24, 2021
|
|
<http://discuss.python.org/t/accepting-pep-654-exception-groups-and-except/10813>`_.
|
|
|
|
|
|
References
|
|
==========
|
|
|
|
.. [1] https://docs.python.org/3/library/asyncio-task.html#asyncio.gather
|
|
|
|
.. [2] https://trio.readthedocs.io/en/stable/
|
|
|
|
.. [3] https://github.com/python-trio/trio/issues/611
|
|
|
|
.. [4] https://bugs.python.org/issue29980
|
|
|
|
.. [5] https://docs.python.org/3/library/atexit.html#atexit.register
|
|
|
|
.. [6] https://github.com/pytest-dev/pytest/issues/8217
|
|
|
|
.. [7] https://hypothesis.readthedocs.io/en/latest/settings.html#hypothesis.settings.report_multiple_bugs
|
|
|
|
.. [8] https://bugs.python.org/issue40857
|
|
|
|
.. [9] https://trio.readthedocs.io/en/stable/reference-core.html#trio.MultiError
|
|
|
|
.. [10] https://github.com/python/exceptiongroups/issues/3#issuecomment-716203284
|
|
|
|
.. [11] https://github.com/iritkatriel/cpython/tree/exceptionGroup-stage5
|
|
|
|
.. [12] https://github.com/python/exceptiongroups/issues/4
|
|
|
|
.. [13] https://trio.readthedocs.io/en/stable/reference-core.html#nurseries-and-spawning
|
|
|
|
Copyright
|
|
=========
|
|
|
|
This document is placed in the public domain or under the
|
|
CC0-1.0-Universal license, whichever is more permissive.
|
|
|
|
|
|
..
|
|
Local Variables:
|
|
mode: indented-text
|
|
indent-tabs-mode: nil
|
|
sentence-end-double-space: t
|
|
fill-column: 70
|
|
coding: utf-8
|
|
End:
|