python-peps/pep-0490.txt

PEP: 490
Title: Chain exceptions at C level
Version: $Revision$
Last-Modified: $Date$
Author: Victor Stinner <victor.stinner@gmail.com>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 25-March-2015
Python-Version: 3.5


Abstract
========

Chain exceptions at C level, as already done at Python level.


Rationale
=========

Python 3 introduced a new killer feature: exceptions are chained by default,
PEP 3134.

Example::

    try:
        raise TypeError("err1")
    except TypeError:
        raise ValueError("err2")

Output::

    Traceback (most recent call last):
      File "test.py", line 2, in <module>
        raise TypeError("err1")
    TypeError: err1

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "test.py", line 4, in <module>
        raise ValueError("err2")
    ValueError: err2

Exceptions are chained by default in Python code, but not in extensions written
in C. This PEP proposes to also chain exceptions automatically at C level to
stay consistent.


Proposal
========

Modify PyErr_*() functions to chain exceptions
----------------------------------------------

Modify C functions raising exceptions of the Python C API to automatically
chain exceptions.


Modify functions to not chain exceptions
----------------------------------------

To remove the previous exception, PyErr_Clear() can be called before
PyErr_SetString() (or other functions raising exceptions).

Keeping the original exception is not always interesting.

Example of Python code (``os._Environ.__getitem__``) suppressing the context::

    try:
        value = self._data[self.encodekey(key)]
    except KeyError:
        # raise KeyError with the original key value
        raise KeyError(key) from None

Example of an exception chain raised by the struct module with exceptions
chained by default::

    $ python3.5 -c 'import struct; struct.pack("b", 2**100)'
    OverflowError: Python int too large to convert to C long

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "<string>", line 1, in <module>
    struct.error: argument out of range

The ``OverflowError`` exception is not very useful, it can be hidden.


Modify functions to chain exceptions
------------------------------------

Some functions save and then restore the current exception. If a new exception
is raised, the exception is displayed or ignored depending on the function.

Maybe some of these functions should be modified to chain exceptions instead?

Examples of function displaying the new exception:

* atexit_callfuncs(): display exceptions with PyErr_Display() and return the
  latest exception
* sock_dealloc(): log warning exception with PyErr_WriteUnraisable()
* slot_tp_del(): display exception with PyErr_WriteUnraisable()
* _PyGen_Finalize(): display gen_close() exception with PyErr_WriteUnraisable()
* slot_tp_finalize(): display exception raised by __del__() with
  PyErr_WriteUnraisable()
* PyErr_GivenExceptionMatches(): display exception raised by PyType_IsSubtype()
  with PyErr_WriteUnraisable()

Examples of function ignoring the new exception(s):

* ptrace_enter_call(): ignore exception
* subprocess_fork_exec(): ignore exception raised by enable_gc()
* t_bootstrap() of the _thread module: ignore raised exception when trying to
  display the bootstrap function to sys.stderr
* PyDict_GetItem(), _PyDict_GetItem_KnownHash(): ignore exception raiesd when
  looking for a key
* _PyErr_TrySetFromCause(): ignore ecxeption
* PyFrame_LocalsToFast(): ignore exception raised by dict_to_map()
* _PyObject_Dump(): ignore exception. _PyObject_Dump() is used to debug, to
  inspect a running process, it should not modify the Python state.
* Py_ReprLeave(): ignore exception "because there is no way to report them"
* type_dealloc(): ignore exception raised by remove_all_subclasses()
* PyObject_ClearWeakRefs(): ignore exceptions?
* call_exc_trace(), call_trace_protected(): ignore exception
* remove_importlib_frames(): ignore exception
* do_mktuple(), helped used by Py_BuildValue() for example: ignore exception?
* flush_io(): ignore exception
* sys_write(), sys_format(): ignore exception
* _PyTraceback_Add(): ignore exception
* PyTraceBack_Print(): ignore exception


Python C API
------------

Include/pyerror.h declares functions related to exceptions.

Functions raising exceptions:

* PyErr_SetNone(exc_type)
* PyErr_SetObject(exc_type, exc_value)
* PyErr_SetString(exc_type, message)
* PyErr_Format(exc, format, ...)

Helpers to raise specific exceptions:

* PyErr_BadArgument()
* PyErr_BadInternalCall()
* PyErr_NoMemory()
* PyErr_SetFromErrno(exc)
* PyErr_SetFromWindowsErr(err)
* PyErr_SetImportError(message, name, path)
* _PyErr_SetKeyError(key)
* _PyErr_TrySetFromCause(prefix_format, ...)

Manage the current exception:

* PyErr_Clear(): clear the current exception, like "except: pass"
* PyErr_Fetch(exc_type, exc_value, exc_tb)
* PyErr_Restore(exc_type, exc_value, exc_tb)
* PyErr_GetExcInfo(exc_type, exc_value, exc_tb)
* PyErr_SetExcInfo(exc_type, exc_value, exc_tb)

Others function to handle exceptions:

* PyErr_ExceptionMatches(exc): check to implement "except exc: ..."
* PyErr_GivenExceptionMatches(exc1, exc2)
* PyErr_NormalizeException(exc_type, exc_value, exc_tb)
* _PyErr_ChainExceptions(exc_type, exc_value, exc_tb)


Backward compatibility
======================

A side effect of chaining exceptions is that exceptions store traceback
objects. Traceback objects store frame objects which store local variables.
Local variables are kept alive by exceptions. A common issue is a reference
cycle between local variables and exceptions. The cycle only impacts
applications storing exceptions.

The reference cycle can now be fixed with the new traceback.TracebackException
object introduced in Python 3.5 which store informations required to format a
full textual traceback without storing local variables.

The asyncio is impacted by the reference cycle issue, but this module is also
maintained outside Python standard library to release a version for Python 3.3.
traceback.TracebackException will probably be backward in a private asyncio
module to fix the issue.


Alternatives
============

No change
---------

Python 3.5 introduces a new private ``_PyErr_ChainExceptions()`` function which
is enough to chain manually exceptions.


New helpers to chain exceptions
-------------------------------

Functions like ``PyErr_SetString()`` don't chain automatically exceptions. To
make usage of ``_PyErr_ChainExceptions()`` easier, new functions are added:

* PyErr_SetStringChain(exc_type, message)
* PyErr_FormatChaine(exc_type, format, ...)
* PyErr_SetNoneChain(exc_type)
* PyErr_SetObjectChain(exc_type, exc_value)

Helper functions like _PyErr_SetKeyError(key) or PyErr_SetImportError(message,
name, path) don't chain exceptions. The generic
_PyErr_ChainExceptions(exc_type, exc_value, exc_tb) should be used.


Appendix
========

Python Issues
-------------

Chain exceptions:

* `Issue #23763: Chain exceptions in C
  <http://bugs.python.org/issue23763>`_
* `Issue #23696: zipimport: chain ImportError to OSError
  <http://bugs.python.org/issue23696>`_
* `Issue #21715: Chaining exceptions at C level
  <http://bugs.python.org/issue21715>`_: added _PyErr_ChainExceptions()
* `Issue #18488: sqlite: finalize() method of user function may be called with
  an exception set if a call to step() method failed
  <http://bugs.python.org/issue18488>`_
* `Issue #23781: Add private _PyErr_ReplaceException() in 2.7
  <http://bugs.python.org/issue23781>`_

Changes preventing loosing exceptions:

* `Issue #23571: Raise SystemError if a function returns a result with an
  exception set <http://bugs.python.org/issue23571>`_
* `Issue #18408: Fixes crashes found by pyfailmalloc
  <http://bugs.python.org/issue18408>`_


PEPs
----

* `PEP 3134 -- Exception Chaining and Embedded Tracebacks
  <https://www.python.org/dev/peps/pep-3134/>`_ (Python 3.0):
  new ``__context__`` and ``__cause__`` attributes for exceptions
* `PEP 415 - Implement context suppression with exception attributes
  <https://www.python.org/dev/peps/pep-0415/>`_ (Python 3.3):
  ``raise exc from None``
* `PEP 409 - Suppressing exception context
  <https://www.python.org/dev/peps/pep-0409/>`_ (superseded by the PEP 415)


Copyright
=========

This document has been placed in the public domain.



..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   coding: utf-8
   End: