python-peps/pep-0620.rst

PEP: 620
Title: Hide implementation details from the C API
Author: Victor Stinner <vstinner@python.org>,
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 19-June-2020
Python-Version: 3.10

Abstract
========

Introduce C API incompatible changes.

Evolving CPython internals and adding new features is made simpler by
reducing backward compatibility issues.

It becomes possible to experiment in CPython more advanced optimizations
than just micro-optimizations.

Define a process to reduce the number of broken C extensions.

The implementation of this PEP is expected to be done slowly over
multiple Python versions. It already started in Python 3.7 and most
changes are already completed. The `Process to introduce incompatible
changes to the C API`_ didactes the rythm.


Motivation
==========

The C API blocks CPython evolutions
-----------------------------------

Adding or removing members of C structures is causing multiple
implementation issues.

Adding a new member breaks the stable ABI (PEP 384), especially for
types declared statically with::

    static PyTypeObject MyType = {...};

In Python 3.4, the PEP 442 "Safe object finalization" added
``tp_finalize`` member at the end of the ``PyTypeObject`` structure. For
ABI backward compatibility, a new ``Py_TPFLAGS_HAVE_FINALIZE`` type flag
was required to announce if the type structure contains the
``tp_finalize`` member. The flag was removed in Python 3.8 (`bpo-32388
<https://bugs.python.org/issue32388>`_).

The ``PyTypeObject.tp_print`` member, deprecated since Python 3.0
released in 2009, has been removed in the Python 3.8 development cycle.
But the change broke too many C extensions and had to be reverted before
3.8 final release. Finally, the member was removed again in Python 3.9.

Modifying other structures, like ``PyListObject``, cannot be even
considered right now, since it would break way too many C extensions.

The ``PyTypeObject`` structure is the one which evolved the most, simply
because there was no other way to evolve CPython than modifying it.

Same CPython design since 1990: structures and reference counting
-----------------------------------------------------------------

When the CPython project was created, it was written with one principle:
keep the implementation simple enough so it can be maintained by a
single developer. CPython complexity grew a lot and many
micro-optimizations have been implemented, but CPython core design has
not changed.

Members of ``PyObject`` and ``PyTupleObject`` structures have not
changed since the "Initial revision" commit (1990)::

    #define OB_HEAD \
        unsigned int ob_refcnt; \
        struct _typeobject *ob_type;

    typedef struct _object {
        OB_HEAD
    } object;

    typedef struct {
           OB_VARHEAD
           object *ob_item[1];
    } tupleobject;

Only names changed: ``object`` was renamed to ``PyObject`` and
``tupleobject`` was renamed to ``PyTupleObject``.

CPython still tracks Python objects lifetime using reference counting
internally and for third party C extensions (through the Python C API).

All Python objects must be allocated on the heap and cannot be moved.

Why is PyPy more efficient than CPython?
----------------------------------------

The PyPy project is a Python implementation which is 4.2x faster than
CPython in average. PyPy developers chose to not fork CPython, but start
from scratch to have more freedom in terms of optimization choices.

PyPy does not use reference counting, but a tracing garbage collector
which moves objects. Objects can be allocated on the stack, rather than
always having to be allocated on the heap.

Objects layouts are designed with performance in mind. For example, a
list strategy stores integers directly as integers, rather than objects.

Moreover, PyPy also has a JIT compiler which emits fast code thanks to
the efficient PyPy design.

PyPy bottleneck: the Python C API
---------------------------------

While PyPy is way more efficient than CPython to run pure Python code,
it is as efficient or slower than CPython to run C extensions.

Since the C API requires ``PyObject*`` and allows to access directly
structure members, PyPy has to associate a CPython object to PyPy
objects and maintain both consistent. Converting a PyPy object to a
CPython object is inefficient. Moreover, reference counting also has to
be implemented on top on PyPy tracing garbage collector.

These conversions are required because the Python C API is too close to
the CPython implementation: there is no high-level abstraction.
example, structures members are part of the public C API and nothing
prevents a C extension to get or set directly
``PyTupleObject.ob_item[0]`` (the first item of a tuple).

See `Inside cpyext: Why emulating CPython C API is so Hard
<https://morepypy.blogspot.com/2018/09/inside-cpyext-why-emulating-cpython-c.html>`_
(Sept 2018) by Antonio Cuni for more details.


Rationale
=========

Hide implementation details
---------------------------

Hiding implementation details from the C API has multiple advantages:

* It becomes possible to experiment in CPython more advanced
  optimizations than just micro-optimizations.
* Adding new features in CPython becomes easier.
* PyPy should be able to avoid conversions to CPython objects in more
  cases: keep efficient PyPy objects.
* It becomes easier to implement the C API for a new Python
  implementation.
* More C extensions will be compatible with Python implementations other
  than CPython.

Relationship with the limited C API
-----------------------------------

The PEP 384 "Defining a Stable ABI" is in Python 3.4. It introduces the
"limited C API": a subset of the C API. When the limited C API is used,
it becomes possible to build a C extensions only once and uses it on
multiple Python versions: that's the stable ABI.

The main limitation of the PEP 384 is that C extensions have to opt-in
for the limited C API. Only very few projects made this choice,
usually to ease distribution of binaries, especially on Windows.

This PEP moves the C API towards the limited C API.

Ideally, the C API will become the limited C API and all C extensions
will use the stable ABI, but this is out of this PEP scope.


Specification
=============

Summary
-------

* (**Completed**) Reorganize the C API header files: create ``Include/cpython/`` and
  ``Include/internal/`` subdirectories.
* (**Completed**) Move private functions exposing implementation details to the internal
  C API.
* (**Completed**) Convert macros to static inline functions.
* (**Completed**) Add new functions ``Py_SET_TYPE()``, ``Py_SET_REFCNT()`` and
  ``Py_SET_SIZE()``. The ``Py_TYPE()``, ``Py_REFCNT()`` and
  ``Py_SIZE()`` macros become functions which cannot be used as l-value.
* (**Completed**) New C API functions must not return borrowed
  references.
* (**In Progress**) Provide ``pythoncapi_compat.h`` header file.
* (**In Progress**) Make structures opaque, add getter and setter
  functions.
* (**Not Started**) Deprecate ``PySequence_Fast_ITEMS()``.
* (**Not Started**) Convert ``PyTuple_GET_ITEM()`` and
  ``PyList_GET_ITEM()`` macros to static inline functions.

Reorganize the C API header files
---------------------------------

The first consumer of the C API was Python itself. There is no clear
separation between APIs which must not be used outside Python, and API
which are public on purpose.

Header files must be reorganized in 3 API:

* ``Include/`` directory is the limited C API: no implementation
  details, structures are opaque. C extensions using it get a stable
  ABI.
* ``Include/cpython/`` directory is the CPython C API: less "portable"
  API, depends more on the Python version, expose some implementation
  details, few incompatible changes can happen.
* ``Internal/internal/`` directory is the internal C API: implementation
  details, incompatible changes are likely at each Python release.

The creation of the ``Include/cpython/`` directory is fully backward
compatible. ``Include/cpython/`` header files cannot be included
directly and are included automatically by ``Include/`` header files
when the ``Py_LIMITED_API`` macro is not defined.

The internal C API is installed and can be used for specific usage like
debuggers and profilers which must access structures members without
executing code. C extensions using the internal C API are tightly
coupled to a Python version and must be recompiled at each Python
version.

**STATUS**: Completed (in Python 3.8)

The reorganization of header files started in Python 3.7 and was
completed in Python 3.8:

* `bpo-35134 <https://bugs.python.org/issue35134>`_: Add a new
  Include/cpython/ subdirectory for the "CPython API" with
  implementation details.
* `bpo-35081 <https://bugs.python.org/issue35081>`_: Move internal
  headers to ``Include/internal/``

Move private functions to the internal C API
--------------------------------------------

Private functions which exposes implementation details must be moved to
the internal C API.

If a C extension relies on a CPython private function which exposes
CPython implementation details, other Python implementations have to
re-implement this private function to support this C extension.

**STATUS**: Completed (in Python 3.9)

In Python 3.9, 4 private functions have been moved to the internal C API
and are no longer exported:

* ``_PyDebug_PrintTotalRefs()``
* ``_Py_AddToAllObjects()``
* ``_Py_PrintReferenceAddresses()``
* ``_Py_PrintReferences()``

In Python 3.9, the public "clear free list" functions have been renamed
to private functions and moved to the internal C API:

* ``PyAsyncGen_ClearFreeLists()()``
* ``PyContext_ClearFreeList()()``
* ``PyDict_ClearFreeList()()``
* ``PyFloat_ClearFreeList()()``
* ``PyFrame_ClearFreeList()()``
* ``PyList_ClearFreeList()()``
* ``PyTuple_ClearFreeList()()``

And the following functions have been simply removed:

* ``PyMethod_ClearFreeList()`` and ``PyCFunction_ClearFreeList()``:
  the free lists of bound method objects have been removed (in Python 3.9).
* ``PySet_ClearFreeList()``: the set free list has been removed in
  Python 3.4.
* ``PyUnicode_ClearFreeList()``: the Unicode free list has been removed
  in Python 3.3.


Convert macros to static inline functions
-----------------------------------------

Converting macros to static inline functions have multiple advantages:

* Functions have well defined parameter types and return type.
* Functions can use variables with a well defined scope (the function).
* Debugger can be put breakpoints on functions and profilers can display
  the function name in the call stacks. In most cases, it works even
  when a static inline function is inlined.
* Functions don't have `macros pitfalls
  <https://gcc.gnu.org/onlinedocs/cpp/Macro-Pitfalls.html>`_.

Converting macros to static inline functions should only impact very few
C extensions which use macros in unusual ways.

For backward compatibility, functions must continue to accept any type,
not only ``PyObject*``, to avoid compiler warnings, since most macros
cast their parameters to ``PyObject*``.

Python 3.6 requires C compilers to support static inline functions: the
PEP 7 requires a subset of C99.

**STATUS**: Completed (in Python 3.8)

Macros converted to static inline functions in Python 3.8:

* ``Py_INCREF()``, ``Py_DECREF()``
* ``Py_XINCREF()``, ``Py_XDECREF()``
* ``PyObject_INIT()``, ``PyObject_INIT_VAR()``
* ``_PyObject_GC_TRACK()``, ``_PyObject_GC_UNTRACK()``, ``_Py_Dealloc()``

Macros converted to regular functions in Python 3.9:

* ``Py_EnterRecursiveCall()``, ``Py_LeaveRecursiveCall()``
* ``PyObject_INIT()``, ``PyObject_INIT_VAR()``

In Python 3.9, the trashcan macros are now calling functions which hide
implementation details, rather than accessing directly members of the
``PyThreadState`` structure.

Make structures opaque
----------------------

All structures of the C API should become opaque: C extensions must
use getter or setter functions to get or set structure members. For
example, ``tuple->ob_item[0]`` must be replaced with
``PyTuple_GET_ITEM(tuple, 0)``.

To be able to move away from reference counting, ``PyObject`` must
become opaque. Currently, the reference counter ``PyObject.ob_refcnt``
is exposed in the C API. All structures must become opaque, since they
"inherit" from PyObject. For, ``PyFloatObject`` inherits from
``PyObject``::

    typedef struct {
        PyObject ob_base;
        double ob_fval;
    } PyFloatObject;

Making ``PyObject`` fully opaque requires to convert ``Py_INCREF()`` and
``Py_DECREF()`` macros to function calls. This change has an impact on
performance. It is likely to be one of the very last change when making
structures opaque.

Making ``PyTypeObject`` structure opaque breaks C extensions declaring
types statically (e.g. ``static PyTypeObject MyType = {...};``). C
extensions must use ``PyType_FromSpec()`` to allocate types on the heap
instead. Using heap types have other advantages like being compatible
with subinterpreters. Combined with PEP 489 "Multi-phase extension
module initialization", it makes a C extension behavior closer to a
Python module, like allowing to create more than one module instance.

Making ``PyThreadState`` structure opaque requires to add getter and
setter functions for members used by C extensions.

**STATUS**: In Progress (started in Python 3.8)

The ``PyInterpreterState`` structure was made opaque in Python 3.8
(`bpo-35886 <https://bugs.python.org/issue35886>`_) and the
``PyGC_Head`` structure (`bpo-40241
<https://bugs.python.org/issue40241>`_) was made opaque in Python 3.9.

Issues tracking the work to prepare the C API to make following
structures opaque:

* ``PyObject``: `bpo-39573 <https://bugs.python.org/issue39573>`_
* ``PyTypeObject``: `bpo-40170 <https://bugs.python.org/issue40170>`_
* ``PyFrameObject``: `bpo-40421 <https://bugs.python.org/issue40421>`_.
  Python 3.9 adds ``PyFrame_GetCode()`` and ``PyFrame_GetBack()``
  getter functions, and moves ``PyFrame_GetLineNumber`` to the limited C
  API.
* ``PyThreadState``: `bpo-39947 <https://bugs.python.org/issue39947>`_.
  Python 3.9 adds 3 getter functions: ``PyThreadState_GetFrame()``,
  ``PyThreadState_GetID()`` and ``PyThreadState_GetInterpreter()``.

Disallow using Py_TYPE() as l-value
-----------------------------------

The ``Py_TYPE()`` function gets an object type, its ``PyObject.ob_type``
member. It is implemented as a macro which can be used as an l-value to
set the type: ``Py_TYPE(obj) = new_type``. This code relies on the
assumption that ``PyObject.ob_type`` can be modified directly. It
prevents to make the ``PyObject`` structure opaque.

New setter functions ``Py_SET_TYPE()``, ``Py_SET_REFCNT()`` and
``Py_SET_SIZE()`` are added and must be used instead.

The ``Py_TYPE()``, ``Py_REFCNT()`` and ``Py_SIZE()`` macros must be
converted to static inline functions which can not be used as l-value.

For example, the ``Py_TYPE()`` macro::

    #define Py_TYPE(ob)             (((PyObject*)(ob))->ob_type)

becomes::

    #define _PyObject_CAST_CONST(op) ((const PyObject*)(op))

    static inline PyTypeObject* _Py_TYPE(const PyObject *ob) {
        return ob->ob_type;
    }

    #define Py_TYPE(ob) _Py_TYPE(_PyObject_CAST_CONST(ob))

**STATUS**: Completed (in Python 3.10)

New functions ``Py_SET_TYPE()``, ``Py_SET_REFCNT()`` and
``Py_SET_SIZE()`` were added to Python 3.9.

In Python 3.10, ``Py_TYPE()``, ``Py_REFCNT()`` and ``Py_SIZE()`` can no
longer be used as l-value and the new setter functions must be used
instead.

New C API functions must not return borrowed references
-------------------------------------------------------

When a function returns a borrowed reference, Python cannot track when
the caller stops using this reference.

For example, if the Python ``list`` type is specialized for small
integers, store directly "raw" numbers rather than Python objects,
``PyList_GetItem()`` has to create a temporary Python object. The
problem is to decide when it is safe to delete the temporary object.

The general guidelines is to avoid returning borrowed references for new
C API functions.

No function returning borrowed functions is scheduled for removal by
this PEP.

**STATUS**: Completed (in Python 3.9)

In Python 3.9, new C API functions returning Python objects only return
strong references:

* ``PyFrame_GetBack()``
* ``PyFrame_GetCode()``
* ``PyObject_CallNoArgs()``
* ``PyObject_CallOneArg()``
* ``PyThreadState_GetFrame()``

Avoid functions returning PyObject**
------------------------------------

The ``PySequence_Fast_ITEMS()`` function gives a direct access to an
array of ``PyObject*`` objects. The function is deprecated in favor of
``PyTuple_GetItem()`` and ``PyList_GetItem()``.

``PyTuple_GET_ITEM()`` can be abused to access directly the
``PyTupleObject.ob_item`` member::

    PyObject **items = &PyTuple_GET_ITEM(0);

The ``PyTuple_GET_ITEM()`` and ``PyList_GET_ITEM()`` macros are
converted to static inline functions to disallow that.

**STATUS**: Not Started

New pythoncapi_compat.h header file
-----------------------------------

Making structures opaque require to add getter and setter functions. C
extensions must be modified to use these new functions. The practical
issue is how to handle backward compatibility.

In Python 3.10, it is no longer possible to use ``Py_TYPE()`` as an
l-value. The new ``Py_SET_TYPE()`` function must be used instead.
Example::

    #if PY_VERSION_HEX >= 0x030900A4
        Py_SET_TYPE(&MyType, &PyType_Type);
    #else
        Py_TYPE(&MyType) = &PyType_Type;
    #endif

This code may ring a bell to developers who ported their Python code
base from Python 2 to Python 3.

Python will distribute a new ``pythoncapi_compat.h`` header file which
provides new C API functions to old Python versions. Example::

    #if PY_VERSION_HEX < 0x030900A4
    static inline void
    _Py_SET_TYPE(PyObject *ob, PyTypeObject *type)
    {
        ob->ob_type = type;
    }
    #define Py_SET_TYPE(ob, type) _Py_SET_TYPE((PyObject*)(ob), type)
    #endif  // PY_VERSION_HEX < 0x030900A4

Using this header file, ``Py_SET_TYPE()`` can be used on old Python
versions as well.

Developers can decide to copy this file in their project, or even to
only copy/paste the few functions needed by the C extension.

**STATUS**: In Progress (implemented but not shiped by CPython yet)

The ``pythoncapi_compat.h`` header file is currently developer at:
https://github.com/pythoncapi/pythoncapi_compat

Process to introduce incompatible changes to the C API
======================================================

* Estimate how many popular C extensions are affected by the
  incompatible change.
* Coordinate with maintainers of broken C extensions to prepare their
  code for the future incompatible change.
* Introduce the incompatible changes in Python. The documentation must
  explain how to port existing code. It is recommended to merge such
  changes at the beginning of a development cycle to have more time to
  test.
* Changes which are the most likely to break a large number of C
  extensions should be announced on the capi-sig mailing list to notify
  C extensions maintainers to prepare their project for the next Python.
* If the change breaks too many projects, reverting the change should be
  discussed, taking in account the number of broken packages, their
  importance in the Python commmunity, and the importance of the change.

The coordination usually means reporting issues to the projects, or even
propose changes. It does not require waiting for a new release including
fixes for every broken project.

Future incompatible changes can be announced by deprecating a function
in the documentation and by annotating the function with
``Py_DEPRECATED()``. Making a structure opaque and preventing the usage
of a macro as l-value cannot be deprecated with ``Py_DEPRECATED()``.

The important part is coordination and balance the tradeoff between
CPython evolutions and backward compatibility. For example, breaking a
random old, obscure and unmaintained C extension on PyPI is less severe
than breaking numpy.

If a change is reverted, we move back to the coordination step to better
prepare the change. Once more C extensions are ready, the incompatible
change can be reconsidered.


Copyright
=========

This document has been placed in the public domain.