PEP: 743 Title: Add Py_COMPAT_API_VERSION to the Python C API Author: Victor Stinner Status: Draft Type: Standards Track Created: 11-Mar-2024 Python-Version: 3.13 .. highlight:: c Abstract ======== Add ``Py_COMPAT_API_VERSION`` and ``Py_COMPAT_API_VERSION_MAX`` macros to opt-in for planned incompatible C API changes in a C extension. Maintainers can decide when they make their C extension compatible and also decide which future Python version they want to be compatible with. Rationale ========= Python releases enforce C API changes ------------------------------------- Every Python 3.x release has a long list of C API changes, including incompatible changes. C extensions have to be updated to work on the newly released Python. Some incompatible changes are driven by new features: they cannot be avoided, unless we decide to not add these features. Other reasons: * Remove deprecated API (see :pep:`387`). * Ease the implementation of another change. * Change or remove error-prone API. Currently, there is no middle ground between "not change the C API" and "incompatible C API changes impact everybody". Either a C extension is updated or the new Python version cannot be used. Such all-or-nothing deal does not satisfy C extension maintainers nor C extensions users. Limited C API ------------- The limited C API is versioned: the ``Py_LIMITED_API`` macro can be set to a Python version to select which API is available. On the Python side, it allows introducing incompatible changes at a specific ``Py_LIMITED_API`` version. For example, if ``Py_LIMITED_API`` is set to Python 3.11 or newer, the ```` is no longer included by ``Python.h``, whereas C extensions targeting Python 3.10 are not affected. The difference here is that upgrading Python does not change if ```` is included or not, but updating ``Py_LIMITED_API`` does. Updating ``Py_LIMITED_API`` is an deliberate action made by the C extension maintainer. It gives more freedom to decide **when** the maintainer is ready to deal with the latest batch of incompatible changes. A similar version can be used with the regular (non-limited) C API. Deprecation and compiler warnings --------------------------------- Deprecated functions are marked with ``Py_DEPRECATED()``. Using a deprecated function emits a compiler warning. The problem is that ``pip`` and ``build`` tools hide compiler logs by default, unless a build fails. Moreover, it's easy to miss a single warning in the middle of hundred lines of logs. Schedule changes ---------------- Currently, there is no way to schedule a C API change: announce it but also provide a way to maintainers to test their C extensions with the change. Either a change is not made, or everybody must update their code if they want to update Python. Specification ============= New macros ---------- Add new ``Py_COMPAT_API_VERSION`` and ``Py_COMPAT_API_VERSION_MAX`` macros. They can be set to test if a C extension is prepared for future C API changes: compatible with future Python versions. The ``Py_COMPAT_API_VERSION`` macro can be set to a specific Python version. For example, ``Py_COMPAT_API_VERSION=0x030e0000`` tests C API changes scheduled in Python 3.14. If the ``Py_COMPAT_API_VERSION`` macro is set to ``Py_COMPAT_API_VERSION_MAX``, all scheduled C API changes are tested at once. If the ``Py_COMPAT_API_VERSION`` macro is not set, it is to ``PY_VERSION_HEX`` by default. The ``Py_COMPAT_API_VERSION`` macro can be set in a single C file or for a whole project in compiler flags. The macro does not affected other projects or Python itself. Example in Python ----------------- For example, the ``PyImport_ImportModuleNoBlock()`` function is deprecated in Python 3.13 and scheduled for removal in Python 3.15. The function can be declared in the Python C API with the following declaration: .. code-block:: c #if Py_COMPAT_API_VERSION < 0x030f0000 Py_DEPRECATED(3.13) PyAPI_FUNC(PyObject *) PyImport_ImportModuleNoBlock( const char *name /* UTF-8 encoded string */ ); #endif If ``if Py_COMPAT_API_VERSION`` is equal to or greater than Python 3.15 (``0x030f0000``), the ``PyImport_ImportModuleNoBlock()`` function is not declared, and so using it fails with a build error. Goals ----- * Reduce the number of C API changes affecting C extensions when updating Python. * When testing C extensions (for example, optional CI test), ``Py_COMPAT_API_VERSION`` can be set to ``Py_COMPAT_API_VERSION_MAX`` to detect future incompatibilities. For mandatory tests, it is recommended to set ``Py_COMPAT_API_VERSION`` to a specific Python version. * For core developers, make sure that the C API can still evolve without being afraid of breaking an unknown number of C extensions. Non-goals --------- * Freeze the API forever: this is not the stable ABI. For example, deprecated functions will continue to be removed on a regular basis. * C extensions maintainers not using ``Py_COMPAT_API_VERSION`` will still be affected by C API changes when updating Python. * Provide a stable ABI: the macro only impacts the regular (non-limited) API. * Silver bullet solving all C API issues. Examples of ``Py_COMPAT_API_VERSION`` usages ============================================ * Remove deprecated functions. * Remove deprecated structure members, such as ``PyBytesObject.ob_shash``. * Remove a standard ``#include``, such as ``#include ``, from ````. * Change the behavior of a function or a macro. For example, calling ``PyObject_SetAttr(obj, name, NULL)`` can fail, to enforce the usage of the ``PyObject_DelAttr()`` function instead to delete an attribute. Implementation ============== * `Issue gh-116587 `_ * PR: `Add Py_COMPAT_API_VERSION and Py_COMPAT_API_VERSION_MAX macros `_ Backwards Compatibility ======================= There is no impact on backward compatibility. Adding ``Py_COMPAT_API_VERSION`` and ``Py_COMPAT_API_VERSION_MAX`` macros has no effect on backward compatibility. Only developers setting the ``Py_COMPAT_API_VERSION`` macro in their project will be impacted by effects of this macro which is the expected behavior. Discussions =========== * C API Evolutions: `Macro to hide deprecated functions `_ (October 2023) * C API Problems: `Opt-in macro for a new clean API? Subset of functions with no known issues `_ (June 2023) Prior Art ========= * ``Py_LIMITED_API`` macro of :pep:`384` "Defining a Stable ABI". * Rejected :pep:`606` "Python Compatibility Version" which has a global scope. Copyright ========= This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive.