PEP: 743 Title: Add Py_COMPAT_API_VERSION to the Python C API Author: Victor Stinner , Petr Viktorin , PEP-Delegate: C API Working Group Discussions-To: https://discuss.python.org/t/pep-743-add-py-compat-api-version-to-the-python-c-api/48243 Status: Draft Type: Standards Track Created: 11-Mar-2024 Python-Version: 3.14 .. highlight:: c Abstract ======== Add ``Py_COMPAT_API_VERSION`` C macro that hides some deprecated and soft-deprecated symbols, allowing users to opt out of using API with known issues that other API solves. The macro is versioned, allowing users to update (or not) on their own pace. Also, add namespaced alternatives for API without the ``Py_`` prefix, and soft-deprecate the original names. Motivation ========== Some of Python's C API has flaws that are only obvious in hindsight. If an API prevents adding features or optimizations, or presents a serious security risk or maintenance burden, we can deprecate and remove it as described in :pep:`387`. However, this leaves us with some API that has “sharp edges” -- it works fine for its current users, but should be avoided in new code. For example: - API that cannot signal an exception, so failures are either ignored or exit the process with a fatal error. For example ``PyObject_HasAttr``. - API that is not thread-safe, for example by borrowing references from mutable objects, or exposing unfinished mutable objects. For example ``PyDict_GetItemWithError``. - API with names that don't use the ``Py``/``_Py`` prefix, and so can clash with other code. For example: ``setter``. It is important to note that despite such flaws, it's usually possible to use the API correctly. For example, in a single-threaded environment, thread safety is not an issue. We do not want to break working code, even if it uses API that would be wrong in some -- or even *most* -- other contexts. On the other hand, we want to steer users away from such “undesirable” API in *new* code, especially if a safer alternative exists. Adding the ``Py`` prefix ------------------------ Some names defined in CPython headers is not namespaced: it that lacks the ``Py`` prefix (or a variant: ``_Py``, and alternative capitalizations). For example, we declare a function type named simply ``setter``. While such names are not exported in the ABI (as checked by ``make smelly``), they can clash with user code and, more importantly, with libraries linked to third-party extensions. While it would be possible to provide namespaced aliases and (soft-)deprecate these names, the only way to make them not clash with third-party code is to not define them in Python headers at all. Rationale ========= We want to allow an easy way for users to avoid “undesirable” API if they choose to do so. It might be be sufficient to leave this to third-party linters. For that we'd need a good way to expose a list of (soft-)deprecated API to such linters. While adding that, we can -- rather easily -- do the linter's job directly in CPython headers, avoiding the neel for an extra tool. Unlike Python, C makes it rather easy to limit available API -- for a whole project or for each individual source file -- by having users define an “opt-in” macro. We already do something similar with ``Py_LIMITED_API``, which limits the available API to a subset that compiles to stable ABI. (In hindsight, we should have used a different macro name for that particular kind of limiting, but it's too late to change that now.) To prevent working code from breaking as we identify more “undesirable” API and add safer alternatives to it, the opt-in macro should be *versioned*. Users can choose a version they need based on their compatibility requirements, and update it at their own pace. To be clear, this mechanism is *not* a replacement for deprecation. Deprecation is for API that prevents new features or optimizations, or presents a security risk or maintenance burden. This mechanism, on the other hand, is meant for cases where “we found a slightly better way of doing things” -- perhaps one that's harder to misuse, or just has a less misleading name. (On a lighter note: many people configure a code quality checker to shout at them about the number of blank lines between functions. Let's help them identify more substantial “code smells”!) The proposed macro does not *change* any API definitions; it only *hides* them. So, if code compiles with the macro, it'll also compile without it, with identical behaviour. This has implications for core devs: to deal with undesirable behaviour, we'll need to introduce new, better API, and *then* discourage the old one. In turn, this implies that we should look at an individual API and fix all its known issues at once, rather than do codebase-wide sweeps for a single kind of issue, so that we avoid multiple renames of the same function. Adding the ``Py`` prefix ------------------------ An opt-in macro allows us to omit definitions that could clash with third-party libraries. Specification ============= We introduce a ``Py_COMPAT_API_VERSION`` macro. If this macro is defined before ``#include ``, some API definitions -- as described below -- will be omitted from the Python header files. The macro only omits complete top-level definitions exposed from ````. Other things (the ABI, structure definitions, macro expansions, static inline function bodies, etc.) are not affected. The C API working group (:pep:`731`) has authority over the set of omitted definitions. The set of omitted definitions will be tied to a particular feature release of CPython, and is finalized in each 3.x.0 Beta 1 release. In rare cases, entries can be removed (i.e. made available for use) at any time. The macro should be defined to a version in the format used by ``PY_VERSION_HEX``, with the “micro”, “release” and “serial” fields set to zero. For example, to omit API deemed undesirable in 3.14.0b1, users should define ``Py_COMPAT_API_VERSION`` to ``0x030e0000``. Requirements for omitted API ---------------------------- An API that is omitted with ``Py_COMPAT_API_VERSION`` must: - be soft-deprecated (see :pep:`387`); - for all known use cases of the API, have a documented alternative or workaround; - have tests to ensure it keeps working (except for 1:1 renames using ``#define`` or ``typedef``); - be documented (except if it was never mentioned in previous versions of the documentation); and - be approved by the C API working group. (The WG may give blanket approvals for groups of related API; see *Initial set* below for examples.) Note that ``Py_COMPAT_API_VERSION`` is meant for API that can be trivially replaced by a better alternative. API without a replacement should generally be deprecated instead. Location -------- All API definitions omitted by ``Py_COMPAT_API_VERSION`` will be moved to a new header, ``Include/legacy.h``. This is meant to help linter authors compile lists, so they can flag the API with warnings rather than errors. Note that for simple renaming of source-only constructs (macros, types), we expect names to be omitted in the same version -- or the same PR -- that adds a replacement. This means that the original definition will be renamed, and a ``typedef`` or ``#define`` for the old name added to ``Include/legacy.h``. Documentation ------------- Documentation for omitted API should generally: - appear after the recommended replacement, - reference the replacement (e.g. “Similar to X, but…”), and - focus on differences from the replacement and migration advice. Exceptions are possible if there is a good reason for them. Initial set ----------- The following API will be omitted with ``Py_COMPAT_API_VERSION`` set to ``0x030e0000`` (3.14) or greater: - Omit API returning borrowed references: ==================================== ============================== Omitted API Replacement ==================================== ============================== ``PyDict_GetItem()`` ``PyDict_GetItemRef()`` ``PyDict_GetItemString()`` ``PyDict_GetItemStringRef()`` ``PyImport_AddModule()`` ``PyImport_AddModuleRef()`` ``PyList_GetItem()`` ``PyList_GetItemRef()`` ==================================== ============================== - Omit deprecated APIs: ==================================== ============================== Omitted Deprecated API Replacement ==================================== ============================== ``PY_FORMAT_SIZE_T`` ``"z"`` ``PY_UNICODE_TYPE`` ``wchar_t`` ``PyCode_GetFirstFree()`` ``PyUnstable_Code_GetFirstFree()`` ``PyCode_New()`` ``PyUnstable_Code_New()`` ``PyCode_NewWithPosOnlyArgs()`` ``PyUnstable_Code_NewWithPosOnlyArgs()`` ``PyImport_ImportModuleNoBlock()`` ``PyImport_ImportModule()`` ``PyMem_DEL()`` ``PyMem_Free()`` ``PyMem_Del()`` ``PyMem_Free()`` ``PyMem_FREE()`` ``PyMem_Free()`` ``PyMem_MALLOC()`` ``PyMem_Malloc()`` ``PyMem_NEW()`` ``PyMem_New()`` ``PyMem_REALLOC()`` ``PyMem_Realloc()`` ``PyMem_RESIZE()`` ``PyMem_Resize()`` ``PyModule_GetFilename()`` ``PyModule_GetFilenameObject()`` ``PyOS_AfterFork()`` ``PyOS_AfterFork_Child()`` ``PyObject_DEL()`` ``PyObject_Free()`` ``PyObject_Del()`` ``PyObject_Free()`` ``PyObject_FREE()`` ``PyObject_Free()`` ``PyObject_MALLOC()`` ``PyObject_Malloc()`` ``PyObject_REALLOC()`` ``PyObject_Realloc()`` ``PySlice_GetIndicesEx()`` (two calls; see current docs) ``PyThread_ReInitTLS()`` (no longer needed) ``PyThread_create_key()`` ``PyThread_tss_alloc()`` ``PyThread_delete_key()`` ``PyThread_tss_free()`` ``PyThread_delete_key_value()`` ``PyThread_tss_delete()`` ``PyThread_get_key_value()`` ``PyThread_tss_get()`` ``PyThread_set_key_value()`` ``PyThread_tss_set()`` ``PyUnicode_AsDecodedObject()`` ``PyUnicode_Decode()`` ``PyUnicode_AsDecodedUnicode()`` ``PyUnicode_Decode()`` ``PyUnicode_AsEncodedObject()`` ``PyUnicode_AsEncodedString()`` ``PyUnicode_AsEncodedUnicode()`` ``PyUnicode_AsEncodedString()`` ``PyUnicode_IS_READY()`` (no longer needed) ``PyUnicode_READY()`` (no longer needed) ``PyWeakref_GET_OBJECT()`` ``PyWeakref_GetRef()`` ``PyWeakref_GetObject()`` ``PyWeakref_GetRef()`` ``Py_UNICODE`` ``wchar_t`` ``_PyCode_GetExtra()`` ``PyUnstable_Code_GetExtra()`` ``_PyCode_SetExtra()`` ``PyUnstable_Code_SetExtra()`` ``_PyDict_GetItemStringWithError()`` ``PyDict_GetItemStringRef()`` ``_PyEval_RequestCodeExtraIndex()`` ``PyUnstable_Eval_RequestCodeExtraIndex()`` ``_PyHASH_BITS`` ``PyHASH_BITS`` ``_PyHASH_IMAG`` ``PyHASH_IMAG`` ``_PyHASH_INF`` ``PyHASH_INF`` ``_PyHASH_MODULUS`` ``PyHASH_MODULUS`` ``_PyHASH_MULTIPLIER`` ``PyHASH_MULTIPLIER`` ``_PyObject_EXTRA_INIT`` (no longer needed) ``_PyThreadState_UncheckedGet()`` ``PyThreadState_GetUnchecked()`` ``_PyUnicode_AsString()`` ``PyUnicode_AsUTF8()`` ``_Py_HashPointer()`` ``Py_HashPointer()`` ``_Py_T_OBJECT`` ``Py_T_OBJECT_EX`` ``_Py_WRITE_RESTRICTED`` (no longer needed) ==================================== ============================== - Soft-deprecate and omit APIs: ==================================== ============================== Omitted Deprecated API Replacement ==================================== ============================== ``PyDict_GetItemWithError()`` ``PyDict_GetItemRef()`` ``PyDict_SetDefault()`` ``PyDict_SetDefaultRef()`` ``PyMapping_HasKey()`` ``PyMapping_HasKeyWithError()`` ``PyMapping_HasKeyString()`` ``PyMapping_HasKeyStringWithError()`` ``PyObject_HasAttr()`` ``PyObject_HasAttrWithError()`` ``PyObject_HasAttrString()`` ``PyObject_HasAttrStringWithError()`` ==================================== ============================== - Omit ```` legacy API: The header file ``structmember.h``, which is not included from ```` and must be included separately, will ``#error`` if ``Py_COMPAT_API_VERSION`` is defined. This affects the following API: ==================================== ============================== Omitted Deprecated API Replacement ==================================== ============================== ``T_SHORT`` ``Py_T_SHORT`` ``T_INT`` ``Py_T_INT`` ``T_LONG`` ``Py_T_LONG`` ``T_FLOAT`` ``Py_T_FLOAT`` ``T_DOUBLE`` ``Py_T_DOUBLE`` ``T_STRING`` ``Py_T_STRING`` ``T_OBJECT`` (``tp_getset``; docs to be written) ``T_CHAR`` ``Py_T_CHAR`` ``T_BYTE`` ``Py_T_BYTE`` ``T_UBYTE`` ``Py_T_UBYTE`` ``T_USHORT`` ``Py_T_USHORT`` ``T_UINT`` ``Py_T_UINT`` ``T_ULONG`` ``Py_T_ULONG`` ``T_STRING_INPLACE`` ``Py_T_STRING_INPLACE`` ``T_BOOL`` ``Py_T_BOOL`` ``T_OBJECT_EX`` ``Py_T_OBJECT_EX`` ``T_LONGLONG`` ``Py_T_LONGLONG`` ``T_ULONGLONG`` ``Py_T_ULONGLONG`` ``T_PYSSIZET`` ``Py_T_PYSSIZET`` ``T_NONE`` (``tp_getset``; docs to be written) ``READONLY`` ``Py_READONLY`` ``PY_AUDIT_READ`` ``Py_AUDIT_READ`` ``READ_RESTRICTED`` ``Py_AUDIT_READ`` ``PY_WRITE_RESTRICTED`` (no longer needed) ``RESTRICTED`` ``Py_AUDIT_READ`` ==================================== ============================== - Omit soft deprecated macros: ====================== ===================================== Omitted Macros Replacement ====================== ===================================== ``Py_IS_NAN()`` ``isnan()`` (C99+ ````) ``Py_IS_INFINITY()`` ``isinf(X)`` (C99+ ````) ``Py_IS_FINITE()`` ``isfinite(X)`` (C99+ ````) ``Py_MEMCPY()`` ``memcpy()`` (C ````) ====================== ===================================== - Soft-deprecate and omit typedefs without the ``Py``/``_Py`` prefix (``getter``, ``setter``, ``allocfunc``, …), in favour of *new* ones that add the prefix (``Py_getter`` , etc.) - Soft-deprecate and omit macros without the ``Py``/``_Py`` prefix (``METH_O``, ``CO_COROUTINE``, ``FUTURE_ANNOTATIONS``, ``WAIT_LOCK``, …), favour of *new* ones that add the prefix (``Py_METH_O`` , etc.). - Any others approved by the C API workgroup If any of these proposed replacements, or associated documentation, are not added in time for 3.14.0b1, they'll be omitted with later versions of ``Py_COMPAT_API_VERSION``. (We expect this for macros generated by ``configure``: ``HAVE_*``, ``WITH_*``, ``ALIGNOF_*``, ``SIZEOF_*``, and several without a common prefix.) Implementation ============== TBD Open issues =========== The name ``Py_COMPAT_API_VERSION`` was taken from the earlier PEP; it doesn't fit this version. Backwards Compatibility ======================= The macro is backwards compatible. Developers can introduce and update the macro on their own pace, potentially for one source file at a time. Discussions =========== * C API Evolutions: `Macro to hide deprecated functions `_ (October 2023) * C API Problems: `Opt-in macro for a new clean API? Subset of functions with no known issues `_ (June 2023) * `Finishing the Great Renaming `_ (May 2024) Prior Art ========= * ``Py_LIMITED_API`` macro of :pep:`384` "Defining a Stable ABI". * Rejected :pep:`606` "Python Compatibility Version" which has a global scope. Copyright ========= This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive.