From 2eb4fe531393a2a4799cdfed4da986aa971bf152 Mon Sep 17 00:00:00 2001 From: Petr Viktorin Date: Thu, 25 Jul 2024 14:32:18 +0200 Subject: [PATCH] PEP 743: Rewrite to hide (soft-)deprecated API (GH-3869) Co-authored-by: Victor Stinner --- .github/CODEOWNERS | 2 +- peps/pep-0743.rst | 456 ++++++++++++++++++++++++++++++++------------- 2 files changed, 323 insertions(+), 135 deletions(-) diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 1db3def14..6151162cd 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -621,7 +621,7 @@ peps/pep-0738.rst @encukou peps/pep-0740.rst @dstufft peps/pep-0741.rst @vstinner peps/pep-0742.rst @JelleZijlstra -peps/pep-0743.rst @vstinner +peps/pep-0743.rst @vstinner @encukou peps/pep-0744.rst @brandtbucher peps/pep-0745.rst @hugovk peps/pep-0746.rst @JelleZijlstra diff --git a/peps/pep-0743.rst b/peps/pep-0743.rst index 39d9039eb..3e202c88b 100644 --- a/peps/pep-0743.rst +++ b/peps/pep-0743.rst @@ -1,10 +1,11 @@ PEP: 743 Title: Add Py_COMPAT_API_VERSION to the Python C API -Author: Victor Stinner +Author: Victor Stinner , + Petr Viktorin , Status: Draft Type: Standards Track Created: 11-Mar-2024 -Python-Version: 3.13 +Python-Version: 3.14 .. highlight:: c @@ -12,177 +13,361 @@ Python-Version: 3.13 Abstract ======== -Add ``Py_COMPAT_API_VERSION`` and ``Py_COMPAT_API_VERSION_MAX`` macros -to opt-in for planned incompatible C API changes in a C extension. -Maintainers can decide when they make their C extension compatible -and also decide which future Python version they want to be compatible -with. +Add ``Py_COMPAT_API_VERSION`` C macro that hides some deprecated and +soft-deprecated symbols, allowing users to opt out of using API with known +issues that other API solves. +The macro is versioned, allowing users to update (or not) on their own pace. + +Also, add namespaced alternatives for API without the ``Py_`` prefix, +and soft-deprecate the original names. + + +Motivation +========== + +Some of Python's C API has flaws that are only obvious in hindsight. + +If an API prevents adding features or optimizations, or presents a serious +security risk or maintenance burden, we can deprecate and remove it as +described in :pep:`387`. + +However, this leaves us with some API that has “sharp edges” -- it works fine +for its current users, but should be avoided in new code. +For example: + +- API that cannot signal an exception, so failures are either ignored or + exit the process with a fatal error. For example ``PyObject_HasAttr``. +- API that is not thread-safe, for example by borrowing references from + mutable objects, or exposing unfinished mutable objects. For example + ``PyDict_GetItemWithError``. +- API with names that don't use the ``Py``/``_Py`` prefix, and so can clash + with other code. For example: ``setter``. + +It is important to note that despite such flaws, it's usually possible +to use the API correctly. For example, in a single-threaded environment, +thread safety is not an issue. +We do not want to break working code, even if it uses API that would be wrong +in some -- or even *most* -- other contexts. + +On the other hand, we want to steer users away from such “undesirable” API +in *new* code, especially if a safer alternative exists. + + +Adding the ``Py`` prefix +------------------------ + +Some names defined in CPython headers is not namespaced: it that lacks the +``Py`` prefix (or a variant: ``_Py``, and alternative capitalizations). +For example, we declare a function type named simply ``setter``. + +While such names are not exported in the ABI (as checked by ``make smelly``), +they can clash with user code and, more importantly, with libraries linked +to third-party extensions. + +While it would be possible to provide namespaced aliases and (soft-)deprecate +these names, the only way to make them not clash with third-party code is to +not define them in Python headers at all. Rationale ========= -Python releases enforce C API changes -------------------------------------- +We want to allow an easy way for users to avoid “undesirable” API if they +choose to do so. -Every Python 3.x release has a long list of C API changes, including -incompatible changes. C extensions have to be updated to work on the -newly released Python. +It might be be sufficient to leave this to third-party linters. +For that we'd need a good way to expose a list of (soft-)deprecated +API to such linters. +While adding that, we can -- rather easily -- do the linter's job directly +in CPython headers, avoiding the neel for an extra tool. +Unlike Python, C makes it rather easy to limit available API -- for a whole +project or for each individual source file -- by having users define +an “opt-in” macro. -Some incompatible changes are driven by new features: they cannot be -avoided, unless we decide to not add these features. Other reasons: +We already do something similar with ``Py_LIMITED_API``, which limits the +available API to a subset that compiles to stable ABI. (In hindsight, we should +have used a different macro name for that particular kind of limiting, but it's +too late to change that now.) -* Remove deprecated API (see :pep:`387`). -* Ease the implementation of another change. -* Change or remove error-prone API. +To prevent working code from breaking as we identify more “undesirable” API +and add safer alternatives to it, the opt-in macro should be *versioned*. +Users can choose a version they need based on their compatibility requirements, +and update it at their own pace. -Currently, there is no middle ground between "not change the C API" and -"incompatible C API changes impact everybody". Either a C extension is -updated or the new Python version cannot be used. Such all-or-nothing -deal does not satisfy C extension maintainers nor C extensions users. +To be clear, this mechanism is *not* a replacement for deprecation. +Deprecation is for API that prevents new features or optimizations, or +presents a security risk or maintenance burden. +This mechanism, on the other hand, is meant for cases where “we found +a slightly better way of doing things” -- perhaps one that's harder to misuse, +or just has a less misleading name. +(On a lighter note: many people configure a code quality checker to shout at +them about the number of blank lines between functions. Let's help them +identify more substantial “code smells”!) + +The proposed macro does not *change* any API definitions; it only *hides* them. +So, if code compiles with the macro, it'll also compile without it, with +identical behaviour. +This has implications for core devs: to deal with undesirable behaviour, +we'll need to introduce new, better API, and *then* discourage the old one. +In turn, this implies that we should look at an individual API and fix all its +known issues at once, rather than do codebase-wide sweeps for a single kind of +issue, so that we avoid multiple renames of the same function. -Limited C API -------------- +Adding the ``Py`` prefix +------------------------ -The limited C API is versioned: the ``Py_LIMITED_API`` macro can be set -to a Python version to select which API is available. On the Python -side, it allows introducing incompatible changes at a specific -``Py_LIMITED_API`` version. For example, if ``Py_LIMITED_API`` is set to -Python 3.11 or newer, the ```` is no longer included by -``Python.h``, whereas C extensions targeting Python 3.10 are not -affected. - -The difference here is that upgrading Python does not change if -```` is included or not, but updating ``Py_LIMITED_API`` does. -Updating ``Py_LIMITED_API`` is an deliberate action made by the C -extension maintainer. It gives more freedom to decide **when** the -maintainer is ready to deal with the latest batch of incompatible -changes. - -A similar version can be used with the regular (non-limited) C API. - - -Deprecation and compiler warnings ---------------------------------- - -Deprecated functions are marked with ``Py_DEPRECATED()``. Using a -deprecated function emits a compiler warning. - -The problem is that ``pip`` and ``build`` tools hide compiler logs by -default, unless a build fails. Moreover, it's easy to miss a single -warning in the middle of hundred lines of logs. - -Schedule changes ----------------- - -Currently, there is no way to schedule a C API change: announce it but -also provide a way to maintainers to test their C extensions with the -change. Either a change is not made, or everybody must update their code -if they want to update Python. +An opt-in macro allows us to omit definitions that could clash with +third-party libraries. Specification ============= -New macros ----------- +We introduce a ``Py_COMPAT_API_VERSION`` macro. +If this macro is defined before ``#include ``, some API definitions +-- as described below -- will be omitted from the Python header files. -Add new ``Py_COMPAT_API_VERSION`` and ``Py_COMPAT_API_VERSION_MAX`` -macros. They can be set to test if a C extension is prepared for future -C API changes: compatible with future Python versions. +The macro only omits complete top-level definitions exposed from ````. +Other things (the ABI, structure definitions, macro expansions, static inline +function bodies, etc.) are not affected. -The ``Py_COMPAT_API_VERSION`` macro can be set to a specific Python -version. For example, ``Py_COMPAT_API_VERSION=0x030e0000`` tests C API -changes scheduled in Python 3.14. +The C API working group (:pep:`731`) has authority over the set of omitted +definitions. -If the ``Py_COMPAT_API_VERSION`` macro is set to -``Py_COMPAT_API_VERSION_MAX``, all scheduled C API changes are tested at -once. +The set of omitted definitions will be tied to a particular feature release +of CPython, and is finalized in each 3.x.0 Beta 1 release. +In rare cases, entries can be removed (i.e. made available for use) at any +time. -If the ``Py_COMPAT_API_VERSION`` macro is not set, it is to -``PY_VERSION_HEX`` by default. - -The ``Py_COMPAT_API_VERSION`` macro can be set in a single C file or for -a whole project in compiler flags. The macro does not affected other -projects or Python itself. +The macro should be defined to a version in the format used by +``PY_VERSION_HEX``, with the “micro”, “release” and “serial” fields +set to zero. +For example, to omit API deemed undesirable in 3.14.0b1, users should define +``Py_COMPAT_API_VERSION`` to ``0x030e0000``. -Example in Python ------------------ +Requirements for omitted API +---------------------------- -For example, the ``PyImport_ImportModuleNoBlock()`` function is -deprecated in Python 3.13 and scheduled for removal in Python 3.15. The -function can be declared in the Python C API with the following -declaration: +An API that is omitted with ``Py_COMPAT_API_VERSION`` must: -.. code-block:: c +- be soft-deprecated (see :pep:`387`); +- for all known use cases of the API, have a documented alternative + or workaround; +- have tests to ensure it keeps working (except for 1:1 renames using + ``#define`` or ``typedef``); +- be documented (except if it was never mentioned in previous versions of the + documentation); and +- be approved by the C API working group. (The WG may give blanket approvals + for groups of related API; see *Initial set* below for examples.) - #if Py_COMPAT_API_VERSION < 0x030f0000 - Py_DEPRECATED(3.13) PyAPI_FUNC(PyObject *) PyImport_ImportModuleNoBlock( - const char *name /* UTF-8 encoded string */ - ); - #endif - -If ``if Py_COMPAT_API_VERSION`` is equal to or greater than Python 3.15 -(``0x030f0000``), the ``PyImport_ImportModuleNoBlock()`` function is not -declared, and so using it fails with a build error. - -Goals ------ - -* Reduce the number of C API changes affecting C extensions when - updating Python. -* When testing C extensions (for example, optional CI test), - ``Py_COMPAT_API_VERSION`` can be set to ``Py_COMPAT_API_VERSION_MAX`` - to detect future incompatibilities. For mandatory tests, it is - recommended to set ``Py_COMPAT_API_VERSION`` to a specific Python - version. -* For core developers, make sure that the C API can still evolve - without being afraid of breaking an unknown number of C extensions. - -Non-goals ---------- - -* Freeze the API forever: this is not the stable ABI. For example, - deprecated functions will continue to be removed on a regular basis. -* C extensions maintainers not using ``Py_COMPAT_API_VERSION`` will - still be affected by C API changes when updating Python. -* Provide a stable ABI: the macro only impacts the regular (non-limited) - API. -* Silver bullet solving all C API issues. +Note that ``Py_COMPAT_API_VERSION`` is meant for API that can be trivially +replaced by a better alternative. +API without a replacement should generally be deprecated instead. -Examples of ``Py_COMPAT_API_VERSION`` usages -============================================ +Location +-------- -* Remove deprecated functions. -* Remove deprecated structure members, such as - ``PyBytesObject.ob_shash``. -* Remove a standard ``#include``, such as ``#include ``, - from ````. -* Change the behavior of a function or a macro. For example, calling - ``PyObject_SetAttr(obj, name, NULL)`` can fail, to enforce the usage - of the ``PyObject_DelAttr()`` function instead to delete an attribute. +All API definitions omitted by ``Py_COMPAT_API_VERSION`` will be moved to +a new header, ``Include/legacy.h``. + +This is meant to help linter authors compile lists, so they can flag the API +with warnings rather than errors. + +Note that for simple renaming of source-only constructs (macros, types), we +expect names to be omitted in the same version -- or the same PR -- that adds +a replacement. +This means that the original definition will be renamed, and a ``typedef`` +or ``#define`` for the old name added to ``Include/legacy.h``. + + +Documentation +------------- + +Documentation for omitted API should generally: + +- appear after the recommended replacement, +- reference the replacement (e.g. “Similar to X, but…”), and +- focus on differences from the replacement and migration advice. + +Exceptions are possible if there is a good reason for them. + + +Initial set +----------- + +The following API will be omitted with ``Py_COMPAT_API_VERSION`` set to +``0x030e0000`` (3.14) or greater: + +- Omit API returning borrowed references: + + ==================================== ============================== + Omitted API Replacement + ==================================== ============================== + ``PyDict_GetItem()`` ``PyDict_GetItemRef()`` + ``PyDict_GetItemString()`` ``PyDict_GetItemStringRef()`` + ``PyImport_AddModule()`` ``PyImport_AddModuleRef()`` + ``PyList_GetItem()`` ``PyList_GetItemRef()`` + ==================================== ============================== + +- Omit deprecated APIs: + + ==================================== ============================== + Omitted Deprecated API Replacement + ==================================== ============================== + ``PY_FORMAT_SIZE_T`` ``"z"`` + ``PY_UNICODE_TYPE`` ``wchar_t`` + ``PyCode_GetFirstFree()`` ``PyUnstable_Code_GetFirstFree()`` + ``PyCode_New()`` ``PyUnstable_Code_New()`` + ``PyCode_NewWithPosOnlyArgs()`` ``PyUnstable_Code_NewWithPosOnlyArgs()`` + ``PyImport_ImportModuleNoBlock()`` ``PyImport_ImportModule()`` + ``PyMem_DEL()`` ``PyMem_Free()`` + ``PyMem_Del()`` ``PyMem_Free()`` + ``PyMem_FREE()`` ``PyMem_Free()`` + ``PyMem_MALLOC()`` ``PyMem_Malloc()`` + ``PyMem_NEW()`` ``PyMem_New()`` + ``PyMem_REALLOC()`` ``PyMem_Realloc()`` + ``PyMem_RESIZE()`` ``PyMem_Resize()`` + ``PyModule_GetFilename()`` ``PyModule_GetFilenameObject()`` + ``PyOS_AfterFork()`` ``PyOS_AfterFork_Child()`` + ``PyObject_DEL()`` ``PyObject_Free()`` + ``PyObject_Del()`` ``PyObject_Free()`` + ``PyObject_FREE()`` ``PyObject_Free()`` + ``PyObject_MALLOC()`` ``PyObject_Malloc()`` + ``PyObject_REALLOC()`` ``PyObject_Realloc()`` + ``PySlice_GetIndicesEx()`` (two calls; see current docs) + ``PyThread_ReInitTLS()`` (no longer needed) + ``PyThread_create_key()`` ``PyThread_tss_alloc()`` + ``PyThread_delete_key()`` ``PyThread_tss_free()`` + ``PyThread_delete_key_value()`` ``PyThread_tss_delete()`` + ``PyThread_get_key_value()`` ``PyThread_tss_get()`` + ``PyThread_set_key_value()`` ``PyThread_tss_set()`` + ``PyUnicode_AsDecodedObject()`` ``PyUnicode_Decode()`` + ``PyUnicode_AsDecodedUnicode()`` ``PyUnicode_Decode()`` + ``PyUnicode_AsEncodedObject()`` ``PyUnicode_AsEncodedString()`` + ``PyUnicode_AsEncodedUnicode()`` ``PyUnicode_AsEncodedString()`` + ``PyUnicode_IS_READY()`` (no longer needed) + ``PyUnicode_READY()`` (no longer needed) + ``PyWeakref_GET_OBJECT()`` ``PyWeakref_GetRef()`` + ``PyWeakref_GetObject()`` ``PyWeakref_GetRef()`` + ``Py_UNICODE`` ``wchar_t`` + ``_PyCode_GetExtra()`` ``PyUnstable_Code_GetExtra()`` + ``_PyCode_SetExtra()`` ``PyUnstable_Code_SetExtra()`` + ``_PyDict_GetItemStringWithError()`` ``PyDict_GetItemStringRef()`` + ``_PyEval_RequestCodeExtraIndex()`` ``PyUnstable_Eval_RequestCodeExtraIndex()`` + ``_PyHASH_BITS`` ``PyHASH_BITS`` + ``_PyHASH_IMAG`` ``PyHASH_IMAG`` + ``_PyHASH_INF`` ``PyHASH_INF`` + ``_PyHASH_MODULUS`` ``PyHASH_MODULUS`` + ``_PyHASH_MULTIPLIER`` ``PyHASH_MULTIPLIER`` + ``_PyObject_EXTRA_INIT`` (no longer needed) + ``_PyThreadState_UncheckedGet()`` ``PyThreadState_GetUnchecked()`` + ``_PyUnicode_AsString()`` ``PyUnicode_AsUTF8()`` + ``_Py_HashPointer()`` ``Py_HashPointer()`` + ``_Py_T_OBJECT`` ``Py_T_OBJECT_EX`` + ``_Py_WRITE_RESTRICTED`` (no longer needed) + ==================================== ============================== + +- Soft-deprecate and omit APIs: + + ==================================== ============================== + Omitted Deprecated API Replacement + ==================================== ============================== + ``PyDict_GetItemWithError()`` ``PyDict_GetItemRef()`` + ``PyDict_SetDefault()`` ``PyDict_SetDefaultRef()`` + ``PyMapping_HasKey()`` ``PyMapping_HasKeyWithError()`` + ``PyMapping_HasKeyString()`` ``PyMapping_HasKeyStringWithError()`` + ``PyObject_HasAttr()`` ``PyObject_HasAttrWithError()`` + ``PyObject_HasAttrString()`` ``PyObject_HasAttrStringWithError()`` + ==================================== ============================== + +- Omit ```` legacy API: + + The header file ``structmember.h``, which is not included from ```` + and must be included separately, will ``#error`` if + ``Py_COMPAT_API_VERSION`` is defined. + This affects the following API: + + ==================================== ============================== + Omitted Deprecated API Replacement + ==================================== ============================== + ``T_SHORT`` ``Py_T_SHORT`` + ``T_INT`` ``Py_T_INT`` + ``T_LONG`` ``Py_T_LONG`` + ``T_FLOAT`` ``Py_T_FLOAT`` + ``T_DOUBLE`` ``Py_T_DOUBLE`` + ``T_STRING`` ``Py_T_STRING`` + ``T_OBJECT`` (``tp_getset``; docs to be written) + ``T_CHAR`` ``Py_T_CHAR`` + ``T_BYTE`` ``Py_T_BYTE`` + ``T_UBYTE`` ``Py_T_UBYTE`` + ``T_USHORT`` ``Py_T_USHORT`` + ``T_UINT`` ``Py_T_UINT`` + ``T_ULONG`` ``Py_T_ULONG`` + ``T_STRING_INPLACE`` ``Py_T_STRING_INPLACE`` + ``T_BOOL`` ``Py_T_BOOL`` + ``T_OBJECT_EX`` ``Py_T_OBJECT_EX`` + ``T_LONGLONG`` ``Py_T_LONGLONG`` + ``T_ULONGLONG`` ``Py_T_ULONGLONG`` + ``T_PYSSIZET`` ``Py_T_PYSSIZET`` + ``T_NONE`` (``tp_getset``; docs to be written) + ``READONLY`` ``Py_READONLY`` + ``PY_AUDIT_READ`` ``Py_AUDIT_READ`` + ``READ_RESTRICTED`` ``Py_AUDIT_READ`` + ``PY_WRITE_RESTRICTED`` (no longer needed) + ``RESTRICTED`` ``Py_AUDIT_READ`` + ==================================== ============================== + +- Omit soft deprecated macros: + + ====================== ===================================== + Omitted Macros Replacement + ====================== ===================================== + ``Py_IS_NAN()`` ``isnan()`` (C99+ ````) + ``Py_IS_INFINITY()`` ``isinf(X)`` (C99+ ````) + ``Py_IS_FINITE()`` ``isfinite(X)`` (C99+ ````) + ``Py_MEMCPY()`` ``memcpy()`` (C ````) + ====================== ===================================== + +- Soft-deprecate and omit typedefs without the ``Py``/``_Py`` prefix + (``getter``, ``setter``, ``allocfunc``, …), in favour of *new* ones + that add the prefix (``Py_getter`` , etc.) + +- Soft-deprecate and omit macros without the ``Py``/``_Py`` prefix + (``METH_O``, ``CO_COROUTINE``, ``FUTURE_ANNOTATIONS``, ``WAIT_LOCK``, …), + favour of *new* ones that add the prefix (``Py_METH_O`` , etc.). + +- Any others approved by the C API workgroup + + +If any of these proposed replacements, or associated documentation, +are not added in time for 3.14.0b1, they'll be omitted with later versions +of ``Py_COMPAT_API_VERSION``. +(We expect this for macros generated by ``configure``: ``HAVE_*``, ``WITH_*``, +``ALIGNOF_*``, ``SIZEOF_*``, and several without a common prefix.) Implementation ============== -* `Issue gh-116587 `_ -* PR: `Add Py_COMPAT_API_VERSION and Py_COMPAT_API_VERSION_MAX macros - `_ +TBD + + +Open issues +=========== + +The name ``Py_COMPAT_API_VERSION`` was taken from the earlier PEP; +it doesn't fit this version. Backwards Compatibility ======================= -There is no impact on backward compatibility. - -Adding ``Py_COMPAT_API_VERSION`` and ``Py_COMPAT_API_VERSION_MAX`` -macros has no effect on backward compatibility. Only developers setting -the ``Py_COMPAT_API_VERSION`` macro in their project will be impacted by -effects of this macro which is the expected behavior. +The macro is backwards compatible. +Developers can introduce and update the macro on their own pace, potentially +for one source file at a time. Discussions @@ -195,6 +380,9 @@ Discussions with no known issues `_ (June 2023) +* `Finishing the Great Renaming + `_ + (May 2024) Prior Art