PEP 743: Rewrite to hide (soft-)deprecated API (GH-3869)

Co-authored-by: Victor Stinner <vstinner@python.org>

.github/CODEOWNERS

@@ -621,7 +621,7 @@ peps/pep-0738.rst  @encukou
 peps/pep-0740.rst  @dstufft
 peps/pep-0741.rst  @vstinner
 peps/pep-0742.rst  @JelleZijlstra
-peps/pep-0743.rst  @vstinner
+peps/pep-0743.rst  @vstinner @encukou
 peps/pep-0744.rst  @brandtbucher
 peps/pep-0745.rst  @hugovk
 peps/pep-0746.rst  @JelleZijlstra

peps/pep-0743.rst

@@ -1,10 +1,11 @@
 PEP: 743
 Title: Add Py_COMPAT_API_VERSION to the Python C API
-Author: Victor Stinner <vstinner@python.org>
+Author: Victor Stinner <vstinner@python.org>,
+        Petr Viktorin <encukou@gmail.com>,
 Status: Draft
 Type: Standards Track
 Created: 11-Mar-2024
-Python-Version: 3.13
+Python-Version: 3.14
 
 .. highlight:: c
 
@@ -12,177 +13,361 @@ Python-Version: 3.13
 Abstract
 ========
 
-Add ``Py_COMPAT_API_VERSION`` and ``Py_COMPAT_API_VERSION_MAX`` macros
+Add ``Py_COMPAT_API_VERSION`` C macro that hides some deprecated and
-to opt-in for planned incompatible C API changes in a C extension.
+soft-deprecated symbols, allowing users to opt out of using API with known
-Maintainers can decide when they make their C extension compatible
+issues that other API solves.
-and also decide which future Python version they want to be compatible
+The macro is versioned, allowing users to update (or not) on their own pace.
-with.
+
+Also, add namespaced alternatives for API without the ``Py_`` prefix,
+and soft-deprecate the original names.
+
+
+Motivation
+==========
+
+Some of Python's C API has flaws that are only obvious in hindsight.
+
+If an API prevents adding features or optimizations, or presents a serious
+security risk or maintenance burden, we can deprecate and remove it as
+described in :pep:`387`.
+
+However, this leaves us with some API that has “sharp edges” -- it works fine
+for its current users, but should be avoided in new code.
+For example:
+
+- API that cannot signal an exception, so failures are either ignored or
+  exit the process with a fatal error. For example ``PyObject_HasAttr``.
+- API that is not thread-safe, for example by borrowing references from
+  mutable objects, or exposing unfinished mutable objects. For example
+  ``PyDict_GetItemWithError``.
+- API with names that don't use the ``Py``/``_Py`` prefix, and so can clash
+  with other code. For example: ``setter``.
+
+It is important to note that despite such flaws, it's usually possible
+to use the API correctly. For example, in a single-threaded environment,
+thread safety is not an issue.
+We do not want to break working code, even if it uses API that would be wrong
+in some -- or even *most* -- other contexts.
+
+On the other hand, we want to steer users away from such “undesirable” API
+in *new* code, especially if a safer alternative exists.
+
+
+Adding the ``Py`` prefix
+------------------------
+
+Some names defined in CPython headers is not namespaced: it that lacks the
+``Py`` prefix (or a variant: ``_Py``, and alternative capitalizations).
+For example, we declare a function type named simply ``setter``.
+
+While such names are not exported in the ABI (as checked by ``make smelly``),
+they can clash with user code and, more importantly, with libraries linked
+to third-party extensions.
+
+While it would be possible to provide namespaced aliases and (soft-)deprecate
+these names, the only way to make them not clash with third-party code is to
+not define them in Python headers at all.
 
 
 Rationale
 =========
 
-Python releases enforce C API changes
+We want to allow an easy way for users to avoid “undesirable” API if they
--------------------------------------
+choose to do so.
 
-Every Python 3.x release has a long list of C API changes, including
+It might be be sufficient to leave this to third-party linters.
-incompatible changes. C extensions have to be updated to work on the
+For that we'd need a good way to expose a list of (soft-)deprecated
-newly released Python.
+API to such linters.
+While adding that, we can -- rather easily -- do the linter's job directly
+in CPython headers, avoiding the neel for an extra tool.
+Unlike Python, C makes it rather easy to limit available API -- for a whole
+project or for each individual source file -- by having users define
+an “opt-in” macro.
 
-Some incompatible changes are driven by new features: they cannot be
+We already do something similar with ``Py_LIMITED_API``, which limits the
-avoided, unless we decide to not add these features. Other reasons:
+available API to a subset that compiles to stable ABI. (In hindsight, we should
+have used a different macro name for that particular kind of limiting, but it's
+too late to change that now.)
 
-* Remove deprecated API (see :pep:`387`).
+To prevent working code from breaking as we identify more “undesirable” API
-* Ease the implementation of another change.
+and add safer alternatives to it, the opt-in macro should be *versioned*.
-* Change or remove error-prone API.
+Users can choose a version they need based on their compatibility requirements,
+and update it at their own pace.
 
-Currently, there is no middle ground between "not change the C API" and
+To be clear, this mechanism is *not* a replacement for deprecation.
-"incompatible C API changes impact everybody". Either a C extension is
+Deprecation is for API that prevents new features or optimizations, or
-updated or the new Python version cannot be used. Such all-or-nothing
+presents a security risk or maintenance burden.
-deal does not satisfy C extension maintainers nor C extensions users.
+This mechanism, on the other hand, is meant for cases where “we found
+a slightly better way of doing things” -- perhaps one that's harder to misuse,
+or just has a less misleading name.
+(On a lighter note: many people configure a code quality checker to shout at
+them about the number of blank lines between functions. Let's help them
+identify more substantial “code smells”!)
+
+The proposed macro does not *change* any API definitions; it only *hides* them.
+So, if code compiles with the macro, it'll also compile without it, with
+identical behaviour.
+This has implications for core devs: to deal with undesirable behaviour,
+we'll need to introduce new, better API, and *then* discourage the old one.
+In turn, this implies that we should look at an individual API and fix all its
+known issues at once, rather than do codebase-wide sweeps for a single kind of
+issue, so that we avoid multiple renames of the same function.
 
 
-Limited C API
+Adding the ``Py`` prefix
--------------
+------------------------
 
-The limited C API is versioned: the ``Py_LIMITED_API`` macro can be set
+An opt-in macro allows us to omit definitions that could clash with
-to a Python version to select which API is available. On the Python
+third-party libraries.
-side, it allows introducing incompatible changes at a specific
-``Py_LIMITED_API`` version. For example, if ``Py_LIMITED_API`` is set to
-Python 3.11 or newer, the ``<stdio.h>`` is no longer included by
-``Python.h``, whereas C extensions targeting Python 3.10 are not
-affected.
-
-The difference here is that upgrading Python does not change if
-``<stdio.h>`` is included or not, but updating ``Py_LIMITED_API`` does.
-Updating ``Py_LIMITED_API`` is an deliberate action made by the C
-extension maintainer. It gives more freedom to decide **when** the
-maintainer is ready to deal with the latest batch of incompatible
-changes.
-
-A similar version can be used with the regular (non-limited) C API.
-
-
-Deprecation and compiler warnings
----------------------------------
-
-Deprecated functions are marked with ``Py_DEPRECATED()``. Using a
-deprecated function emits a compiler warning.
-
-The problem is that ``pip`` and ``build`` tools hide compiler logs by
-default, unless a build fails.  Moreover, it's easy to miss a single
-warning in the middle of hundred lines of logs.
-
-Schedule changes
-----------------
-
-Currently, there is no way to schedule a C API change: announce it but
-also provide a way to maintainers to test their C extensions with the
-change. Either a change is not made, or everybody must update their code
-if they want to update Python.
 
 
 Specification
 =============
 
-New macros
+We introduce a ``Py_COMPAT_API_VERSION`` macro.
-----------
+If this macro is defined before ``#include <Python.h>``, some API definitions
+-- as described below -- will be omitted from the Python header files.
 
-Add new ``Py_COMPAT_API_VERSION`` and ``Py_COMPAT_API_VERSION_MAX``
+The macro only omits complete top-level definitions exposed from ``<Python.h>``.
-macros. They can be set to test if a C extension is prepared for future
+Other things (the ABI, structure definitions, macro expansions, static inline
-C API changes: compatible with future Python versions.
+function bodies, etc.) are not affected.
 
-The ``Py_COMPAT_API_VERSION`` macro can be set to a specific Python
+The C API working group (:pep:`731`) has authority over the set of omitted
-version. For example, ``Py_COMPAT_API_VERSION=0x030e0000`` tests C API
+definitions.
-changes scheduled in Python 3.14.
 
-If the ``Py_COMPAT_API_VERSION`` macro is set to
+The set of omitted definitions will be tied to a particular feature release
-``Py_COMPAT_API_VERSION_MAX``, all scheduled C API changes are tested at
+of CPython, and is finalized in each 3.x.0 Beta 1 release.
-once.
+In rare cases, entries can be removed (i.e. made available for use) at any
+time.
 
-If the ``Py_COMPAT_API_VERSION`` macro is not set, it is to
+The macro should be defined to a version in the format used by
-``PY_VERSION_HEX`` by default.
+``PY_VERSION_HEX``, with the “micro”, “release” and “serial” fields
-
+set to zero.
-The ``Py_COMPAT_API_VERSION`` macro can be set in a single C file or for
+For example, to omit API deemed undesirable in 3.14.0b1, users should define
-a whole project in compiler flags. The macro does not affected other
+``Py_COMPAT_API_VERSION`` to ``0x030e0000``.
-projects or Python itself.
 
 
-Example in Python
+Requirements for omitted API
------------------
+----------------------------
 
-For example, the ``PyImport_ImportModuleNoBlock()`` function is
+An API that is omitted with ``Py_COMPAT_API_VERSION`` must:
-deprecated in Python 3.13 and scheduled for removal in Python 3.15. The
-function can be declared in the Python C API with the following
-declaration:
 
-.. code-block:: c
+- be soft-deprecated (see :pep:`387`);
+- for all known use cases of the API, have a documented alternative
+  or workaround;
+- have tests to ensure it keeps working (except for 1:1 renames using
+  ``#define`` or ``typedef``);
+- be documented (except if it was never mentioned in previous versions of the
+  documentation); and
+- be approved by the C API working group. (The WG may give blanket approvals
+  for groups of related API; see *Initial set* below for examples.)
 
-    #if Py_COMPAT_API_VERSION < 0x030f0000
+Note that ``Py_COMPAT_API_VERSION`` is meant for API that can be trivially
-    Py_DEPRECATED(3.13) PyAPI_FUNC(PyObject *) PyImport_ImportModuleNoBlock(
+replaced by a better alternative.
-        const char *name            /* UTF-8 encoded string */
+API without a replacement should generally be deprecated instead.
-        );
-    #endif
-
-If ``if Py_COMPAT_API_VERSION`` is equal to or greater than Python 3.15
-(``0x030f0000``), the ``PyImport_ImportModuleNoBlock()`` function is not
-declared, and so using it fails with a build error.
-
-Goals
------
-
-* Reduce the number of C API changes affecting C extensions when
-  updating Python.
-* When testing C extensions (for example, optional CI test),
-  ``Py_COMPAT_API_VERSION`` can be set to ``Py_COMPAT_API_VERSION_MAX``
-  to detect future incompatibilities. For mandatory tests, it is
-  recommended to set ``Py_COMPAT_API_VERSION`` to a specific Python
-  version.
-* For core developers, make sure that the C API can still evolve
-  without being afraid of breaking an unknown number of C extensions.
-
-Non-goals
----------
-
-* Freeze the API forever: this is not the stable ABI. For example,
-  deprecated functions will continue to be removed on a regular basis.
-* C extensions maintainers not using ``Py_COMPAT_API_VERSION`` will
-  still be affected by C API changes when updating Python.
-* Provide a stable ABI: the macro only impacts the regular (non-limited)
-  API.
-* Silver bullet solving all C API issues.
 
 
-Examples of ``Py_COMPAT_API_VERSION`` usages
+Location
-============================================
+--------
 
-* Remove deprecated functions.
+All API definitions omitted by ``Py_COMPAT_API_VERSION`` will be moved to
-* Remove deprecated structure members, such as
+a new header, ``Include/legacy.h``.
-  ``PyBytesObject.ob_shash``.
+
-* Remove a standard ``#include``, such as ``#include <string.h>``,
+This is meant to help linter authors compile lists, so they can flag the API
-  from ``<Python.h>``.
+with warnings rather than errors.
-* Change the behavior of a function or a macro. For example, calling
+
-  ``PyObject_SetAttr(obj, name, NULL)`` can fail, to enforce the usage
+Note that for simple renaming of source-only constructs (macros, types), we
-  of the ``PyObject_DelAttr()`` function instead to delete an attribute.
+expect names to be omitted in the same version -- or the same PR -- that adds
+a replacement.
+This means that the original definition will be renamed, and a ``typedef``
+or ``#define`` for the old name added to ``Include/legacy.h``.
+
+
+Documentation
+-------------
+
+Documentation for omitted API should generally:
+
+- appear after the recommended replacement,
+- reference the replacement (e.g. “Similar to X, but…”), and
+- focus on differences from the replacement and migration advice.
+
+Exceptions are possible if there is a good reason for them.
+
+
+Initial set
+-----------
+
+The following API will be omitted with ``Py_COMPAT_API_VERSION`` set to
+``0x030e0000`` (3.14) or greater:
+
+- Omit API returning borrowed references:
+
+  ====================================  ==============================
+  Omitted API                           Replacement
+  ====================================  ==============================
+  ``PyDict_GetItem()``                  ``PyDict_GetItemRef()``
+  ``PyDict_GetItemString()``            ``PyDict_GetItemStringRef()``
+  ``PyImport_AddModule()``              ``PyImport_AddModuleRef()``
+  ``PyList_GetItem()``                  ``PyList_GetItemRef()``
+  ====================================  ==============================
+
+- Omit deprecated APIs:
+
+  ====================================  ==============================
+  Omitted Deprecated API                Replacement
+  ====================================  ==============================
+  ``PY_FORMAT_SIZE_T``                  ``"z"``
+  ``PY_UNICODE_TYPE``                   ``wchar_t``
+  ``PyCode_GetFirstFree()``             ``PyUnstable_Code_GetFirstFree()``
+  ``PyCode_New()``                      ``PyUnstable_Code_New()``
+  ``PyCode_NewWithPosOnlyArgs()``       ``PyUnstable_Code_NewWithPosOnlyArgs()``
+  ``PyImport_ImportModuleNoBlock()``    ``PyImport_ImportModule()``
+  ``PyMem_DEL()``                       ``PyMem_Free()``
+  ``PyMem_Del()``                       ``PyMem_Free()``
+  ``PyMem_FREE()``                      ``PyMem_Free()``
+  ``PyMem_MALLOC()``                    ``PyMem_Malloc()``
+  ``PyMem_NEW()``                       ``PyMem_New()``
+  ``PyMem_REALLOC()``                   ``PyMem_Realloc()``
+  ``PyMem_RESIZE()``                    ``PyMem_Resize()``
+  ``PyModule_GetFilename()``            ``PyModule_GetFilenameObject()``
+  ``PyOS_AfterFork()``                  ``PyOS_AfterFork_Child()``
+  ``PyObject_DEL()``                    ``PyObject_Free()``
+  ``PyObject_Del()``                    ``PyObject_Free()``
+  ``PyObject_FREE()``                   ``PyObject_Free()``
+  ``PyObject_MALLOC()``                 ``PyObject_Malloc()``
+  ``PyObject_REALLOC()``                ``PyObject_Realloc()``
+  ``PySlice_GetIndicesEx()``            (two calls; see current docs)
+  ``PyThread_ReInitTLS()``              (no longer needed)
+  ``PyThread_create_key()``             ``PyThread_tss_alloc()``
+  ``PyThread_delete_key()``             ``PyThread_tss_free()``
+  ``PyThread_delete_key_value()``       ``PyThread_tss_delete()``
+  ``PyThread_get_key_value()``          ``PyThread_tss_get()``
+  ``PyThread_set_key_value()``          ``PyThread_tss_set()``
+  ``PyUnicode_AsDecodedObject()``       ``PyUnicode_Decode()``
+  ``PyUnicode_AsDecodedUnicode()``      ``PyUnicode_Decode()``
+  ``PyUnicode_AsEncodedObject()``       ``PyUnicode_AsEncodedString()``
+  ``PyUnicode_AsEncodedUnicode()``      ``PyUnicode_AsEncodedString()``
+  ``PyUnicode_IS_READY()``              (no longer needed)
+  ``PyUnicode_READY()``                 (no longer needed)
+  ``PyWeakref_GET_OBJECT()``            ``PyWeakref_GetRef()``
+  ``PyWeakref_GetObject()``             ``PyWeakref_GetRef()``
+  ``Py_UNICODE``                        ``wchar_t``
+  ``_PyCode_GetExtra()``                ``PyUnstable_Code_GetExtra()``
+  ``_PyCode_SetExtra()``                ``PyUnstable_Code_SetExtra()``
+  ``_PyDict_GetItemStringWithError()``  ``PyDict_GetItemStringRef()``
+  ``_PyEval_RequestCodeExtraIndex()``   ``PyUnstable_Eval_RequestCodeExtraIndex()``
+  ``_PyHASH_BITS``                      ``PyHASH_BITS``
+  ``_PyHASH_IMAG``                      ``PyHASH_IMAG``
+  ``_PyHASH_INF``                       ``PyHASH_INF``
+  ``_PyHASH_MODULUS``                   ``PyHASH_MODULUS``
+  ``_PyHASH_MULTIPLIER``                ``PyHASH_MULTIPLIER``
+  ``_PyObject_EXTRA_INIT``              (no longer needed)
+  ``_PyThreadState_UncheckedGet()``     ``PyThreadState_GetUnchecked()``
+  ``_PyUnicode_AsString()``             ``PyUnicode_AsUTF8()``
+  ``_Py_HashPointer()``                 ``Py_HashPointer()``
+  ``_Py_T_OBJECT``                      ``Py_T_OBJECT_EX``
+  ``_Py_WRITE_RESTRICTED``              (no longer needed)
+  ====================================  ==============================
+
+- Soft-deprecate and omit APIs:
+
+  ====================================  ==============================
+  Omitted Deprecated API                Replacement
+  ====================================  ==============================
+  ``PyDict_GetItemWithError()``         ``PyDict_GetItemRef()``
+  ``PyDict_SetDefault()``               ``PyDict_SetDefaultRef()``
+  ``PyMapping_HasKey()``                ``PyMapping_HasKeyWithError()``
+  ``PyMapping_HasKeyString()``          ``PyMapping_HasKeyStringWithError()``
+  ``PyObject_HasAttr()``                ``PyObject_HasAttrWithError()``
+  ``PyObject_HasAttrString()``          ``PyObject_HasAttrStringWithError()``
+  ====================================  ==============================
+
+- Omit ``<structmember.h>`` legacy API:
+
+  The header file ``structmember.h``, which is not included from ``<Python.h>``
+  and must be included separately, will ``#error`` if
+  ``Py_COMPAT_API_VERSION`` is defined.
+  This affects the following API:
+
+  ====================================  ==============================
+  Omitted Deprecated API                Replacement
+  ====================================  ==============================
+  ``T_SHORT``                           ``Py_T_SHORT``
+  ``T_INT``                             ``Py_T_INT``
+  ``T_LONG``                            ``Py_T_LONG``
+  ``T_FLOAT``                           ``Py_T_FLOAT``
+  ``T_DOUBLE``                          ``Py_T_DOUBLE``
+  ``T_STRING``                          ``Py_T_STRING``
+  ``T_OBJECT``                          (``tp_getset``; docs to be written)
+  ``T_CHAR``                            ``Py_T_CHAR``
+  ``T_BYTE``                            ``Py_T_BYTE``
+  ``T_UBYTE``                           ``Py_T_UBYTE``
+  ``T_USHORT``                          ``Py_T_USHORT``
+  ``T_UINT``                            ``Py_T_UINT``
+  ``T_ULONG``                           ``Py_T_ULONG``
+  ``T_STRING_INPLACE``                  ``Py_T_STRING_INPLACE``
+  ``T_BOOL``                            ``Py_T_BOOL``
+  ``T_OBJECT_EX``                       ``Py_T_OBJECT_EX``
+  ``T_LONGLONG``                        ``Py_T_LONGLONG``
+  ``T_ULONGLONG``                       ``Py_T_ULONGLONG``
+  ``T_PYSSIZET``                        ``Py_T_PYSSIZET``
+  ``T_NONE``                            (``tp_getset``; docs to be written)
+  ``READONLY``                          ``Py_READONLY``
+  ``PY_AUDIT_READ``                     ``Py_AUDIT_READ``
+  ``READ_RESTRICTED``                   ``Py_AUDIT_READ``
+  ``PY_WRITE_RESTRICTED``               (no longer needed)
+  ``RESTRICTED``                        ``Py_AUDIT_READ``
+  ====================================  ==============================
+
+- Omit soft deprecated macros:
+
+  ======================  =====================================
+  Omitted Macros          Replacement
+  ======================  =====================================
+  ``Py_IS_NAN()``         ``isnan()`` (C99+ ``<math.h>``)
+  ``Py_IS_INFINITY()``    ``isinf(X)`` (C99+ ``<math.h>``)
+  ``Py_IS_FINITE()``      ``isfinite(X)`` (C99+ ``<math.h>``)
+  ``Py_MEMCPY()``         ``memcpy()`` (C ``<string.h>``)
+  ======================  =====================================
+
+- Soft-deprecate and omit typedefs without the ``Py``/``_Py`` prefix
+  (``getter``, ``setter``, ``allocfunc``, …), in favour of *new* ones
+  that add the prefix (``Py_getter`` , etc.)
+
+- Soft-deprecate and omit macros without the ``Py``/``_Py`` prefix
+  (``METH_O``, ``CO_COROUTINE``, ``FUTURE_ANNOTATIONS``, ``WAIT_LOCK``, …),
+  favour of *new* ones that add the prefix  (``Py_METH_O`` , etc.).
+
+- Any others approved by the C API workgroup
+
+
+If any of these proposed replacements, or associated documentation,
+are not added in time for 3.14.0b1, they'll be omitted with later versions
+of ``Py_COMPAT_API_VERSION``.
+(We expect this for macros generated by ``configure``: ``HAVE_*``, ``WITH_*``,
+``ALIGNOF_*``, ``SIZEOF_*``, and several without a common prefix.)
 
 
 Implementation
 ==============
 
-* `Issue gh-116587 <https://github.com/python/cpython/issues/116587>`_
+TBD
-* PR: `Add Py_COMPAT_API_VERSION and Py_COMPAT_API_VERSION_MAX macros
+
-  <https://github.com/python/cpython/pull/116588>`_
+
+Open issues
+===========
+
+The name ``Py_COMPAT_API_VERSION`` was taken from the earlier PEP;
+it doesn't fit this version.
 
 
 Backwards Compatibility
 =======================
 
-There is no impact on backward compatibility.
+The macro is backwards compatible.
-
+Developers can introduce and update the macro on their own pace, potentially
-Adding ``Py_COMPAT_API_VERSION`` and ``Py_COMPAT_API_VERSION_MAX``
+for one source file at a time.
-macros has no effect on backward compatibility. Only developers setting
-the ``Py_COMPAT_API_VERSION`` macro in their project will be impacted by
-effects of this macro which is the expected behavior.
 
 
 Discussions
@@ -195,6 +380,9 @@ Discussions
   with no known issues
   <https://github.com/capi-workgroup/problems/issues/54>`_
   (June 2023)
+* `Finishing the Great Renaming
+  <https://discuss.python.org/t/finishing-the-great-renaming/54082>`_
+  (May 2024)
 
 
 Prior Art