PEP 743: Rewrite to hide (soft-)deprecated API (GH-3869)

Co-authored-by: Victor Stinner <vstinner@python.org>
This commit is contained in:
Petr Viktorin 2024-07-25 14:32:18 +02:00 committed by GitHub
parent 92634ee01f
commit 2eb4fe5313
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
2 changed files with 323 additions and 135 deletions

2
.github/CODEOWNERS vendored
View File

@ -621,7 +621,7 @@ peps/pep-0738.rst @encukou
peps/pep-0740.rst @dstufft
peps/pep-0741.rst @vstinner
peps/pep-0742.rst @JelleZijlstra
peps/pep-0743.rst @vstinner
peps/pep-0743.rst @vstinner @encukou
peps/pep-0744.rst @brandtbucher
peps/pep-0745.rst @hugovk
peps/pep-0746.rst @JelleZijlstra

View File

@ -1,10 +1,11 @@
PEP: 743
Title: Add Py_COMPAT_API_VERSION to the Python C API
Author: Victor Stinner <vstinner@python.org>
Author: Victor Stinner <vstinner@python.org>,
Petr Viktorin <encukou@gmail.com>,
Status: Draft
Type: Standards Track
Created: 11-Mar-2024
Python-Version: 3.13
Python-Version: 3.14
.. highlight:: c
@ -12,177 +13,361 @@ Python-Version: 3.13
Abstract
========
Add ``Py_COMPAT_API_VERSION`` and ``Py_COMPAT_API_VERSION_MAX`` macros
to opt-in for planned incompatible C API changes in a C extension.
Maintainers can decide when they make their C extension compatible
and also decide which future Python version they want to be compatible
with.
Add ``Py_COMPAT_API_VERSION`` C macro that hides some deprecated and
soft-deprecated symbols, allowing users to opt out of using API with known
issues that other API solves.
The macro is versioned, allowing users to update (or not) on their own pace.
Also, add namespaced alternatives for API without the ``Py_`` prefix,
and soft-deprecate the original names.
Motivation
==========
Some of Python's C API has flaws that are only obvious in hindsight.
If an API prevents adding features or optimizations, or presents a serious
security risk or maintenance burden, we can deprecate and remove it as
described in :pep:`387`.
However, this leaves us with some API that has “sharp edges” -- it works fine
for its current users, but should be avoided in new code.
For example:
- API that cannot signal an exception, so failures are either ignored or
exit the process with a fatal error. For example ``PyObject_HasAttr``.
- API that is not thread-safe, for example by borrowing references from
mutable objects, or exposing unfinished mutable objects. For example
``PyDict_GetItemWithError``.
- API with names that don't use the ``Py``/``_Py`` prefix, and so can clash
with other code. For example: ``setter``.
It is important to note that despite such flaws, it's usually possible
to use the API correctly. For example, in a single-threaded environment,
thread safety is not an issue.
We do not want to break working code, even if it uses API that would be wrong
in some -- or even *most* -- other contexts.
On the other hand, we want to steer users away from such “undesirable” API
in *new* code, especially if a safer alternative exists.
Adding the ``Py`` prefix
------------------------
Some names defined in CPython headers is not namespaced: it that lacks the
``Py`` prefix (or a variant: ``_Py``, and alternative capitalizations).
For example, we declare a function type named simply ``setter``.
While such names are not exported in the ABI (as checked by ``make smelly``),
they can clash with user code and, more importantly, with libraries linked
to third-party extensions.
While it would be possible to provide namespaced aliases and (soft-)deprecate
these names, the only way to make them not clash with third-party code is to
not define them in Python headers at all.
Rationale
=========
Python releases enforce C API changes
-------------------------------------
We want to allow an easy way for users to avoid “undesirable” API if they
choose to do so.
Every Python 3.x release has a long list of C API changes, including
incompatible changes. C extensions have to be updated to work on the
newly released Python.
It might be be sufficient to leave this to third-party linters.
For that we'd need a good way to expose a list of (soft-)deprecated
API to such linters.
While adding that, we can -- rather easily -- do the linter's job directly
in CPython headers, avoiding the neel for an extra tool.
Unlike Python, C makes it rather easy to limit available API -- for a whole
project or for each individual source file -- by having users define
an “opt-in” macro.
Some incompatible changes are driven by new features: they cannot be
avoided, unless we decide to not add these features. Other reasons:
We already do something similar with ``Py_LIMITED_API``, which limits the
available API to a subset that compiles to stable ABI. (In hindsight, we should
have used a different macro name for that particular kind of limiting, but it's
too late to change that now.)
* Remove deprecated API (see :pep:`387`).
* Ease the implementation of another change.
* Change or remove error-prone API.
To prevent working code from breaking as we identify more “undesirable” API
and add safer alternatives to it, the opt-in macro should be *versioned*.
Users can choose a version they need based on their compatibility requirements,
and update it at their own pace.
Currently, there is no middle ground between "not change the C API" and
"incompatible C API changes impact everybody". Either a C extension is
updated or the new Python version cannot be used. Such all-or-nothing
deal does not satisfy C extension maintainers nor C extensions users.
To be clear, this mechanism is *not* a replacement for deprecation.
Deprecation is for API that prevents new features or optimizations, or
presents a security risk or maintenance burden.
This mechanism, on the other hand, is meant for cases where “we found
a slightly better way of doing things” -- perhaps one that's harder to misuse,
or just has a less misleading name.
(On a lighter note: many people configure a code quality checker to shout at
them about the number of blank lines between functions. Let's help them
identify more substantial “code smells”!)
The proposed macro does not *change* any API definitions; it only *hides* them.
So, if code compiles with the macro, it'll also compile without it, with
identical behaviour.
This has implications for core devs: to deal with undesirable behaviour,
we'll need to introduce new, better API, and *then* discourage the old one.
In turn, this implies that we should look at an individual API and fix all its
known issues at once, rather than do codebase-wide sweeps for a single kind of
issue, so that we avoid multiple renames of the same function.
Limited C API
-------------
Adding the ``Py`` prefix
------------------------
The limited C API is versioned: the ``Py_LIMITED_API`` macro can be set
to a Python version to select which API is available. On the Python
side, it allows introducing incompatible changes at a specific
``Py_LIMITED_API`` version. For example, if ``Py_LIMITED_API`` is set to
Python 3.11 or newer, the ``<stdio.h>`` is no longer included by
``Python.h``, whereas C extensions targeting Python 3.10 are not
affected.
The difference here is that upgrading Python does not change if
``<stdio.h>`` is included or not, but updating ``Py_LIMITED_API`` does.
Updating ``Py_LIMITED_API`` is an deliberate action made by the C
extension maintainer. It gives more freedom to decide **when** the
maintainer is ready to deal with the latest batch of incompatible
changes.
A similar version can be used with the regular (non-limited) C API.
Deprecation and compiler warnings
---------------------------------
Deprecated functions are marked with ``Py_DEPRECATED()``. Using a
deprecated function emits a compiler warning.
The problem is that ``pip`` and ``build`` tools hide compiler logs by
default, unless a build fails. Moreover, it's easy to miss a single
warning in the middle of hundred lines of logs.
Schedule changes
----------------
Currently, there is no way to schedule a C API change: announce it but
also provide a way to maintainers to test their C extensions with the
change. Either a change is not made, or everybody must update their code
if they want to update Python.
An opt-in macro allows us to omit definitions that could clash with
third-party libraries.
Specification
=============
New macros
----------
We introduce a ``Py_COMPAT_API_VERSION`` macro.
If this macro is defined before ``#include <Python.h>``, some API definitions
-- as described below -- will be omitted from the Python header files.
Add new ``Py_COMPAT_API_VERSION`` and ``Py_COMPAT_API_VERSION_MAX``
macros. They can be set to test if a C extension is prepared for future
C API changes: compatible with future Python versions.
The macro only omits complete top-level definitions exposed from ``<Python.h>``.
Other things (the ABI, structure definitions, macro expansions, static inline
function bodies, etc.) are not affected.
The ``Py_COMPAT_API_VERSION`` macro can be set to a specific Python
version. For example, ``Py_COMPAT_API_VERSION=0x030e0000`` tests C API
changes scheduled in Python 3.14.
The C API working group (:pep:`731`) has authority over the set of omitted
definitions.
If the ``Py_COMPAT_API_VERSION`` macro is set to
``Py_COMPAT_API_VERSION_MAX``, all scheduled C API changes are tested at
once.
The set of omitted definitions will be tied to a particular feature release
of CPython, and is finalized in each 3.x.0 Beta 1 release.
In rare cases, entries can be removed (i.e. made available for use) at any
time.
If the ``Py_COMPAT_API_VERSION`` macro is not set, it is to
``PY_VERSION_HEX`` by default.
The ``Py_COMPAT_API_VERSION`` macro can be set in a single C file or for
a whole project in compiler flags. The macro does not affected other
projects or Python itself.
The macro should be defined to a version in the format used by
``PY_VERSION_HEX``, with the “micro”, “release” and “serial” fields
set to zero.
For example, to omit API deemed undesirable in 3.14.0b1, users should define
``Py_COMPAT_API_VERSION`` to ``0x030e0000``.
Example in Python
-----------------
Requirements for omitted API
----------------------------
For example, the ``PyImport_ImportModuleNoBlock()`` function is
deprecated in Python 3.13 and scheduled for removal in Python 3.15. The
function can be declared in the Python C API with the following
declaration:
An API that is omitted with ``Py_COMPAT_API_VERSION`` must:
.. code-block:: c
- be soft-deprecated (see :pep:`387`);
- for all known use cases of the API, have a documented alternative
or workaround;
- have tests to ensure it keeps working (except for 1:1 renames using
``#define`` or ``typedef``);
- be documented (except if it was never mentioned in previous versions of the
documentation); and
- be approved by the C API working group. (The WG may give blanket approvals
for groups of related API; see *Initial set* below for examples.)
#if Py_COMPAT_API_VERSION < 0x030f0000
Py_DEPRECATED(3.13) PyAPI_FUNC(PyObject *) PyImport_ImportModuleNoBlock(
const char *name /* UTF-8 encoded string */
);
#endif
If ``if Py_COMPAT_API_VERSION`` is equal to or greater than Python 3.15
(``0x030f0000``), the ``PyImport_ImportModuleNoBlock()`` function is not
declared, and so using it fails with a build error.
Goals
-----
* Reduce the number of C API changes affecting C extensions when
updating Python.
* When testing C extensions (for example, optional CI test),
``Py_COMPAT_API_VERSION`` can be set to ``Py_COMPAT_API_VERSION_MAX``
to detect future incompatibilities. For mandatory tests, it is
recommended to set ``Py_COMPAT_API_VERSION`` to a specific Python
version.
* For core developers, make sure that the C API can still evolve
without being afraid of breaking an unknown number of C extensions.
Non-goals
---------
* Freeze the API forever: this is not the stable ABI. For example,
deprecated functions will continue to be removed on a regular basis.
* C extensions maintainers not using ``Py_COMPAT_API_VERSION`` will
still be affected by C API changes when updating Python.
* Provide a stable ABI: the macro only impacts the regular (non-limited)
API.
* Silver bullet solving all C API issues.
Note that ``Py_COMPAT_API_VERSION`` is meant for API that can be trivially
replaced by a better alternative.
API without a replacement should generally be deprecated instead.
Examples of ``Py_COMPAT_API_VERSION`` usages
============================================
Location
--------
* Remove deprecated functions.
* Remove deprecated structure members, such as
``PyBytesObject.ob_shash``.
* Remove a standard ``#include``, such as ``#include <string.h>``,
from ``<Python.h>``.
* Change the behavior of a function or a macro. For example, calling
``PyObject_SetAttr(obj, name, NULL)`` can fail, to enforce the usage
of the ``PyObject_DelAttr()`` function instead to delete an attribute.
All API definitions omitted by ``Py_COMPAT_API_VERSION`` will be moved to
a new header, ``Include/legacy.h``.
This is meant to help linter authors compile lists, so they can flag the API
with warnings rather than errors.
Note that for simple renaming of source-only constructs (macros, types), we
expect names to be omitted in the same version -- or the same PR -- that adds
a replacement.
This means that the original definition will be renamed, and a ``typedef``
or ``#define`` for the old name added to ``Include/legacy.h``.
Documentation
-------------
Documentation for omitted API should generally:
- appear after the recommended replacement,
- reference the replacement (e.g. “Similar to X, but…”), and
- focus on differences from the replacement and migration advice.
Exceptions are possible if there is a good reason for them.
Initial set
-----------
The following API will be omitted with ``Py_COMPAT_API_VERSION`` set to
``0x030e0000`` (3.14) or greater:
- Omit API returning borrowed references:
==================================== ==============================
Omitted API Replacement
==================================== ==============================
``PyDict_GetItem()`` ``PyDict_GetItemRef()``
``PyDict_GetItemString()`` ``PyDict_GetItemStringRef()``
``PyImport_AddModule()`` ``PyImport_AddModuleRef()``
``PyList_GetItem()`` ``PyList_GetItemRef()``
==================================== ==============================
- Omit deprecated APIs:
==================================== ==============================
Omitted Deprecated API Replacement
==================================== ==============================
``PY_FORMAT_SIZE_T`` ``"z"``
``PY_UNICODE_TYPE`` ``wchar_t``
``PyCode_GetFirstFree()`` ``PyUnstable_Code_GetFirstFree()``
``PyCode_New()`` ``PyUnstable_Code_New()``
``PyCode_NewWithPosOnlyArgs()`` ``PyUnstable_Code_NewWithPosOnlyArgs()``
``PyImport_ImportModuleNoBlock()`` ``PyImport_ImportModule()``
``PyMem_DEL()`` ``PyMem_Free()``
``PyMem_Del()`` ``PyMem_Free()``
``PyMem_FREE()`` ``PyMem_Free()``
``PyMem_MALLOC()`` ``PyMem_Malloc()``
``PyMem_NEW()`` ``PyMem_New()``
``PyMem_REALLOC()`` ``PyMem_Realloc()``
``PyMem_RESIZE()`` ``PyMem_Resize()``
``PyModule_GetFilename()`` ``PyModule_GetFilenameObject()``
``PyOS_AfterFork()`` ``PyOS_AfterFork_Child()``
``PyObject_DEL()`` ``PyObject_Free()``
``PyObject_Del()`` ``PyObject_Free()``
``PyObject_FREE()`` ``PyObject_Free()``
``PyObject_MALLOC()`` ``PyObject_Malloc()``
``PyObject_REALLOC()`` ``PyObject_Realloc()``
``PySlice_GetIndicesEx()`` (two calls; see current docs)
``PyThread_ReInitTLS()`` (no longer needed)
``PyThread_create_key()`` ``PyThread_tss_alloc()``
``PyThread_delete_key()`` ``PyThread_tss_free()``
``PyThread_delete_key_value()`` ``PyThread_tss_delete()``
``PyThread_get_key_value()`` ``PyThread_tss_get()``
``PyThread_set_key_value()`` ``PyThread_tss_set()``
``PyUnicode_AsDecodedObject()`` ``PyUnicode_Decode()``
``PyUnicode_AsDecodedUnicode()`` ``PyUnicode_Decode()``
``PyUnicode_AsEncodedObject()`` ``PyUnicode_AsEncodedString()``
``PyUnicode_AsEncodedUnicode()`` ``PyUnicode_AsEncodedString()``
``PyUnicode_IS_READY()`` (no longer needed)
``PyUnicode_READY()`` (no longer needed)
``PyWeakref_GET_OBJECT()`` ``PyWeakref_GetRef()``
``PyWeakref_GetObject()`` ``PyWeakref_GetRef()``
``Py_UNICODE`` ``wchar_t``
``_PyCode_GetExtra()`` ``PyUnstable_Code_GetExtra()``
``_PyCode_SetExtra()`` ``PyUnstable_Code_SetExtra()``
``_PyDict_GetItemStringWithError()`` ``PyDict_GetItemStringRef()``
``_PyEval_RequestCodeExtraIndex()`` ``PyUnstable_Eval_RequestCodeExtraIndex()``
``_PyHASH_BITS`` ``PyHASH_BITS``
``_PyHASH_IMAG`` ``PyHASH_IMAG``
``_PyHASH_INF`` ``PyHASH_INF``
``_PyHASH_MODULUS`` ``PyHASH_MODULUS``
``_PyHASH_MULTIPLIER`` ``PyHASH_MULTIPLIER``
``_PyObject_EXTRA_INIT`` (no longer needed)
``_PyThreadState_UncheckedGet()`` ``PyThreadState_GetUnchecked()``
``_PyUnicode_AsString()`` ``PyUnicode_AsUTF8()``
``_Py_HashPointer()`` ``Py_HashPointer()``
``_Py_T_OBJECT`` ``Py_T_OBJECT_EX``
``_Py_WRITE_RESTRICTED`` (no longer needed)
==================================== ==============================
- Soft-deprecate and omit APIs:
==================================== ==============================
Omitted Deprecated API Replacement
==================================== ==============================
``PyDict_GetItemWithError()`` ``PyDict_GetItemRef()``
``PyDict_SetDefault()`` ``PyDict_SetDefaultRef()``
``PyMapping_HasKey()`` ``PyMapping_HasKeyWithError()``
``PyMapping_HasKeyString()`` ``PyMapping_HasKeyStringWithError()``
``PyObject_HasAttr()`` ``PyObject_HasAttrWithError()``
``PyObject_HasAttrString()`` ``PyObject_HasAttrStringWithError()``
==================================== ==============================
- Omit ``<structmember.h>`` legacy API:
The header file ``structmember.h``, which is not included from ``<Python.h>``
and must be included separately, will ``#error`` if
``Py_COMPAT_API_VERSION`` is defined.
This affects the following API:
==================================== ==============================
Omitted Deprecated API Replacement
==================================== ==============================
``T_SHORT`` ``Py_T_SHORT``
``T_INT`` ``Py_T_INT``
``T_LONG`` ``Py_T_LONG``
``T_FLOAT`` ``Py_T_FLOAT``
``T_DOUBLE`` ``Py_T_DOUBLE``
``T_STRING`` ``Py_T_STRING``
``T_OBJECT`` (``tp_getset``; docs to be written)
``T_CHAR`` ``Py_T_CHAR``
``T_BYTE`` ``Py_T_BYTE``
``T_UBYTE`` ``Py_T_UBYTE``
``T_USHORT`` ``Py_T_USHORT``
``T_UINT`` ``Py_T_UINT``
``T_ULONG`` ``Py_T_ULONG``
``T_STRING_INPLACE`` ``Py_T_STRING_INPLACE``
``T_BOOL`` ``Py_T_BOOL``
``T_OBJECT_EX`` ``Py_T_OBJECT_EX``
``T_LONGLONG`` ``Py_T_LONGLONG``
``T_ULONGLONG`` ``Py_T_ULONGLONG``
``T_PYSSIZET`` ``Py_T_PYSSIZET``
``T_NONE`` (``tp_getset``; docs to be written)
``READONLY`` ``Py_READONLY``
``PY_AUDIT_READ`` ``Py_AUDIT_READ``
``READ_RESTRICTED`` ``Py_AUDIT_READ``
``PY_WRITE_RESTRICTED`` (no longer needed)
``RESTRICTED`` ``Py_AUDIT_READ``
==================================== ==============================
- Omit soft deprecated macros:
====================== =====================================
Omitted Macros Replacement
====================== =====================================
``Py_IS_NAN()`` ``isnan()`` (C99+ ``<math.h>``)
``Py_IS_INFINITY()`` ``isinf(X)`` (C99+ ``<math.h>``)
``Py_IS_FINITE()`` ``isfinite(X)`` (C99+ ``<math.h>``)
``Py_MEMCPY()`` ``memcpy()`` (C ``<string.h>``)
====================== =====================================
- Soft-deprecate and omit typedefs without the ``Py``/``_Py`` prefix
(``getter``, ``setter``, ``allocfunc``, …), in favour of *new* ones
that add the prefix (``Py_getter`` , etc.)
- Soft-deprecate and omit macros without the ``Py``/``_Py`` prefix
(``METH_O``, ``CO_COROUTINE``, ``FUTURE_ANNOTATIONS``, ``WAIT_LOCK``, …),
favour of *new* ones that add the prefix (``Py_METH_O`` , etc.).
- Any others approved by the C API workgroup
If any of these proposed replacements, or associated documentation,
are not added in time for 3.14.0b1, they'll be omitted with later versions
of ``Py_COMPAT_API_VERSION``.
(We expect this for macros generated by ``configure``: ``HAVE_*``, ``WITH_*``,
``ALIGNOF_*``, ``SIZEOF_*``, and several without a common prefix.)
Implementation
==============
* `Issue gh-116587 <https://github.com/python/cpython/issues/116587>`_
* PR: `Add Py_COMPAT_API_VERSION and Py_COMPAT_API_VERSION_MAX macros
<https://github.com/python/cpython/pull/116588>`_
TBD
Open issues
===========
The name ``Py_COMPAT_API_VERSION`` was taken from the earlier PEP;
it doesn't fit this version.
Backwards Compatibility
=======================
There is no impact on backward compatibility.
Adding ``Py_COMPAT_API_VERSION`` and ``Py_COMPAT_API_VERSION_MAX``
macros has no effect on backward compatibility. Only developers setting
the ``Py_COMPAT_API_VERSION`` macro in their project will be impacted by
effects of this macro which is the expected behavior.
The macro is backwards compatible.
Developers can introduce and update the macro on their own pace, potentially
for one source file at a time.
Discussions
@ -195,6 +380,9 @@ Discussions
with no known issues
<https://github.com/capi-workgroup/problems/issues/54>`_
(June 2023)
* `Finishing the Great Renaming
<https://discuss.python.org/t/finishing-the-great-renaming/54082>`_
(May 2024)
Prior Art