403 lines
17 KiB
ReStructuredText
403 lines
17 KiB
ReStructuredText
PEP: 743
|
|
Title: Add Py_COMPAT_API_VERSION to the Python C API
|
|
Author: Victor Stinner <vstinner@python.org>,
|
|
Petr Viktorin <encukou@gmail.com>,
|
|
PEP-Delegate: C API Working Group
|
|
Discussions-To: https://discuss.python.org/t/pep-743-add-py-compat-api-version-to-the-python-c-api/48243
|
|
Status: Draft
|
|
Type: Standards Track
|
|
Created: 11-Mar-2024
|
|
Python-Version: 3.14
|
|
|
|
.. highlight:: c
|
|
|
|
|
|
Abstract
|
|
========
|
|
|
|
Add ``Py_COMPAT_API_VERSION`` C macro that hides some deprecated and
|
|
soft-deprecated symbols, allowing users to opt out of using API with known
|
|
issues that other API solves.
|
|
The macro is versioned, allowing users to update (or not) on their own pace.
|
|
|
|
Also, add namespaced alternatives for API without the ``Py_`` prefix,
|
|
and soft-deprecate the original names.
|
|
|
|
|
|
Motivation
|
|
==========
|
|
|
|
Some of Python's C API has flaws that are only obvious in hindsight.
|
|
|
|
If an API prevents adding features or optimizations, or presents a serious
|
|
security risk or maintenance burden, we can deprecate and remove it as
|
|
described in :pep:`387`.
|
|
|
|
However, this leaves us with some API that has “sharp edges” -- it works fine
|
|
for its current users, but should be avoided in new code.
|
|
For example:
|
|
|
|
- API that cannot signal an exception, so failures are either ignored or
|
|
exit the process with a fatal error. For example ``PyObject_HasAttr``.
|
|
- API that is not thread-safe, for example by borrowing references from
|
|
mutable objects, or exposing unfinished mutable objects. For example
|
|
``PyDict_GetItemWithError``.
|
|
- API with names that don't use the ``Py``/``_Py`` prefix, and so can clash
|
|
with other code. For example: ``setter``.
|
|
|
|
It is important to note that despite such flaws, it's usually possible
|
|
to use the API correctly. For example, in a single-threaded environment,
|
|
thread safety is not an issue.
|
|
We do not want to break working code, even if it uses API that would be wrong
|
|
in some -- or even *most* -- other contexts.
|
|
|
|
On the other hand, we want to steer users away from such “undesirable” API
|
|
in *new* code, especially if a safer alternative exists.
|
|
|
|
|
|
Adding the ``Py`` prefix
|
|
------------------------
|
|
|
|
Some names defined in CPython headers is not namespaced: it that lacks the
|
|
``Py`` prefix (or a variant: ``_Py``, and alternative capitalizations).
|
|
For example, we declare a function type named simply ``setter``.
|
|
|
|
While such names are not exported in the ABI (as checked by ``make smelly``),
|
|
they can clash with user code and, more importantly, with libraries linked
|
|
to third-party extensions.
|
|
|
|
While it would be possible to provide namespaced aliases and (soft-)deprecate
|
|
these names, the only way to make them not clash with third-party code is to
|
|
not define them in Python headers at all.
|
|
|
|
|
|
Rationale
|
|
=========
|
|
|
|
We want to allow an easy way for users to avoid “undesirable” API if they
|
|
choose to do so.
|
|
|
|
It might be be sufficient to leave this to third-party linters.
|
|
For that we'd need a good way to expose a list of (soft-)deprecated
|
|
API to such linters.
|
|
While adding that, we can -- rather easily -- do the linter's job directly
|
|
in CPython headers, avoiding the neel for an extra tool.
|
|
Unlike Python, C makes it rather easy to limit available API -- for a whole
|
|
project or for each individual source file -- by having users define
|
|
an “opt-in” macro.
|
|
|
|
We already do something similar with ``Py_LIMITED_API``, which limits the
|
|
available API to a subset that compiles to stable ABI. (In hindsight, we should
|
|
have used a different macro name for that particular kind of limiting, but it's
|
|
too late to change that now.)
|
|
|
|
To prevent working code from breaking as we identify more “undesirable” API
|
|
and add safer alternatives to it, the opt-in macro should be *versioned*.
|
|
Users can choose a version they need based on their compatibility requirements,
|
|
and update it at their own pace.
|
|
|
|
To be clear, this mechanism is *not* a replacement for deprecation.
|
|
Deprecation is for API that prevents new features or optimizations, or
|
|
presents a security risk or maintenance burden.
|
|
This mechanism, on the other hand, is meant for cases where “we found
|
|
a slightly better way of doing things” -- perhaps one that's harder to misuse,
|
|
or just has a less misleading name.
|
|
(On a lighter note: many people configure a code quality checker to shout at
|
|
them about the number of blank lines between functions. Let's help them
|
|
identify more substantial “code smells”!)
|
|
|
|
The proposed macro does not *change* any API definitions; it only *hides* them.
|
|
So, if code compiles with the macro, it'll also compile without it, with
|
|
identical behaviour.
|
|
This has implications for core devs: to deal with undesirable behaviour,
|
|
we'll need to introduce new, better API, and *then* discourage the old one.
|
|
In turn, this implies that we should look at an individual API and fix all its
|
|
known issues at once, rather than do codebase-wide sweeps for a single kind of
|
|
issue, so that we avoid multiple renames of the same function.
|
|
|
|
|
|
Adding the ``Py`` prefix
|
|
------------------------
|
|
|
|
An opt-in macro allows us to omit definitions that could clash with
|
|
third-party libraries.
|
|
|
|
|
|
Specification
|
|
=============
|
|
|
|
We introduce a ``Py_COMPAT_API_VERSION`` macro.
|
|
If this macro is defined before ``#include <Python.h>``, some API definitions
|
|
-- as described below -- will be omitted from the Python header files.
|
|
|
|
The macro only omits complete top-level definitions exposed from ``<Python.h>``.
|
|
Other things (the ABI, structure definitions, macro expansions, static inline
|
|
function bodies, etc.) are not affected.
|
|
|
|
The C API working group (:pep:`731`) has authority over the set of omitted
|
|
definitions.
|
|
|
|
The set of omitted definitions will be tied to a particular feature release
|
|
of CPython, and is finalized in each 3.x.0 Beta 1 release.
|
|
In rare cases, entries can be removed (i.e. made available for use) at any
|
|
time.
|
|
|
|
The macro should be defined to a version in the format used by
|
|
``PY_VERSION_HEX``, with the “micro”, “release” and “serial” fields
|
|
set to zero.
|
|
For example, to omit API deemed undesirable in 3.14.0b1, users should define
|
|
``Py_COMPAT_API_VERSION`` to ``0x030e0000``.
|
|
|
|
|
|
Requirements for omitted API
|
|
----------------------------
|
|
|
|
An API that is omitted with ``Py_COMPAT_API_VERSION`` must:
|
|
|
|
- be soft-deprecated (see :pep:`387`);
|
|
- for all known use cases of the API, have a documented alternative
|
|
or workaround;
|
|
- have tests to ensure it keeps working (except for 1:1 renames using
|
|
``#define`` or ``typedef``);
|
|
- be documented (except if it was never mentioned in previous versions of the
|
|
documentation); and
|
|
- be approved by the C API working group. (The WG may give blanket approvals
|
|
for groups of related API; see *Initial set* below for examples.)
|
|
|
|
Note that ``Py_COMPAT_API_VERSION`` is meant for API that can be trivially
|
|
replaced by a better alternative.
|
|
API without a replacement should generally be deprecated instead.
|
|
|
|
|
|
Location
|
|
--------
|
|
|
|
All API definitions omitted by ``Py_COMPAT_API_VERSION`` will be moved to
|
|
a new header, ``Include/legacy.h``.
|
|
|
|
This is meant to help linter authors compile lists, so they can flag the API
|
|
with warnings rather than errors.
|
|
|
|
Note that for simple renaming of source-only constructs (macros, types), we
|
|
expect names to be omitted in the same version -- or the same PR -- that adds
|
|
a replacement.
|
|
This means that the original definition will be renamed, and a ``typedef``
|
|
or ``#define`` for the old name added to ``Include/legacy.h``.
|
|
|
|
|
|
Documentation
|
|
-------------
|
|
|
|
Documentation for omitted API should generally:
|
|
|
|
- appear after the recommended replacement,
|
|
- reference the replacement (e.g. “Similar to X, but…”), and
|
|
- focus on differences from the replacement and migration advice.
|
|
|
|
Exceptions are possible if there is a good reason for them.
|
|
|
|
|
|
Initial set
|
|
-----------
|
|
|
|
The following API will be omitted with ``Py_COMPAT_API_VERSION`` set to
|
|
``0x030e0000`` (3.14) or greater:
|
|
|
|
- Omit API returning borrowed references:
|
|
|
|
==================================== ==============================
|
|
Omitted API Replacement
|
|
==================================== ==============================
|
|
``PyDict_GetItem()`` ``PyDict_GetItemRef()``
|
|
``PyDict_GetItemString()`` ``PyDict_GetItemStringRef()``
|
|
``PyImport_AddModule()`` ``PyImport_AddModuleRef()``
|
|
``PyList_GetItem()`` ``PyList_GetItemRef()``
|
|
==================================== ==============================
|
|
|
|
- Omit deprecated APIs:
|
|
|
|
==================================== ==============================
|
|
Omitted Deprecated API Replacement
|
|
==================================== ==============================
|
|
``PY_FORMAT_SIZE_T`` ``"z"``
|
|
``PY_UNICODE_TYPE`` ``wchar_t``
|
|
``PyCode_GetFirstFree()`` ``PyUnstable_Code_GetFirstFree()``
|
|
``PyCode_New()`` ``PyUnstable_Code_New()``
|
|
``PyCode_NewWithPosOnlyArgs()`` ``PyUnstable_Code_NewWithPosOnlyArgs()``
|
|
``PyImport_ImportModuleNoBlock()`` ``PyImport_ImportModule()``
|
|
``PyMem_DEL()`` ``PyMem_Free()``
|
|
``PyMem_Del()`` ``PyMem_Free()``
|
|
``PyMem_FREE()`` ``PyMem_Free()``
|
|
``PyMem_MALLOC()`` ``PyMem_Malloc()``
|
|
``PyMem_NEW()`` ``PyMem_New()``
|
|
``PyMem_REALLOC()`` ``PyMem_Realloc()``
|
|
``PyMem_RESIZE()`` ``PyMem_Resize()``
|
|
``PyModule_GetFilename()`` ``PyModule_GetFilenameObject()``
|
|
``PyOS_AfterFork()`` ``PyOS_AfterFork_Child()``
|
|
``PyObject_DEL()`` ``PyObject_Free()``
|
|
``PyObject_Del()`` ``PyObject_Free()``
|
|
``PyObject_FREE()`` ``PyObject_Free()``
|
|
``PyObject_MALLOC()`` ``PyObject_Malloc()``
|
|
``PyObject_REALLOC()`` ``PyObject_Realloc()``
|
|
``PySlice_GetIndicesEx()`` (two calls; see current docs)
|
|
``PyThread_ReInitTLS()`` (no longer needed)
|
|
``PyThread_create_key()`` ``PyThread_tss_alloc()``
|
|
``PyThread_delete_key()`` ``PyThread_tss_free()``
|
|
``PyThread_delete_key_value()`` ``PyThread_tss_delete()``
|
|
``PyThread_get_key_value()`` ``PyThread_tss_get()``
|
|
``PyThread_set_key_value()`` ``PyThread_tss_set()``
|
|
``PyUnicode_AsDecodedObject()`` ``PyUnicode_Decode()``
|
|
``PyUnicode_AsDecodedUnicode()`` ``PyUnicode_Decode()``
|
|
``PyUnicode_AsEncodedObject()`` ``PyUnicode_AsEncodedString()``
|
|
``PyUnicode_AsEncodedUnicode()`` ``PyUnicode_AsEncodedString()``
|
|
``PyUnicode_IS_READY()`` (no longer needed)
|
|
``PyUnicode_READY()`` (no longer needed)
|
|
``PyWeakref_GET_OBJECT()`` ``PyWeakref_GetRef()``
|
|
``PyWeakref_GetObject()`` ``PyWeakref_GetRef()``
|
|
``Py_UNICODE`` ``wchar_t``
|
|
``_PyCode_GetExtra()`` ``PyUnstable_Code_GetExtra()``
|
|
``_PyCode_SetExtra()`` ``PyUnstable_Code_SetExtra()``
|
|
``_PyDict_GetItemStringWithError()`` ``PyDict_GetItemStringRef()``
|
|
``_PyEval_RequestCodeExtraIndex()`` ``PyUnstable_Eval_RequestCodeExtraIndex()``
|
|
``_PyHASH_BITS`` ``PyHASH_BITS``
|
|
``_PyHASH_IMAG`` ``PyHASH_IMAG``
|
|
``_PyHASH_INF`` ``PyHASH_INF``
|
|
``_PyHASH_MODULUS`` ``PyHASH_MODULUS``
|
|
``_PyHASH_MULTIPLIER`` ``PyHASH_MULTIPLIER``
|
|
``_PyObject_EXTRA_INIT`` (no longer needed)
|
|
``_PyThreadState_UncheckedGet()`` ``PyThreadState_GetUnchecked()``
|
|
``_PyUnicode_AsString()`` ``PyUnicode_AsUTF8()``
|
|
``_Py_HashPointer()`` ``Py_HashPointer()``
|
|
``_Py_T_OBJECT`` ``Py_T_OBJECT_EX``
|
|
``_Py_WRITE_RESTRICTED`` (no longer needed)
|
|
==================================== ==============================
|
|
|
|
- Soft-deprecate and omit APIs:
|
|
|
|
==================================== ==============================
|
|
Omitted Deprecated API Replacement
|
|
==================================== ==============================
|
|
``PyDict_GetItemWithError()`` ``PyDict_GetItemRef()``
|
|
``PyDict_SetDefault()`` ``PyDict_SetDefaultRef()``
|
|
``PyMapping_HasKey()`` ``PyMapping_HasKeyWithError()``
|
|
``PyMapping_HasKeyString()`` ``PyMapping_HasKeyStringWithError()``
|
|
``PyObject_HasAttr()`` ``PyObject_HasAttrWithError()``
|
|
``PyObject_HasAttrString()`` ``PyObject_HasAttrStringWithError()``
|
|
==================================== ==============================
|
|
|
|
- Omit ``<structmember.h>`` legacy API:
|
|
|
|
The header file ``structmember.h``, which is not included from ``<Python.h>``
|
|
and must be included separately, will ``#error`` if
|
|
``Py_COMPAT_API_VERSION`` is defined.
|
|
This affects the following API:
|
|
|
|
==================================== ==============================
|
|
Omitted Deprecated API Replacement
|
|
==================================== ==============================
|
|
``T_SHORT`` ``Py_T_SHORT``
|
|
``T_INT`` ``Py_T_INT``
|
|
``T_LONG`` ``Py_T_LONG``
|
|
``T_FLOAT`` ``Py_T_FLOAT``
|
|
``T_DOUBLE`` ``Py_T_DOUBLE``
|
|
``T_STRING`` ``Py_T_STRING``
|
|
``T_OBJECT`` (``tp_getset``; docs to be written)
|
|
``T_CHAR`` ``Py_T_CHAR``
|
|
``T_BYTE`` ``Py_T_BYTE``
|
|
``T_UBYTE`` ``Py_T_UBYTE``
|
|
``T_USHORT`` ``Py_T_USHORT``
|
|
``T_UINT`` ``Py_T_UINT``
|
|
``T_ULONG`` ``Py_T_ULONG``
|
|
``T_STRING_INPLACE`` ``Py_T_STRING_INPLACE``
|
|
``T_BOOL`` ``Py_T_BOOL``
|
|
``T_OBJECT_EX`` ``Py_T_OBJECT_EX``
|
|
``T_LONGLONG`` ``Py_T_LONGLONG``
|
|
``T_ULONGLONG`` ``Py_T_ULONGLONG``
|
|
``T_PYSSIZET`` ``Py_T_PYSSIZET``
|
|
``T_NONE`` (``tp_getset``; docs to be written)
|
|
``READONLY`` ``Py_READONLY``
|
|
``PY_AUDIT_READ`` ``Py_AUDIT_READ``
|
|
``READ_RESTRICTED`` ``Py_AUDIT_READ``
|
|
``PY_WRITE_RESTRICTED`` (no longer needed)
|
|
``RESTRICTED`` ``Py_AUDIT_READ``
|
|
==================================== ==============================
|
|
|
|
- Omit soft deprecated macros:
|
|
|
|
====================== =====================================
|
|
Omitted Macros Replacement
|
|
====================== =====================================
|
|
``Py_IS_NAN()`` ``isnan()`` (C99+ ``<math.h>``)
|
|
``Py_IS_INFINITY()`` ``isinf(X)`` (C99+ ``<math.h>``)
|
|
``Py_IS_FINITE()`` ``isfinite(X)`` (C99+ ``<math.h>``)
|
|
``Py_MEMCPY()`` ``memcpy()`` (C ``<string.h>``)
|
|
====================== =====================================
|
|
|
|
- Soft-deprecate and omit typedefs without the ``Py``/``_Py`` prefix
|
|
(``getter``, ``setter``, ``allocfunc``, …), in favour of *new* ones
|
|
that add the prefix (``Py_getter`` , etc.)
|
|
|
|
- Soft-deprecate and omit macros without the ``Py``/``_Py`` prefix
|
|
(``METH_O``, ``CO_COROUTINE``, ``FUTURE_ANNOTATIONS``, ``WAIT_LOCK``, …),
|
|
favour of *new* ones that add the prefix (``Py_METH_O`` , etc.).
|
|
|
|
- Any others approved by the C API workgroup
|
|
|
|
|
|
If any of these proposed replacements, or associated documentation,
|
|
are not added in time for 3.14.0b1, they'll be omitted with later versions
|
|
of ``Py_COMPAT_API_VERSION``.
|
|
(We expect this for macros generated by ``configure``: ``HAVE_*``, ``WITH_*``,
|
|
``ALIGNOF_*``, ``SIZEOF_*``, and several without a common prefix.)
|
|
|
|
|
|
Implementation
|
|
==============
|
|
|
|
TBD
|
|
|
|
|
|
Open issues
|
|
===========
|
|
|
|
The name ``Py_COMPAT_API_VERSION`` was taken from the earlier PEP;
|
|
it doesn't fit this version.
|
|
|
|
|
|
Backwards Compatibility
|
|
=======================
|
|
|
|
The macro is backwards compatible.
|
|
Developers can introduce and update the macro on their own pace, potentially
|
|
for one source file at a time.
|
|
|
|
|
|
Discussions
|
|
===========
|
|
|
|
* C API Evolutions: `Macro to hide deprecated functions
|
|
<https://github.com/capi-workgroup/api-evolution/issues/24>`_
|
|
(October 2023)
|
|
* C API Problems: `Opt-in macro for a new clean API? Subset of functions
|
|
with no known issues
|
|
<https://github.com/capi-workgroup/problems/issues/54>`_
|
|
(June 2023)
|
|
* `Finishing the Great Renaming
|
|
<https://discuss.python.org/t/finishing-the-great-renaming/54082>`_
|
|
(May 2024)
|
|
|
|
|
|
Prior Art
|
|
=========
|
|
|
|
* ``Py_LIMITED_API`` macro of :pep:`384` "Defining a Stable ABI".
|
|
* Rejected :pep:`606` "Python Compatibility Version" which has a global
|
|
scope.
|
|
|
|
|
|
Copyright
|
|
=========
|
|
|
|
This document is placed in the public domain or under the
|
|
CC0-1.0-Universal license, whichever is more permissive.
|