PEP 652: Maintaining the Stable ABI (#1810)
This commit is contained in:
Petr Viktorin
GitHub
parent
c19863770f
commit
4e2254a52a
|
|
@ -0,0 +1,462 @@
|
||||||
|
PEP: 652
|
||||||
|
Title: Maintaining the Stable ABI
|
||||||
|
Author: Petr Viktorin <encukou@gmail.com>
|
||||||
|
Discussions-To: https://discuss.python.org/t/pre-pep-maintaining-the-stable-abi/6986/
|
||||||
|
Status: Draft
|
||||||
|
Type: Standards Track
|
||||||
|
Content-Type: text/x-rst
|
||||||
|
Created: 09-Feb-2021
|
||||||
|
Python-Version: 3.10
|
||||||
|
|
||||||
|
|
||||||
|
Abstract
|
||||||
|
========
|
||||||
|
|
||||||
|
CPython's Limited C-API and Stable ABI, introduced in :pep:`384`,
|
||||||
|
will be formalized in a single definitive file, tested, and documented.
|
||||||
|
|
||||||
|
|
||||||
|
Motivation
|
||||||
|
==========
|
||||||
|
|
||||||
|
:pep:`384` defined a Limited API and Stable ABI, which allow extenders and
|
||||||
|
embedders of CPython to compile extension modules that are binary-compatible
|
||||||
|
with any subsequent version of 3.x.
|
||||||
|
In theory, this brings several advantages:
|
||||||
|
|
||||||
|
* A module can be built only once per platform and support multiple versions
|
||||||
|
of Python, reducing time, power and maintainer attention needed for builds
|
||||||
|
(in exchange for potentially worse performance).
|
||||||
|
|
||||||
|
* Binary wheels using the Stable ABI work with new versions of CPython
|
||||||
|
throughout the pre-release period, and can be tested in environments where
|
||||||
|
building from source is not practical.
|
||||||
|
|
||||||
|
* As a welcome side effect of the Limited API's hiding of implementation
|
||||||
|
details, this API is becoming a viable target for alternate Python
|
||||||
|
implementations that would be incompatible with the full C API.
|
||||||
|
|
||||||
|
However, in hindsight, PEP 384 and its implementation has several issues:
|
||||||
|
|
||||||
|
* It is ill-defined. According to PEP 384, functions are *opt-out*:
|
||||||
|
all functions not specially marked are part of the Stable ABI.
|
||||||
|
In practice, for Windows there's a list that's *opt-in*.
|
||||||
|
For users there is a ``#define`` that should make only the Stable ABI
|
||||||
|
available, but there is no process that ensures it is kept up-to date.
|
||||||
|
Neither is there a process for updating the documentation.
|
||||||
|
* Until recently, the Stable ABI was not tested at all. It tends to break.
|
||||||
|
For example, changing a function to a macro can break the Stable ABI as the
|
||||||
|
function symbol is removed.
|
||||||
|
* There is no way to deprecate parts of the Limited API.
|
||||||
|
* It is incomplete. Some operations are not available in the Stable ABI,
|
||||||
|
with little reason except "we forgot".
|
||||||
|
(This last point is one the PEP will not help with, however.)
|
||||||
|
|
||||||
|
This PEP defines the Limited API more clearly and introducess process
|
||||||
|
designed to make the Stable ABI and Limited API more useful and robust.
|
||||||
|
|
||||||
|
|
||||||
|
Rationale
|
||||||
|
=========
|
||||||
|
|
||||||
|
This PEP contains a lot of clarifications and definitions, but just one big
|
||||||
|
technical change: the Stable ABI will be explicitly listed in
|
||||||
|
a human-maintained “manifest” file.
|
||||||
|
|
||||||
|
There have been efforts to collect such lists automatically, e.g. by scanning
|
||||||
|
the symbols exported from Python.
|
||||||
|
Such automation might seem easier to maintain than a handcrafted file,
|
||||||
|
but has major issues: for example, the set exported symbols has
|
||||||
|
platform-specific variations.
|
||||||
|
Also, the cost of updating an explicit manifest is small compared
|
||||||
|
to the overall work that should go into changing API that will need to
|
||||||
|
be suppported forever (or until Python 3 reaches end of life, if that
|
||||||
|
comes sooner).
|
||||||
|
|
||||||
|
This PEP proposes automatically generating things *from* the manifest:
|
||||||
|
initially documentation and DLL contents, with later possibilities
|
||||||
|
for also automating tests.
|
||||||
|
|
||||||
|
|
||||||
|
Stable ABI vs. Limited API
|
||||||
|
==========================
|
||||||
|
|
||||||
|
:pep:`384` and this document deal with the *Limited API* and the *Stable ABI*,
|
||||||
|
two related but distinct concepts. In short:
|
||||||
|
|
||||||
|
* The *Stable ABI* is a promise that certain extensions compiled with
|
||||||
|
CPython 3.x will be binary compatible with all subsequent versions
|
||||||
|
of CPython 3.x.
|
||||||
|
* The *Limited API* is a subset of CPython's C API that produces such extensions.
|
||||||
|
|
||||||
|
This section clarifies these terms and defines some of their semantics
|
||||||
|
(either pre-existing or newly proposed here).
|
||||||
|
|
||||||
|
The word “Extensions” is used as a shorthand for all code that uses the
|
||||||
|
Python API, e.g. extension modules or software that embeds Python.
|
||||||
|
|
||||||
|
|
||||||
|
Stable ABI
|
||||||
|
----------
|
||||||
|
|
||||||
|
The CPython *Stable ABI* is a promise that extensions built against
|
||||||
|
a specific Stable ABI version will be compatible with any newer
|
||||||
|
interpreter of the same major version.
|
||||||
|
|
||||||
|
The Stable ABI does not define a complete binary interface:
|
||||||
|
important details like the layout of structures in memory or function
|
||||||
|
calling conventions are determined by the platform and the compiler and
|
||||||
|
its settings.
|
||||||
|
The Stable ABI promise only applies if these lower-details are also stable.
|
||||||
|
|
||||||
|
For example, an extension built with the CPython 3.10 Stable ABI will be usable
|
||||||
|
with CPython 3.11, 3.12, etc.
|
||||||
|
It will not necessarily be compatible with CPython 4.0, nor with CPython 3.10
|
||||||
|
on a different platform.
|
||||||
|
|
||||||
|
The Stable ABI is not generally forward-compatible: an extension built and
|
||||||
|
tested with CPython 3.10 will not generally be compatible with CPython 3.9.
|
||||||
|
|
||||||
|
.. note::
|
||||||
|
For example, starting in Python 3.10, the ``Py_tp_doc`` slot may be set to
|
||||||
|
``NULL``, while in older versions, a ``NULL`` value will likely crash the
|
||||||
|
interpreter.
|
||||||
|
|
||||||
|
The Stable ABI trades performance for its stability.
|
||||||
|
For example, extensions built for a specific CPython version will automatically
|
||||||
|
use faster macros instead of functions in the Stable ABI.
|
||||||
|
|
||||||
|
Future Python versions may deprecate some members of the Stable ABI.
|
||||||
|
Deprecated members will still work, but may suffer from issues like reduced
|
||||||
|
performance or, in the most extreme cases, memory/resource leaks.
|
||||||
|
|
||||||
|
Limited API
|
||||||
|
-----------
|
||||||
|
|
||||||
|
The Stable ABI promise holds for extensions compiled from code that restricts
|
||||||
|
itself to the *Limited API* (application programming interface).
|
||||||
|
The Limited API is a subset of CPython's C API.
|
||||||
|
|
||||||
|
Extensions that target the Limited API should define the preprocessor macro
|
||||||
|
``Py_LIMITED_API`` to either ``3`` or the current ``PYTHON_API_VERSION``.
|
||||||
|
This will enable Stable ABI versions of several functions and limit definitions
|
||||||
|
to the Limited API.
|
||||||
|
(However, note that the macro is not perfect: due to technical issues or
|
||||||
|
oversights, some non-limited API might be exposed even with it defined.)
|
||||||
|
|
||||||
|
The Limited API is not guaranteed to be *stable*.
|
||||||
|
In the future, parts of the Limited API may be deprecated.
|
||||||
|
They may even be removed, as long as the *Stable ABI* is kept
|
||||||
|
stable and Python's general backwards compatibility policy, :pep:`387`,
|
||||||
|
is followed.
|
||||||
|
|
||||||
|
.. note::
|
||||||
|
|
||||||
|
For example, a function declaration might be removed from public header
|
||||||
|
files but kept in the library.
|
||||||
|
This is currently a possibility for the future; this PEP does not to propose
|
||||||
|
a concrete process for deprecations and removals.
|
||||||
|
|
||||||
|
The goal for the Limited API is to cover everything needed to interact
|
||||||
|
with the interpreter.
|
||||||
|
The main reason to not include a public API in the Limited subset
|
||||||
|
should be that it needs implementation details that change between CPython
|
||||||
|
versions (like struct memory layouts) – usually for performance reasons.
|
||||||
|
|
||||||
|
The Limited API is not limited to CPython. Other implementations are
|
||||||
|
encouraged to implement it and help drive its design.
|
||||||
|
|
||||||
|
|
||||||
|
Specification
|
||||||
|
=============
|
||||||
|
|
||||||
|
To make the Stable ABI more useful and robust, the following changes
|
||||||
|
are proposed.
|
||||||
|
|
||||||
|
|
||||||
|
Stable ABI Manifest
|
||||||
|
-------------------
|
||||||
|
|
||||||
|
All members of the Stable ABI – functions, typedefs, structs, data, macros,
|
||||||
|
and constants – will be explicitly listed in a single "manifest" file,
|
||||||
|
``Misc/stable_abi.txt``.
|
||||||
|
|
||||||
|
For structs, any fields that users of the Stable ABI are allowed to access
|
||||||
|
will be listed explicitly.
|
||||||
|
|
||||||
|
The manifest will also serve as the definitive list of the Limited API.
|
||||||
|
Members that are not part of the Limited API, but are part of the Stable ABI
|
||||||
|
(e.g. ``PyObject.ob_type``, which is accessible by the ``Py_TYPE`` macro),
|
||||||
|
will be annotated as such.
|
||||||
|
|
||||||
|
For items that are only available on some systems, the manifest will record the
|
||||||
|
feature macro that determines their presence (such as ``MS_WINDOWS`` or
|
||||||
|
``HAVE_FORK``).
|
||||||
|
To make the implementation (and usage from non-C languages) easier,
|
||||||
|
all such macros will be simple names.
|
||||||
|
If a future item needs a “negative” macro or complex expression (such as a
|
||||||
|
hypothetical ``#ifndef MACOSX`` or ``#if defined(POSIX) && !defined(LINUX)``),
|
||||||
|
a new feature macro will be derived.
|
||||||
|
|
||||||
|
The format of the manifest will be subject to change whenever needed.
|
||||||
|
It should be consumed only by scripts in the CPython repository.
|
||||||
|
If a stable list is needed, a script can be added to generate it.
|
||||||
|
|
||||||
|
The following wil be generated from the ABI manifest:
|
||||||
|
|
||||||
|
* Source for the Windows shared library, ``PC/python3dll.c``.
|
||||||
|
* Input for documentation (see below).
|
||||||
|
* Test case that checks the runtime availablility of symbols (see below).
|
||||||
|
|
||||||
|
The following will be checked against the Stable ABI manifest as part of
|
||||||
|
continuous integration:
|
||||||
|
|
||||||
|
* The reference count summary, ``Doc/data/refcounts.txt``, includes all
|
||||||
|
function in the Stable ABI (among others).
|
||||||
|
* The functions/structs declared and constants/macros defined
|
||||||
|
when ``Python.h`` is included with ``Py_LIMITED_API`` set.
|
||||||
|
(Initially Linux only; checks on other systems may be added in the future.)
|
||||||
|
|
||||||
|
After the initial implementation, details such as function arguments will be
|
||||||
|
added and the manifest will be checked for internal consistency (e.g. all
|
||||||
|
types used in function signatures are part of the API).
|
||||||
|
|
||||||
|
|
||||||
|
Contents of the Stable ABI
|
||||||
|
--------------------------
|
||||||
|
|
||||||
|
The initial Stable ABI manifest will include:
|
||||||
|
|
||||||
|
* The Stable ABI specified in :pep:`384`.
|
||||||
|
* Everything listed in ``PC/python3dll.c``.
|
||||||
|
* All structs (struct typedefs) which these functions return or take as
|
||||||
|
arguments. (Fields of such structs will not necessarily be added.)
|
||||||
|
* New type slots, such as ``Py_am_aiter``.
|
||||||
|
* The type flags ``Py_TPFLAGS_DEFAULT``, ``Py_TPFLAGS_BASETYPE``,
|
||||||
|
``Py_TPFLAGS_HAVE_GC``, ``Py_TPFLAGS_METHOD_DESCRIPTOR``.
|
||||||
|
* The calling conventions ``METH_*`` (except deprecated ones).
|
||||||
|
* All API needed by macros is the Stable ABI (annotated as not being part of
|
||||||
|
the Limited API).
|
||||||
|
|
||||||
|
Items that are no longer in CPython when this PEP is accepted will be removed
|
||||||
|
from the list.
|
||||||
|
|
||||||
|
Additional items may be aded to the initial manifest according to
|
||||||
|
the checklist below.
|
||||||
|
|
||||||
|
|
||||||
|
Documenting the Limited API
|
||||||
|
---------------------------
|
||||||
|
|
||||||
|
Notes saying “Part of the Limited API” will be added to Python's documentation
|
||||||
|
automatically, in a way similar to the notes on functions that return borrowed
|
||||||
|
references.
|
||||||
|
|
||||||
|
A complete list of all members of the Limited API will also be added to
|
||||||
|
the documentation.
|
||||||
|
|
||||||
|
|
||||||
|
Testing the Stable ABI
|
||||||
|
----------------------
|
||||||
|
|
||||||
|
An automatically generated test module will be added to ensure that all symbols
|
||||||
|
included in the Stable ABI are available at compile time.
|
||||||
|
|
||||||
|
|
||||||
|
Changing the Limited API
|
||||||
|
------------------------
|
||||||
|
|
||||||
|
A checklist for changing the Limited API, including adding new items to it
|
||||||
|
and removing existing ones, will be added to the `Devguide`_.
|
||||||
|
The checklist will 1) mention best practices and common pitfalls in Python
|
||||||
|
C API design and 2) guide the developer around the files that need changing and
|
||||||
|
scripts that need running when the Limited API is changed.
|
||||||
|
|
||||||
|
Below is the initial proposal for the checklist.
|
||||||
|
(After the PEP is accepted, see the Devguide for the current version.)
|
||||||
|
|
||||||
|
Note that the checklist applies to new changes; several items
|
||||||
|
in the *existing* Limited API are grandfathered and couldn't be added today.
|
||||||
|
|
||||||
|
Design considerations:
|
||||||
|
|
||||||
|
* Make sure the change does not break the Stable ABI of any version of Python
|
||||||
|
since 3.5.
|
||||||
|
* Make sure no exposed names are private (i.e. begin with an underscore).
|
||||||
|
* Make sure the new API is well documented.
|
||||||
|
* Make sure the types of all parameters and return values of the added
|
||||||
|
function(s) and all fields of the added struct(s) are be part of the
|
||||||
|
Limited API (or standard C).
|
||||||
|
|
||||||
|
* Make sure the new API and its intended use follows standard C, not just
|
||||||
|
features of currently supported platforms.
|
||||||
|
Specifically, follow the C dialect specified in :pep:`7`.
|
||||||
|
|
||||||
|
* Do not cast a function pointer to ``void*`` (a data pointer) or vice versa.
|
||||||
|
|
||||||
|
* Make sure the new API follows reference counting conventions. (Following them
|
||||||
|
makes the API easier to reason about, and easier use in other Python
|
||||||
|
implementations.)
|
||||||
|
|
||||||
|
* Do not return borrowed references from functions.
|
||||||
|
* Do not steal references to function arguments.
|
||||||
|
|
||||||
|
* Make sure the ownership rules and lifetimes of all applicable struct fields,
|
||||||
|
arguments and return values are well defined.
|
||||||
|
* Think about ease of use for the user. (In C, ease of use itself is not very
|
||||||
|
important; what *is* useful is reducing boilerplate code needed to use the
|
||||||
|
API. Bugs like to hide in boiler plates.)
|
||||||
|
|
||||||
|
* If a function will be often called with specific value for an argument,
|
||||||
|
consider making it default (used when ``NULL`` is passed in).
|
||||||
|
|
||||||
|
* Think about future extensions: for example, if it's possible that future
|
||||||
|
Python versions will need to add a new field to your struct,
|
||||||
|
how will that be done?
|
||||||
|
|
||||||
|
* Make as few assumptions as possible about details that might change in
|
||||||
|
future CPython versions or differ across C API implementations:
|
||||||
|
|
||||||
|
* The GIL
|
||||||
|
* Garbage collection
|
||||||
|
* Memory layout of PyObject, lists/tuples and other structures
|
||||||
|
|
||||||
|
If following these guidelines would hurt performance, add a fast function
|
||||||
|
(or macro) to the non-limited API and a stable equivalent to the Limited API.
|
||||||
|
|
||||||
|
If anything is unclear, or you have a good reason to break the guidelines,
|
||||||
|
consider discussing the change at the `capi-sig`_ mailing list.
|
||||||
|
|
||||||
|
.. _capi-sig: https://mail.python.org/mailman3/lists/capi-sig.python.org/
|
||||||
|
|
||||||
|
Procedure:
|
||||||
|
|
||||||
|
* Move the declaration to a header file directly under ``Include/``, into a
|
||||||
|
``#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 >= 0x03yy0000`` block
|
||||||
|
(with the ``yy`` corresponding to the target CPython version).
|
||||||
|
* Make an entry in the Stable ABI manifest, ``Misc/stable_abi.txt``.
|
||||||
|
* Regenerate the autogenerated files using ``make regen-all``.
|
||||||
|
(or the alternative for non-``make`` platforms)
|
||||||
|
* Build Python and run checks using ``make check-abi``.
|
||||||
|
(or the alternative for non-``make`` platforms)
|
||||||
|
|
||||||
|
|
||||||
|
Advice for Extenders and Embedders
|
||||||
|
----------------------------------
|
||||||
|
|
||||||
|
The following notes will be added to documentation.
|
||||||
|
|
||||||
|
Extension authors should test with all Python versions they support,
|
||||||
|
and preferably build with the lowest such version.
|
||||||
|
|
||||||
|
Compiling with ``Py_LIMITED_API`` defined is *not* a guarantee that your code
|
||||||
|
conforms to the Limited API or the Stable ABI.
|
||||||
|
``Py_LIMITED_API`` only covers definitions, but an API also includes other
|
||||||
|
issues, such as expected semantics.
|
||||||
|
|
||||||
|
Examples of issues that ``Py_LIMITED_API`` does not guard against are:
|
||||||
|
|
||||||
|
* Calling a function with invalid arguments
|
||||||
|
* A function that started accepting ``NULL`` values for an argument
|
||||||
|
in Python 3.9 will fail if ``NULL`` is passed to it under Python 3.8.
|
||||||
|
Only testing with 3.8 (or lower versions) will uncover this issue.
|
||||||
|
* Some structs include a few fields that are part of the Stable ABI and other
|
||||||
|
fields that aren't.
|
||||||
|
``Py_LIMITED_API`` does not filter out such “private” fields.
|
||||||
|
* Code that uses something that is not documented as part of the Stable ABI,
|
||||||
|
but exposed even with ``Py_LIMITED_API`` defined, may break in the future.
|
||||||
|
Despite the team's best efforts, such issues may happen.
|
||||||
|
|
||||||
|
|
||||||
|
Note for Redistributors of Python
|
||||||
|
---------------------------------
|
||||||
|
|
||||||
|
The Stable ABI promise relies on stable underlying ABI details, such as the
|
||||||
|
layout of structures in memory and function calling conventions, which
|
||||||
|
are affected by the compiler and its settings.
|
||||||
|
For the promise to hold, these details must not change between CPython 3.x
|
||||||
|
releases on a particular platform.
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
Backwards Compatibility
|
||||||
|
=======================
|
||||||
|
|
||||||
|
Backwards compatibility is one honking great idea!
|
||||||
|
|
||||||
|
This PEP aims at full compatibility with the existing Stable ABI and Limited
|
||||||
|
API, but defines them terms more explicitly.
|
||||||
|
It might not be consistent with some interpretations of what the existing
|
||||||
|
Stable ABI/Limited API is.
|
||||||
|
|
||||||
|
|
||||||
|
Security Implications
|
||||||
|
=====================
|
||||||
|
|
||||||
|
None known.
|
||||||
|
|
||||||
|
|
||||||
|
How to Teach This
|
||||||
|
=================
|
||||||
|
|
||||||
|
Technical documentation will be provided in ``Doc/c-api/stable``
|
||||||
|
and linked from the *What's New* document.
|
||||||
|
Docs for CPython core developers will be added to the devguide.
|
||||||
|
|
||||||
|
|
||||||
|
Reference Implementation
|
||||||
|
========================
|
||||||
|
|
||||||
|
Nothing presentable yet.
|
||||||
|
|
||||||
|
|
||||||
|
Ideas for the Future
|
||||||
|
====================
|
||||||
|
|
||||||
|
The following issues are out of scope of this PEP, but show possible
|
||||||
|
future directions.
|
||||||
|
|
||||||
|
Defining a process for deprecations/removals
|
||||||
|
--------------------------------------------
|
||||||
|
|
||||||
|
While this PEP acknowledges that parts of the Limited API might be deprecated
|
||||||
|
or removed in the future, a process to do this is not in scope, and is left
|
||||||
|
to a possible future PEP.
|
||||||
|
|
||||||
|
|
||||||
|
C syntax for the ABI manifest
|
||||||
|
-----------------------------
|
||||||
|
|
||||||
|
It might be useful to have the ABI manifest be a C header file, or to
|
||||||
|
generate header files from the manifest.
|
||||||
|
Again, either are options for the future.
|
||||||
|
|
||||||
|
|
||||||
|
Open Issues
|
||||||
|
===========
|
||||||
|
|
||||||
|
None so far.
|
||||||
|
|
||||||
|
|
||||||
|
References
|
||||||
|
==========
|
||||||
|
|
||||||
|
.. _Devguide: https://devguide.python.org/
|
||||||
|
|
||||||
|
|
||||||
|
Copyright
|
||||||
|
=========
|
||||||
|
|
||||||
|
This document is placed in the public domain or under the
|
||||||
|
CC0-1.0-Universal license, whichever is more permissive.
|
||||||
|
|
||||||
|
|
||||||
|
..
|
||||||
|
Local Variables:
|
||||||
|
mode: indented-text
|
||||||
|
indent-tabs-mode: nil
|
||||||
|
sentence-end-double-space: t
|
||||||
|
fill-column: 70
|
||||||
|
coding: utf-8
|
||||||
|
End:
|
||||||
Loading…
Reference in New Issue