PEP 652: Maintaining the Stable ABI (#1810)
This commit is contained in:
Petr Viktorin
GitHub
parent
c19863770f
commit
4e2254a52a
|
|
@ -0,0 +1,462 @@
|
|||
PEP: 652
|
||||
Title: Maintaining the Stable ABI
|
||||
Author: Petr Viktorin <encukou@gmail.com>
|
||||
Discussions-To: https://discuss.python.org/t/pre-pep-maintaining-the-stable-abi/6986/
|
||||
Status: Draft
|
||||
Type: Standards Track
|
||||
Content-Type: text/x-rst
|
||||
Created: 09-Feb-2021
|
||||
Python-Version: 3.10
|
||||
|
||||
|
||||
Abstract
|
||||
========
|
||||
|
||||
CPython's Limited C-API and Stable ABI, introduced in :pep:`384`,
|
||||
will be formalized in a single definitive file, tested, and documented.
|
||||
|
||||
|
||||
Motivation
|
||||
==========
|
||||
|
||||
:pep:`384` defined a Limited API and Stable ABI, which allow extenders and
|
||||
embedders of CPython to compile extension modules that are binary-compatible
|
||||
with any subsequent version of 3.x.
|
||||
In theory, this brings several advantages:
|
||||
|
||||
* A module can be built only once per platform and support multiple versions
|
||||
of Python, reducing time, power and maintainer attention needed for builds
|
||||
(in exchange for potentially worse performance).
|
||||
|
||||
* Binary wheels using the Stable ABI work with new versions of CPython
|
||||
throughout the pre-release period, and can be tested in environments where
|
||||
building from source is not practical.
|
||||
|
||||
* As a welcome side effect of the Limited API's hiding of implementation
|
||||
details, this API is becoming a viable target for alternate Python
|
||||
implementations that would be incompatible with the full C API.
|
||||
|
||||
However, in hindsight, PEP 384 and its implementation has several issues:
|
||||
|
||||
* It is ill-defined. According to PEP 384, functions are *opt-out*:
|
||||
all functions not specially marked are part of the Stable ABI.
|
||||
In practice, for Windows there's a list that's *opt-in*.
|
||||
For users there is a ``#define`` that should make only the Stable ABI
|
||||
available, but there is no process that ensures it is kept up-to date.
|
||||
Neither is there a process for updating the documentation.
|
||||
* Until recently, the Stable ABI was not tested at all. It tends to break.
|
||||
For example, changing a function to a macro can break the Stable ABI as the
|
||||
function symbol is removed.
|
||||
* There is no way to deprecate parts of the Limited API.
|
||||
* It is incomplete. Some operations are not available in the Stable ABI,
|
||||
with little reason except "we forgot".
|
||||
(This last point is one the PEP will not help with, however.)
|
||||
|
||||
This PEP defines the Limited API more clearly and introducess process
|
||||
designed to make the Stable ABI and Limited API more useful and robust.
|
||||
|
||||
|
||||
Rationale
|
||||
=========
|
||||
|
||||
This PEP contains a lot of clarifications and definitions, but just one big
|
||||
technical change: the Stable ABI will be explicitly listed in
|
||||
a human-maintained “manifest” file.
|
||||
|
||||
There have been efforts to collect such lists automatically, e.g. by scanning
|
||||
the symbols exported from Python.
|
||||
Such automation might seem easier to maintain than a handcrafted file,
|
||||
but has major issues: for example, the set exported symbols has
|
||||
platform-specific variations.
|
||||
Also, the cost of updating an explicit manifest is small compared
|
||||
to the overall work that should go into changing API that will need to
|
||||
be suppported forever (or until Python 3 reaches end of life, if that
|
||||
comes sooner).
|
||||
|
||||
This PEP proposes automatically generating things *from* the manifest:
|
||||
initially documentation and DLL contents, with later possibilities
|
||||
for also automating tests.
|
||||
|
||||
|
||||
Stable ABI vs. Limited API
|
||||
==========================
|
||||
|
||||
:pep:`384` and this document deal with the *Limited API* and the *Stable ABI*,
|
||||
two related but distinct concepts. In short:
|
||||
|
||||
* The *Stable ABI* is a promise that certain extensions compiled with
|
||||
CPython 3.x will be binary compatible with all subsequent versions
|
||||
of CPython 3.x.
|
||||
* The *Limited API* is a subset of CPython's C API that produces such extensions.
|
||||
|
||||
This section clarifies these terms and defines some of their semantics
|
||||
(either pre-existing or newly proposed here).
|
||||
|
||||
The word “Extensions” is used as a shorthand for all code that uses the
|
||||
Python API, e.g. extension modules or software that embeds Python.
|
||||
|
||||
|
||||
Stable ABI
|
||||
----------
|
||||
|
||||
The CPython *Stable ABI* is a promise that extensions built against
|
||||
a specific Stable ABI version will be compatible with any newer
|
||||
interpreter of the same major version.
|
||||
|
||||
The Stable ABI does not define a complete binary interface:
|
||||
important details like the layout of structures in memory or function
|
||||
calling conventions are determined by the platform and the compiler and
|
||||
its settings.
|
||||
The Stable ABI promise only applies if these lower-details are also stable.
|
||||
|
||||
For example, an extension built with the CPython 3.10 Stable ABI will be usable
|
||||
with CPython 3.11, 3.12, etc.
|
||||
It will not necessarily be compatible with CPython 4.0, nor with CPython 3.10
|
||||
on a different platform.
|
||||
|
||||
The Stable ABI is not generally forward-compatible: an extension built and
|
||||
tested with CPython 3.10 will not generally be compatible with CPython 3.9.
|
||||
|
||||
.. note::
|
||||
For example, starting in Python 3.10, the ``Py_tp_doc`` slot may be set to
|
||||
``NULL``, while in older versions, a ``NULL`` value will likely crash the
|
||||
interpreter.
|
||||
|
||||
The Stable ABI trades performance for its stability.
|
||||
For example, extensions built for a specific CPython version will automatically
|
||||
use faster macros instead of functions in the Stable ABI.
|
||||
|
||||
Future Python versions may deprecate some members of the Stable ABI.
|
||||
Deprecated members will still work, but may suffer from issues like reduced
|
||||
performance or, in the most extreme cases, memory/resource leaks.
|
||||
|
||||
Limited API
|
||||
-----------
|
||||
|
||||
The Stable ABI promise holds for extensions compiled from code that restricts
|
||||
itself to the *Limited API* (application programming interface).
|
||||
The Limited API is a subset of CPython's C API.
|
||||
|
||||
Extensions that target the Limited API should define the preprocessor macro
|
||||
``Py_LIMITED_API`` to either ``3`` or the current ``PYTHON_API_VERSION``.
|
||||
This will enable Stable ABI versions of several functions and limit definitions
|
||||
to the Limited API.
|
||||
(However, note that the macro is not perfect: due to technical issues or
|
||||
oversights, some non-limited API might be exposed even with it defined.)
|
||||
|
||||
The Limited API is not guaranteed to be *stable*.
|
||||
In the future, parts of the Limited API may be deprecated.
|
||||
They may even be removed, as long as the *Stable ABI* is kept
|
||||
stable and Python's general backwards compatibility policy, :pep:`387`,
|
||||
is followed.
|
||||
|
||||
.. note::
|
||||
|
||||
For example, a function declaration might be removed from public header
|
||||
files but kept in the library.
|
||||
This is currently a possibility for the future; this PEP does not to propose
|
||||
a concrete process for deprecations and removals.
|
||||
|
||||
The goal for the Limited API is to cover everything needed to interact
|
||||
with the interpreter.
|
||||
The main reason to not include a public API in the Limited subset
|
||||
should be that it needs implementation details that change between CPython
|
||||
versions (like struct memory layouts) – usually for performance reasons.
|
||||
|
||||
The Limited API is not limited to CPython. Other implementations are
|
||||
encouraged to implement it and help drive its design.
|
||||
|
||||
|
||||
Specification
|
||||
=============
|
||||
|
||||
To make the Stable ABI more useful and robust, the following changes
|
||||
are proposed.
|
||||
|
||||
|
||||
Stable ABI Manifest
|
||||
-------------------
|
||||
|
||||
All members of the Stable ABI – functions, typedefs, structs, data, macros,
|
||||
and constants – will be explicitly listed in a single "manifest" file,
|
||||
``Misc/stable_abi.txt``.
|
||||
|
||||
For structs, any fields that users of the Stable ABI are allowed to access
|
||||
will be listed explicitly.
|
||||
|
||||
The manifest will also serve as the definitive list of the Limited API.
|
||||
Members that are not part of the Limited API, but are part of the Stable ABI
|
||||
(e.g. ``PyObject.ob_type``, which is accessible by the ``Py_TYPE`` macro),
|
||||
will be annotated as such.
|
||||
|
||||
For items that are only available on some systems, the manifest will record the
|
||||
feature macro that determines their presence (such as ``MS_WINDOWS`` or
|
||||
``HAVE_FORK``).
|
||||
To make the implementation (and usage from non-C languages) easier,
|
||||
all such macros will be simple names.
|
||||
If a future item needs a “negative” macro or complex expression (such as a
|
||||
hypothetical ``#ifndef MACOSX`` or ``#if defined(POSIX) && !defined(LINUX)``),
|
||||
a new feature macro will be derived.
|
||||
|
||||
The format of the manifest will be subject to change whenever needed.
|
||||
It should be consumed only by scripts in the CPython repository.
|
||||
If a stable list is needed, a script can be added to generate it.
|
||||
|
||||
The following wil be generated from the ABI manifest:
|
||||
|
||||
* Source for the Windows shared library, ``PC/python3dll.c``.
|
||||
* Input for documentation (see below).
|
||||
* Test case that checks the runtime availablility of symbols (see below).
|
||||
|
||||
The following will be checked against the Stable ABI manifest as part of
|
||||
continuous integration:
|
||||
|
||||
* The reference count summary, ``Doc/data/refcounts.txt``, includes all
|
||||
function in the Stable ABI (among others).
|
||||
* The functions/structs declared and constants/macros defined
|
||||
when ``Python.h`` is included with ``Py_LIMITED_API`` set.
|
||||
(Initially Linux only; checks on other systems may be added in the future.)
|
||||
|
||||
After the initial implementation, details such as function arguments will be
|
||||
added and the manifest will be checked for internal consistency (e.g. all
|
||||
types used in function signatures are part of the API).
|
||||
|
||||
|
||||
Contents of the Stable ABI
|
||||
--------------------------
|
||||
|
||||
The initial Stable ABI manifest will include:
|
||||
|
||||
* The Stable ABI specified in :pep:`384`.
|
||||
* Everything listed in ``PC/python3dll.c``.
|
||||
* All structs (struct typedefs) which these functions return or take as
|
||||
arguments. (Fields of such structs will not necessarily be added.)
|
||||
* New type slots, such as ``Py_am_aiter``.
|
||||
* The type flags ``Py_TPFLAGS_DEFAULT``, ``Py_TPFLAGS_BASETYPE``,
|
||||
``Py_TPFLAGS_HAVE_GC``, ``Py_TPFLAGS_METHOD_DESCRIPTOR``.
|
||||
* The calling conventions ``METH_*`` (except deprecated ones).
|
||||
* All API needed by macros is the Stable ABI (annotated as not being part of
|
||||
the Limited API).
|
||||
|
||||
Items that are no longer in CPython when this PEP is accepted will be removed
|
||||
from the list.
|
||||
|
||||
Additional items may be aded to the initial manifest according to
|
||||
the checklist below.
|
||||
|
||||
|
||||
Documenting the Limited API
|
||||
---------------------------
|
||||
|
||||
Notes saying “Part of the Limited API” will be added to Python's documentation
|
||||
automatically, in a way similar to the notes on functions that return borrowed
|
||||
references.
|
||||
|
||||
A complete list of all members of the Limited API will also be added to
|
||||
the documentation.
|
||||
|
||||
|
||||
Testing the Stable ABI
|
||||
----------------------
|
||||
|
||||
An automatically generated test module will be added to ensure that all symbols
|
||||
included in the Stable ABI are available at compile time.
|
||||
|
||||
|
||||
Changing the Limited API
|
||||
------------------------
|
||||
|
||||
A checklist for changing the Limited API, including adding new items to it
|
||||
and removing existing ones, will be added to the `Devguide`_.
|
||||
The checklist will 1) mention best practices and common pitfalls in Python
|
||||
C API design and 2) guide the developer around the files that need changing and
|
||||
scripts that need running when the Limited API is changed.
|
||||
|
||||
Below is the initial proposal for the checklist.
|
||||
(After the PEP is accepted, see the Devguide for the current version.)
|
||||
|
||||
Note that the checklist applies to new changes; several items
|
||||
in the *existing* Limited API are grandfathered and couldn't be added today.
|
||||
|
||||
Design considerations:
|
||||
|
||||
* Make sure the change does not break the Stable ABI of any version of Python
|
||||
since 3.5.
|
||||
* Make sure no exposed names are private (i.e. begin with an underscore).
|
||||
* Make sure the new API is well documented.
|
||||
* Make sure the types of all parameters and return values of the added
|
||||
function(s) and all fields of the added struct(s) are be part of the
|
||||
Limited API (or standard C).
|
||||
|
||||
* Make sure the new API and its intended use follows standard C, not just
|
||||
features of currently supported platforms.
|
||||
Specifically, follow the C dialect specified in :pep:`7`.
|
||||
|
||||
* Do not cast a function pointer to ``void*`` (a data pointer) or vice versa.
|
||||
|
||||
* Make sure the new API follows reference counting conventions. (Following them
|
||||
makes the API easier to reason about, and easier use in other Python
|
||||
implementations.)
|
||||
|
||||
* Do not return borrowed references from functions.
|
||||
* Do not steal references to function arguments.
|
||||
|
||||
* Make sure the ownership rules and lifetimes of all applicable struct fields,
|
||||
arguments and return values are well defined.
|
||||
* Think about ease of use for the user. (In C, ease of use itself is not very
|
||||
important; what *is* useful is reducing boilerplate code needed to use the
|
||||
API. Bugs like to hide in boiler plates.)
|
||||
|
||||
* If a function will be often called with specific value for an argument,
|
||||
consider making it default (used when ``NULL`` is passed in).
|
||||
|
||||
* Think about future extensions: for example, if it's possible that future
|
||||
Python versions will need to add a new field to your struct,
|
||||
how will that be done?
|
||||
|
||||
* Make as few assumptions as possible about details that might change in
|
||||
future CPython versions or differ across C API implementations:
|
||||
|
||||
* The GIL
|
||||
* Garbage collection
|
||||
* Memory layout of PyObject, lists/tuples and other structures
|
||||
|
||||
If following these guidelines would hurt performance, add a fast function
|
||||
(or macro) to the non-limited API and a stable equivalent to the Limited API.
|
||||
|
||||
If anything is unclear, or you have a good reason to break the guidelines,
|
||||
consider discussing the change at the `capi-sig`_ mailing list.
|
||||
|
||||
.. _capi-sig: https://mail.python.org/mailman3/lists/capi-sig.python.org/
|
||||
|
||||
Procedure:
|
||||
|
||||
* Move the declaration to a header file directly under ``Include/``, into a
|
||||
``#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 >= 0x03yy0000`` block
|
||||
(with the ``yy`` corresponding to the target CPython version).
|
||||
* Make an entry in the Stable ABI manifest, ``Misc/stable_abi.txt``.
|
||||
* Regenerate the autogenerated files using ``make regen-all``.
|
||||
(or the alternative for non-``make`` platforms)
|
||||
* Build Python and run checks using ``make check-abi``.
|
||||
(or the alternative for non-``make`` platforms)
|
||||
|
||||
|
||||
Advice for Extenders and Embedders
|
||||
----------------------------------
|
||||
|
||||
The following notes will be added to documentation.
|
||||
|
||||
Extension authors should test with all Python versions they support,
|
||||
and preferably build with the lowest such version.
|
||||
|
||||
Compiling with ``Py_LIMITED_API`` defined is *not* a guarantee that your code
|
||||
conforms to the Limited API or the Stable ABI.
|
||||
``Py_LIMITED_API`` only covers definitions, but an API also includes other
|
||||
issues, such as expected semantics.
|
||||
|
||||
Examples of issues that ``Py_LIMITED_API`` does not guard against are:
|
||||
|
||||
* Calling a function with invalid arguments
|
||||
* A function that started accepting ``NULL`` values for an argument
|
||||
in Python 3.9 will fail if ``NULL`` is passed to it under Python 3.8.
|
||||
Only testing with 3.8 (or lower versions) will uncover this issue.
|
||||
* Some structs include a few fields that are part of the Stable ABI and other
|
||||
fields that aren't.
|
||||
``Py_LIMITED_API`` does not filter out such “private” fields.
|
||||
* Code that uses something that is not documented as part of the Stable ABI,
|
||||
but exposed even with ``Py_LIMITED_API`` defined, may break in the future.
|
||||
Despite the team's best efforts, such issues may happen.
|
||||
|
||||
|
||||
Note for Redistributors of Python
|
||||
---------------------------------
|
||||
|
||||
The Stable ABI promise relies on stable underlying ABI details, such as the
|
||||
layout of structures in memory and function calling conventions, which
|
||||
are affected by the compiler and its settings.
|
||||
For the promise to hold, these details must not change between CPython 3.x
|
||||
releases on a particular platform.
|
||||
|
||||
|
||||
|
||||
|
||||
Backwards Compatibility
|
||||
=======================
|
||||
|
||||
Backwards compatibility is one honking great idea!
|
||||
|
||||
This PEP aims at full compatibility with the existing Stable ABI and Limited
|
||||
API, but defines them terms more explicitly.
|
||||
It might not be consistent with some interpretations of what the existing
|
||||
Stable ABI/Limited API is.
|
||||
|
||||
|
||||
Security Implications
|
||||
=====================
|
||||
|
||||
None known.
|
||||
|
||||
|
||||
How to Teach This
|
||||
=================
|
||||
|
||||
Technical documentation will be provided in ``Doc/c-api/stable``
|
||||
and linked from the *What's New* document.
|
||||
Docs for CPython core developers will be added to the devguide.
|
||||
|
||||
|
||||
Reference Implementation
|
||||
========================
|
||||
|
||||
Nothing presentable yet.
|
||||
|
||||
|
||||
Ideas for the Future
|
||||
====================
|
||||
|
||||
The following issues are out of scope of this PEP, but show possible
|
||||
future directions.
|
||||
|
||||
Defining a process for deprecations/removals
|
||||
--------------------------------------------
|
||||
|
||||
While this PEP acknowledges that parts of the Limited API might be deprecated
|
||||
or removed in the future, a process to do this is not in scope, and is left
|
||||
to a possible future PEP.
|
||||
|
||||
|
||||
C syntax for the ABI manifest
|
||||
-----------------------------
|
||||
|
||||
It might be useful to have the ABI manifest be a C header file, or to
|
||||
generate header files from the manifest.
|
||||
Again, either are options for the future.
|
||||
|
||||
|
||||
Open Issues
|
||||
===========
|
||||
|
||||
None so far.
|
||||
|
||||
|
||||
References
|
||||
==========
|
||||
|
||||
.. _Devguide: https://devguide.python.org/
|
||||
|
||||
|
||||
Copyright
|
||||
=========
|
||||
|
||||
This document is placed in the public domain or under the
|
||||
CC0-1.0-Universal license, whichever is more permissive.
|
||||
|
||||
|
||||
..
|
||||
Local Variables:
|
||||
mode: indented-text
|
||||
indent-tabs-mode: nil
|
||||
sentence-end-double-space: t
|
||||
fill-column: 70
|
||||
coding: utf-8
|
||||
End:
|
||||
Loading…
Reference in New Issue