From 4e2254a52ad9e23d6b8217886b7bc11603722ea0 Mon Sep 17 00:00:00 2001 From: Petr Viktorin Date: Wed, 10 Feb 2021 00:52:48 +0100 Subject: [PATCH] PEP 652: Maintaining the Stable ABI (#1810) --- pep-0652.rst | 462 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 462 insertions(+) create mode 100644 pep-0652.rst diff --git a/pep-0652.rst b/pep-0652.rst new file mode 100644 index 000000000..5f0c72f70 --- /dev/null +++ b/pep-0652.rst @@ -0,0 +1,462 @@ +PEP: 652 +Title: Maintaining the Stable ABI +Author: Petr Viktorin +Discussions-To: https://discuss.python.org/t/pre-pep-maintaining-the-stable-abi/6986/ +Status: Draft +Type: Standards Track +Content-Type: text/x-rst +Created: 09-Feb-2021 +Python-Version: 3.10 + + +Abstract +======== + +CPython's Limited C-API and Stable ABI, introduced in :pep:`384`, +will be formalized in a single definitive file, tested, and documented. + + +Motivation +========== + +:pep:`384` defined a Limited API and Stable ABI, which allow extenders and +embedders of CPython to compile extension modules that are binary-compatible +with any subsequent version of 3.x. +In theory, this brings several advantages: + +* A module can be built only once per platform and support multiple versions + of Python, reducing time, power and maintainer attention needed for builds + (in exchange for potentially worse performance). + +* Binary wheels using the Stable ABI work with new versions of CPython + throughout the pre-release period, and can be tested in environments where + building from source is not practical. + +* As a welcome side effect of the Limited API's hiding of implementation + details, this API is becoming a viable target for alternate Python + implementations that would be incompatible with the full C API. + +However, in hindsight, PEP 384 and its implementation has several issues: + +* It is ill-defined. According to PEP 384, functions are *opt-out*: + all functions not specially marked are part of the Stable ABI. + In practice, for Windows there's a list that's *opt-in*. + For users there is a ``#define`` that should make only the Stable ABI + available, but there is no process that ensures it is kept up-to date. + Neither is there a process for updating the documentation. +* Until recently, the Stable ABI was not tested at all. It tends to break. + For example, changing a function to a macro can break the Stable ABI as the + function symbol is removed. +* There is no way to deprecate parts of the Limited API. +* It is incomplete. Some operations are not available in the Stable ABI, + with little reason except "we forgot". + (This last point is one the PEP will not help with, however.) + +This PEP defines the Limited API more clearly and introducess process +designed to make the Stable ABI and Limited API more useful and robust. + + +Rationale +========= + +This PEP contains a lot of clarifications and definitions, but just one big +technical change: the Stable ABI will be explicitly listed in +a human-maintained “manifest” file. + +There have been efforts to collect such lists automatically, e.g. by scanning +the symbols exported from Python. +Such automation might seem easier to maintain than a handcrafted file, +but has major issues: for example, the set exported symbols has +platform-specific variations. +Also, the cost of updating an explicit manifest is small compared +to the overall work that should go into changing API that will need to +be suppported forever (or until Python 3 reaches end of life, if that +comes sooner). + +This PEP proposes automatically generating things *from* the manifest: +initially documentation and DLL contents, with later possibilities +for also automating tests. + + +Stable ABI vs. Limited API +========================== + +:pep:`384` and this document deal with the *Limited API* and the *Stable ABI*, +two related but distinct concepts. In short: + +* The *Stable ABI* is a promise that certain extensions compiled with + CPython 3.x will be binary compatible with all subsequent versions + of CPython 3.x. +* The *Limited API* is a subset of CPython's C API that produces such extensions. + +This section clarifies these terms and defines some of their semantics +(either pre-existing or newly proposed here). + +The word “Extensions” is used as a shorthand for all code that uses the +Python API, e.g. extension modules or software that embeds Python. + + +Stable ABI +---------- + +The CPython *Stable ABI* is a promise that extensions built against +a specific Stable ABI version will be compatible with any newer +interpreter of the same major version. + +The Stable ABI does not define a complete binary interface: +important details like the layout of structures in memory or function +calling conventions are determined by the platform and the compiler and +its settings. +The Stable ABI promise only applies if these lower-details are also stable. + +For example, an extension built with the CPython 3.10 Stable ABI will be usable +with CPython 3.11, 3.12, etc. +It will not necessarily be compatible with CPython 4.0, nor with CPython 3.10 +on a different platform. + +The Stable ABI is not generally forward-compatible: an extension built and +tested with CPython 3.10 will not generally be compatible with CPython 3.9. + +.. note:: + For example, starting in Python 3.10, the ``Py_tp_doc`` slot may be set to + ``NULL``, while in older versions, a ``NULL`` value will likely crash the + interpreter. + +The Stable ABI trades performance for its stability. +For example, extensions built for a specific CPython version will automatically +use faster macros instead of functions in the Stable ABI. + +Future Python versions may deprecate some members of the Stable ABI. +Deprecated members will still work, but may suffer from issues like reduced +performance or, in the most extreme cases, memory/resource leaks. + +Limited API +----------- + +The Stable ABI promise holds for extensions compiled from code that restricts +itself to the *Limited API* (application programming interface). +The Limited API is a subset of CPython's C API. + +Extensions that target the Limited API should define the preprocessor macro +``Py_LIMITED_API`` to either ``3`` or the current ``PYTHON_API_VERSION``. +This will enable Stable ABI versions of several functions and limit definitions +to the Limited API. +(However, note that the macro is not perfect: due to technical issues or +oversights, some non-limited API might be exposed even with it defined.) + +The Limited API is not guaranteed to be *stable*. +In the future, parts of the Limited API may be deprecated. +They may even be removed, as long as the *Stable ABI* is kept +stable and Python's general backwards compatibility policy, :pep:`387`, +is followed. + +.. note:: + + For example, a function declaration might be removed from public header + files but kept in the library. + This is currently a possibility for the future; this PEP does not to propose + a concrete process for deprecations and removals. + +The goal for the Limited API is to cover everything needed to interact +with the interpreter. +The main reason to not include a public API in the Limited subset +should be that it needs implementation details that change between CPython +versions (like struct memory layouts) – usually for performance reasons. + +The Limited API is not limited to CPython. Other implementations are +encouraged to implement it and help drive its design. + + +Specification +============= + +To make the Stable ABI more useful and robust, the following changes +are proposed. + + +Stable ABI Manifest +------------------- + +All members of the Stable ABI – functions, typedefs, structs, data, macros, +and constants – will be explicitly listed in a single "manifest" file, +``Misc/stable_abi.txt``. + +For structs, any fields that users of the Stable ABI are allowed to access +will be listed explicitly. + +The manifest will also serve as the definitive list of the Limited API. +Members that are not part of the Limited API, but are part of the Stable ABI +(e.g. ``PyObject.ob_type``, which is accessible by the ``Py_TYPE`` macro), +will be annotated as such. + +For items that are only available on some systems, the manifest will record the +feature macro that determines their presence (such as ``MS_WINDOWS`` or +``HAVE_FORK``). +To make the implementation (and usage from non-C languages) easier, +all such macros will be simple names. +If a future item needs a “negative” macro or complex expression (such as a +hypothetical ``#ifndef MACOSX`` or ``#if defined(POSIX) && !defined(LINUX)``), +a new feature macro will be derived. + +The format of the manifest will be subject to change whenever needed. +It should be consumed only by scripts in the CPython repository. +If a stable list is needed, a script can be added to generate it. + +The following wil be generated from the ABI manifest: + +* Source for the Windows shared library, ``PC/python3dll.c``. +* Input for documentation (see below). +* Test case that checks the runtime availablility of symbols (see below). + +The following will be checked against the Stable ABI manifest as part of +continuous integration: + +* The reference count summary, ``Doc/data/refcounts.txt``, includes all + function in the Stable ABI (among others). +* The functions/structs declared and constants/macros defined + when ``Python.h`` is included with ``Py_LIMITED_API`` set. + (Initially Linux only; checks on other systems may be added in the future.) + +After the initial implementation, details such as function arguments will be +added and the manifest will be checked for internal consistency (e.g. all +types used in function signatures are part of the API). + + +Contents of the Stable ABI +-------------------------- + +The initial Stable ABI manifest will include: + +* The Stable ABI specified in :pep:`384`. +* Everything listed in ``PC/python3dll.c``. +* All structs (struct typedefs) which these functions return or take as + arguments. (Fields of such structs will not necessarily be added.) +* New type slots, such as ``Py_am_aiter``. +* The type flags ``Py_TPFLAGS_DEFAULT``, ``Py_TPFLAGS_BASETYPE``, + ``Py_TPFLAGS_HAVE_GC``, ``Py_TPFLAGS_METHOD_DESCRIPTOR``. +* The calling conventions ``METH_*`` (except deprecated ones). +* All API needed by macros is the Stable ABI (annotated as not being part of + the Limited API). + +Items that are no longer in CPython when this PEP is accepted will be removed +from the list. + +Additional items may be aded to the initial manifest according to +the checklist below. + + +Documenting the Limited API +--------------------------- + +Notes saying “Part of the Limited API” will be added to Python's documentation +automatically, in a way similar to the notes on functions that return borrowed +references. + +A complete list of all members of the Limited API will also be added to +the documentation. + + +Testing the Stable ABI +---------------------- + +An automatically generated test module will be added to ensure that all symbols +included in the Stable ABI are available at compile time. + + +Changing the Limited API +------------------------ + +A checklist for changing the Limited API, including adding new items to it +and removing existing ones, will be added to the `Devguide`_. +The checklist will 1) mention best practices and common pitfalls in Python +C API design and 2) guide the developer around the files that need changing and +scripts that need running when the Limited API is changed. + +Below is the initial proposal for the checklist. +(After the PEP is accepted, see the Devguide for the current version.) + +Note that the checklist applies to new changes; several items +in the *existing* Limited API are grandfathered and couldn't be added today. + +Design considerations: + +* Make sure the change does not break the Stable ABI of any version of Python + since 3.5. +* Make sure no exposed names are private (i.e. begin with an underscore). +* Make sure the new API is well documented. +* Make sure the types of all parameters and return values of the added + function(s) and all fields of the added struct(s) are be part of the + Limited API (or standard C). + +* Make sure the new API and its intended use follows standard C, not just + features of currently supported platforms. + Specifically, follow the C dialect specified in :pep:`7`. + + * Do not cast a function pointer to ``void*`` (a data pointer) or vice versa. + +* Make sure the new API follows reference counting conventions. (Following them + makes the API easier to reason about, and easier use in other Python + implementations.) + + * Do not return borrowed references from functions. + * Do not steal references to function arguments. + +* Make sure the ownership rules and lifetimes of all applicable struct fields, + arguments and return values are well defined. +* Think about ease of use for the user. (In C, ease of use itself is not very + important; what *is* useful is reducing boilerplate code needed to use the + API. Bugs like to hide in boiler plates.) + + * If a function will be often called with specific value for an argument, + consider making it default (used when ``NULL`` is passed in). + +* Think about future extensions: for example, if it's possible that future + Python versions will need to add a new field to your struct, + how will that be done? + +* Make as few assumptions as possible about details that might change in + future CPython versions or differ across C API implementations: + + * The GIL + * Garbage collection + * Memory layout of PyObject, lists/tuples and other structures + +If following these guidelines would hurt performance, add a fast function +(or macro) to the non-limited API and a stable equivalent to the Limited API. + +If anything is unclear, or you have a good reason to break the guidelines, +consider discussing the change at the `capi-sig`_ mailing list. + +.. _capi-sig: https://mail.python.org/mailman3/lists/capi-sig.python.org/ + +Procedure: + +* Move the declaration to a header file directly under ``Include/``, into a + ``#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 >= 0x03yy0000`` block + (with the ``yy`` corresponding to the target CPython version). +* Make an entry in the Stable ABI manifest, ``Misc/stable_abi.txt``. +* Regenerate the autogenerated files using ``make regen-all``. + (or the alternative for non-``make`` platforms) +* Build Python and run checks using ``make check-abi``. + (or the alternative for non-``make`` platforms) + + +Advice for Extenders and Embedders +---------------------------------- + +The following notes will be added to documentation. + +Extension authors should test with all Python versions they support, +and preferably build with the lowest such version. + +Compiling with ``Py_LIMITED_API`` defined is *not* a guarantee that your code +conforms to the Limited API or the Stable ABI. +``Py_LIMITED_API`` only covers definitions, but an API also includes other +issues, such as expected semantics. + +Examples of issues that ``Py_LIMITED_API`` does not guard against are: + +* Calling a function with invalid arguments +* A function that started accepting ``NULL`` values for an argument + in Python 3.9 will fail if ``NULL`` is passed to it under Python 3.8. + Only testing with 3.8 (or lower versions) will uncover this issue. +* Some structs include a few fields that are part of the Stable ABI and other + fields that aren't. + ``Py_LIMITED_API`` does not filter out such “private” fields. +* Code that uses something that is not documented as part of the Stable ABI, + but exposed even with ``Py_LIMITED_API`` defined, may break in the future. + Despite the team's best efforts, such issues may happen. + + +Note for Redistributors of Python +--------------------------------- + +The Stable ABI promise relies on stable underlying ABI details, such as the +layout of structures in memory and function calling conventions, which +are affected by the compiler and its settings. +For the promise to hold, these details must not change between CPython 3.x +releases on a particular platform. + + + + +Backwards Compatibility +======================= + +Backwards compatibility is one honking great idea! + +This PEP aims at full compatibility with the existing Stable ABI and Limited +API, but defines them terms more explicitly. +It might not be consistent with some interpretations of what the existing +Stable ABI/Limited API is. + + +Security Implications +===================== + +None known. + + +How to Teach This +================= + +Technical documentation will be provided in ``Doc/c-api/stable`` +and linked from the *What's New* document. +Docs for CPython core developers will be added to the devguide. + + +Reference Implementation +======================== + +Nothing presentable yet. + + +Ideas for the Future +==================== + +The following issues are out of scope of this PEP, but show possible +future directions. + +Defining a process for deprecations/removals +-------------------------------------------- + +While this PEP acknowledges that parts of the Limited API might be deprecated +or removed in the future, a process to do this is not in scope, and is left +to a possible future PEP. + + +C syntax for the ABI manifest +----------------------------- + +It might be useful to have the ABI manifest be a C header file, or to +generate header files from the manifest. +Again, either are options for the future. + + +Open Issues +=========== + +None so far. + + +References +========== + +.. _Devguide: https://devguide.python.org/ + + +Copyright +========= + +This document is placed in the public domain or under the +CC0-1.0-Universal license, whichever is more permissive. + + +.. + Local Variables: + mode: indented-text + indent-tabs-mode: nil + sentence-end-double-space: t + fill-column: 70 + coding: utf-8 + End: