python-peps/peps/pep-0689.rst

PEP: 689
Title: Unstable C API tier
Author: Petr Viktorin <encukou@gmail.com>
Discussions-To: https://discuss.python.org/t/pep-689-unstable-c-api-tier/20452
Status: Final
Type: Standards Track
Content-Type: text/x-rst
Requires: 523
Created: 22-Apr-2022
Python-Version: 3.12
Post-History: `27-Apr-2022 <https://mail.python.org/archives/list/python-dev@python.org/thread/PQXSP7E2B6KNXTJ2AERWMKKX42YP5D6O/>`__,
              `25-Aug-2022 <https://discuss.python.org/t/c-api-what-should-the-leading-underscore-py-mean/18486>`__,
              `27-Oct-2022 <https://discuss.python.org/t/pep-689-unstable-c-api-tier/20452>`__,
Resolution: https://discuss.python.org/t/pep-689-unstable-c-api-tier/20452/13

.. canonical-doc:: :ref:`devguide:c-api`

   User-facing documentation is at :ref:`py3.12:unstable-c-api`.


Abstract
========

Some functions and types of the C-API are designated *unstable*,
meaning that they will not change in patch (bugfix/security) releases,
but may change between minor releases (e.g. between 3.11 and 3.12) without
deprecation warnings.

Any C API with a leading underscore is designated *internal*, meaning that it
may change or disappear without any notice.


Motivation & Rationale
======================

Unstable C API tier
-------------------

The Python C-API is currently divided into `three stability tiers <https://devguide.python.org/developer-workflow/c-api/index.html>`__:

- Limited API, with high compatibility expectations
- Public API, which follows the :pep:`backwards compatibility policy
  <387>`, and requires deprecation warnings before changes
- Internal (private) API, which can change at any time.

Tools requiring access to CPython internals (e.g. advanced
debuggers and JIT compilers) are often built for minor series releases
of CPython, and assume that the C-API internals used do not change
in patch releases. To support these tools, we need a tier between the
Public and Private C-API, with guarantees on stability throughout
the minor-series release: the proposed *Unstable tier*.

Some functions, like ``PyCode_New()``, are documented as unstable
(“Calling [it] directly can bind you to a precise Python version”),
and also often change in practice.
The unstable tier should make their status obvious even to people who don't
read the docs carefully enough, making them hard to use accidentally.


Reserving leading underscores for Private API
---------------------------------------------

Currently, CPython developers don't agree on the exact meaning of a leading
underscore in API names.
It is used to mean two different things:

- API that may change between minor releases, as in the Unstable tier proposed
  here (e.g. functions introduced in :pep:`523`).
- API that is *private* and should not be used outside of CPython at all
  (e.g. because it may change without notice, or it relies on undocumented
  assumptions that non-CPython code cannot guarantee).

The unclear meaning makes the underscore less useful than it could be.
If it only marked *private* API, CPython developers could change underscored
functions, or remove unused ones, without researching how they're
documented or used outside CPython.

With the introduction of a dedicated unstable tier, we can clarify the meaning
of the leading underscore. It should mark private API only.


Not breaking code unnecessarily
-------------------------------

This PEP specifies that API in the unstable tier should have a special name
prefix. This means functions (macros, etc.) will need to be renamed.
After a rename, the old name should continue to be available until
an incompatible change is made (i.e. until call sites need to be updated
anyway).
In other words, just changing the tier of a function shouldn't break users'
code.


Specification
=============

The C API is divided by stability expectations into `three “sections” <https://devguide.python.org/developer-workflow/c-api/index.html>`__
(internal, public, and limited).
We'll now call these *stability tiers*, or *tiers* for short.

An *Unstable tier* will be added.

APIs (functions, types, etc.) in this tier will named with the ``PyUnstable_``
prefix, with no leading underscore.

They will be declared in headers used for public API (``Include/*.h``,
rather than in a subdirectory like ``Include/unstable/``).

Several rules for dealing with the unstable tier will be introduced:

-  Unstable API should have no backwards-incompatible
   changes across patch releases, but may change or be removed in minor
   releases (3.x.0, including Alpha and Beta releases of 3.x.0).
   Such changes must be documented and mentioned in the What's New document.

-  Backwards-incompatible changes to these APIs should be made so that
   code that uses them will need to be updated to compile with
   the new version (e.g. arguments should be added/removed, or a function should
   be renamed, but the semantic meaning of an argument should not change).

-  Unstable API should be documented and tested.

-  To move an API from the public tier to the unstable tier, it should be
   exposed under the new ``PyUnstable_*`` name.

   The old name should be deprecated (e.g. with ``Py_DEPRECATED``), but
   continue to be available until an incompatible change is made to the API.
   Per Python's backwards compatibility policy (:pep:`387`), this deprecation
   needs to last *at least* two releases (without an SC exceptions).
   But it can also last indefinitely -- for example, if :pep:`590`'s
   :pep:`“provisional” <590#finalizing-the-api>`
   ``_PyObject_Vectorcall`` was added today, it would be initially named
   ``PyUnstable_Object_Vectorcall`` and there would be no plan to remove
   this name.

   In the following cases, an incompatible change (and thus removing the
   deprecated name) is allowed without an SC exception, as if the function was
   already part of the Unstable tier:

   -  Any API introduced before Python 3.12 that is *documented* to be less
      stable than default.
   -  Any API introduced before Python 3.12 that was named with a leading
      underscore.

   For examples, see the `initial unstable API`_
   specified in this PEP.

-  To move an *internal* API to the unstable tier, it should be
   exposed under the new ``PyUnstable_*`` name.

   If the old name is documented, or widely used externally,
   it should continue to be available until an
   incompatible change is made (and call sites need to be updated).
   It should start raising deprecation warnings (e.g. using ``Py_DEPRECATED``).

-  To move an API from the unstable tier to the public tier, it should be
   exposed without the ``PyUnstable_*`` prefix.

   The old name should remain available until the API is deprecated or removed.

-  Adding new unstable API *for existing features* is allowed even after
   Beta feature freeze, up until the first Release Candidate.
   Consensus on Core Development Discourse or is needed in the Beta period.

These rules will be documented in the `devguide <https://devguide.python.org/developer-workflow/c-api/index.html>`__,
and `user documentation <https://docs.python.org/3/c-api/stable.html>`__
will be updated accordingly.

Reference docs for C API named ``PyUnstable_*`` will automatically show
notes with links to the unstable tier documentation.


Leading underscore
------------------

C API named with a leading underscore, as well as API only available with
``Py_BUILD_CORE``, will be considered *internal*.
This means:

-  It may change or be removed *without notice* in minor
   releases (3.x.0, including Alpha and Beta releases of 3.x.0).
   API changes in patch releases or Release Candidates should only be done if
   absolutely necessary.

-  It should be documented in source comments or Devguide only, not in the
   public documentation.

-  API introduced before Python 3.12 that is documented or widely used
   externally should be moved to the Unstable tier as explained above.

   This might happen long after this PEP is accepted.
   Consequently, for a few years core devs should do some research before
   changing underscored API, especially if it doesn't need ``Py_BUILD_CORE``.

Users of the C API are encouraged to search their codebase for ``_Py`` and
``_PY`` identifier prefixes, and treat any hits as issues to be eventually
fixed -- either by switching to an existing alternative, or by opening
a CPython issue to request exposing public API for their use case,
and eventually switching to that.


Initial unstable API
--------------------

The following API will be moved to the Unstable tier in the initial
implementation as proof of the concept.

Code object constructors:

- ``PyUnstable_Code_New()`` (renamed from ``PyCode_New``)
- ``PyUnstable_Code_NewWithPosOnlyArgs()`` (renamed from ``PyCode_NewWithPosOnlyArgs``)

Code extra information (:pep:`523`):

- ``PyUnstable_Eval_RequestCodeExtraIndex()`` (renamed from ``_PyEval_RequestCodeExtraIndex``)
- ``PyUnstable_Code_GetExtra()`` (renamed from ``_PyCode_GetExtra``)
- ``PyUnstable_Code_SetExtra()`` (renamed from ``_PyCode_SetExtra``)

More are expected in Python 3.12, without the need for another PEP.


Backwards Compatibility
=======================

The C API backwards compatibility expectations will be made clearer.

All renamed API will be available under old names for as long as feasible.


How to Teach This
=================

The changes affect advanced C programmers, who should consult the
updated reference documentation, devguide and/or What's New document.


Reference Implementation
========================

https://github.com/python/cpython/compare/main...encukou:unstable-tier


Rejected Ideas
==============

No special prefix
-----------------

In the initial version of this PEP, unstable API didn't have the ``PyUnstable``
prefix.
Instead, defining ``Py_USING_UNSTABLE_API`` made the API available in a given
source file, signifying acknowledgement that the file as a whole will
potentially need to be revisited for each Python release.

However, it was decided that unstable-ness needs to be exposed
in the individual names.

Underscore prefix
-----------------

It would be possible to mark both private and unstable API with
leading underscores.
However, that would dilute the meaning of ``_Py`` prefix.
Reserving the prefix for internal API only makes it trivial to search for.


New header directory
--------------------

Other API tiers have dedicated directories for headers
(``Include/cpython/``, ``Include/internal/``).

Since the unstable tier uses a very obvious naming convention
and the names are always available,
a directory like ``Include/unstable/`` is unnecessary.


Python API
----------

It might be good to add a similar tier in the Python (not C) API,
e.g. for ``types.CodeType``.
However, the mechanism for that would need to be different.
This is outside the scope of the PEP.


Copyright
=========

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.