iSharkFly-Open
/
python-peps
mirror of https://github.com/python/peps.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

PEP 689: Rewrite, with PyUnstable_ prefix (GH-2847) 

Co-authored-by: C.A.M. Gerlach <CAM.Gerlach@Gerlach.CAM>
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
This commit is contained in:
Petr Viktorin 2022-10-27 17:16:40 +02:00 committed by GitHub
parent 13c89dc8cb
commit e707124176
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 192 additions and 131 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

323
pep-0689.rst
View File

@ -2,22 +2,14 @@ PEP: 689
Title: Unstable C API tier
Author: Petr Viktorin <encukou@gmail.com>
Discussions-To: https://mail.python.org/archives/list/python-dev@python.org/thread/PQXSP7E2B6KNXTJ2AERWMKKX42YP5D6O/
Status: Deferred
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Requires: 523
Created: 22-Apr-2022
Python-Version: 3.12
Post-History: `27-Apr-2022 <https://mail.python.org/archives/list/python-dev@python.org/thread/PQXSP7E2B6KNXTJ2AERWMKKX42YP5D6O/>`__,
Deferral note
===============
This PEP was accepted provided that function names in the unstable API start
with leading underscores. Unfortunately, this uncovered wider disagreement
about the meaning of leading underscores. The PEP's author is not comfortable
pushing the PEP forward before we agree on what a leading underscore means.
`25-Aug-2022 <https://discuss.python.org/t/c-api-what-should-the-leading-underscore-py-mean/18486>`__,
Abstract
@ -28,183 +20,228 @@ meaning that they will not change in patch (bugfix/security) releases,
but may change between minor releases (e.g. between 3.11 and 3.12) without
deprecation warnings.
Any C API with a leading underscore is designated *internal*, meaning that it
may change or disappear without any notice.
Motivation & Rationale
======================
The Python C-API is currently divided into `three tiers <https://devguide.python.org/c-api/>`__:
Unstable C API tier
-------------------
The Python C-API is currently divided into `three stability tiers <https://devguide.python.org/developer-workflow/c-api/index.html>`__:
- Limited API, with high compatibility expectations
- Public API, which follows the :pep:`backwards compatibility policy
<387>`, and requires deprecation warnings before changes
- Private (internal) API, which can change at any time.
- Internal (private) API, which can change at any time.
Tools requring access to CPython internals (e.g. advanced
debuggers and JIT compilers) are often built for minor series releases
of CPython, and assume that the C-API internals used do not change
in patch releases. To support these tools, we need a tier between the
Public and Private C-API, with guarantees on stability throughout
the minor-series release.
the minor-series release: the proposed *Unstable tier*.
Some functions, like ``PyCode_New()``, are documented as unstable
(“Calling [it] directly can bind you to a precise Python version”),
and also often change in practice.
The unstable tier should make their status obvious even to people who don't
read the docs carefully enough, making them hard to use accidentally.
Setting Stability Expectations
------------------------------
Reserving leading underscores for Private API
---------------------------------------------
Currently, there are no guarantees for the internal API -- that is, anything
that requires ``Py_BUILD_CORE`` or is named with a leading underscore.
This API can change without warning at any time, and code that uses it
is pinned to a specific build of Python.
Currently, CPython developers don't agree on the exact meaning of a leading
underscore in API names.
It is used to mean two different things:
However, in practice, even the internal API usually happens to be stable
in patch releases:
- API that may change between minor releases, as in the Unstable tier proposed
here (e.g. functions introduced in :pep:`523`).
- API that is *private* and should not be used outside of CPython at all
(e.g. because it may change without notice, or it relies on undocumented
assumptions that non-CPython code cannot guarantee).
- Some CPython core developers take this as an an unwritten rule.
- Patch releases only contain bugfixes, which are unlikely to
change the API.
The unclear meaning makes the underscore less useful than it could be.
If it only marked *private* API, CPython developers could change underscored
functions, or remove unused ones, without researching how they're
documented or used outside CPython.
Unstable API will make the stability expectations more explicit.
It will also hopefully encourage existing users of the private API to
reach out to python-dev, so we can expose, standardize and test an API
for some of their use cases.
With the introduction of an unstable tier, we can clarify the meaning
of the leading underscore, eventually making it OK to skip that research.
Reserving underscores for Private API
-------------------------------------
Not breaking code unnecessarily
-------------------------------
:pep:`523` introduced functions for use by debuggers and JIT compilers,
which are stable only across minor releases.
The functions names have leading underscores to suggest their limited
stability.
However, leading underscores usually mark *fully private* API.
CPython developers familiar with the “underscore means internal”
convention are unlikely to check if underscored functions they are
changing are documented and used outside CPython itself.
This proposal brings us a bit closer to reserving underscores
only for truly internal, private, hands-off API.
Warning about API that is changed often
---------------------------------------
The ``PyCode_New()`` family is an example of functions that are
documented as unstable (“Calling [it] directly can bind you to a precise
Python version”), and also often change in practice.
Moving it to the unstable tier will make its status obvious even
to people who don't read the docs carefully enough, and will make it
hard to use accidentally.
Changes during the Beta period
------------------------------
Since the API itself can change continuously up until Beta 1 (feature freeze)
of a minor version, major users of this API are unlikely to test
Alpha releases and provide feedback.
It is very difficult to determine what needs to be exposed as unstable.
Additions to the unstable tier will count as *stabilization*,
and will be allowed up to Release Candidate 1.
This PEP specifies that API should be renamed so that the
public/unstable/internal stability tier is expressed in function names.
Whenever this happens, the old name should continue to be available until
an incompatible change is made (i.e. until call sites need to be updated
anyway).
In other words, just changing tiers shouldn't break users' code.
Specification
=============
Several functions and types (“APIs”) will be moved to a new *unstable* tier.
The C API is divided by stability expectations into `three “sections” <https://devguide.python.org/developer-workflow/c-api/index.html>`__
(internal, public, and limited).
We'll now call these *stability tiers*, or *tiers* for short.
They will be expected to stay stable across patch releases,
but may change or be removed without warning in minor releases (3.x.0),
including Alpha and Beta releases of 3.x.0.
An *Unstable tier* will be added.
When they change significantly, code that uses them should no longer compile
(e.g. arguments should be added/removed, or a function should be renamed,
but the semantic meaning of an argument should not change).
APIs (functions, types, etc.) in this tier will named with the ``PyUnstable_``
prefix, with no leading underscore.
Their definitions will be moved to a new directory, ``Include/unstable/``,
and will be included from ``Python.h``.
Internally, they will be declared in headers in a new directory,
``Include/unstable/``.
Users should include ``Python.h`` rather than using these headers directly.
From Python 3.12 on, these APIs will only be usable when the
These APIs will only be declared when the
``Py_USING_UNSTABLE_API`` macro is defined.
CPython will only define the macro for building CPython itself
CPython will define the macro for building CPython itself
(``Py_BUILD_CORE``).
To make transition to unstable API easier,
in Python 3.11 the APIs will be available without ``Py_USING_UNSTABLE_API``
defined. In this case, using them will generate a deprecation warning on
compilers that support ``Py_DEPRECATED``.
Several rules for dealing with the unstable tier will be introduced:
A similar deprecation period will be used when making more APIs unstable
in the future:
- Unstable API should have no backwards-incompatible
changes across patch releases, but may change or be removed in minor
releases (3.x.0, including Alpha and Beta releases of 3.x.0).
Such changes must be documented and mentioned in the What's New document.
- When moving from public API, the deprecation period should follow Python's
backwards compatibility policy (currently, it should last at least
two releases).
- When moving from public API that is documented as unstable,
the deprecation period can only last one release.
- When moving from private API or adding new API, no deprecation period
is necessary.
- Backwards-incompatible changes to these APIs should be made so that
code that uses them will need to be updated to compile with
the new version (e.g. arguments should be added/removed, or a function should
be renamed, but the semantic meaning of an argument should not change).
Leading underscores will be removed from the names of the moved APIs.
The old underscored name of a renamed API will be available (as an alias
using ``#define``) at least until that API changes.
- To move an API from the public tier to the unstable tier, it should be
exposed under the new ``PyUnstable_*`` name and the definition should be
guarded with ``Py_USING_UNSTABLE_API``.
The unstable C-API tier and ``Py_USING_UNSTABLE_API`` will be documented,
and documentation of each unstable API will be updated.
The old name should be deprecated (e.g. with ``Py_DEPRECATED``), but
continue to be available until an incompatible change is made to the API.
Per Python's backwards compatibility policy (:pep:`387`), this deprecation
needs to last *at least* two releases (without an SC exceptions).
But it can also last indefinitely -- for example, if :pep:`590`'s
:pep:`“provisional” <590#finalizing-the-api>`
``_PyObject_Vectorcall`` was added today, it would be initially named
``PyUnstable_Object_Vectorcall`` and there would be no plan to ever remove
this name.
In the following cases, an incompatible change (and thus removing the
deprecated name) is allowed without an SC exeption, as if the function was
already part of the Unstable tier:
- Any API introduced before Python 3.12 that is *documented* to be less
stable than default.
- Any API introduced before Python 3.12 that was named with a leading
underscore.
For examples, see the :ref:`initial unstaple API <pep689-initial-list>`
specified in this PEP.
- To move an *internal* API to the unstable tier, it should be
exposed under the new ``PyUnstable_*`` name and the definition should be
guarded with ``Py_USING_UNSTABLE_API``.
If the old name is documented, or widely used externally,
it should continue to be available until an
incompatible change is made (and call sites need to be updated).
It should start raising deprecation warnings.
- To move an API from the unstable tier to the public tier, it should be
exposed without the ``PyUnstable_*`` prefix.
The old name should remain available, possibly without requiring
``Py_USING_UNSTABLE_API``, until the first incompatible change is made
otr the API is removed.
- Adding new unstable API *for existing features* is allowed even after
the feature freeze, up until the first Release Candidate.
Consensus on Core Development Discourse or ``capi-sig`` is needed in the
Beta period.
These rules will be documented in the `devguide <https://devguide.python.org/developer-workflow/c-api/index.html>`__,
and `user documentation <https://docs.python.org/3/c-api/stable.html>`__
will be updated accordingly.
Reference docs for C API named ``PyUnstable_*`` will automatically show
notes with links to the unstable tier documentation.
Adjustments during Beta periods
-------------------------------
Leading underscore
------------------
New APIs can be added to the unstable tier, and private APIs can be moved
to it, up to the first release candidate of a new minor version.
Consensus on the ``capi-sig`` or ``python-dev`` is needed in the Beta period.
C API named with a leading underscore, as well as API only available with
``Py_BUILD_CORE``, will be considered *internal*.
This means:
In the Beta period, no API may be moved to more private tier, e.g.
what is public in Beta 1 must stay public until the final release.
- It may change or be removed *without notice* in minor
releases (3.x.0, including Alpha and Beta releases of 3.x.0).
API changes in patch releases or Release Candidates should only be done if
absolutely necessary.
- It should be documented in source comments or Devguide only, not in the
public documentation.
- API introduced before Python 3.12 that is documented or widely used
externally should be moved to the Unstable tier as explained above.
This might happen long after this PEP is accepted.
Consequently, for a few years core devs should do some research before
changing underscored API, especially if it doesn't need ``Py_BUILD_CORE``.
Users of the C API are encouraged to search their codebase for ``_Py`` and
``_PY`` identifier prefixes, and treat any hits as issues to be eventually
fixed -- either by switching to an existing alternative, or by opening
a CPython issue to request exposing public API for their use case,
and eventually switching to that.
.. _pep689-initial-list:
Initial unstable API
--------------------
The following API will initially be unstable.
The set may be adjusted for 3.11.
The following API will be moved to the Unstable tier in the initial
implementation as proof of the concept.
Code object constructors:
- ``PyCode_New()``
- ``PyCode_NewWithPosOnlyArgs()``
- ``PyUnstable_Code_New()`` (renamed from ``PyCode_New``)
- ``PyUnstable_Code_NewWithPosOnlyArgs()`` (renamed from ``PyCode_NewWithPosOnlyArgs``)
Frame evaluation API (PEP 523):
Frame evaluation API (:pep:`523`):
- ``_PyFrameEvalFunction``
- ``_PyInterpreterState_GetEvalFrameFunc()``
- ``_PyInterpreterState_SetEvalFrameFunc()``
- ``_PyEval_RequestCodeExtraIndex()``
- ``_PyCode_GetExtra()``
- ``_PyCode_SetExtra()``
- ``struct _PyInterpreterFrame`` (as an incomplete, opaque struct)
- ``_PyFrame_GetFrameObject``
- ``PyEval_EvalFrameDefault``
- ``PyUnstable_FrameEvalFunction`` (renamed from ``_PyFrameEvalFunction``)
- ``PyUnstable_InterpreterState_GetEvalFrameFunc()`` (renamed from ``_PyInterpreterState_GetEvalFrameFunc``)
- ``PyUnstable_InterpreterState_SetEvalFrameFunc()`` (renamed from ``_PyInterpreterState_SetEvalFrameFunc``)
- ``PyUnstable_Eval_RequestCodeExtraIndex()`` (renamed from ``_PyEval_RequestCodeExtraIndex``)
- ``PyUnstable_Code_GetExtra()`` (renamed from ``_PyCode_GetExtra``)
- ``PyUnstable_Code_SetExtra()`` (renamed from ``_PyCode_SetExtra``)
- ``PyUnstable_InterpreterFrame`` (typedef for ``_PyInterpreterFrame``, as an opaque struct)
- ``PyUnstable_Frame_GetFrameObject`` (renamed from ``_PyFrame_GetFrameObject``)
- ``PyUnstable_EvalFrameDefault``
(new function that calls ``_PyEval_EvalFrameDefault``, but takes
``PyFrameObject`` rather than ``_PyInterpreterFrame``)
(Leading underscores will be removed as mentioned above.)
Backwards Compatibility
=======================
The C API backwards compatibility story will be made clearer.
The C API backwards compatibility expectations will be made clearer.
All renamed API will be available under old names for as long as feasible.
How to Teach This
=================
The changes affect advanced C programmers, who should consult the
updated reference documentation, devguide and/or What's New document·.
updated reference documentation, devguide and/or What's New document.
Reference Implementation
@ -213,19 +250,43 @@ Reference Implementation
https://github.com/python/cpython/issues/91744
Rejected Ideas
==============
It might be good to add a similar tier in the Python (not C) API,
e.g. for ``types.CodeType``.
However, the opt-in mechanism would need to be different (if any).
This is outside the scope of the PEP.
Open Issues
===========
The exact set of exposed API may change.
With the ``PyUnstable_`` prefix, is the opt-in macro necessary?
Rejected Ideas
==============
No special prefix
-----------------
In the initial version of this PEP, unstable API didn't have the ``PyUnstable``
prefix.
Instead, defining ``Py_USING_UNSTABLE_API`` made the API available in a given
source file, signifying acknowledgement that the file as a whole will
potentially need to be revisited for each Python release.
However, it was decided that unstable-ness needs to be exposed
in the individual names.
Underscore prefix
-----------------
It would be possible to mark both private and unstable API with
leading underscores.
However, that would dilute the meaning of ``_Py`` prefix.
Reserving the prefix for internal API only makes it trivial to search for.
Python API
----------
It might be good to add a similar tier in the Python (not C) API,
e.g. for ``types.CodeType``.
However, the mechanism for that would need to be different.
This is outside the scope of the PEP.
Copyright