iSharkFly-Open
/
python-peps
mirror of https://github.com/python/peps.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

PEP 741: Address Steering Council's review (#3789) 

* Remove string types other than UTF-8.
* Exclude the API from the limited C API.
* Remove the explicit preconfiguration.
* Remove the rationale about the limited C API / stable ABI.
This commit is contained in:
Victor Stinner 2024-05-25 15:06:53 +02:00 committed by GitHub
parent 2d2cdf1bd3
commit 0f45dbd37b
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 23 additions and 336 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

359
peps/pep-0741.rst
View File

@ -5,15 +5,15 @@ Discussions-To: https://discuss.python.org/t/pep-741-python-configuration-c-api-
Status: Draft Status: Draft
Type: Standards Track Type: Standards Track
Created: 18-Jan-2024 Created: 18-Jan-2024
Python-Version: 3.13 Python-Version: 3.14
Post-History: `19-Jan-2024 <https://discuss.python.org/t/pep-741-python-configuration-c-api/43637>`__, Post-History: `19-Jan-2024 <https://discuss.python.org/t/pep-741-python-configuration-c-api/43637>`__,
`08-Feb-2024 <https://discuss.python.org/t/pep-741-python-configuration-c-api-second-version/45403>`__, `08-Feb-2024 <https://discuss.python.org/t/pep-741-python-configuration-c-api-second-version/45403>`__,
Abstract Abstract
======== ========
Add a C API to the limited C API to configure the Python initialization. Add a C API to configure the Python initialization without relying on C
It can be used with the stable ABI. structures and the ability to make ABI-compatible changes in the future.
Complete :pep:`587` API by adding ``PyInitConfig_AddModule()`` which can be Complete :pep:`587` API by adding ``PyInitConfig_AddModule()`` which can be
used to add a built-in extension module; feature previously referred to used to add a built-in extension module; feature previously referred to
@ -23,11 +23,9 @@ Add ``PyConfig_Get()`` and ``PyConfig_Set()`` functions to
get and set the current runtime configuration at runtime. get and set the current runtime configuration at runtime.
:pep:`587` "Python Initialization Configuration" unified all the ways to :pep:`587` "Python Initialization Configuration" unified all the ways to
configure the Python **initialization**. This PEP (almost fully) unifies configure the Python **initialization**. This PEP unifies also the
also the configuration of the Python **preinitialization** and the configuration of the Python **preinitialization** and the Python
Python **initialization** in a single API, even if the **initialization** in a single API.
**preinitialization** is still required to decode strings from the
locale encoding.
This new API replaces the deprecated and incomplete legacy API which is This new API replaces the deprecated and incomplete legacy API which is
scheduled for removal between Python 3.13 and Python 3.15. scheduled for removal between Python 3.13 and Python 3.15.
@ -36,69 +34,6 @@ scheduled for removal between Python 3.13 and Python 3.15.
Rationale Rationale
========= =========
PyConfig is not part of the limited C API
-----------------------------------------
When the first versions of :pep:`587` "Python Initialization Configuration"
were discussed, there was a private field ``_config_version`` (``int``):
the configuration version, used for ABI compatibility. It was decided
that if an application embeds Python, it sticks to a Python version
anyway, and so there is no need to bother with the ABI compatibility.
The final PyConfig API of :pep:`587` is excluded from the limited C API
since its main ``PyConfig`` structure is not versioned. Python cannot
guarantee ABI backward and forward compatibility, it's incompatible with
the stable ABI.
Since PyConfig was added to Python 3.8, the limited C API and the stable
ABI are getting more popular. For example, Rust bindings such as the
`PyO3 project <https://pyo3.rs/>`_ can target the limited C API to embed
Python in Rust (but it's not the default). In practice, PyO3 can use
non-limited C API for specific needs, but using them avoids the stable
ABI advantages.
Limitations of the legacy API
-----------------------------
The legacy API to configure the Python initialization is based on the
legacy ``Py_Initialize()`` function. It is now mostly deprecated:
* Set the initialization configuration such as ``Py_SetPath()``:
deprecated in Python 3.11 and removed in Python 3.13.
* Global configuration variables such as ``Py_VerboseFlag``:
deprecated in Python 3.12 and scheduled for removal in Python 3.14.
* Get the current configuration such as ``Py_GetPath()``:
deprecated in Python 3.13 and scheduled for removal in Python 3.15.
The legacy API doesn't support the "Python Configuration" and the
"Isolated Configuration" of PEP 587 PyConfig API. It only provides a
"legacy configuration" of ``Py_Initialize()`` which is in the between,
and also uses the legacy global configuration variables (such as
``Py_VerboseFlag``).
Some options can only by set by environment variables, such as ``home``
set by the ``PYTHONHOME`` environment variable. The problem is that
environment variables are inherited by child processes which can be a
surprising and unwanted behavior.
Some configuration options, such as ``configure_locale``, simply cannot
be set.
Limitations of the limited C API
--------------------------------
The limited C API is a subset of the legacy API. For example,
global configuration variables, such as ``Py_VerboseFlag``, are not
part of the limited C API.
While some functions were removed from the limited C API version 3.13,
they are still part of the stable ABI. For example, building a
application with the limited C API version 3.12 can still run with
Python 3.13 stable ABI.
Get the runtime configuration Get the runtime configuration
----------------------------- -----------------------------
@ -214,116 +149,6 @@ Utilities creating standalone applications
These utilities create standalone applications, they are not linked to These utilities create standalone applications, they are not linked to
libpython. libpython.
Usage of a stable ABI
---------------------
`Ronald Oussoren
<https://discuss.python.org/t/fr-allow-private-runtime-config-to-enable-extending-without-breaking-the-pyconfig-abi/18004/9>`__:
For tools like py2app/py2exe/pyinstaller, it is pretty
**inconvenient to have to rebuild the launcher executable** that’s
used to start the packaged application when there’s a bug fix
release of Python.
`Gregory P. Smith
<https://discuss.python.org/t/fr-allow-private-runtime-config-to-enable-extending-without-breaking-the-pyconfig-abi/18004/10>`__:
You can’t **extend a struct** and **assume embedding people all
rebuild**. They don’t. Real world embedding uses exist that use an
installed Python minor version as a **shared library**. Update that to
use a different sized struct in a public API and someone is going to
have a bad time. That’s why I consider the struct frozen at rc1
time, even when only for use in the embedding / writing their own
launcher case.
`Colton Murphy
<https://discuss.python.org/t/fr-allow-private-runtime-config-to-enable-extending-without-breaking-the-pyconfig-abi/18004/11>`__:
I am trying to **embed the Python interpreter** using a **non C
language**. I have to stick with the limited API and private
structures for configuration in headers files is a no-no. Basically,
I need to be able to allocate and configure everything using only
exportable functions and the heap… no private structure details.
(...)
I am strictly limited to what’s in the **shared library** (DLL). I
**don’t have headers**, I can’t statically “recompile” every time a
new version of python comes out. That’s unmaintainable for me.
Quotes of Milian Wolff's `message
<https://discuss.python.org/t/pep-741-python-configuration-c-api-second-version/45403/4>`__:
Our application is a large complex C++ code base with lots of
dependencies targeting all three major desktop platforms.
Originally, we hoped to be able to use the **stable python ABI** to
allow biologists to **“bring your own python”**. The idea was that
they probably have a custom set of python libraries and code that
they would like to continue using. Our integrated API - so we
thought - was a tiny addition that should work with any Python out
there, so we used the stable ABI.
This turned out to be a dead end, and I believe we can (should?) now
use the non-stable ABI of python. Allowing end users to BYO Python
caused far too much setup problems and support issues for us that it
was not worth it in the end. Instead, we now rather want to ship a
custom Python with a custom prefix that they can pip install custom
libraries into as needed.
The problems we faced are not directly related to the stable ABI -
quite the contrary. Rather, it was due to thirdparty python
libraries that we shipped which themselves are not compatible across
python version increments. E.g. for the integrated console we use
qtconsole/jupyter, which worked in an archaic version with python
3.9 but requires newer versions for python 3.11+.
The ton of dependencies pulled in by UMAP was even worse, with numba
and pydnndescent and llvmlite often taking months to support newer
Python versions.
`David Hewitt <https://discuss.python.org/t/pep-741-python-configuration-c-api/43637/38>`__ of the PyO3 project:
I think making the configuration structure opaque and using an API
to set/get configuration by name is a welcome simplification:
* It’s a **smaller API** for language bindings like PyO3 to wrap and
re-expose, and
* It’s **easier** for people to **support multiple Python versions
to embed into their application**; no need to conditionally
compile structure field access, can just use normal error handling
if configuration values are not available for a specific version
at runtime.
Quotes of `Paul P. message <https://discuss.python.org/t/pep-741-python-configuration-c-api-second-version/45403/5>`__:
I cannot agree more, it is the same story everywhere/every time
CPython must be **embedded**. I maintened a runtime+ecosystem for
Android 4.4+ for some time (in order more comfortably use Panda3D
standalone than with Kivy), patching CPython and making a CI for it
was ok.
But I had to give up, because I had often to recompile every known
modules: this is not sustainable for one individual.
So I dropped the Android arch to only go WebAssembly (Emscripten).
But same (hard and boring) problem as always: have to rebuild
numerous packages that are commonly used with 2D/3D framework. (...)
Except for ONE, Harfang3d. I did not rebuild this one since Python
3.11 initial port… Guess why? it is a limited C API - **abi3
module**!
Limited API abi3 are fresh air, fast and portable. And associated
with a **stable config runtime**, it would be just perfect way!
See also `issue gh-116139
<https://github.com/python/cpython/issues/116139>`_ building an
application embedding Python 3.11 and attempting to run it with Python
3.10: it does crash because the ``PyConfig`` structure ABI is not stable
between two Python 3.x minor releases.
Set the runtime configuration Set the runtime configuration
----------------------------- -----------------------------
@ -368,26 +193,18 @@ initialization:
* ``PyInitConfig_HasOption(config, name)``. * ``PyInitConfig_HasOption(config, name)``.
* ``PyInitConfig_GetInt(config, name, &value)``. * ``PyInitConfig_GetInt(config, name, &value)``.
* ``PyInitConfig_GetStr(config, name, &value)``. * ``PyInitConfig_GetStr(config, name, &value)``.
* ``PyInitConfig_GetWStr(config, name, &value)``.
* ``PyInitConfig_GetStrList(config, name, &length, &items)``. * ``PyInitConfig_GetStrList(config, name, &length, &items)``.
* ``PyInitConfig_FreeStrList()``. * ``PyInitConfig_FreeStrList()``.
* ``PyInitConfig_GetWStrList(config, name, &length, &items)``.
* ``PyInitConfig_FreeWStrList()``.
* Set options: * Set options:
* ``PyInitConfig_SetInt(config, name, value)``. * ``PyInitConfig_SetInt(config, name, value)``.
* ``PyInitConfig_SetStr(config, name, value)``. * ``PyInitConfig_SetStr(config, name, value)``.
* ``PyInitConfig_SetStrLocale(config, name, value)``.
* ``PyInitConfig_SetWStr(config, name, value)``.
* ``PyInitConfig_SetStrList(config, name, length, items)``. * ``PyInitConfig_SetStrList(config, name, length, items)``.
* ``PyInitConfig_SetStrLocaleList(config, name, length, items)``.
* ``PyInitConfig_SetWStrList(config, name, length, items)``.
* ``PyInitConfig_AddModule(config, name, initfunc)`` * ``PyInitConfig_AddModule(config, name, initfunc)``
* Initialize: * Initialize:
* ``Py_PreInitializeFromInitConfig(config)``.
* ``Py_InitializeFromInitConfig(config)``. * ``Py_InitializeFromInitConfig(config)``.
* Error handling: * Error handling:
@ -397,15 +214,18 @@ initialization:
Add C API functions to get and set the current runtime configuration: Add C API functions to get and set the current runtime configuration:
* ``PyConfig_Get(name)`` → ``object``. * ``PyConfig_Get(name)``.
* ``PyConfig_GetInt(name, &value)``. * ``PyConfig_GetInt(name, &value)``.
* ``PyConfig_Set(name)``. * ``PyConfig_Set(name)``.
* ``PyConfig_Names()`` → ``frozenset``. * ``PyConfig_Names()``.
The C API uses null-terminated UTF-8 encoded strings to refer to a The C API uses null-terminated UTF-8 encoded strings to refer to a
configuration option. configuration option name.
All C API functions are added to the limited C API version 3.13. These C API functions are excluded from the limited C API.
PyInitConfig structure
----------------------
The ``PyInitConfig`` structure is implemented by combining the four The ``PyInitConfig`` structure is implemented by combining the four
structures of the ``PyConfig`` API and has an ``inittab`` member as structures of the ``PyConfig`` API and has an ``inittab`` member as
@ -697,32 +517,6 @@ Following options can be get ``PyConfig_Get()``, but cannot be set by
Need a ``Py_STATS`` build. Need a ``Py_STATS`` build.
Preinitialization
-----------------
Calling ``Py_PreInitializeFromInitConfig()`` preinitializes Python. For
example, it sets the memory allocator, and can configure the
``LC_CTYPE`` locale and configure the standard C streams such as
``stdin`` and ``stdout``.
The following options can only be set during the Python
preinitialization:
* ``allocator``,
* ``coerce_c_locale``,
* ``coerce_c_locale_warn``,
* ``configure_locale``,
* ``legacy_windows_fs_encoding``,
* ``utf8_mode``.
Trying to set these options after Python preinitialization fails with an
error.
``PyInitConfig_SetStrLocale()`` and ``PyInitConfig_SetStrLocaleList()``
functions call ``Py_PreInitializeFromInitConfig()`` if Python is not
already preinitialized.
Create Config Create Config
------------- -------------
@ -773,14 +567,6 @@ null-terminated UTF-8 encoded string.
On success, the string must be released with ``free(value)``. On success, the string must be released with ``free(value)``.
``int PyInitConfig_GetWStr(PyInitConfig *config, const char *name, wchar_t **value)``:
Get a string configuration option as a null-terminated wide string.
* Set *\*value* and return ``0`` on success.
* Set an error in *config* and return ``-1`` on error.
On success, the string must be released with ``free(value)``.
``int PyInitConfig_GetStrList(PyInitConfig *config, const char *name, size_t *length, char ***items)``: ``int PyInitConfig_GetStrList(PyInitConfig *config, const char *name, size_t *length, char ***items)``:
Get a string list configuration option as an array of Get a string list configuration option as an array of
null-terminated UTF-8 encoded strings. null-terminated UTF-8 encoded strings.
@ -795,20 +581,6 @@ null-terminated UTF-8 encoded string.
Free memory of a string list created by Free memory of a string list created by
``PyInitConfig_GetStrList()``. ``PyInitConfig_GetStrList()``.
``int PyInitConfig_GetWStrList(PyInitConfig *config, const char *name, size_t *length, wchar_t ***items)``:
Get a string list configuration option as an array of
null-terminated wide strings.
* Set *\*length* and *\*value*, and return ``0`` on success.
* Set an error in *config* and return ``-1`` on error.
On success, the string list must be released with
``PyInitConfig_FreeWStrList(length, items)``.
``void PyInitConfig_FreeWStrList(size_t length, wchar_t **items)``:
Free memory of a string list created by
``PyInitConfig_GetWStrList()``.
Set Options Set Options
----------- -----------
@ -834,24 +606,6 @@ called, not by the "Set" functions below. For example, setting
* Return ``0`` on success. * Return ``0`` on success.
* Set an error in *config* and return ``-1`` on error. * Set an error in *config* and return ``-1`` on error.
``int PyInitConfig_SetStrLocale(PyInitConfig *config, const char *name, const char *value)``:
Set a string configuration option from a null-terminated bytes
string encoded in the locale encoding. The string is copied.
The bytes string is decoded by ``Py_DecodeLocale()``.
``Py_PreInitializeFromInitConfig()`` must be called before calling
this function.
* Return ``0`` on success.
* Set an error in *config* and return ``-1`` on error.
``int PyInitConfig_SetWStr(PyInitConfig *config, const char *name, const wchar_t *value)``:
Set a string configuration option from a null-terminated wide
string. The string is copied.
* Return ``0`` on success.
* Set an error in *config* and return ``-1`` on error.
``int PyInitConfig_SetStrList(PyInitConfig *config, const char *name, size_t length, char * const *items)``: ``int PyInitConfig_SetStrList(PyInitConfig *config, const char *name, size_t length, char * const *items)``:
Set a string list configuration option from an array of Set a string list configuration option from an array of
null-terminated UTF-8 encoded strings. The string list is copied. null-terminated UTF-8 encoded strings. The string list is copied.
@ -859,25 +613,6 @@ called, not by the "Set" functions below. For example, setting
* Return ``0`` on success. * Return ``0`` on success.
* Set an error in *config* and return ``-1`` on error. * Set an error in *config* and return ``-1`` on error.
``int PyInitConfig_SetStrLocaleList(PyInitConfig *config, const char *name, size_t length, char * const *items)``:
Set a string list configuration option from an array of
null-terminated bytes strings encoded in the locale encoding.
The string list is copied.
The bytes string is decoded by :c:func:`Py_DecodeLocale`.
``Py_PreInitializeFromInitConfig()`` must be called before calling
this function.
* Return ``0`` on success.
* Set an error in *config* and return ``-1`` on error.
``int PyInitConfig_SetWStrList(PyInitConfig *config, const char *name, size_t length, wchar_t * const *items)``:
Set a string list configuration option from an error of
null-terminated wide strings. The string list is copied.
* Return ``0`` on success.
* Set an error in *config* and return ``-1`` on error.
``int PyInitConfig_AddModule(PyInitConfig *config, const char *name, PyObject* (*initfunc)(void))``: ``int PyInitConfig_AddModule(PyInitConfig *config, const char *name, PyObject* (*initfunc)(void))``:
Add a built-in extension module to the table of built-in modules. Add a built-in extension module to the table of built-in modules.
@ -898,12 +633,6 @@ called, not by the "Set" functions below. For example, setting
Initialize Python Initialize Python
----------------- -----------------
``int Py_PreInitializeFromInitConfig(PyInitConfig *config)``:
Preinitialize Python from the initialization configuration.
* Return ``0`` on success.
* Set an error in *config* and return ``-1`` on error.
``int Py_InitializeFromInitConfig(PyInitConfig *config)``: ``int Py_InitializeFromInitConfig(PyInitConfig *config)``:
Initialize Python from the initialization configuration. Initialize Python from the initialization configuration.
@ -956,7 +685,7 @@ The configuration option *name* parameter must be a non-NULL
null-terminated UTF-8 encoded string. null-terminated UTF-8 encoded string.
``PyObject* PyConfig_Get(const char *name)``: ``PyObject* PyConfig_Get(const char *name)``:
Get the current runtime value of a configuration option as an Get the current runtime value of a configuration option as a Python
object. object.
* Return a new reference on success. * Return a new reference on success.
@ -997,11 +726,8 @@ null-terminated UTF-8 encoded string.
Python initialization nor after Python finalization. Python initialization nor after Python finalization.
Scope of the stable ABI Stability
----------------------- ---------
The limited C API and the stable ABI added by this PEP only provide a
stable interface to program the Python initialization.
The behavior of options, the default option values, and the Python The behavior of options, the default option values, and the Python
behavior can change at each Python version: they are not "stable". behavior can change at each Python version: they are not "stable".
@ -1034,20 +760,20 @@ return -1 on error:
goto error; goto error;
} }
// Set a list of wide strings (argv) // Set a list of UTF-8 strings (argv)
wchar_t *argv[] = {L"my_program", L"-c", L"pass"}; // Preinitialize implicitly Python to decode the bytes string.
if (PyInitConfig_SetWStrList(config, "argv", char *argv[] = {"my_program", "-c", "pass"};
if (PyInitConfig_SetStrList(config, "argv",
Py_ARRAY_LENGTH(argv), argv) < 0) { Py_ARRAY_LENGTH(argv), argv) < 0) {
goto error; goto error;
} }
// Set a wide string (program name) // Set a UTF-8 string (program name)
if (PyInitConfig_SetWStr(config, "program_name", L"my_program") < 0) { if (PyInitConfig_SetStr(config, "program_name", L"my_program") < 0) {
goto error; goto error;
} }
// Set a list of bytes strings (xoptions). // Set a list of UTF-8 strings (xoptions).
// Preinitialize implicitly Python to decode the bytes string.
char* xoptions[] = {"faulthandler"}; char* xoptions[] = {"faulthandler"};
if (PyInitConfig_SetStrList(config, "xoptions", if (PyInitConfig_SetStrList(config, "xoptions",
Py_ARRAY_LENGTH(xoptions), xoptions) < 0) { Py_ARRAY_LENGTH(xoptions), xoptions) < 0) {
@ -1215,45 +941,6 @@ read once and cached. By the way, most configuration options cannot be
changed at runtime. changed at runtime.
Fully remove the preinitialization
----------------------------------
Delay decoding
^^^^^^^^^^^^^^
Without ``PyInitConfig_Get*()`` functions, it would be possible to store
``PyInitConfig_SetStrLocale()`` and ``PyInitConfig_SetStrLocaleList()``
strings encoded and only initialize the ``LC_CTYPE`` locale and
decode the strings in ``Py_InitializeFromInitConfig()``.
The problem is that users asked for ``PyInitConfig_Get*()`` functions.
For example, ``PyInitConfig_GetStr()`` must decode the string from the
locale encoding and then encode it to the UTF-8 encoding.
However, if ``PyInitConfig_SetStrLocale()`` and
``PyInitConfig_SetStrLocaleList()`` strings are decoded as designed by
the PEP, there is no risk of mojibake: ``PyInitConfig_GetStr()`` returns
the expected decoded strings.
Remove the Python configuration
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
If ``PyInitConfig_CreatePython()`` is removed, the preinitialization is
no longer needed since the ``LC_CTYPE`` is not configured by default by
``PyInitConfig_CreateIsolated()`` and setting ``"configure_locale"``
option can always fail.
The problem is that users asked to be able to write their own customized
Python, so have a Python-like program but with a different default
configuration. The ``PyInitConfig_CreatePython()`` function is needed
for that.
Moreover, the Python configuration is also part of the :pep:`587`
design, implemented in Python 3.8.
Disallow setting the ``"configure_locale"`` option has similar issues.
Multi-phase initialization (similar to PEP 432) Multi-phase initialization (similar to PEP 432)
----------------------------------------------- -----------------------------------------------