From 0f45dbd37b33e75ab36e73f531318e96ec3e4fe7 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Sat, 25 May 2024 15:06:53 +0200 Subject: [PATCH] PEP 741: Address Steering Council's review (#3789) * Remove string types other than UTF-8. * Exclude the API from the limited C API. * Remove the explicit preconfiguration. * Remove the rationale about the limited C API / stable ABI. --- peps/pep-0741.rst | 359 +++------------------------------------------- 1 file changed, 23 insertions(+), 336 deletions(-) diff --git a/peps/pep-0741.rst b/peps/pep-0741.rst index 6eaacfe56..ac0b4e70b 100644 --- a/peps/pep-0741.rst +++ b/peps/pep-0741.rst @@ -5,15 +5,15 @@ Discussions-To: https://discuss.python.org/t/pep-741-python-configuration-c-api- Status: Draft Type: Standards Track Created: 18-Jan-2024 -Python-Version: 3.13 +Python-Version: 3.14 Post-History: `19-Jan-2024 `__, `08-Feb-2024 `__, Abstract ======== -Add a C API to the limited C API to configure the Python initialization. -It can be used with the stable ABI. +Add a C API to configure the Python initialization without relying on C +structures and the ability to make ABI-compatible changes in the future. Complete :pep:`587` API by adding ``PyInitConfig_AddModule()`` which can be used to add a built-in extension module; feature previously referred to @@ -23,11 +23,9 @@ Add ``PyConfig_Get()`` and ``PyConfig_Set()`` functions to get and set the current runtime configuration at runtime. :pep:`587` "Python Initialization Configuration" unified all the ways to -configure the Python **initialization**. This PEP (almost fully) unifies -also the configuration of the Python **preinitialization** and the -Python **initialization** in a single API, even if the -**preinitialization** is still required to decode strings from the -locale encoding. +configure the Python **initialization**. This PEP unifies also the +configuration of the Python **preinitialization** and the Python +**initialization** in a single API. This new API replaces the deprecated and incomplete legacy API which is scheduled for removal between Python 3.13 and Python 3.15. @@ -36,69 +34,6 @@ scheduled for removal between Python 3.13 and Python 3.15. Rationale ========= -PyConfig is not part of the limited C API ------------------------------------------ - -When the first versions of :pep:`587` "Python Initialization Configuration" -were discussed, there was a private field ``_config_version`` (``int``): -the configuration version, used for ABI compatibility. It was decided -that if an application embeds Python, it sticks to a Python version -anyway, and so there is no need to bother with the ABI compatibility. - -The final PyConfig API of :pep:`587` is excluded from the limited C API -since its main ``PyConfig`` structure is not versioned. Python cannot -guarantee ABI backward and forward compatibility, it's incompatible with -the stable ABI. - -Since PyConfig was added to Python 3.8, the limited C API and the stable -ABI are getting more popular. For example, Rust bindings such as the -`PyO3 project `_ can target the limited C API to embed -Python in Rust (but it's not the default). In practice, PyO3 can use -non-limited C API for specific needs, but using them avoids the stable -ABI advantages. - - -Limitations of the legacy API ------------------------------ - -The legacy API to configure the Python initialization is based on the -legacy ``Py_Initialize()`` function. It is now mostly deprecated: - -* Set the initialization configuration such as ``Py_SetPath()``: - deprecated in Python 3.11 and removed in Python 3.13. -* Global configuration variables such as ``Py_VerboseFlag``: - deprecated in Python 3.12 and scheduled for removal in Python 3.14. -* Get the current configuration such as ``Py_GetPath()``: - deprecated in Python 3.13 and scheduled for removal in Python 3.15. - -The legacy API doesn't support the "Python Configuration" and the -"Isolated Configuration" of PEP 587 PyConfig API. It only provides a -"legacy configuration" of ``Py_Initialize()`` which is in the between, -and also uses the legacy global configuration variables (such as -``Py_VerboseFlag``). - -Some options can only by set by environment variables, such as ``home`` -set by the ``PYTHONHOME`` environment variable. The problem is that -environment variables are inherited by child processes which can be a -surprising and unwanted behavior. - -Some configuration options, such as ``configure_locale``, simply cannot -be set. - - -Limitations of the limited C API --------------------------------- - -The limited C API is a subset of the legacy API. For example, -global configuration variables, such as ``Py_VerboseFlag``, are not -part of the limited C API. - -While some functions were removed from the limited C API version 3.13, -they are still part of the stable ABI. For example, building a -application with the limited C API version 3.12 can still run with -Python 3.13 stable ABI. - - Get the runtime configuration ----------------------------- @@ -214,116 +149,6 @@ Utilities creating standalone applications These utilities create standalone applications, they are not linked to libpython. -Usage of a stable ABI ---------------------- - -`Ronald Oussoren -`__: - - For tools like py2app/py2exe/pyinstaller, it is pretty - **inconvenient to have to rebuild the launcher executable** that’s - used to start the packaged application when there’s a bug fix - release of Python. - -`Gregory P. Smith -`__: - - You can’t **extend a struct** and **assume embedding people all - rebuild**. They don’t. Real world embedding uses exist that use an - installed Python minor version as a **shared library**. Update that to - use a different sized struct in a public API and someone is going to - have a bad time. That’s why I consider the struct frozen at rc1 - time, even when only for use in the embedding / writing their own - launcher case. - -`Colton Murphy -`__: - - I am trying to **embed the Python interpreter** using a **non C - language**. I have to stick with the limited API and private - structures for configuration in headers files is a no-no. Basically, - I need to be able to allocate and configure everything using only - exportable functions and the heap… no private structure details. - - (...) - - I am strictly limited to what’s in the **shared library** (DLL). I - **don’t have headers**, I can’t statically “recompile” every time a - new version of python comes out. That’s unmaintainable for me. - -Quotes of Milian Wolff's `message -`__: - - Our application is a large complex C++ code base with lots of - dependencies targeting all three major desktop platforms. - - Originally, we hoped to be able to use the **stable python ABI** to - allow biologists to **“bring your own python”**. The idea was that - they probably have a custom set of python libraries and code that - they would like to continue using. Our integrated API - so we - thought - was a tiny addition that should work with any Python out - there, so we used the stable ABI. - - This turned out to be a dead end, and I believe we can (should?) now - use the non-stable ABI of python. Allowing end users to BYO Python - caused far too much setup problems and support issues for us that it - was not worth it in the end. Instead, we now rather want to ship a - custom Python with a custom prefix that they can pip install custom - libraries into as needed. - - The problems we faced are not directly related to the stable ABI - - quite the contrary. Rather, it was due to thirdparty python - libraries that we shipped which themselves are not compatible across - python version increments. E.g. for the integrated console we use - qtconsole/jupyter, which worked in an archaic version with python - 3.9 but requires newer versions for python 3.11+. - - The ton of dependencies pulled in by UMAP was even worse, with numba - and pydnndescent and llvmlite often taking months to support newer - Python versions. - -`David Hewitt `__ of the PyO3 project: - - I think making the configuration structure opaque and using an API - to set/get configuration by name is a welcome simplification: - - * It’s a **smaller API** for language bindings like PyO3 to wrap and - re-expose, and - * It’s **easier** for people to **support multiple Python versions - to embed into their application**; no need to conditionally - compile structure field access, can just use normal error handling - if configuration values are not available for a specific version - at runtime. - -Quotes of `Paul P. message `__: - - I cannot agree more, it is the same story everywhere/every time - CPython must be **embedded**. I maintened a runtime+ecosystem for - Android 4.4+ for some time (in order more comfortably use Panda3D - standalone than with Kivy), patching CPython and making a CI for it - was ok. - - But I had to give up, because I had often to recompile every known - modules: this is not sustainable for one individual. - - So I dropped the Android arch to only go WebAssembly (Emscripten). - But same (hard and boring) problem as always: have to rebuild - numerous packages that are commonly used with 2D/3D framework. (...) - - Except for ONE, Harfang3d. I did not rebuild this one since Python - 3.11 initial port… Guess why? it is a limited C API - **abi3 - module**! - - Limited API abi3 are fresh air, fast and portable. And associated - with a **stable config runtime**, it would be just perfect way! - -See also `issue gh-116139 -`_ building an -application embedding Python 3.11 and attempting to run it with Python -3.10: it does crash because the ``PyConfig`` structure ABI is not stable -between two Python 3.x minor releases. - - Set the runtime configuration ----------------------------- @@ -368,26 +193,18 @@ initialization: * ``PyInitConfig_HasOption(config, name)``. * ``PyInitConfig_GetInt(config, name, &value)``. * ``PyInitConfig_GetStr(config, name, &value)``. - * ``PyInitConfig_GetWStr(config, name, &value)``. * ``PyInitConfig_GetStrList(config, name, &length, &items)``. * ``PyInitConfig_FreeStrList()``. - * ``PyInitConfig_GetWStrList(config, name, &length, &items)``. - * ``PyInitConfig_FreeWStrList()``. * Set options: * ``PyInitConfig_SetInt(config, name, value)``. * ``PyInitConfig_SetStr(config, name, value)``. - * ``PyInitConfig_SetStrLocale(config, name, value)``. - * ``PyInitConfig_SetWStr(config, name, value)``. * ``PyInitConfig_SetStrList(config, name, length, items)``. - * ``PyInitConfig_SetStrLocaleList(config, name, length, items)``. - * ``PyInitConfig_SetWStrList(config, name, length, items)``. * ``PyInitConfig_AddModule(config, name, initfunc)`` * Initialize: - * ``Py_PreInitializeFromInitConfig(config)``. * ``Py_InitializeFromInitConfig(config)``. * Error handling: @@ -397,15 +214,18 @@ initialization: Add C API functions to get and set the current runtime configuration: -* ``PyConfig_Get(name)`` → ``object``. +* ``PyConfig_Get(name)``. * ``PyConfig_GetInt(name, &value)``. * ``PyConfig_Set(name)``. -* ``PyConfig_Names()`` → ``frozenset``. +* ``PyConfig_Names()``. The C API uses null-terminated UTF-8 encoded strings to refer to a -configuration option. +configuration option name. -All C API functions are added to the limited C API version 3.13. +These C API functions are excluded from the limited C API. + +PyInitConfig structure +---------------------- The ``PyInitConfig`` structure is implemented by combining the four structures of the ``PyConfig`` API and has an ``inittab`` member as @@ -697,32 +517,6 @@ Following options can be get ``PyConfig_Get()``, but cannot be set by Need a ``Py_STATS`` build. -Preinitialization ------------------ - -Calling ``Py_PreInitializeFromInitConfig()`` preinitializes Python. For -example, it sets the memory allocator, and can configure the -``LC_CTYPE`` locale and configure the standard C streams such as -``stdin`` and ``stdout``. - -The following options can only be set during the Python -preinitialization: - -* ``allocator``, -* ``coerce_c_locale``, -* ``coerce_c_locale_warn``, -* ``configure_locale``, -* ``legacy_windows_fs_encoding``, -* ``utf8_mode``. - -Trying to set these options after Python preinitialization fails with an -error. - -``PyInitConfig_SetStrLocale()`` and ``PyInitConfig_SetStrLocaleList()`` -functions call ``Py_PreInitializeFromInitConfig()`` if Python is not -already preinitialized. - - Create Config ------------- @@ -773,14 +567,6 @@ null-terminated UTF-8 encoded string. On success, the string must be released with ``free(value)``. -``int PyInitConfig_GetWStr(PyInitConfig *config, const char *name, wchar_t **value)``: - Get a string configuration option as a null-terminated wide string. - - * Set *\*value* and return ``0`` on success. - * Set an error in *config* and return ``-1`` on error. - - On success, the string must be released with ``free(value)``. - ``int PyInitConfig_GetStrList(PyInitConfig *config, const char *name, size_t *length, char ***items)``: Get a string list configuration option as an array of null-terminated UTF-8 encoded strings. @@ -795,20 +581,6 @@ null-terminated UTF-8 encoded string. Free memory of a string list created by ``PyInitConfig_GetStrList()``. -``int PyInitConfig_GetWStrList(PyInitConfig *config, const char *name, size_t *length, wchar_t ***items)``: - Get a string list configuration option as an array of - null-terminated wide strings. - - * Set *\*length* and *\*value*, and return ``0`` on success. - * Set an error in *config* and return ``-1`` on error. - - On success, the string list must be released with - ``PyInitConfig_FreeWStrList(length, items)``. - -``void PyInitConfig_FreeWStrList(size_t length, wchar_t **items)``: - Free memory of a string list created by - ``PyInitConfig_GetWStrList()``. - Set Options ----------- @@ -834,24 +606,6 @@ called, not by the "Set" functions below. For example, setting * Return ``0`` on success. * Set an error in *config* and return ``-1`` on error. -``int PyInitConfig_SetStrLocale(PyInitConfig *config, const char *name, const char *value)``: - Set a string configuration option from a null-terminated bytes - string encoded in the locale encoding. The string is copied. - - The bytes string is decoded by ``Py_DecodeLocale()``. - ``Py_PreInitializeFromInitConfig()`` must be called before calling - this function. - - * Return ``0`` on success. - * Set an error in *config* and return ``-1`` on error. - -``int PyInitConfig_SetWStr(PyInitConfig *config, const char *name, const wchar_t *value)``: - Set a string configuration option from a null-terminated wide - string. The string is copied. - - * Return ``0`` on success. - * Set an error in *config* and return ``-1`` on error. - ``int PyInitConfig_SetStrList(PyInitConfig *config, const char *name, size_t length, char * const *items)``: Set a string list configuration option from an array of null-terminated UTF-8 encoded strings. The string list is copied. @@ -859,25 +613,6 @@ called, not by the "Set" functions below. For example, setting * Return ``0`` on success. * Set an error in *config* and return ``-1`` on error. -``int PyInitConfig_SetStrLocaleList(PyInitConfig *config, const char *name, size_t length, char * const *items)``: - Set a string list configuration option from an array of - null-terminated bytes strings encoded in the locale encoding. - The string list is copied. - - The bytes string is decoded by :c:func:`Py_DecodeLocale`. - ``Py_PreInitializeFromInitConfig()`` must be called before calling - this function. - - * Return ``0`` on success. - * Set an error in *config* and return ``-1`` on error. - -``int PyInitConfig_SetWStrList(PyInitConfig *config, const char *name, size_t length, wchar_t * const *items)``: - Set a string list configuration option from an error of - null-terminated wide strings. The string list is copied. - - * Return ``0`` on success. - * Set an error in *config* and return ``-1`` on error. - ``int PyInitConfig_AddModule(PyInitConfig *config, const char *name, PyObject* (*initfunc)(void))``: Add a built-in extension module to the table of built-in modules. @@ -898,12 +633,6 @@ called, not by the "Set" functions below. For example, setting Initialize Python ----------------- -``int Py_PreInitializeFromInitConfig(PyInitConfig *config)``: - Preinitialize Python from the initialization configuration. - - * Return ``0`` on success. - * Set an error in *config* and return ``-1`` on error. - ``int Py_InitializeFromInitConfig(PyInitConfig *config)``: Initialize Python from the initialization configuration. @@ -956,7 +685,7 @@ The configuration option *name* parameter must be a non-NULL null-terminated UTF-8 encoded string. ``PyObject* PyConfig_Get(const char *name)``: - Get the current runtime value of a configuration option as an + Get the current runtime value of a configuration option as a Python object. * Return a new reference on success. @@ -997,11 +726,8 @@ null-terminated UTF-8 encoded string. Python initialization nor after Python finalization. -Scope of the stable ABI ------------------------ - -The limited C API and the stable ABI added by this PEP only provide a -stable interface to program the Python initialization. +Stability +--------- The behavior of options, the default option values, and the Python behavior can change at each Python version: they are not "stable". @@ -1034,20 +760,20 @@ return -1 on error: goto error; } - // Set a list of wide strings (argv) - wchar_t *argv[] = {L"my_program", L"-c", L"pass"}; - if (PyInitConfig_SetWStrList(config, "argv", + // Set a list of UTF-8 strings (argv) + // Preinitialize implicitly Python to decode the bytes string. + char *argv[] = {"my_program", "-c", "pass"}; + if (PyInitConfig_SetStrList(config, "argv", Py_ARRAY_LENGTH(argv), argv) < 0) { goto error; } - // Set a wide string (program name) - if (PyInitConfig_SetWStr(config, "program_name", L"my_program") < 0) { + // Set a UTF-8 string (program name) + if (PyInitConfig_SetStr(config, "program_name", L"my_program") < 0) { goto error; } - // Set a list of bytes strings (xoptions). - // Preinitialize implicitly Python to decode the bytes string. + // Set a list of UTF-8 strings (xoptions). char* xoptions[] = {"faulthandler"}; if (PyInitConfig_SetStrList(config, "xoptions", Py_ARRAY_LENGTH(xoptions), xoptions) < 0) { @@ -1215,45 +941,6 @@ read once and cached. By the way, most configuration options cannot be changed at runtime. -Fully remove the preinitialization ----------------------------------- - -Delay decoding -^^^^^^^^^^^^^^ - -Without ``PyInitConfig_Get*()`` functions, it would be possible to store -``PyInitConfig_SetStrLocale()`` and ``PyInitConfig_SetStrLocaleList()`` -strings encoded and only initialize the ``LC_CTYPE`` locale and -decode the strings in ``Py_InitializeFromInitConfig()``. - -The problem is that users asked for ``PyInitConfig_Get*()`` functions. -For example, ``PyInitConfig_GetStr()`` must decode the string from the -locale encoding and then encode it to the UTF-8 encoding. - -However, if ``PyInitConfig_SetStrLocale()`` and -``PyInitConfig_SetStrLocaleList()`` strings are decoded as designed by -the PEP, there is no risk of mojibake: ``PyInitConfig_GetStr()`` returns -the expected decoded strings. - -Remove the Python configuration -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -If ``PyInitConfig_CreatePython()`` is removed, the preinitialization is -no longer needed since the ``LC_CTYPE`` is not configured by default by -``PyInitConfig_CreateIsolated()`` and setting ``"configure_locale"`` -option can always fail. - -The problem is that users asked to be able to write their own customized -Python, so have a Python-like program but with a different default -configuration. The ``PyInitConfig_CreatePython()`` function is needed -for that. - -Moreover, the Python configuration is also part of the :pep:`587` -design, implemented in Python 3.8. - -Disallow setting the ``"configure_locale"`` option has similar issues. - - Multi-phase initialization (similar to PEP 432) -----------------------------------------------