From 7e7296bfdc84e9d52696696ccdeda26cd732683c Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Fri, 8 Mar 2024 15:35:01 +0100 Subject: [PATCH] PEP 741: Add sys.get_config_names() (#3686) * PEP 741: Add sys.get_config_names() * Add sys.get_config_names() function. * Add PyInitConfig_HasOption() function. * Remove Py_ExitWithInitConfig() function. * Add "Fully remove the preinitialization" section. * Mention when the caller must hold the GIL. * Add example increasing an initialization configuration option. * "Usage of the stable ABI" section: add more quotes. * Document that options side effects are not taken in account by PyInitConfig_Set*() functions. * Add "Spawnw process" section. * Add Cython rationale. * Add myself as PEP 741 code owner. --- .github/CODEOWNERS | 1 + peps/pep-0741.rst | 291 ++++++++++++++++++++++++++++++++++++++------- 2 files changed, 248 insertions(+), 44 deletions(-) diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index cbdf45b6d..813952843 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -619,6 +619,7 @@ peps/pep-0736.rst @gvanrossum @Rosuav peps/pep-0737.rst @vstinner peps/pep-0738.rst @encukou peps/pep-0740.rst @dstufft +peps/pep-0741.rst @vstinner peps/pep-0742.rst @JelleZijlstra # ... # peps/pep-0754.rst diff --git a/peps/pep-0741.rst b/peps/pep-0741.rst index 631a6f557..9e01d0149 100644 --- a/peps/pep-0741.rst +++ b/peps/pep-0741.rst @@ -16,10 +16,10 @@ Add a C API to the limited C API to configure the Python initialization, and to get the current configuration. It can be used with the stable ABI. -Add also ``sys.get_config(name)`` function to get the current -configuration. +Add also ``sys.get_config(name)`` and ``sys.get_config_names()`` +functions to get the current configuration. -Complete PEP 587 API by adding ``PyInitConfig_AddModule()`` which can be +Complete :pep:`587` API by adding ``PyInitConfig_AddModule()`` which can be used to add a built-in extension module; feature previously referred to as the "inittab". @@ -58,6 +58,7 @@ Python in Rust (but it's not the default). In practice, PyO3 can use non-limited C API for specific needs, but using them avoids the stable ABI advantages. + Limitations of the legacy API ----------------------------- @@ -85,6 +86,7 @@ surprising and unwanted behavior. Some configuration options, such as ``configure_locale``, simply cannot be set. + Limitations of the limited C API -------------------------------- @@ -97,6 +99,7 @@ they are still part of the stable ABI. For example, building a application with the limited C API version 3.12 can still run with Python 3.13 stable ABI. + Get the current configuration ----------------------------- @@ -112,6 +115,16 @@ configure Python, there is no public API to get Users of the limited C API are asking for a public API to get the current configuration. +Cython needs to access the ``optimization_level`` configuration option: +`issue `_. + +When global configuration variables were deprecated in 2022, `Marc-André +Lemburg requested +`_ +an API to access these configuration variables at runtime (not only +during Python initialization). + + Security fix ------------ @@ -133,6 +146,7 @@ member to the development branch (which became Python 3.12). A dedicated private global variable (unrelated to ``PyConfig``) is used in stable branches. + Redundancy between PyPreConfig and PyConfig ------------------------------------------- @@ -145,8 +159,12 @@ The redundancy is caused by the fact that the two structures are separated, whereas some ``PyConfig`` members are needed by the preinitialization. + +Embedding Python +---------------- + Applications embedding Python ------------------------------ +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Examples: @@ -160,8 +178,8 @@ Examples: On Linux, FreeBSD and macOS, applications are usually either statically linked to a ``libpython``, or load dynamically a ``libpython`` . The -``libpython`` shared library is versionned, example: -``libpython3.12.so`` for Python 3.13 on Linux. +``libpython`` shared library is versioned, example: +``libpython3.12.so`` for Python 3.12 on Linux. The vim project can target the stable ABI. Usually, the "system Python" version is used. It's not currently possible to select which Python @@ -174,7 +192,7 @@ such as GIMP, is to include Python in Flatpack, AppImage or Snap version with the container. Libraries embedding Python --------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^^ Examples: @@ -186,7 +204,7 @@ Examples: Rust bindings for the Python interpreter. Utilities creating standalone applications ------------------------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ * `py2app `_ for macOS. * `py2exe `_ for Windows. @@ -213,7 +231,7 @@ Usage of a stable ABI You can’t **extend a struct** and **assume embedding people all rebuild**. They don’t. Real world embedding uses exist that use an - installed Python minor version as a shared library. Update that to + installed Python minor version as a **shared library**. Update that to use a different sized struct in a public API and someone is going to have a bad time. That’s why I consider the struct frozen at rc1 time, even when only for use in the embedding / writing their own @@ -230,17 +248,40 @@ Usage of a stable ABI (...) - I am strictly limited to what’s in the shared library (DLL). I don’t - have headers, I can’t statically “recompile” every time a new - version of python comes out. That’s unmaintainable for me. + I am strictly limited to what’s in the **shared library** (DLL). I + **don’t have headers**, I can’t statically “recompile” every time a + new version of python comes out. That’s unmaintainable for me. -`Milian Wolff -`__: +Quotes of Milian Wolff's `message +`__: - IIUC then there's still no non-deprecated API in the **limited C API - to customize the initialization**, right? Can you then please reopen - this task to indicate that this? Or should I report a separate issue - to track this? Thank you + Our application is a large complex C++ code base with lots of + dependencies targeting all three major desktop platforms. + + Originally, we hoped to be able to use the **stable python ABI** to + allow biologists to **“bring your own python”**. The idea was that + they probably have a custom set of python libraries and code that + they would like to continue using. Our integrated API - so we + thought - was a tiny addition that should work with any Python out + there, so we used the stable ABI. + + This turned out to be a dead end, and I believe we can (should?) now + use the non-stable ABI of python. Allowing end users to BYO Python + caused far too much setup problems and support issues for us that it + was not worth it in the end. Instead, we now rather want to ship a + custom Python with a custom prefix that they can pip install custom + libraries into as needed. + + The problems we faced are not directly related to the stable ABI - + quite the contrary. Rather, it was due to thirdparty python + libraries that we shipped which themselves are not compatible across + python version increments. E.g. for the integrated console we use + qtconsole/jupyter, which worked in an archaic version with python + 3.9 but requires newer versions for python 3.11+. + + The ton of dependencies pulled in by UMAP was even worse, with numba + and pydnndescent and llvmlite often taking months to support newer + Python versions. `David Hewitt `__ of the PyO3 project: @@ -255,6 +296,69 @@ Usage of a stable ABI if configuration values are not available for a specific version at runtime. +Quotes of `Paul P. message `__: + + I cannot agree more, it is the same story everywhere/every time + CPython must be **embedded**. I maintened a runtime+ecosystem for + Android 4.4+ for some time (in order more comfortably use Panda3D + standalone than with Kivy), patching CPython and making a CI for it + was ok. + + But I had to give up, because I had often to recompile every known + modules: this is not sustainable for one individual. + + So I dropped the Android arch to only go WebAssembly (Emscripten). + But same (hard and boring) problem as always: have to rebuild + numerous packages that are commonly used with 2D/3D framework. (...) + + Except for ONE, Harfang3d. I did not rebuild this one since Python + 3.11 initial port… Guess why? it is a limited C API - **abi3 + module**! + + Limited API abi3 are fresh air, fast and portable. And associated + with a **stable config runtime**, it would be just perfect way! + + +Spawn a new Python process with the same configuration +------------------------------------------------------ + +The Python test suite runner spawns test worker processes by creating a +new command line with the same configuration options. For that, it calls +the ``args_from_interpreter_flags()`` of ``test.support`` module which +calls ``_args_from_interpreter_flags()`` of the ``subprocess`` module. +These functions inspect ``sys.flags``, ``sys.warnoptions`` and +``sys._xoptions``. + +The problem is that every time a new configuration is added, it should +be exposed to ``sys.flags``. Otherwise, +``args_from_interpreter_flags()`` ignores the option and so the option +is not inherited by child processes. + +Another problem is that inspecting ``sys._xoptions`` doesn't take +environment variables in account. + +Python 3.10 added ``sys.orig_argv`` for a specific implementation: copy +and then modify the original command line option to add or remove +command line optioins. This method also has a limitation, it ignores +environment variables. + +Examples of ``-X`` options which are not exposed in ``sys.flags``: + +* ``code_debug_ranges``, +* ``cpu_count``, +* ``import_time``, +* ``int_max_str_digits``, +* ``perf_profiling``, +* ``pycache_prefix``, +* ``run_presite`` (only on a Python debug build), +* ``show_ref_count``. + +Some of these options are inherited by inspecting ``sys._xoptions`` +which doesn't take in account environment variables. + +The ``sys.get_int_max_str_digits()`` function can be used to get the +``int_max_str_digits`` option. + Specification ============= @@ -266,6 +370,7 @@ initialization: * ``PyInitConfig_CreatePython()``. * ``PyInitConfig_CreateIsolated()``. * ``PyInitConfig_Free(config)``. +* ``PyInitConfig_HasOption(config, name)``. * ``PyInitConfig_GetInt(config, name, &value)``. * ``PyInitConfig_GetStr(config, name, &value)``. * ``PyInitConfig_GetWStr(config, name, &value)``. @@ -284,7 +389,6 @@ initialization: * ``Py_PreInitializeFromInitConfig(config)``. * ``Py_InitializeFromInitConfig(config)``. * ``PyInitConfig_GetError(config, &err_msg)``. -* ``Py_ExitWithInitConfig(config)``. Add C API and Python functions to get the current configuration: @@ -292,6 +396,7 @@ Add C API and Python functions to get the current configuration: * ``PyConfig_GetInt(name, &value)``. * ``PyConfig_Keys()``. * ``sys.get_config(name)``. +* ``sys.get_config_names()``. The C API uses null-terminated UTF-8 encoded strings to refer to a configuration option. @@ -402,7 +507,7 @@ Preconfiguration ---------------- Calling ``Py_PreInitializeFromInitConfig()`` preinitializes Python. For -example, it sets the memory allocation, and can configure the +example, it sets the memory allocator, and can configure the ``LC_CTYPE`` locale and configure the standard C streams such as ``stdin`` and ``stdout``. @@ -420,7 +525,7 @@ Trying to set these options after Python preinitialization fails with an error. ``PyInitConfig_SetStrLocale()`` and ``PyInitConfig_SetStrLocaleList()`` -functions cannot be called Python preinitialization. +functions cannot be called before the Python preinitialization. Create PyInitConfig @@ -450,6 +555,14 @@ Create PyInitConfig Get PyInitConfig Options ------------------------ +The configuration option *name* parameter must be a non-NULL +null-terminated UTF-8 encoded string. + +``int PyInitConfig_HasOption(PyInitConfig *config, const char *name)``: + Test if the configuration has an option called *name*. + + Return ``1`` if the option exists, or return ``0`` otherwise. + ``int PyInitConfig_GetInt(PyInitConfig *config, const char *name, int64_t *value)``: Get an integer configuration option. @@ -488,7 +601,7 @@ Get PyInitConfig Options ``PyInitConfig_GetStrList()``. ``int PyInitConfig_GetWStrList(PyInitConfig *config, const char *name, size_t *length, wchar_t ***items)``: - Get a string list configuration option as an error of + Get a string list configuration option as an array of null-terminated wide strings. * Set *\*length* and *\*value*, and return ``0`` on success. @@ -505,6 +618,14 @@ Get PyInitConfig Options Set PyInitConfig Options ------------------------ +The configuration option *name* parameter must be a non-NULL +null-terminated UTF-8 encoded string. + +Some configuration options have side effects on other options. This +logic is only implemented when ``Py_InitializeFromInitConfig()`` is +called, not by the "Set" functions below. For example, setting +``"dev_mode"`` to ``1`` does not set ``"faulthandler"`` to ``1``. + ``int PyInitConfig_SetInt(PyInitConfig *config, const char *name, int64_t value)``: Set an integer configuration option. @@ -610,17 +731,13 @@ Error handling function is called with *config*. The caller doesn't have to free the error message. -``void Py_ExitWithInitConfig(PyInitConfig *config)``: - Exit Python and free memory of an initialization configuration. - - If an error message is set, display the error message. - - The function does not return. - Get current configuration ------------------------- +The configuration option *name* parameter must be a non-NULL +null-terminated UTF-8 encoded string. + ``PyObject* PyConfig_Get(const char *name)``: Get the current value of a configuration option as an object. @@ -650,8 +767,8 @@ Get current configuration Other options are get from internal ``PyPreConfig`` and ``PyConfig`` structures. - The function cannot be called before Python initialization nor - after Python finalization. + The caller must hold the GIL. The function cannot be called before + Python initialization nor after Python finalization. ``int PyConfig_GetInt(const char *name, int *value)``: Similar to ``PyConfig_Get()``, but get the value as an integer. @@ -660,10 +777,12 @@ Get current configuration * Set an exception and return ``-1`` on error. ``PyObject* PyConfig_Keys(void)``: - Get all configuration option names as a tuple. + Get all configuration option names as a ``frozenset``. Set an exception and return ``NULL`` on error. + The caller must hold the GIL. + sys.get_config() ---------------- @@ -675,23 +794,42 @@ Add ``sys.get_config(name: str)`` function which calls * Raise an exception on error. +sys.get_config_names() +---------------------- + +Add ``sys.get_config_names()`` function which gets all configuration +option names as a ``frozenset``. + +Scope of the stable ABI +----------------------- + +The limited C API and the stable ABI added by this PEP only provide a +stable interface to program the Python initialization. + +The behavior of options, the default option values, and the Python +behavior can change at each Python version: they are not "stable". + +Moreover, configuration options can be added, deprecated and removed +following the usual :pep:`387` deprecation process. + + Examples ======== Initialize Python ----------------- -Example setting some configuration options of different types to -initialize Python. +Example initializing Python, set configuration options of different types, +return -1 on error: .. code-block:: c - void init_python(void) + int init_python(void) { PyInitConfig *config = PyInitConfig_CreatePython(); if (config == NULL) { - printf("Init allocation error\n"); - return; + printf("PYTHON INIT ERROR: memory allocation failed\n"); + return -1; } // Set an integer (dev mode) @@ -724,19 +862,46 @@ initialize Python. goto error; } PyInitConfig_Free(config); - return; + return 0; error: - // Display the error message an exit the process - // with a non-zero exit code - Py_ExitWithInitConfig(config); + // Display the error message + const char *err_msg; + (void)PyInitConfig_GetError(config, &err_msg); + printf("PYTHON INIT ERROR: %s\n", err_msg); + PyInitConfig_Free(config); + + return -1; } -Get the verbose option ------------------------ +Increase initialization bytes_warning option +-------------------------------------------- -Example getting the configuration option ``verbose``: +Example increasing the ``bytes_warning`` option of an initialization +configuration: + +.. code-block:: c + + int config_bytes_warning(PyInitConfig *config) + { + int bytes_warning; + if (PyInitConfig_GetInt(config, "bytes_warning", &bytes_warning)) { + return -1; + } + bytes_warning += 1; + if (PyInitConfig_SetInt(config, "bytes_warning", bytes_warning)) { + return -1; + } + return 0; + } + + +Get the current verbose option +------------------------------ + +Example getting the current value of the configuration option +``verbose``: .. code-block:: c @@ -851,6 +1016,44 @@ read once and cached. By the way, most configuration options cannot be changed at runtime. +Fully remove the preinitialization +---------------------------------- + +Delay decoding +^^^^^^^^^^^^^^ + +Without ``PyInitConfig_Get*()`` functions, it would be possible to store +``PyInitConfig_SetStrLocale()`` and ``PyInitConfig_SetStrLocaleList()`` +strings encoded and only initialize the ``LC_CTYPE`` locale and +decode the strings in ``Py_InitializeFromInitConfig()``. + +The problem is that users asked for ``PyInitConfig_Get*()`` functions. + +Also, if ``PyInitConfig_SetStrLocale()`` and +``PyInitConfig_SetStrLocaleList()`` strings are decoded as designed by +the PEP, there is no risk of mojibake: ``PyInitConfig_GetStr()`` returns +the expected decoded strings. + +Remove the Python configuration +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +If ``PyInitConfig_CreatePython()`` is removed, the preinitialization is +no longer needed since the ``LC_CTYPE`` is not configured by default by +``PyInitConfig_CreateIsolated()`` and setting ``"configure_locale"`` +option can always fail. + +The problem is that users asked to be able to write their own customized +Python, so have a Python-like program but with a different default +configuration. The ``PyInitConfig_CreatePython()`` function is needed +for that. + +The Python configuration is also part of the :pep:`587` design, +implemented in Python 3.8. + +Disallow setting the ``"configure_locale"`` option has similar issues. + + + Discussions ===========