PEP 741: Add sys.get_config_names() (#3686)

* PEP 741: Add sys.get_config_names()

* Add sys.get_config_names() function.
* Add PyInitConfig_HasOption() function.
* Remove Py_ExitWithInitConfig() function.
* Add "Fully remove the preinitialization" section.
* Mention when the caller must hold the GIL.
* Add example increasing an initialization configuration option.
* "Usage of the stable ABI" section: add more quotes.
* Document that options side effects are not taken in
  account by PyInitConfig_Set*() functions.
* Add "Spawnw process" section.
* Add Cython rationale.
* Add myself as PEP 741 code owner.

.github/CODEOWNERS

@@ -619,6 +619,7 @@ peps/pep-0736.rst  @gvanrossum @Rosuav
 peps/pep-0737.rst  @vstinner
 peps/pep-0738.rst  @encukou
 peps/pep-0740.rst  @dstufft
+peps/pep-0741.rst  @vstinner
 peps/pep-0742.rst  @JelleZijlstra
 # ...
 # peps/pep-0754.rst

peps/pep-0741.rst

@@ -16,10 +16,10 @@ Add a C API to the limited C API to configure the Python initialization,
 and to get the current configuration. It can be used with the stable
 ABI.
 
-Add also ``sys.get_config(name)`` function to get the current
+Add also ``sys.get_config(name)`` and ``sys.get_config_names()``
-configuration.
+functions to get the current configuration.
 
-Complete PEP 587 API by adding ``PyInitConfig_AddModule()`` which can be
+Complete :pep:`587` API by adding ``PyInitConfig_AddModule()`` which can be
 used to add a built-in extension module; feature previously referred to
 as the "inittab".
 
@@ -58,6 +58,7 @@ Python in Rust (but it's not the default). In practice, PyO3 can use
 non-limited C API for specific needs, but using them avoids the stable
 ABI advantages.
 
+
 Limitations of the legacy API
 -----------------------------
 
@@ -85,6 +86,7 @@ surprising and unwanted behavior.
 Some configuration options, such as ``configure_locale``, simply cannot
 be set.
 
+
 Limitations of the limited C API
 --------------------------------
 
@@ -97,6 +99,7 @@ they are still part of the stable ABI. For example, building a
 application with the limited C API version 3.12 can still run with
 Python 3.13 stable ABI.
 
+
 Get the current configuration
 -----------------------------
 
@@ -112,6 +115,16 @@ configure Python, there is no public API to get
 Users of the limited C API are asking for a public API to get the
 current configuration.
 
+Cython needs to access the ``optimization_level`` configuration option:
+`issue <https://github.com/python/cpython/issues/99872>`_.
+
+When global configuration variables were deprecated in 2022, `Marc-André
+Lemburg requested
+<https://github.com/python/cpython/issues/93103#issuecomment-1136462708>`_
+an API to access these configuration variables at runtime (not only
+during Python initialization).
+
+
 Security fix
 ------------
 
@@ -133,6 +146,7 @@ member to the development branch (which became Python 3.12). A dedicated
 private global variable (unrelated to ``PyConfig``) is used in stable
 branches.
 
+
 Redundancy between PyPreConfig and PyConfig
 -------------------------------------------
 
@@ -145,8 +159,12 @@ The redundancy is caused by the fact that the two structures are
 separated, whereas some ``PyConfig`` members are needed by the
 preinitialization.
 
+
+Embedding Python
+----------------
+
 Applications embedding Python
------------------------------
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 Examples:
 
@@ -160,8 +178,8 @@ Examples:
 
 On Linux, FreeBSD and macOS, applications are usually either statically
 linked to a ``libpython``, or load dynamically a ``libpython`` . The
-``libpython`` shared library is versionned, example:
+``libpython`` shared library is versioned, example:
-``libpython3.12.so`` for Python 3.13 on Linux.
+``libpython3.12.so`` for Python 3.12 on Linux.
 
 The vim project can target the stable ABI. Usually, the "system Python"
 version is used. It's not currently possible to select which Python
@@ -174,7 +192,7 @@ such as GIMP, is to include Python in Flatpack, AppImage or Snap
 version with the container.
 
 Libraries embedding Python
---------------------------
+^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 Examples:
 
@@ -186,7 +204,7 @@ Examples:
   Rust bindings for the Python interpreter.
 
 Utilities creating standalone applications
-------------------------------------------
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 * `py2app <https://py2app.readthedocs.io/>`_ for macOS.
 * `py2exe <http://www.py2exe.org/>`_ for Windows.
@@ -213,7 +231,7 @@ Usage of a stable ABI
 
     You can’t **extend a struct** and **assume embedding people all
     rebuild**.  They don’t. Real world embedding uses exist that use an
-    installed Python minor version as a shared library. Update that to
+    installed Python minor version as a **shared library**. Update that to
     use a different sized struct in a public API and someone is going to
     have a bad time. That’s why I consider the struct frozen at rc1
     time, even when only for use in the embedding / writing their own
@@ -230,17 +248,40 @@ Usage of a stable ABI
 
     (...)
 
-    I am strictly limited to what’s in the shared library (DLL). I don’t
+    I am strictly limited to what’s in the **shared library** (DLL). I
-    have headers, I can’t statically “recompile” every time a new
+    **don’t have headers**, I can’t statically “recompile” every time a
-    version of python comes out. That’s unmaintainable for me.
+    new version of python comes out. That’s unmaintainable for me.
 
-`Milian Wolff
+Quotes of Milian Wolff's `message
-<https://github.com/python/cpython/issues/107954#issuecomment-1893988614>`__:
+<https://discuss.python.org/t/pep-741-python-configuration-c-api-second-version/45403/4>`__:
 
-    IIUC then there's still no non-deprecated API in the **limited C API
+    Our application is a large complex C++ code base with lots of
-    to customize the initialization**, right? Can you then please reopen
+    dependencies targeting all three major desktop platforms.
-    this task to indicate that this? Or should I report a separate issue
+
-    to track this? Thank you
+    Originally, we hoped to be able to use the **stable python ABI** to
+    allow biologists to **“bring your own python”**. The idea was that
+    they probably have a custom set of python libraries and code that
+    they would like to continue using. Our integrated API - so we
+    thought - was a tiny addition that should work with any Python out
+    there, so we used the stable ABI.
+
+    This turned out to be a dead end, and I believe we can (should?) now
+    use the non-stable ABI of python. Allowing end users to BYO Python
+    caused far too much setup problems and support issues for us that it
+    was not worth it in the end.  Instead, we now rather want to ship a
+    custom Python with a custom prefix that they can pip install custom
+    libraries into as needed.
+
+    The problems we faced are not directly related to the stable ABI -
+    quite the contrary. Rather, it was due to thirdparty python
+    libraries that we shipped which themselves are not compatible across
+    python version increments. E.g. for the integrated console we use
+    qtconsole/jupyter, which worked in an archaic version with python
+    3.9 but requires newer versions for python 3.11+.
+
+    The ton of dependencies pulled in by UMAP was even worse, with numba
+    and pydnndescent and llvmlite often taking months to support newer
+    Python versions.
 
 `David Hewitt <https://discuss.python.org/t/pep-741-python-configuration-c-api/43637/38>`__ of the PyO3 project:
 
@@ -255,6 +296,69 @@ Usage of a stable ABI
       if configuration values are not available for a specific version
       at runtime.
 
+Quotes of `Paul P. message <https://discuss.python.org/t/pep-741-python-configuration-c-api-second-version/45403/5>`__:
+
+    I cannot agree more, it is the same story everywhere/every time
+    CPython must be **embedded**. I maintened a runtime+ecosystem for
+    Android 4.4+ for some time (in order more comfortably use Panda3D
+    standalone than with Kivy), patching CPython and making a CI for it
+    was ok.
+
+    But I had to give up, because I had often to recompile every known
+    modules: this is not sustainable for one individual.
+
+    So I dropped the Android arch to only go WebAssembly (Emscripten).
+    But same (hard and boring) problem as always: have to rebuild
+    numerous packages that are commonly used with 2D/3D framework. (...)
+
+    Except for ONE, Harfang3d. I did not rebuild this one since Python
+    3.11 initial port… Guess why? it is a limited C API - **abi3
+    module**!
+
+    Limited API abi3 are fresh air, fast and portable. And associated
+    with a **stable config runtime**, it would be just perfect way!
+
+
+Spawn a new Python process with the same configuration
+------------------------------------------------------
+
+The Python test suite runner spawns test worker processes by creating a
+new command line with the same configuration options. For that, it calls
+the ``args_from_interpreter_flags()`` of ``test.support`` module which
+calls ``_args_from_interpreter_flags()`` of the ``subprocess`` module.
+These functions inspect ``sys.flags``, ``sys.warnoptions`` and
+``sys._xoptions``.
+
+The problem is that every time a new configuration is added, it should
+be exposed to ``sys.flags``. Otherwise,
+``args_from_interpreter_flags()`` ignores the option and so the option
+is not inherited by child processes.
+
+Another problem is that inspecting ``sys._xoptions`` doesn't take
+environment variables in account.
+
+Python 3.10 added ``sys.orig_argv`` for a specific implementation: copy
+and then modify the original command line option to add or remove
+command line optioins. This method also has a limitation, it ignores
+environment variables.
+
+Examples of ``-X`` options which are not exposed in ``sys.flags``:
+
+* ``code_debug_ranges``,
+* ``cpu_count``,
+* ``import_time``,
+* ``int_max_str_digits``,
+* ``perf_profiling``,
+* ``pycache_prefix``,
+* ``run_presite`` (only on a Python debug build),
+* ``show_ref_count``.
+
+Some of these options are inherited by inspecting ``sys._xoptions``
+which doesn't take in account environment variables.
+
+The ``sys.get_int_max_str_digits()`` function can be used to get the
+``int_max_str_digits`` option.
+
 
 Specification
 =============
@@ -266,6 +370,7 @@ initialization:
 * ``PyInitConfig_CreatePython()``.
 * ``PyInitConfig_CreateIsolated()``.
 * ``PyInitConfig_Free(config)``.
+* ``PyInitConfig_HasOption(config, name)``.
 * ``PyInitConfig_GetInt(config, name, &value)``.
 * ``PyInitConfig_GetStr(config, name, &value)``.
 * ``PyInitConfig_GetWStr(config, name, &value)``.
@@ -284,7 +389,6 @@ initialization:
 * ``Py_PreInitializeFromInitConfig(config)``.
 * ``Py_InitializeFromInitConfig(config)``.
 * ``PyInitConfig_GetError(config, &err_msg)``.
-* ``Py_ExitWithInitConfig(config)``.
 
 Add C API and Python functions to get the current configuration:
 
@@ -292,6 +396,7 @@ Add C API and Python functions to get the current configuration:
 * ``PyConfig_GetInt(name, &value)``.
 * ``PyConfig_Keys()``.
 * ``sys.get_config(name)``.
+* ``sys.get_config_names()``.
 
 The C API uses null-terminated UTF-8 encoded strings to refer to a
 configuration option.
@@ -402,7 +507,7 @@ Preconfiguration
 ----------------
 
 Calling ``Py_PreInitializeFromInitConfig()`` preinitializes Python. For
-example, it sets the memory allocation, and can configure the
+example, it sets the memory allocator, and can configure the
 ``LC_CTYPE`` locale and configure the standard C streams such as
 ``stdin`` and ``stdout``.
 
@@ -420,7 +525,7 @@ Trying to set these options after Python preinitialization fails with an
 error.
 
 ``PyInitConfig_SetStrLocale()`` and ``PyInitConfig_SetStrLocaleList()``
-functions cannot be called Python preinitialization.
+functions cannot be called before the Python preinitialization.
 
 
 Create PyInitConfig
@@ -450,6 +555,14 @@ Create PyInitConfig
 Get PyInitConfig Options
 ------------------------
 
+The configuration option *name* parameter must be a non-NULL
+null-terminated UTF-8 encoded string.
+
+``int PyInitConfig_HasOption(PyInitConfig *config, const char *name)``:
+    Test if the configuration has an option called *name*.
+
+    Return ``1`` if the option exists, or return ``0`` otherwise.
+
 ``int PyInitConfig_GetInt(PyInitConfig *config, const char *name, int64_t *value)``:
     Get an integer configuration option.
 
@@ -488,7 +601,7 @@ Get PyInitConfig Options
     ``PyInitConfig_GetStrList()``.
 
 ``int PyInitConfig_GetWStrList(PyInitConfig *config, const char *name, size_t *length, wchar_t ***items)``:
-    Get a string list configuration option as an error of
+    Get a string list configuration option as an array of
     null-terminated wide strings.
 
     * Set *\*length* and *\*value*, and return ``0`` on success.
@@ -505,6 +618,14 @@ Get PyInitConfig Options
 Set PyInitConfig Options
 ------------------------
 
+The configuration option *name* parameter must be a non-NULL
+null-terminated UTF-8 encoded string.
+
+Some configuration options have side effects on other options. This
+logic is only implemented when ``Py_InitializeFromInitConfig()`` is
+called, not by the "Set" functions below. For example, setting
+``"dev_mode"`` to ``1`` does not set ``"faulthandler"`` to ``1``.
+
 ``int PyInitConfig_SetInt(PyInitConfig *config, const char *name, int64_t value)``:
     Set an integer configuration option.
 
@@ -610,17 +731,13 @@ Error handling
    function is called with *config*. The caller doesn't have to free the
    error message.
 
-``void Py_ExitWithInitConfig(PyInitConfig *config)``:
-    Exit Python and free memory of an initialization configuration.
-
-    If an error message is set, display the error message.
-
-    The function does not return.
-
 
 Get current configuration
 -------------------------
 
+The configuration option *name* parameter must be a non-NULL
+null-terminated UTF-8 encoded string.
+
 ``PyObject* PyConfig_Get(const char *name)``:
     Get the current value of a configuration option as an object.
 
@@ -650,8 +767,8 @@ Get current configuration
 
     Other options are get from internal ``PyPreConfig`` and ``PyConfig`` structures.
 
-    The function cannot be called before Python initialization nor
+    The caller must hold the GIL. The function cannot be called before
-    after Python finalization.
+    Python initialization nor after Python finalization.
 
 ``int PyConfig_GetInt(const char *name, int *value)``:
     Similar to ``PyConfig_Get()``, but get the value as an integer.
@@ -660,10 +777,12 @@ Get current configuration
     * Set an exception and return ``-1`` on error.
 
 ``PyObject* PyConfig_Keys(void)``:
-    Get all configuration option names as a tuple.
+    Get all configuration option names as a ``frozenset``.
 
     Set an exception and return ``NULL`` on error.
 
+    The caller must hold the GIL.
+
 
 sys.get_config()
 ----------------
@@ -675,23 +794,42 @@ Add ``sys.get_config(name: str)`` function which calls
 * Raise an exception on error.
 
 
+sys.get_config_names()
+----------------------
+
+Add ``sys.get_config_names()`` function which gets all configuration
+option names as a ``frozenset``.
+
+Scope of the stable ABI
+-----------------------
+
+The limited C API and the stable ABI added by this PEP only provide a
+stable interface to program the Python initialization.
+
+The behavior of options, the default option values, and the Python
+behavior can change at each Python version: they are not "stable".
+
+Moreover, configuration options can be added, deprecated and removed
+following the usual :pep:`387` deprecation process.
+
+
 Examples
 ========
 
 Initialize Python
 -----------------
 
-Example setting some configuration options of different types to
+Example initializing Python, set configuration options of different types,
-initialize Python.
+return -1 on error:
 
 .. code-block:: c
 
-    void init_python(void)
+    int init_python(void)
     {
         PyInitConfig *config = PyInitConfig_CreatePython();
         if (config == NULL) {
-            printf("Init allocation error\n");
+            printf("PYTHON INIT ERROR: memory allocation failed\n");
-            return;
+            return -1;
         }
 
         // Set an integer (dev mode)
@@ -724,19 +862,46 @@ initialize Python.
             goto error;
         }
         PyInitConfig_Free(config);
-        return;
+        return 0;
 
     error:
-        // Display the error message an exit the process
+        // Display the error message
-        // with a non-zero exit code
+        const char *err_msg;
-        Py_ExitWithInitConfig(config);
+        (void)PyInitConfig_GetError(config, &err_msg);
+        printf("PYTHON INIT ERROR: %s\n", err_msg);
+        PyInitConfig_Free(config);
+
+        return -1;
     }
 
 
-Get the verbose option
+Increase initialization bytes_warning option
------------------------
+--------------------------------------------
 
-Example getting the configuration option ``verbose``:
+Example increasing the ``bytes_warning`` option of an initialization
+configuration:
+
+.. code-block:: c
+
+    int config_bytes_warning(PyInitConfig *config)
+    {
+        int bytes_warning;
+        if (PyInitConfig_GetInt(config, "bytes_warning", &bytes_warning)) {
+            return -1;
+        }
+        bytes_warning += 1;
+        if (PyInitConfig_SetInt(config, "bytes_warning", bytes_warning)) {
+            return -1;
+        }
+        return 0;
+    }
+
+
+Get the current verbose option
+------------------------------
+
+Example getting the current value of the configuration option
+``verbose``:
 
 .. code-block:: c
 
@@ -851,6 +1016,44 @@ read once and cached. By the way, most configuration options cannot be
 changed at runtime.
 
 
+Fully remove the preinitialization
+----------------------------------
+
+Delay decoding
+^^^^^^^^^^^^^^
+
+Without ``PyInitConfig_Get*()`` functions, it would be possible to store
+``PyInitConfig_SetStrLocale()`` and ``PyInitConfig_SetStrLocaleList()``
+strings encoded and only initialize the ``LC_CTYPE`` locale and
+decode the strings in ``Py_InitializeFromInitConfig()``.
+
+The problem is that users asked for ``PyInitConfig_Get*()`` functions.
+
+Also, if ``PyInitConfig_SetStrLocale()`` and
+``PyInitConfig_SetStrLocaleList()`` strings are decoded as designed by
+the PEP, there is no risk of mojibake: ``PyInitConfig_GetStr()`` returns
+the expected decoded strings.
+
+Remove the Python configuration
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+If ``PyInitConfig_CreatePython()`` is removed, the preinitialization is
+no longer needed since the ``LC_CTYPE`` is not configured by default by
+``PyInitConfig_CreateIsolated()`` and setting ``"configure_locale"``
+option can always fail.
+
+The problem is that users asked to be able to write their own customized
+Python, so have a Python-like program but with a different default
+configuration. The ``PyInitConfig_CreatePython()`` function is needed
+for that.
+
+The Python configuration is also part of the :pep:`587` design,
+implemented in Python 3.8.
+
+Disallow setting the ``"configure_locale"`` option has similar issues.
+
+
+
 Discussions
 ===========