iSharkFly-Open
/
python-peps
mirror of https://github.com/python/peps.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

PEP 741: Add sys.get_config_names() (#3686) 

* PEP 741: Add sys.get_config_names()

* Add sys.get_config_names() function.
* Add PyInitConfig_HasOption() function.
* Remove Py_ExitWithInitConfig() function.
* Add "Fully remove the preinitialization" section.
* Mention when the caller must hold the GIL.
* Add example increasing an initialization configuration option.
* "Usage of the stable ABI" section: add more quotes.
* Document that options side effects are not taken in
  account by PyInitConfig_Set*() functions.
* Add "Spawnw process" section.
* Add Cython rationale.
* Add myself as PEP 741 code owner.
This commit is contained in:
Victor Stinner 2024-03-08 15:35:01 +01:00 committed by GitHub
parent 95305ef99a
commit 7e7296bfdc
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
2 changed files with 248 additions and 44 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
.github/CODEOWNERS vendored
View File

@ -619,6 +619,7 @@ peps/pep-0736.rst @gvanrossum @Rosuav
peps/pep-0737.rst @vstinner
peps/pep-0738.rst @encukou
peps/pep-0740.rst @dstufft
peps/pep-0741.rst @vstinner
peps/pep-0742.rst @JelleZijlstra
# ...
# peps/pep-0754.rst

291
peps/pep-0741.rst
View File

@ -16,10 +16,10 @@ Add a C API to the limited C API to configure the Python initialization,
and to get the current configuration. It can be used with the stable
ABI.
Add also ``sys.get_config(name)`` function to get the current
configuration.
Add also ``sys.get_config(name)`` and ``sys.get_config_names()``
functions to get the current configuration.
Complete PEP 587 API by adding ``PyInitConfig_AddModule()`` which can be
Complete :pep:`587` API by adding ``PyInitConfig_AddModule()`` which can be
used to add a built-in extension module; feature previously referred to
as the "inittab".
@ -58,6 +58,7 @@ Python in Rust (but it's not the default). In practice, PyO3 can use
non-limited C API for specific needs, but using them avoids the stable
ABI advantages.
Limitations of the legacy API
-----------------------------
@ -85,6 +86,7 @@ surprising and unwanted behavior.
Some configuration options, such as ``configure_locale``, simply cannot
be set.
Limitations of the limited C API
--------------------------------
@ -97,6 +99,7 @@ they are still part of the stable ABI. For example, building a
application with the limited C API version 3.12 can still run with
Python 3.13 stable ABI.
Get the current configuration
-----------------------------
@ -112,6 +115,16 @@ configure Python, there is no public API to get
Users of the limited C API are asking for a public API to get the
current configuration.
Cython needs to access the ``optimization_level`` configuration option:
`issue <https://github.com/python/cpython/issues/99872>`_.
When global configuration variables were deprecated in 2022, `Marc-André
Lemburg requested
<https://github.com/python/cpython/issues/93103#issuecomment-1136462708>`_
an API to access these configuration variables at runtime (not only
during Python initialization).
Security fix
------------
@ -133,6 +146,7 @@ member to the development branch (which became Python 3.12). A dedicated
private global variable (unrelated to ``PyConfig``) is used in stable
branches.
Redundancy between PyPreConfig and PyConfig
-------------------------------------------
@ -145,8 +159,12 @@ The redundancy is caused by the fact that the two structures are
separated, whereas some ``PyConfig`` members are needed by the
preinitialization.
Embedding Python
----------------
Applications embedding Python
-----------------------------
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Examples:
@ -160,8 +178,8 @@ Examples:
On Linux, FreeBSD and macOS, applications are usually either statically
linked to a ``libpython``, or load dynamically a ``libpython`` . The
``libpython`` shared library is versionned, example:
``libpython3.12.so`` for Python 3.13 on Linux.
``libpython`` shared library is versioned, example:
``libpython3.12.so`` for Python 3.12 on Linux.
The vim project can target the stable ABI. Usually, the "system Python"
version is used. It's not currently possible to select which Python
@ -174,7 +192,7 @@ such as GIMP, is to include Python in Flatpack, AppImage or Snap
version with the container.
Libraries embedding Python
--------------------------
^^^^^^^^^^^^^^^^^^^^^^^^^^
Examples:
@ -186,7 +204,7 @@ Examples:
Rust bindings for the Python interpreter.
Utilities creating standalone applications
------------------------------------------
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
* `py2app <https://py2app.readthedocs.io/>`_ for macOS.
* `py2exe <http://www.py2exe.org/>`_ for Windows.
@ -213,7 +231,7 @@ Usage of a stable ABI
You can’t **extend a struct** and **assume embedding people all
rebuild**. They don’t. Real world embedding uses exist that use an
installed Python minor version as a shared library. Update that to
installed Python minor version as a **shared library**. Update that to
use a different sized struct in a public API and someone is going to
have a bad time. That’s why I consider the struct frozen at rc1
time, even when only for use in the embedding / writing their own
@ -230,17 +248,40 @@ Usage of a stable ABI
(...)
I am strictly limited to what’s in the shared library (DLL). I don’t
have headers, I can’t statically “recompile” every time a new
version of python comes out. That’s unmaintainable for me.
I am strictly limited to what’s in the **shared library** (DLL). I
**don’t have headers**, I can’t statically “recompile” every time a
new version of python comes out. That’s unmaintainable for me.
`Milian Wolff
<https://github.com/python/cpython/issues/107954#issuecomment-1893988614>`__:
Quotes of Milian Wolff's `message
<https://discuss.python.org/t/pep-741-python-configuration-c-api-second-version/45403/4>`__:
IIUC then there's still no non-deprecated API in the **limited C API
to customize the initialization**, right? Can you then please reopen
this task to indicate that this? Or should I report a separate issue
to track this? Thank you
Our application is a large complex C++ code base with lots of
dependencies targeting all three major desktop platforms.
Originally, we hoped to be able to use the **stable python ABI** to
allow biologists to **“bring your own python”**. The idea was that
they probably have a custom set of python libraries and code that
they would like to continue using. Our integrated API - so we
thought - was a tiny addition that should work with any Python out
there, so we used the stable ABI.
This turned out to be a dead end, and I believe we can (should?) now
use the non-stable ABI of python. Allowing end users to BYO Python
caused far too much setup problems and support issues for us that it
was not worth it in the end. Instead, we now rather want to ship a
custom Python with a custom prefix that they can pip install custom
libraries into as needed.
The problems we faced are not directly related to the stable ABI -
quite the contrary. Rather, it was due to thirdparty python
libraries that we shipped which themselves are not compatible across
python version increments. E.g. for the integrated console we use
qtconsole/jupyter, which worked in an archaic version with python
3.9 but requires newer versions for python 3.11+.
The ton of dependencies pulled in by UMAP was even worse, with numba
and pydnndescent and llvmlite often taking months to support newer
Python versions.
`David Hewitt <https://discuss.python.org/t/pep-741-python-configuration-c-api/43637/38>`__ of the PyO3 project:
@ -255,6 +296,69 @@ Usage of a stable ABI
if configuration values are not available for a specific version
at runtime.
Quotes of `Paul P. message <https://discuss.python.org/t/pep-741-python-configuration-c-api-second-version/45403/5>`__:
I cannot agree more, it is the same story everywhere/every time
CPython must be **embedded**. I maintened a runtime+ecosystem for
Android 4.4+ for some time (in order more comfortably use Panda3D
standalone than with Kivy), patching CPython and making a CI for it
was ok.
But I had to give up, because I had often to recompile every known
modules: this is not sustainable for one individual.
So I dropped the Android arch to only go WebAssembly (Emscripten).
But same (hard and boring) problem as always: have to rebuild
numerous packages that are commonly used with 2D/3D framework. (...)
Except for ONE, Harfang3d. I did not rebuild this one since Python
3.11 initial port… Guess why? it is a limited C API - **abi3
module**!
Limited API abi3 are fresh air, fast and portable. And associated
with a **stable config runtime**, it would be just perfect way!
Spawn a new Python process with the same configuration
------------------------------------------------------
The Python test suite runner spawns test worker processes by creating a
new command line with the same configuration options. For that, it calls
the ``args_from_interpreter_flags()`` of ``test.support`` module which
calls ``_args_from_interpreter_flags()`` of the ``subprocess`` module.
These functions inspect ``sys.flags``, ``sys.warnoptions`` and
``sys._xoptions``.
The problem is that every time a new configuration is added, it should
be exposed to ``sys.flags``. Otherwise,
``args_from_interpreter_flags()`` ignores the option and so the option
is not inherited by child processes.
Another problem is that inspecting ``sys._xoptions`` doesn't take
environment variables in account.
Python 3.10 added ``sys.orig_argv`` for a specific implementation: copy
and then modify the original command line option to add or remove
command line optioins. This method also has a limitation, it ignores
environment variables.
Examples of ``-X`` options which are not exposed in ``sys.flags``:
* ``code_debug_ranges``,
* ``cpu_count``,
* ``import_time``,
* ``int_max_str_digits``,
* ``perf_profiling``,
* ``pycache_prefix``,
* ``run_presite`` (only on a Python debug build),
* ``show_ref_count``.
Some of these options are inherited by inspecting ``sys._xoptions``
which doesn't take in account environment variables.
The ``sys.get_int_max_str_digits()`` function can be used to get the
``int_max_str_digits`` option.
Specification
=============
@ -266,6 +370,7 @@ initialization:
* ``PyInitConfig_CreatePython()``.
* ``PyInitConfig_CreateIsolated()``.
* ``PyInitConfig_Free(config)``.
* ``PyInitConfig_HasOption(config, name)``.
* ``PyInitConfig_GetInt(config, name, &value)``.
* ``PyInitConfig_GetStr(config, name, &value)``.
* ``PyInitConfig_GetWStr(config, name, &value)``.
@ -284,7 +389,6 @@ initialization:
* ``Py_PreInitializeFromInitConfig(config)``.
* ``Py_InitializeFromInitConfig(config)``.
* ``PyInitConfig_GetError(config, &err_msg)``.
* ``Py_ExitWithInitConfig(config)``.
Add C API and Python functions to get the current configuration:
@ -292,6 +396,7 @@ Add C API and Python functions to get the current configuration:
* ``PyConfig_GetInt(name, &value)``.
* ``PyConfig_Keys()``.
* ``sys.get_config(name)``.
* ``sys.get_config_names()``.
The C API uses null-terminated UTF-8 encoded strings to refer to a
configuration option.
@ -402,7 +507,7 @@ Preconfiguration
----------------
Calling ``Py_PreInitializeFromInitConfig()`` preinitializes Python. For
example, it sets the memory allocation, and can configure the
example, it sets the memory allocator, and can configure the
``LC_CTYPE`` locale and configure the standard C streams such as
``stdin`` and ``stdout``.
@ -420,7 +525,7 @@ Trying to set these options after Python preinitialization fails with an
error.
``PyInitConfig_SetStrLocale()`` and ``PyInitConfig_SetStrLocaleList()``
functions cannot be called Python preinitialization.
functions cannot be called before the Python preinitialization.
Create PyInitConfig
@ -450,6 +555,14 @@ Create PyInitConfig
Get PyInitConfig Options
------------------------
The configuration option *name* parameter must be a non-NULL
null-terminated UTF-8 encoded string.
``int PyInitConfig_HasOption(PyInitConfig *config, const char *name)``:
Test if the configuration has an option called *name*.
Return ``1`` if the option exists, or return ``0`` otherwise.
``int PyInitConfig_GetInt(PyInitConfig *config, const char *name, int64_t *value)``:
Get an integer configuration option.
@ -488,7 +601,7 @@ Get PyInitConfig Options
``PyInitConfig_GetStrList()``.
``int PyInitConfig_GetWStrList(PyInitConfig *config, const char *name, size_t *length, wchar_t ***items)``:
Get a string list configuration option as an error of
Get a string list configuration option as an array of
null-terminated wide strings.
* Set *\*length* and *\*value*, and return ``0`` on success.
@ -505,6 +618,14 @@ Get PyInitConfig Options
Set PyInitConfig Options
------------------------
The configuration option *name* parameter must be a non-NULL
null-terminated UTF-8 encoded string.
Some configuration options have side effects on other options. This
logic is only implemented when ``Py_InitializeFromInitConfig()`` is
called, not by the "Set" functions below. For example, setting
``"dev_mode"`` to ``1`` does not set ``"faulthandler"`` to ``1``.
``int PyInitConfig_SetInt(PyInitConfig *config, const char *name, int64_t value)``:
Set an integer configuration option.
@ -610,17 +731,13 @@ Error handling
function is called with *config*. The caller doesn't have to free the
error message.
``void Py_ExitWithInitConfig(PyInitConfig *config)``:
Exit Python and free memory of an initialization configuration.
If an error message is set, display the error message.
The function does not return.
Get current configuration
-------------------------
The configuration option *name* parameter must be a non-NULL
null-terminated UTF-8 encoded string.
``PyObject* PyConfig_Get(const char *name)``:
Get the current value of a configuration option as an object.
@ -650,8 +767,8 @@ Get current configuration
Other options are get from internal ``PyPreConfig`` and ``PyConfig`` structures.
The function cannot be called before Python initialization nor
after Python finalization.
The caller must hold the GIL. The function cannot be called before
Python initialization nor after Python finalization.
``int PyConfig_GetInt(const char *name, int *value)``:
Similar to ``PyConfig_Get()``, but get the value as an integer.
@ -660,10 +777,12 @@ Get current configuration
* Set an exception and return ``-1`` on error.
``PyObject* PyConfig_Keys(void)``:
Get all configuration option names as a tuple.
Get all configuration option names as a ``frozenset``.
Set an exception and return ``NULL`` on error.
The caller must hold the GIL.
sys.get_config()
----------------
@ -675,23 +794,42 @@ Add ``sys.get_config(name: str)`` function which calls
* Raise an exception on error.
sys.get_config_names()
----------------------
Add ``sys.get_config_names()`` function which gets all configuration
option names as a ``frozenset``.
Scope of the stable ABI
-----------------------
The limited C API and the stable ABI added by this PEP only provide a
stable interface to program the Python initialization.
The behavior of options, the default option values, and the Python
behavior can change at each Python version: they are not "stable".
Moreover, configuration options can be added, deprecated and removed
following the usual :pep:`387` deprecation process.
Examples
========
Initialize Python
-----------------
Example setting some configuration options of different types to
initialize Python.
Example initializing Python, set configuration options of different types,
return -1 on error:
.. code-block:: c
void init_python(void)
int init_python(void)
{
PyInitConfig *config = PyInitConfig_CreatePython();
if (config == NULL) {
printf("Init allocation error\n");
return;
printf("PYTHON INIT ERROR: memory allocation failed\n");
return -1;
}
// Set an integer (dev mode)
@ -724,19 +862,46 @@ initialize Python.
goto error;
}
PyInitConfig_Free(config);
return;
return 0;
error:
// Display the error message an exit the process
// with a non-zero exit code
Py_ExitWithInitConfig(config);
// Display the error message
const char *err_msg;
(void)PyInitConfig_GetError(config, &err_msg);
printf("PYTHON INIT ERROR: %s\n", err_msg);
PyInitConfig_Free(config);
return -1;
}
Get the verbose option
-----------------------
Increase initialization bytes_warning option
--------------------------------------------
Example getting the configuration option ``verbose``:
Example increasing the ``bytes_warning`` option of an initialization
configuration:
.. code-block:: c
int config_bytes_warning(PyInitConfig *config)
{
int bytes_warning;
if (PyInitConfig_GetInt(config, "bytes_warning", &bytes_warning)) {
return -1;
}
bytes_warning += 1;
if (PyInitConfig_SetInt(config, "bytes_warning", bytes_warning)) {
return -1;
}
return 0;
}
Get the current verbose option
------------------------------
Example getting the current value of the configuration option
``verbose``:
.. code-block:: c
@ -851,6 +1016,44 @@ read once and cached. By the way, most configuration options cannot be
changed at runtime.
Fully remove the preinitialization
----------------------------------
Delay decoding
^^^^^^^^^^^^^^
Without ``PyInitConfig_Get*()`` functions, it would be possible to store
``PyInitConfig_SetStrLocale()`` and ``PyInitConfig_SetStrLocaleList()``
strings encoded and only initialize the ``LC_CTYPE`` locale and
decode the strings in ``Py_InitializeFromInitConfig()``.
The problem is that users asked for ``PyInitConfig_Get*()`` functions.
Also, if ``PyInitConfig_SetStrLocale()`` and
``PyInitConfig_SetStrLocaleList()`` strings are decoded as designed by
the PEP, there is no risk of mojibake: ``PyInitConfig_GetStr()`` returns
the expected decoded strings.
Remove the Python configuration
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
If ``PyInitConfig_CreatePython()`` is removed, the preinitialization is
no longer needed since the ``LC_CTYPE`` is not configured by default by
``PyInitConfig_CreateIsolated()`` and setting ``"configure_locale"``
option can always fail.
The problem is that users asked to be able to write their own customized
Python, so have a Python-like program but with a different default
configuration. The ``PyInitConfig_CreatePython()`` function is needed
for that.
The Python configuration is also part of the :pep:`587` design,
implemented in Python 3.8.
Disallow setting the ``"configure_locale"`` option has similar issues.
Discussions
===========