PEP 587: API v3 (#1052)

* Thomas Wouters is now the BDFL-delegate
* Elaborate the Rationale
* New "Path configuration" section
* New "Python Issues" section
* Add PyWideStringList_Insert() function
* Add PyConfig.configure_c_stdio option.
* Add PyConfig.parse_argv field
* Rename bytes/wide functions. Add "Bytes" to functions taking char*,
  remove "Wide" from functions taking wchar_t*. Rename
  PyConfig_DecodeLocale() to PyConfig_SetBytesString().
* PyConfig: Rename '_frozen' to 'pathconfig_warnings'
* Re-introduce PyConfig._init_main
* Complete tables of global configuration variables; annotate
  variable values which are negated.
* Add default values of all PyPreConfig and PyConfig fields
* Elaborate the Rationale
* Add a few more issues
* Add PyPreConfig.legacy_windows_fs_encoding rules
* Add "Isolate Python" section
* PyConfig_SetString() and PyConfig_SetBytesString() now requires the
  configuration as the first argument.
* Rename Py_UnixMain() to Py_BytesMain()
This commit is contained in:
Victor Stinner 2019-05-15 18:50:11 +02:00 committed by GitHub
parent 81d2b5ccd7
commit 4cefacceba
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 484 additions and 209 deletions

View File

@ -1,6 +1,7 @@
PEP: 587
Title: Python Initialization Configuration
Author: Victor Stinner <vstinner@redhat.com>, Nick Coghlan <ncoghlan@gmail.com>
BDFL-Delegate: Thomas Wouters <thomas@python.org>
Discussions-To: python-dev@python.org
Status: Draft
Type: Standards Track
@ -14,6 +15,16 @@ Abstract
Add a new C API to configure the Python Initialization providing finer
control on the whole configuration and better error reporting.
It becomes possible to read the configuration and modify it before it is
applied. It also becomes possible to completely override how Python
computes the module search paths (``sys.path``).
Building a customized Python which behaves as regular Python becomes
easier using the new ``Py_RunMain()`` function. Moreover, command line
arguments passed to ``PyConfig.argv`` are now parsed as the regular
Python parses command line options, and ``PyConfig.xoptions`` are
handled as ``-X opt`` command line options.
This extracts a subset of the API design from the PEP 432 development and
refactoring work that is now considered sufficiently stable to make public
(allowing 3rd party embedding applications access to the same configuration
@ -23,16 +34,51 @@ APIs that the native CPython CLI is now using).
Rationale
=========
Python is highly configurable but its configuration evolved organically:
configuration parameters is scattered all around the code using
different ways to set them (mostly global configuration variables and
functions). A straightforward and reliable way to configure Python is
needed. Some configuration parameters are not accessible from the C API,
or not easily.
Python is highly configurable but its configuration evolved organically.
The initialization configuration is scattered all around the code using
different ways to set them: global configuration variables (ex:
``Py_IsolatedFlag``), environment variables (ex: ``PYTHONPATH``),
command line arguments (ex: ``-b``), configuration files (ex:
``pyvenv.cfg``), function calls (ex: ``Py_SetProgramName()``). A
straightforward and reliable way to configure Python is needed.
Some configuration parameters are not accessible from the C API, or not
easily. For example, there is no API to override the default values of
``sys.executable``.
Some options like ``PYTHONPATH`` can only be set using an environment
variable which has a side effect on Python child processes.
Some options also depends on other options: see `Priority and Rules`_.
Python 3.7 API does not provide a consistent view of the overall
configuration.
The C API of Python 3.7 Initialization takes ``wchar_t*`` strings as
input whereas the Python filesystem encoding is set during the
initialization.
initialization which can lead to mojibake.
Python 3.7 APIs like ``Py_Initialize()`` aborts the process on memory
allocation failure which is not convenient when Python is embedded.
Moreover, ``Py_Main()`` could exit directly the process rather than
returning an exit code. Proposed new API reports the error or exit code
to the caller which can decide how to handle it.
Implementing the PEP 540 (UTF-8 Mode) and the new ``-X dev`` correctly
was almost impossible in Python 3.6. The code base has been deeply
reworked in Python 3.7 and then in Python 3.8 to read the configuration
into a structure with no side effect. It becomes possible to clear the
configuration (release memory) and read again the configuration if the
encoding changed . It is required to implement properly the UTF-8 which
changes the encoding using ``-X utf8`` command line option. Internally,
bytes ``argv`` strings are decoded from the filesystem encoding. The
``-X dev`` changes the memory allocator (behaves as
``PYTHONMALLOC=debug``), whereas it was not possible to change the
memory allocation *while* parsing the command line arguments. The new
design of the internal implementation not only allowed to implement
properly ``-X utf8`` and ``-X dev``, it also allows to change the Python
behavior way more easily, especially for corner cases like that, and
ensure that the configuration remains consistent: see `Priority and
Rules`_.
This PEP is a partial implementation of PEP 432 which is the overall
design. New fields can be added later to ``PyConfig`` structure to
@ -57,24 +103,25 @@ New structures (4):
* ``PyPreConfig``
* ``PyWideStringList``
New functions (16):
New functions (17):
* ``Py_PreInitialize(config)``
* ``Py_PreInitializeFromBytesArgs(config, argc, argv)``
* ``Py_PreInitializeFromArgs(config, argc, argv)``
* ``Py_PreInitializeFromWideArgs(config, argc, argv)``
* ``PyWideStringList_Append(list, item)``
* ``PyConfig_DecodeLocale(config_str, str)``
* ``PyConfig_SetString(config_str, str)``
* ``PyConfig_Read(config)``
* ``PyWideStringList_Insert(list, index, item)``
* ``PyConfig_SetString(config,config_str, str)``
* ``PyConfig_SetBytesString(config, config_str, str)``
* ``PyConfig_SetBytesArgv(config, argc, argv)``
* ``PyConfig_SetArgv(config, argc, argv)``
* ``PyConfig_SetWideArgv(config, argc, argv)``
* ``PyConfig_Read(config)``
* ``PyConfig_Clear(config)``
* ``Py_InitializeFromConfig(config)``
* ``Py_InitializeFromBytesArgs(config, argc, argv)``
* ``Py_InitializeFromArgs(config, argc, argv)``
* ``Py_InitializeFromWideArgs(config, argc, argv)``
* ``Py_UnixMain(argc, argv)``
* ``Py_ExitInitError(err)``
* ``Py_BytesMain(argc, argv)``
* ``Py_RunMain()``
* ``Py_ExitInitError(err)``
New macros (9):
@ -119,6 +166,9 @@ Methods:
* ``PyInitError PyWideStringList_Append(PyWideStringList *list, const wchar_t *item)``:
Append *item* to *list*.
* ``PyInitError PyWideStringList_Insert(PyWideStringList *list, Py_ssize_t index, const wchar_t *item)``:
Insert *item* into *list* at *index*. If *index* is greater than
*list* length, just append *item* to *list*.
If *length* is non-zero, *items* must be non-NULL and all strings must
be non-NULL.
@ -164,10 +214,10 @@ Example::
Macro to create an error:
* ``Py_INIT_OK()``: success
* ``Py_INIT_ERR(err_msg)``: initialization error with a message
* ``Py_INIT_NO_MEMORY()``: memory allocation failure (out of memory)
* ``Py_INIT_EXIT(exitcode)``: exit Python with the specified exit code
* ``Py_INIT_OK()``: Success.
* ``Py_INIT_ERR(err_msg)``: Initialization error with a message.
* ``Py_INIT_NO_MEMORY()``: Memory allocation failure (out of memory).
* ``Py_INIT_EXIT(exitcode)``: Exit Python with the specified exit code.
Other macros and functions:
@ -208,8 +258,8 @@ Example using the pre-initialization to enable the UTF-8 Mode::
Functions to pre-initialize Python:
* ``PyInitError Py_PreInitialize(const PyPreConfig *config)``
* ``PyInitError Py_PreInitializeFromArgs(const PyPreConfig *config, int argc, char **argv)``
* ``PyInitError Py_PreInitializeFromWideArgs(const PyPreConfig *config, int argc, wchar_t **argv)``
* ``PyInitError Py_PreInitializeFromBytesArgs(const PyPreConfig *config, int argc, char **argv)``
* ``PyInitError Py_PreInitializeFromArgs(const PyPreConfig *config, int argc, wchar_t **argv)``
If Python should be pre-initialized explicitly first and then
initialized with command line arguments, it is possible to pass these
@ -222,27 +272,36 @@ is responsible to handle error using ``Py_INIT_FAILED()`` and
``PyPreConfig`` fields:
* ``allocator`` (``char*``): name of the memory allocator (ex: ``"malloc"``)
* ``coerce_c_locale_warn`` (``int``): if non-zero, emit a warning if the C locale
is coerced.
* ``coerce_c_locale`` (``int``): if equals to 2, coerce the C locale; if equals to
1, read the LC_CTYPE to decide if it should be coerced.
* ``dev_mode`` (``int``): see ``PyConfig.dev_mode``
* ``isolated`` (``int``): see ``PyConfig.isolated``
* ``legacy_windows_fs_encoding`` (``int``, Windows only): if non-zero, set the
Python filesystem encoding to ``"mbcs"``.
* ``use_environment`` (``int``): see ``PyConfig.use_environment``
* ``utf8_mode`` (``int``): if non-zero, enable the UTF-8 mode
* ``allocator`` (``char*``, default: ``NULL``):
Name of the memory allocator (ex: ``"malloc"``).
* ``coerce_c_locale`` (``int``, default: 0):
If equals to 2, coerce the C locale; if equals to 1, read the LC_CTYPE
locale to decide if it should be coerced.
* ``coerce_c_locale_warn`` (``int``, default: 0):
If non-zero, emit a warning if the C locale is coerced.
* ``dev_mode`` (``int``, default: 0):
See ``PyConfig.dev_mode``.
* ``isolated`` (``int``, default: 0):
See ``PyConfig.isolated``.
* ``legacy_windows_fs_encoding`` (``int``, Windows only, default: 0):
If non-zero, disable UTF-8 Mode, set the Python filesystem encoding to
``mbcs``, set the filesystem error handler to ``replace``.
* ``use_environment`` (``int``, default: 1):
See ``PyConfig.use_environment``.
* ``utf8_mode`` (``int``, default: 0):
If non-zero, enable the UTF-8 mode.
There is also a private field which is for internal-usage only:
* ``_config_version`` (``int``): Configuration version, used for ABI
compatibility
* ``_config_version`` (``int``, default: config version):
Configuration version, used for ABI compatibility.
The C locale coercion (PEP 538) and the UTF-8 Mode (PEP 540) are
disabled by default in ``PyPreConfig``. Set ``coerce_c_locale``,
``coerce_c_locale_warn`` and ``utf8_mode`` to ``-1`` to let Python
enable them depending on the user configuration.
enable them depending on the user configuration. In this case, it's
safer to explicitly pre-initialize Python to ensure that encodings are
configured before the Python initialization starts.
Initialization with PyConfig
----------------------------
@ -268,21 +327,22 @@ Example::
``PyConfig`` methods:
* ``PyInitError PyConfig_SetString(wchar_t **config_str, const wchar_t *str)``:
Set a config wide string field from *str* (copy the string)
* ``PyInitError PyConfig_DecodeLocale(wchar_t **config_str, const char *str)``:
* ``PyInitError PyConfig_SetString(PyConfig *config, wchar_t **config_str, const wchar_t *str)``:
Set a config wide string field from *str* (copy the string).
* ``PyInitError PyConfig_SetBytesString(PyConfig *config, wchar_t **config_str, const char *str)``:
Decode *str* using ``Py_DecodeLocale()`` and set the result into
``*config_str``. Pre-initialize Python if needed to ensure that
encodings are properly configured.
* ``PyInitError PyConfig_SetArgv(PyConfig *config, int argc, char **argv)``:
Set command line arguments (decode bytes). Pre-initialize Python if
needed to ensure that encodings are properly configured.
* ``PyInitError PyConfig_SetWideArgv(PyConfig *config, int argc, wchar_t **argv)``:
* ``PyInitError PyConfig_SetArgv(PyConfig *config, int argc, wchar_t **argv)``:
Set command line arguments (wide characters).
* ``PyInitError PyConfig_SetBytesArgv(PyConfig *config, int argc, char **argv)``:
Set command line arguments (decode bytes using ``Py_DecodeLocale()``).
Pre-initialize Python if needed to ensure that encodings are properly
configured.
* ``PyInitError PyConfig_Read(PyConfig *config)``:
Read all Python configuration
Read all Python configuration.
* ``void PyConfig_Clear(PyConfig *config)``:
Release memory
Release configuration memory.
Functions to initialize Python:
@ -294,79 +354,135 @@ is responsible to handler error using ``Py_INIT_FAILED()`` and
PyConfig fields:
* ``argv`` (``PyWideStringList``): ``sys.argv``
* ``base_exec_prefix`` (``wchar_t*``): ``sys.base_exec_prefix``
* ``base_prefix`` (``wchar_t*``): ``sys.base_prefix``
* ``buffered_stdio`` (``int``): if equals to 0, enable unbuffered mode,
make stdout and stderr streams to be unbuffered.
* ``bytes_warning`` (``int``): if equals to 1, issue a warning when
comparing ``bytes`` or ``bytearray`` with ``str``, or comparing
``bytes`` with ``int``. If equal or greater to 2, raise a
``BytesWarning`` exception.
* ``check_hash_pycs_mode`` (``wchar_t*``): ``--check-hash-based-pycs``
command line option value (see PEP 552)
* ``dev_mode`` (``int``): Development mode
* ``dll_path`` (``wchar_t*``, Windows only): Windows DLL path
* ``dump_refs`` (``int``): if non-zero, display all objects still alive
at exit
* ``exec_prefix`` (``wchar_t*``): ``sys.exec_prefix``
* ``executable`` (``wchar_t*``): ``sys.executable``
* ``faulthandler`` (``int``): if non-zero, call
``faulthandler.enable()``
* ``filesystem_encoding`` (``wchar_t*``): Filesystem encoding,
``sys.getfilesystemencoding()``
* ``filesystem_errors`` (``wchar_t*``): Filesystem encoding errors,
``sys.getfilesystemencodeerrors()``
* ``use_hash_seed`` (``int``), ``hash_seed`` (``unsigned long``):
randomized hash function seed
* ``home`` (``wchar_t*``): Python home
* ``import_time`` (``int``): if non-zero, profile import time
* ``inspect`` (``int``): enter interactive mode after executing a script or a
command
* ``install_signal_handlers`` (``int``): install signal handlers?
* ``interactive`` (``int``): interactive mode
* ``legacy_windows_stdio`` (``int``, Windows only): if non-zero, use
``io.FileIO`` instead of ``WindowsConsoleIO`` for ``sys.stdin``,
``sys.stdout`` and ``sys.stderr``.
* ``malloc_stats`` (``int``): if non-zero, dump memory allocation
statistics at exit
* ``module_search_path_env`` (``wchar_t*``): ``PYTHONPATH`` environment variale value
* ``use_module_search_paths`` (``int``), ``module_search_paths``
(``PyWideStringList``): ``sys.path``
* ``optimization_level`` (``int``): compilation optimization level
* ``parser_debug`` (``int``): if non-zero, turn on parser debugging output (for
expert only, depending on compilation options).
* ``prefix`` (``wchar_t*``): ``sys.prefix``
* ``program_name`` (``wchar_t*``): Program name
* ``program`` (``wchar_t*``): ``argv[0]`` or an empty string
* ``pycache_prefix`` (``wchar_t*``): ``.pyc`` cache prefix
* ``quiet`` (``int``): quiet mode (ex: don't display the copyright and version
messages even in interactive mode)
* ``run_command`` (``wchar_t*``): ``-c COMMAND`` argument
* ``run_filename`` (``wchar_t*``): ``python3 SCRIPT`` argument
* ``run_module`` (``wchar_t*``): ``python3 -m MODULE`` argument
* ``show_alloc_count`` (``int``): show allocation counts at exit?
* ``show_ref_count`` (``int``): show total reference count at exit?
* ``site_import`` (``int``): import the ``site`` module at startup?
* ``skip_source_first_line`` (``int``): skip the first line of the source
* ``stdio_encoding`` (``wchar_t*``), ``stdio_errors`` (``wchar_t*``): encoding and encoding errors of
``sys.stdin``, ``sys.stdout`` and ``sys.stderr``
* ``tracemalloc`` (``int``): if non-zero, call
``tracemalloc.start(value)``
* ``user_site_directory`` (``int``): if non-zero, add user site directory to
``sys.path``
* ``verbose`` (``int``): if non-zero, enable verbose mode
* ``warnoptions`` (``PyWideStringList``): options of the ``warnings`` module to build filters
* ``write_bytecode`` (``int``): if non-zero, write ``.pyc`` files
* ``xoptions`` (``PyWideStringList``): ``sys._xoptions``
* ``argv`` (``PyWideStringList``, default: empty):
Command line arguments, ``sys.argv``.
It is parsed and updated by default, set ``parse_argv`` to 0 to avoid
that.
* ``base_exec_prefix`` (``wchar_t*``, default: ``NULL``):
``sys.base_exec_prefix``.
* ``base_prefix`` (``wchar_t*``, default: ``NULL``):
``sys.base_prefix``.
* ``buffered_stdio`` (``int``, default: 1):
If equals to 0, enable unbuffered mode, make stdout and stderr streams
to be unbuffered.
* ``bytes_warning`` (``int``, default: 0):
If equals to 1, issue a warning when comparing ``bytes`` or
``bytearray`` with ``str``, or comparing ``bytes`` with ``int``. If
equal or greater to 2, raise a ``BytesWarning`` exception.
* ``check_hash_pycs_mode`` (``wchar_t*``, default: ``"default"``):
``--check-hash-based-pycs`` command line option value (see PEP 552).
* ``configure_c_stdio`` (``int``, default: 1):
If non-zero, configure C standard streams (``stdio``, ``stdout``,
``stdout``). For example, set their mode to ``O_BINARY`` on Windows.
* ``dev_mode`` (``int``, default: 0):
Development mode
* ``dll_path`` (``wchar_t*``, Windows only, default: ``NULL``):
Windows DLL path.
* ``dump_refs`` (``int``, default: 0):
If non-zero, dump all objects which are still alive at exit
* ``exec_prefix`` (``wchar_t*``, default: ``NULL``):
``sys.exec_prefix``.
* ``executable`` (``wchar_t*``, default: ``NULL``):
``sys.executable``.
* ``faulthandler`` (``int``, default: 0):
If non-zero, call ``faulthandler.enable()``.
* ``filesystem_encoding`` (``wchar_t*``, default: ``NULL``):
Filesystem encoding, ``sys.getfilesystemencoding()``.
* ``filesystem_errors`` (``wchar_t*``, default: ``NULL``):
Filesystem encoding errors, ``sys.getfilesystemencodeerrors()``.
* ``use_hash_seed`` (``int``, default: 0),
``hash_seed`` (``unsigned long``, default: 0):
Randomized hash function seed.
* ``home`` (``wchar_t*``, default: ``NULL``):
Python home directory.
* ``import_time`` (``int``, default: 0):
If non-zero, profile import time.
* ``inspect`` (``int``, default: 0):
Enter interactive mode after executing a script or a command.
* ``install_signal_handlers`` (``int``, default: 1):
Install signal handlers?
* ``interactive`` (``int``, default: 0):
Interactive mode.
* ``legacy_windows_stdio`` (``int``, Windows only, default: 0):
If non-zero, use ``io.FileIO`` instead of ``WindowsConsoleIO`` for
``sys.stdin``, ``sys.stdout`` and ``sys.stderr``.
* ``malloc_stats`` (``int``, default: 0):
If non-zero, dump memory allocation statistics at exit.
* ``module_search_path_env`` (``wchar_t*``, default: ``NULL``):
``PYTHONPATH`` environment variale value.
* ``use_module_search_paths`` (``int``, default: 0),
``module_search_paths`` (``PyWideStringList``, default: empty):
``sys.path``.
* ``optimization_level`` (``int``, default: 0):
Compilation optimization level.
* ``parse_argv`` (``int``, default: 1):
If non-zero, parse ``argv`` command line arguments and update
``argv``.
* ``parser_debug`` (``int``, default: 0):
If non-zero, turn on parser debugging output (for expert only,
depending on compilation options).
* ``pathconfig_warnings`` (``int``, default: 1):
If equal to 0, suppress warnings when computing the path
configuration.
* ``prefix`` (``wchar_t*``, default: ``NULL``):
``sys.prefix``.
* ``program_name`` (``wchar_t*``, default: ``NULL``):
Program name.
* ``program`` (``wchar_t*``, default: ``NULL``):
``argv[0]`` or an empty string.
* ``pycache_prefix`` (``wchar_t*``, default: ``NULL``):
``.pyc`` cache prefix.
* ``quiet`` (``int``, default: 0):
Quiet mode. For example, don't display the copyright and version
messages even in interactive mode.
* ``run_command`` (``wchar_t*``, default: ``NULL``):
``-c COMMAND`` argument.
* ``run_filename`` (``wchar_t*``), default: ``NULL``:
``python3 SCRIPT`` argument.
* ``run_module`` (``wchar_t*``, default: ``NULL``):
``python3 -m MODULE`` argument.
* ``show_alloc_count`` (``int``, default: 0):
Show allocation counts at exit?
* ``show_ref_count`` (``int``, default: 0):
Show total reference count at exit?
* ``site_import`` (``int``, default: 1):
Import the ``site`` module at startup?
* ``skip_source_first_line`` (``int``, default: 0):
Skip the first line of the source?
* ``stdio_encoding`` (``wchar_t*``, default: ``NULL``),
``stdio_errors`` (``wchar_t*``, default: ``NULL``):
Encoding and encoding errors of ``sys.stdin``, ``sys.stdout``
and ``sys.stderr``.
* ``tracemalloc`` (``int``, default: 0):
If non-zero, call ``tracemalloc.start(value)``.
* ``user_site_directory`` (``int``, default: 1):
If non-zero, add user site directory to ``sys.path``.
* ``verbose`` (``int``, default: 0):
If non-zero, enable verbose mode.
* ``warnoptions`` (``PyWideStringList``, default: empty):
Options of the ``warnings`` module to build warnings filters.
* ``write_bytecode`` (``int``, default: 1):
If non-zero, write ``.pyc`` files.
* ``xoptions`` (``PyWideStringList``, default: empty):
``sys._xoptions``.
There are also private fields which are for internal-usage only:
* ``_config_version`` (``int``): Configuration version, used for ABI
compatibility
* ``_frozen`` (``int``): Emit warning when computing the path
configuration?
* ``_install_importlib`` (``int``): Install importlib?
* ``_config_version`` (``int``, default: config version):
Configuration version, used for ABI compatibility.
* ``_install_importlib`` (``int``, default: 1):
Install importlib?
* ``_init_main`` (``int``, default: 1):
If equal to 0, stop Python initialization before the "main" phase.
By default, the ``argv`` arguments are parsed as regular Python command
line arguments and ``argv`` is updated to strip parsed Python arguments:
see `Command Line Arguments`_. Set ``parse_argv`` to 0 to avoid parsing
and updating ``argv``. If ``argv`` is empty, an empty string is added to
ensure that ``sys.argv`` always exists and is never empty.
The ``xoptions`` options are parsed to set other options: see `-X
Options`_.
More complete commented example modifying the configuration before
calling ``PyConfig_Read()`` and then modify the read configuration::
@ -378,8 +494,8 @@ calling ``PyConfig_Read()`` and then modify the read configuration::
/* Set the program name before reading the configuraton
(decode byte string from the locale encoding) */
err = PyConfig_DecodeLocale(&config.program_name,
program_name);
err = PyConfig_SetBytesString(&config.program_name,
program_name);
if (_Py_INIT_FAILED(err)) {
goto fail;
}
@ -398,7 +514,7 @@ calling ``PyConfig_Read()`` and then modify the read configuration::
}
/* Override executable computed by PyConfig_Read() */
err = PyConfig_SetString(&config.executable, L"my_executable");
err = PyConfig_SetString(&config, &config.executable, L"my_executable");
if (_Py_INIT_FAILED(err)) {
goto fail;
}
@ -419,17 +535,18 @@ calling ``PyConfig_Read()`` and then modify the read configuration::
.. note::
``PyConfig`` does not have any field for extra inittab functions:
``PyImport_AppendInittab()`` and ``PyImport_ExtendInittab()``
functions are still relevant.
functions are still relevant (and can be called before Python
initialization).
Initialization with static PyConfig
-----------------------------------
Initialization with constant PyConfig
-------------------------------------
When no ``PyConfig`` method is used but only
``Py_InitializeFromConfig()``, the caller is responsible for managing
``PyConfig`` memory which means that static strings and static string
lists can be used rather than using dynamically allocated memory. It
can be used for most simple configurations.
``PyConfig`` memory which means that constant strings and constant string
lists can be used to avoid dynamically allocated memory. It can be used
for most simple configurations.
Example of Python initialization enabling the isolated mode::
@ -450,56 +567,74 @@ and manage the memory.
For convenience, two other functions are provided:
* ``PyInitError Py_InitializeFromArgs(const PyConfig *config, int argc, char **argv)``
* ``PyInitError Py_InitializeFromWideArgs(const PyConfig *config, int argc, wchar_t **argv)``
* ``PyInitError Py_InitializeFromArgs(const PyConfig *config, int argc, wchar_t **argv)``
* ``PyInitError Py_InitializeFromBytesArgs(const PyConfig *config, int argc, char **argv)``
These functions can be used with static ``PyConfig``.
Pseudo-code of ``Py_InitializeFromArgs()``::
PyInitError init_with_args(const PyConfig *src_config, int argc, char **argv)
{
PyInitError err;
PyConfig config = PyConfig_INIT;
/* Copy strings and string lists
* (memory dynamically allocated on the heap) */
err = _PyConfig_Copy(&config, src_config);
if (Py_INIT_FAILED(err)) {
goto exit;
}
/* Set config.argv: decode argv bytes. Pre-initialize Python
if needed to ensure that the encodings are properly
configured. */
err = PyConfig_SetArgv(&config, argc, argv);
if (Py_INIT_FAILED(err)) {
goto exit;
}
err = Py_InitializeFromConfig(&config);
exit:
PyConfig_Clear(&config);
return err;
}
where ``_PyConfig_Copy()`` is an internal function. The actual
implementation of ``Py_InitializeFromArgs()`` is more complex.
These functions can be used with constant ``PyConfig``.
Py_UnixMain()
-------------
Path Configuration
------------------
``PyConfig`` contains multiple fields for the path configuration:
* Path configuration input fields:
* ``home``
* ``module_search_path_env``
* ``pathconfig_warns``
* Path configuration output fields:
* ``dll_path`` (Windows only)
* ``exec_prefix``
* ``executable``
* ``prefix``
* ``use_module_search_paths``, ``module_search_paths``
It is possible to completely ignore the function computing the default
path configuration by setting explicitly all path configuration output
fields listed above. A string is considered as set even if it's an empty
string. ``module_search_paths`` is considered as set if
``use_module_search_paths`` is set to 1. In this case, path
configuration input fields are ignored as well.
If ``base_prefix`` or ``base_exec_prefix`` fields are not set, they
inherit their value from ``prefix`` and ``exec_prefix`` respectively.
Set ``pathconfig_warnings`` to 0 to suppress warnings when computing the
path configuration.
If ``site_import`` is non-zero, ``sys.path`` can be modified by the
``site`` module. For example, if ``user_site_directory`` is non-zero,
the user site directory is added to ``sys.path`` (if it exists).
Isolate Python
--------------
The default configuration is designed to behave as a regular Python.
To embed Python into an application, it's possible to tune the
configuration to better isolated the embedded Python from the system:
* Set the `Path Configuration`_ ("output fields") to ignore the function
computing the default path configuration.
* Set ``isolated`` to 1 to ignore environment variables and not prepend
the current directory to ``sys.path``.
Py_BytesMain()
--------------
Python 3.7 provides a high-level ``Py_Main()`` function which requires
to pass command line arguments as ``wchar_t*`` strings. It is
non-trivial to use the correct encoding to decode bytes. Python has its
own set of issues with C locale coercion and UTF-8 Mode.
This PEP adds a new ``Py_UnixMain()`` function which takes command line
This PEP adds a new ``Py_BytesMain()`` function which takes command line
arguments as bytes::
int Py_UnixMain(int argc, char **argv)
int Py_BytesMain(int argc, char **argv)
Py_RunMain()
------------
@ -519,7 +654,7 @@ Example of custom Python executable always running in isolated mode::
PyConfig config = PyConfig_INIT;
config.isolated = 1;
PyInitError err = Py_InitializeFromArgs(&config, argc, argv);
PyInitError err = Py_InitializeFromBytesArgs(&config, argc, argv);
if (Py_INIT_FAILED(err)) {
Py_ExitInitError(err);
}
@ -540,16 +675,16 @@ Python memory allocation functions like ``PyMem_RawMalloc()`` must not
be used before Python pre-initialization. Calling directly ``malloc()``
and ``free()`` is always safe.
For ``PyPreConfig`` and static ``PyConfig``, the caller is responsible
to manage dynamically allocated strings, but static strings and static
string lists are fine.
For ``PyPreConfig`` and constant ``PyConfig``, the caller is responsible
to manage dynamically allocated strings; constant strings and constant
string lists can be used to avoid memory allocations.
Dynamic ``PyConfig`` requires to call ``PyConfig_Clear()`` to release
memory.
``Py_DecodeLocale()`` must not be called before the pre-initialization.
When using dynanic configuration, ``PyConfig_DecodeLocale()`` must be
When using dynanic configuration, ``PyConfig_SetBytesString()`` must be
used instead of ``Py_DecodeLocale()``.
@ -584,16 +719,28 @@ Priority of warning options, highest to lowest:
* ``PyConfig.bytes_warning`` (add ``"error::BytesWarning"`` if greater
than 1, or add ``"default::BytesWarning``)
Rules on ``PyConfig`` and ``PyPreConfig`` parameters:
Rules on ``PyConfig`` parameters:
* If ``isolated`` is non-zero, ``use_environment`` and
``user_site_directory`` are set to 0
``user_site_directory`` are set to 0.
* If ``legacy_windows_fs_encoding`` is non-zero, ``utf8_mode`` is set to
0
0.
* If ``dev_mode`` is non-zero, ``allocator`` is set to ``"debug"``,
``faulthandler`` is set to 1, and ``"default"`` filter is added to
``warnoptions``. But ``PYTHONMALLOC`` has the priority over
``dev_mode`` to set the memory allocator.
``warnoptions``. But the ``PYTHONMALLOC`` environment variable has the
priority over ``dev_mode`` to set the memory allocator.
* If ``base_prefix`` is not set, it inherits ``prefix`` value.
* If ``base_exec_prefix`` is not set, it inherits ``exec_prefix`` value.
* If the ``python._pth`` configuration file is present, ``isolated`` is
set to 1 and ``site_import`` is set to 0. ``site_import`` is set to
1 if ``python._pth`` contains ``import site``.
Rules on ``PyConfig`` and ``PyPreConfig`` parameters:
* If ``PyPreConfig.legacy_windows_fs_encoding`` is non-zero,
set ``PyConfig.utf8_mode`` to 0, set ``PyConfig.filesystem_encoding``
to ``mbcs`` (if unset), and set ``PyConfig.filesystem_errors`` to
``replace`` (if unset).
Configuration Files
-------------------
@ -612,9 +759,9 @@ Global configuration variables mapped to ``PyPreConfig`` fields:
======================================== ================================
Variable Field
======================================== ================================
``Py_IgnoreEnvironmentFlag`` ``use_environment`` (NOT)
``Py_IsolatedFlag`` ``isolated``
``Py_LegacyWindowsFSEncodingFlag`` ``legacy_windows_fs_encoding``
``Py_LegacyWindowsFSEncodingFlag`` ``legacy_windows_fs_encoding``
``Py_UTF8Mode`` ``utf8_mode``
``Py_UTF8Mode`` ``utf8_mode``
======================================== ================================
@ -625,47 +772,26 @@ Variable Field
======================================== ================================
``Py_BytesWarningFlag`` ``bytes_warning``
``Py_DebugFlag`` ``parser_debug``
``Py_DontWriteBytecodeFlag`` ``write_bytecode``
``Py_DontWriteBytecodeFlag`` ``write_bytecode`` (NOT)
``Py_FileSystemDefaultEncodeErrors`` ``filesystem_errors``
``Py_FileSystemDefaultEncoding`` ``filesystem_encoding``
``Py_FrozenFlag`` ``_frozen``
``Py_FrozenFlag`` ``pathconfig_warnings`` (NOT)
``Py_HasFileSystemDefaultEncoding`` ``filesystem_encoding``
``Py_HashRandomizationFlag`` ``use_hash_seed``, ``hash_seed``
``Py_IgnoreEnvironmentFlag`` ``use_environment``
``Py_IgnoreEnvironmentFlag`` ``use_environment`` (NOT)
``Py_InspectFlag`` ``inspect``
``Py_InteractiveFlag`` ``interactive``
``Py_IsolatedFlag`` ``isolated``
``Py_LegacyWindowsStdioFlag`` ``legacy_windows_stdio``
``Py_NoSiteFlag`` ``site_import``
``Py_NoUserSiteDirectory`` ``user_site_directory``
``Py_NoSiteFlag`` ``site_import`` (NOT)
``Py_NoUserSiteDirectory`` ``user_site_directory`` (NOT)
``Py_OptimizeFlag`` ``optimization_level``
``Py_QuietFlag`` ``quiet``
``Py_UnbufferedStdioFlag`` ``buffered_stdio``
``Py_VerboseFlag`` ``verbose``
``_Py_HasFileSystemDefaultEncodeErrors`` ``filesystem_errors``
``Py_BytesWarningFlag`` ``bytes_warning``
``Py_DebugFlag`` ``parser_debug``
``Py_DontWriteBytecodeFlag`` ``write_bytecode``
``Py_FileSystemDefaultEncodeErrors`` ``filesystem_errors``
``Py_FileSystemDefaultEncoding`` ``filesystem_encoding``
``Py_FrozenFlag`` ``_frozen``
``Py_HasFileSystemDefaultEncoding`` ``filesystem_encoding``
``Py_HashRandomizationFlag`` ``use_hash_seed``, ``hash_seed``
``Py_IgnoreEnvironmentFlag`` ``use_environment``
``Py_InspectFlag`` ``inspect``
``Py_InteractiveFlag`` ``interactive``
``Py_IsolatedFlag`` ``isolated``
``Py_LegacyWindowsStdioFlag`` ``legacy_windows_stdio``
``Py_NoSiteFlag`` ``site_import``
``Py_NoUserSiteDirectory`` ``user_site_directory``
``Py_OptimizeFlag`` ``optimization_level``
``Py_QuietFlag`` ``quiet``
``Py_UnbufferedStdioFlag`` ``buffered_stdio``
``Py_UnbufferedStdioFlag`` ``buffered_stdio`` (NOT)
``Py_VerboseFlag`` ``verbose``
``_Py_HasFileSystemDefaultEncodeErrors`` ``filesystem_errors``
======================================== ================================
``Py_LegacyWindowsFSEncodingFlag`` and ``Py_LegacyWindowsStdioFlag`` are
only available on Windows.
@ -680,13 +806,16 @@ Usage::
python3 [options] SCRIPT
Command line options mapped to pseudo-action on ``PyConfig`` fields:
Command line options mapped to pseudo-action on ``PyPreConfig`` fields:
================================ ================================
Option ``PyPreConfig`` field
Option ``PyConfig`` field
================================ ================================
``-E`` ``use_environment = 0``
``-I`` ``isolated = 1``
``-X dev`` ``dev_mode = 1``
``-X utf8=N`` ``utf8_mode = N``
``-X utf8`` ``utf8_mode = 1``
``-X utf8=VALUE`` ``utf8_mode = VALUE``
================================ ================================
Command line options mapped to pseudo-action on ``PyConfig`` fields:
@ -714,17 +843,27 @@ Option ``PyConfig`` field
``-W WARNING`` add ``WARNING`` to ``warnoptions``
``-x`` ``skip_source_first_line = 1``
``-X XOPTION`` add ``XOPTION`` to ``xoptions``
================================ ================================
``-h``, ``-?`` and ``-V`` options are handled outside ``PyConfig``.
-X Options
----------
-X options mapped to pseudo-action on ``PyConfig`` fields:
================================ ================================
Option ``PyConfig`` field
================================ ================================
``-X dev`` ``dev_mode = 1``
``-X faulthandler`` ``faulthandler = 1``
``-X importtime`` ``import_time = 1``
``-X pycache_prefix=PREFIX`` ``pycache_prefix = PREFIX``
``-X show_alloc_count`` ``show_alloc_count = 1``
``-X show_ref_count`` ``show_ref_count = 1``
``-X showalloccount`` ``show_alloc_count = 1``
``-X showrefcount`` ``show_ref_count = 1``
``-X tracemalloc=N`` ``tracemalloc = N``
================================ ================================
``-h``, ``-?`` and ``-V`` options are handled outside ``PyConfig``.
Environment Variables
---------------------
@ -803,6 +942,142 @@ Python can be configured using scattered global configuration variables
There is also a high-level ``Py_Main()`` function.
Python Issues
=============
Issues that will be fixed by this PEP, directly or indirectly:
* `bpo-1195571 <https://bugs.python.org/issue1195571>`_: "simple
callback system for Py_FatalError"
* `bpo-11320 <https://bugs.python.org/issue11320>`_:
"Usage of API method Py_SetPath causes errors in Py_Initialize()
(Posix ony)"
* `bpo-13533 <https://bugs.python.org/issue13533>`_: "Would like
Py_Initialize to play friendly with host app"
* `bpo-14956 <https://bugs.python.org/issue14956>`_: "custom PYTHONPATH
may break apps embedding Python"
* `bpo-19983 <https://bugs.python.org/issue19983>`_: "When interrupted
during startup, Python should not call abort() but exit()"
* `bpo-22213 <https://bugs.python.org/issue22213>`_: "Make pyvenv style
virtual environments easier to configure when embedding Python". This
PEP more or
* `bpo-22257 <https://bugs.python.org/issue22257>`_: "PEP 432: Redesign
the interpreter startup sequence"
* `bpo-29778 <https://bugs.python.org/issue29778>`_: "_Py_CheckPython3
uses uninitialized dllpath when embedder sets module path with
Py_SetPath"
* `bpo-30560 <https://bugs.python.org/issue30560>`_: "Add
Py_SetFatalErrorAbortFunc: Allow embedding program to handle fatal
errors".
* `bpo-31745 <https://bugs.python.org/issue31745>`_: "Overloading
"Py_GetPath" does not work"
* `bpo-32573 <https://bugs.python.org/issue32573>`_: "All sys attributes
(.argv, ...) should exist in embedded environments".
* `bpo-34725 <https://bugs.python.org/issue34725>`_:
"Py_GetProgramFullPath() odd behaviour in Windows"
* `bpo-36204 <https://bugs.python.org/issue36204>`_: "Deprecate calling
Py_Main() after Py_Initialize()? Add Py_InitializeFromArgv()?"
* `bpo-33135 <https://bugs.python.org/issue33135>`_: "Define field
prefixes for the various config structs". The PEP now defines well
how warnings options are handled.
Issues of the PEP implementation:
* `bpo-16961 <https://bugs.python.org/issue16961>`_: "No regression
tests for -E and individual environment vars"
* `bpo-20361 <https://bugs.python.org/issue20361>`_: "-W command line
options and PYTHONWARNINGS environmental variable should not override
-b / -bb command line options"
* `bpo-26122 <https://bugs.python.org/issue26122>`_: "Isolated mode
doesn't ignore PYTHONHASHSEED"
* `bpo-29818 <https://bugs.python.org/issue29818>`_:
"Py_SetStandardStreamEncoding leads to a memory error in debug mode"
* `bpo-31845 <https://bugs.python.org/issue31845>`_:
"PYTHONDONTWRITEBYTECODE and PYTHONOPTIMIZE have no effect"
* `bpo-32030 <https://bugs.python.org/issue32030>`_: "PEP 432: Rewrite
Py_Main()"
* `bpo-32124 <https://bugs.python.org/issue32124>`_: "Document functions
safe to be called before Py_Initialize()"
* `bpo-33042 <https://bugs.python.org/issue33042>`_: "New 3.7 startup
sequence crashes PyInstaller"
* `bpo-33932 <https://bugs.python.org/issue33932>`_: "Calling
Py_Initialize() twice now triggers a fatal error (Python 3.7)"
* `bpo-34008 <https://bugs.python.org/issue34008>`_: "Do we support
calling Py_Main() after Py_Initialize()?"
* `bpo-34170 <https://bugs.python.org/issue34170>`_: "Py_Initialize():
computing path configuration must not have side effect (PEP 432)"
* `bpo-34589 <https://bugs.python.org/issue34589>`_: "Py_Initialize()
and Py_Main() should not enable C locale coercion"
* `bpo-34639 <https://bugs.python.org/issue34639>`_:
"PYTHONCOERCECLOCALE is ignored when using -E or -I option"
* `bpo-36142 <https://bugs.python.org/issue36142>`_: "Add a new
_PyPreConfig step to Python initialization to setup memory allocator
and encodings"
* `bpo-36202 <https://bugs.python.org/issue36202>`_: "Calling
Py_DecodeLocale() before _PyPreConfig_Write() can produce mojibake"
* `bpo-36301 <https://bugs.python.org/issue36301>`_: "Add
_Py_PreInitialize() function"
* `bpo-36443 <https://bugs.python.org/issue36443>`_: "Disable
coerce_c_locale and utf8_mode by default in _PyPreConfig?"
* `bpo-36444 <https://bugs.python.org/issue36444>`_: "Python
initialization: remove _PyMainInterpreterConfig"
* `bpo-36471 <https://bugs.python.org/issue36471>`_: "PEP 432, PEP 587:
Add _Py_RunMain()"
* `bpo-36763 <https://bugs.python.org/issue36763>`_: "PEP 587: Rework
initialization API to prepare second version of the PEP"
* `bpo-36775 <https://bugs.python.org/issue36775>`_: "Rework filesystem
codec implementation"
* `bpo-36900 <https://bugs.python.org/issue36900>`_: "Use _PyCoreConfig
rather than global configuration variables"
Issues related to this PEP:
* `bpo-12598 <https://bugs.python.org/issue12598>`_: "Move sys variable
initialization from import.c to sysmodule.c"
* `bpo-15577 <https://bugs.python.org/issue15577>`_: "Real argc and argv
in embedded interpreter"
* `bpo-16202 <https://bugs.python.org/issue16202>`_: "sys.path[0]
security issues"
* `bpo-18309 <https://bugs.python.org/issue18309>`_: "Make python
slightly more relocatable"
* `bpo-25631 <https://bugs.python.org/issue25631>`_: "Segmentation fault
with invalid Unicode command-line arguments in embedded Python"
* `bpo-26007 <https://bugs.python.org/issue26007>`_: "Support embedding
the standard library in an executable"
* `bpo-31210 <https://bugs.python.org/issue31210>`_: "Can not import
modules if sys.prefix contains DELIM".
* `bpo-31349 <https://bugs.python.org/issue31349>`_: "Embedded
initialization ignores Py_SetProgramName()"
* `bpo-33919 <https://bugs.python.org/issue33919>`_: "Expose
_PyCoreConfig structure to Python"
* `bpo-35173 <https://bugs.python.org/issue35173>`_: "Re-use already
existing functionality to allow Python 2.7.x (both embedded and
standalone) to locate the module path according to the shared library"
Version History
===============
* Version 3:
* ``PyConfig``: Add ``configure_c_stdio`` and ``parse_argv``,
rename ``_frozen`` to ``pathconfig_warnings``.
* Rename functions using bytes strings and wide strings. For example,
``Py_PreInitializeFromWideArgs`` becomes
``Py_PreInitializeFromArgs``, and ``PyConfig_SetArgv`` becomes
``PyConfig_SetBytesArgv``.
* Add ``PyWideStringList_Insert()`` function.
* New "Path configuration", "Isolate Python", "Python Issues"
and "Version History" sections.
* ``PyConfig_SetString()`` and ``PyConfig_SetBytesString()`` now
requires the configuration as the first argument.
* Rename ``Py_UnixMain()`` to ``Py_BytesMain()``
* Version 2: Add ``PyConfig`` methods (ex: ``PyConfig_Read()``), add
``PyWideStringList_Append()``, rename ``PyWideCharList`` to
``PyWideStringList``.
* Version 1: Initial version.
Copyright
=========