iSharkFly-Open
/
python-peps
mirror of https://github.com/python/peps.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

PEP 432: update to target 3.7/3.8 

- timeline updates based on completely missing the 3.6/3.7 targets
- API design & implementation strategy updates based on recent
  discussions with Eric Snow, Steve Dower & Brett Cannon
This commit is contained in:
Nick Coghlan 2016-12-31 17:07:04 +10:00
parent 361599ec4b
commit 9f3a93dc85
1 changed files with 148 additions and 145 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

293
pep-0432.txt
View File

@ -1,5 +1,5 @@
PEP: 432 PEP: 432
Title: Simplifying the CPython startup sequence Title: Restructuring the CPython startup sequence
Version: $Revision$ Version: $Revision$
Last-Modified: $Date$ Last-Modified: $Date$
Author: Nick Coghlan <ncoghlan@gmail.com> Author: Nick Coghlan <ncoghlan@gmail.com>
@ -7,19 +7,28 @@ Status: Draft
Type: Standards Track Type: Standards Track
Content-Type: text/x-rst Content-Type: text/x-rst
Created: 28-Dec-2012 Created: 28-Dec-2012
Python-Version: 3.6 Python-Version: 3.8
Post-History: 28-Dec-2012, 2-Jan-2013 Post-History: 28-Dec-2012, 2-Jan-2013
Abstract Abstract
======== ========
This PEP proposes a mechanism for simplifying the startup sequence for This PEP proposes a mechanism for restructuring the startup sequence for
CPython, making it easier to modify the initialization behaviour of the CPython, making it easier to modify the initialization behaviour of the
reference interpreter executable, as well as making it easier to control reference interpreter executable, as well as making it easier to control
CPython's startup behaviour when creating an alternate executable or CPython's startup behaviour when creating an alternate executable or
embedding it as a Python execution engine inside a larger application. embedding it as a Python execution engine inside a larger application.
When implementation of this proposal is completed, interpreter startup will
consist of two clearly distinct and independently configurable phases:
* Python runtime initialization
* Main interpreter configuration
Changes are also proposed that impact main module execution and subinterpreter
initialization.
Note: TBC = To Be Confirmed, TBD = To Be Determined. The appropriate Note: TBC = To Be Confirmed, TBD = To Be Determined. The appropriate
resolution for most of these should become clearer as the reference resolution for most of these should become clearer as the reference
implementation is developed. implementation is developed.
@ -32,7 +41,7 @@ This PEP proposes that initialization of the CPython runtime be split into
two clearly distinct phases: two clearly distinct phases:
* core runtime initialization * core runtime initialization
* main interpreter initialization * main interpreter configuration
The proposed design also has significant implications for: The proposed design also has significant implications for:
@ -43,7 +52,7 @@ In the new design, the interpreter will move through the following
well-defined phases during the initialization sequence: well-defined phases during the initialization sequence:
* Pre-Initialization - no interpreter available * Pre-Initialization - no interpreter available
* Core Initialized - main interpreter partially available, * Runtime Initialized - main interpreter partially available,
subinterpreter creation not yet available subinterpreter creation not yet available
* Initialized - main interpreter fully available, subinterpreter creation * Initialized - main interpreter fully available, subinterpreter creation
available available
@ -94,9 +103,9 @@ in a cross-platform fashion [7_], controlling the configuration of
tracing when launching Python subprocesses [9_]). tracing when launching Python subprocesses [9_]).
Rather than continuing to bolt such behaviour onto an already complicated Rather than continuing to bolt such behaviour onto an already complicated
system, this PEP proposes to start simplifying the status quo by introducing system indefinitely, this PEP proposes to start simplifying the status quo by
a more structured startup sequence, with the aim of making these further introducing a more structured startup sequence, with the aim of making these
feature requests easier to implement. further feature requests easier to implement.
Key Concerns Key Concerns
@ -116,9 +125,9 @@ as lists, dictionaries and Unicode values being created prior to the call
to ``Py_Initialize`` when the ``-X`` or ``-W`` options are used [1_]. to ``Py_Initialize`` when the ``-X`` or ``-W`` options are used [1_].
By moving to an explicitly multi-phase startup sequence, developers should By moving to an explicitly multi-phase startup sequence, developers should
only need to understand which features are not available in the core only need to understand which features are not available in the main
bootstrapping phase, as the vast majority of the configuration process interpreter configuration phase, as the vast majority of the configuration
will now take place during that phase. process will now take place during that phase.
By basing the new design on a combination of C structures and Python By basing the new design on a combination of C structures and Python
data types, it should also be easier to modify the system in the data types, it should also be easier to modify the system in the
@ -150,12 +159,12 @@ tear down the interpreter::
python3 -m timeit -s "from subprocess import call" "call(['./python', '-c', 'pass'])" python3 -m timeit -s "from subprocess import call" "call(['./python', '-c', 'pass'])"
Current numbers on my system for Python 3.6 (using the 3.5 Current numbers on my system for Python 3.7 (using the 3.5
subprocess and timeit modules to execute the check, all with non-debug subprocess and timeit modules to execute the check, all with non-debug
builds):: builds)::
$ python3 -m timeit -s "from subprocess import call" "call(['./python', '-c', 'pass'])" $ python3 -m timeit -s "from subprocess import call" "call(['./python', '-c', 'pass'])"
100 loops, best of 3: 15.1 msec per loop 100 loops, best of 3: 13.9 msec per loop
This PEP is not expected to have any significant effect on the startup time, This PEP is not expected to have any significant effect on the startup time,
as it is aimed primarily at *reordering* the existing initialization as it is aimed primarily at *reordering* the existing initialization
@ -247,7 +256,7 @@ be able to control the following aspects of the final interpreter state:
* Standard input as a script (i.e. a non-interactive stream) * Standard input as a script (i.e. a non-interactive stream)
* Standard input as an interactive interpreter session * Standard input as an interactive interpreter session
<TBD: Did I miss anything?> <TBD: What, if anything, is still missing from this list?>
Note that this just covers settings that are currently configurable in some Note that this just covers settings that are currently configurable in some
manner when using the main CPython executable. While this PEP aims to make manner when using the main CPython executable. While this PEP aims to make
@ -270,18 +279,18 @@ structure a draft implementation that won't be prone to the kinds of merge
conflicts that afflicted the original attempt. conflicts that afflicted the original attempt.
Accordingly, the implementation strategy now being proposed is to implement this Accordingly, the implementation strategy now being proposed is to implement this
refactoring as a private API for CPython 3.6, before exposing the new functions refactoring as a private API for CPython 3.7, before exposing the new functions
and structures as public API elements in CPython 3.7. and structures as public API elements in CPython 3.8.
The affected APIs with a leading underscore, as they would be named in CPython The affected APIs with a leading underscore, as they would be named in CPython
3.6: 3.7:
* ``_Py_IsCoreInitialized`` * ``_Py_IsRuntimeInitialized``
* ``_Py_InitializeCore`` * ``_Py_InitializeRuntime``
* ``_PyCoreConfig`` * ``_PyRuntimeConfig``
* ``_PyCoreConfig_INIT`` * ``_PyRuntimeConfig_INIT``
* ``_Py_ReadHashSeed`` * ``_Py_ReadHashSeed``
* ``_Py_InitializeMainInterpreter`` * ``_Py_ConfigureMainInterpreter``
* ``_PyMainInterpreterConfig`` * ``_PyMainInterpreterConfig``
* ``_PyInterpreterConfig`` * ``_PyInterpreterConfig``
* ``_Py_ReadMainInterpreterConfig`` * ``_Py_ReadMainInterpreterConfig``
@ -294,15 +303,15 @@ intended to be retained permanently as private CPython implementation details.
The principle benefit of this approach is allowing the refactoring to adopt the The principle benefit of this approach is allowing the refactoring to adopt the
new configuration structures to be handled on a setting by setting basis new configuration structures to be handled on a setting by setting basis
over the course of the 3.6 and 3.7 development cycles, rather than having to over the course of the 3.7 and 3.8 development cycles, rather than having to
migrate them in a single monolithic change. It also means any new settings can migrate them in a single monolithic change. It also means any new settings
be handled using the new approach, even if some existing settings have yet to introduced for these releases can be handled using the new approach from the
be migrated. start, even if some existing settings have yet to be migrated.
If all existing settings are successfully migrated to the new initialization If all existing settings are successfully migrated to the new initialization
model in time for the 3.6.0a4 release in August 2016, then the proposal would model in time for the 3.7.0a4 release in December 2017, then the proposal would
be to make the APIs public for the 3.6.0b1 release in September 2016, rather be to make the APIs public for the 3.7.0b1 release in January 2018, rather
than waiting for 3.7. than waiting for 3.8.
Design Details Design Details
@ -311,12 +320,13 @@ Design Details
(Note: details here are still very much in flux, but preliminary feedback (Note: details here are still very much in flux, but preliminary feedback
is appreciated anyway) is appreciated anyway)
The main theme of this proposal is to create the interpreter state for The main theme of this proposal is to initialize the core language runtime
the main interpreter *much* earlier in the startup process. This will allow and create a partially initialized interpreter state for the main interpreter
most of the CPython API to be used during the remainder of the initialization *much* earlier in the startup process. This will allow most of the CPython API
process, potentially simplifying a number of operations that currently need to be used during the remainder of the initialization process, potentially
to rely on basic C functionality rather than being able to use the richer simplifying a number of operations that currently need to rely on basic C
data structures provided by the CPython C API. functionality rather than being able to use the richer data structures provided
by the CPython C API.
In the following, the term "embedding application" also covers the standard In the following, the term "embedding application" also covers the standard
CPython command line application. CPython command line application.
@ -329,29 +339,31 @@ Three distinct interpreter initialisation phases are proposed:
* Pre-Initialization: * Pre-Initialization:
* no interpreter is available. * no interpreter is available
* ``Py_IsCoreInitialized()`` returns ``0`` * ``Py_IsRuntimeInitialized()`` returns ``0``
* ``Py_IsInitialized()`` returns ``0`` * ``Py_IsInitialized()`` returns ``0``
* The embedding application determines the settings required to create the * The embedding application determines the settings required to initialize
main interpreter and moves to the next phase by calling the core CPython runtime and create the main interpreter and moves to the
``Py_InitializeCore``. next phase by calling ``Py_InitializeRuntime``
* Core Initialized: * Initializing:
* the main interpreter is available, but only partially configured. * the builtin data types and other core runtime services are available
* ``Py_IsCoreInitialized()`` returns ``1`` * the main interpreter is available, but only partially configured
* ``Py_IsRuntimeInitialized()`` returns ``1``
* ``Py_IsInitialized()`` returns ``0`` * ``Py_IsInitialized()`` returns ``0``
* The embedding application determines and applies the settings * The embedding application determines and applies the settings
required to complete the initialization process by calling required to complete the initialization process by calling
``Py_ReadMainInterpreterConfig`` and ``Py_InitializeMainInterpreter``. ``Py_ReadMainInterpreterConfig`` and ``Py_ConfigureMainInterpreter``.
* Initialized: * Initialized:
* the main interpreter is available and fully operational, but * the main interpreter is available and fully operational, but
``__main__`` related metadata is incomplete ``__main__`` related metadata is incomplete
* ``Py_IsCoreInitialized()`` returns ``1`` * ``Py_IsRuntimeInitialized()`` returns ``1``
* ``Py_IsInitialized()`` returns ``1`` * ``Py_IsInitialized()`` returns ``1``
Invocation of Phases Invocation of Phases
-------------------- --------------------
@ -366,18 +378,18 @@ grained API, which allows the embedding application greater control
over the initialization process:: over the initialization process::
/* Phase 1: Pre-Initialization */ /* Phase 1: Pre-Initialization */
PyCoreConfig core_config = PyCoreConfig_INIT; PyRuntimeConfig runtime_config = PyRuntimeConfig_INIT;
PyMainInterpreterConfig config; PyMainInterpreterConfig interpreter_config;
/* Easily control the core configuration */ /* Easily control the core configuration */
core_config.ignore_environment = 1; /* Ignore environment variables */ runtime_config.ignore_environment = 1; /* Ignore environment variables */
core_config.use_hash_seed = 0; /* Full hash randomisation */ runtime_config.use_hash_seed = 0; /* Full hash randomisation */
Py_InitializeCore(&core_config); Py_InitializeRuntime(&runtime_config);
/* Phase 2: Initialization */ /* Phase 2: Initializing */
/* Optionally preconfigure some settings here - they will then be /* Optionally preconfigure some settings here - they will then be
* used to derive other settings */ * used to derive other settings */
Py_ReadMainInterpreterConfig(&config); Py_ReadMainInterpreterConfig(&interpreter_config);
/* Can completely override derived settings here */ /* Can completely override derived settings here */
Py_InitializeMainInterpreter(&config); Py_ConfigureMainInterpreter(&interpreter_config);
/* Phase 3: Initialized */ /* Phase 3: Initialized */
/* If an embedding application has no real concept of a main module /* If an embedding application has no real concept of a main module
* it can just stop the initialization process here. * it can just stop the initialization process here.
@ -389,7 +401,7 @@ Pre-Initialization Phase
------------------------ ------------------------
The pre-initialization phase is where an embedding application determines The pre-initialization phase is where an embedding application determines
the settings which are absolutely required before the interpreter can be the settings which are absolutely required before the CPython runtime can be
initialized at all. Currently, the primary configuration settings in this initialized at all. Currently, the primary configuration settings in this
category are those related to the randomised hash algorithm - the hash category are those related to the randomised hash algorithm - the hash
algorithms must be consistent for the lifetime of the process, and so they algorithms must be consistent for the lifetime of the process, and so they
@ -406,7 +418,7 @@ system.
The proposed API for this step in the startup sequence is:: The proposed API for this step in the startup sequence is::
void Py_InitializeCore(const PyCoreConfig *config); void Py_InitializeRuntime(const PyRuntimeConfig *config);
Like ``Py_Initialize``, this part of the new API treats initialization failures Like ``Py_Initialize``, this part of the new API treats initialization failures
as fatal errors. While that's still not particularly embedding friendly, as fatal errors. While that's still not particularly embedding friendly,
@ -414,34 +426,31 @@ the operations in this step *really* shouldn't be failing, and changing them
to return error codes instead of aborting would be an even larger task than to return error codes instead of aborting would be an even larger task than
the one already being proposed. the one already being proposed.
The new ``PyCoreConfig`` struct holds the settings required for preliminary The new ``PyRuntimeConfig`` struct holds the settings required for preliminary
configuration of the core runtime and creation of the main interpreter:: configuration of the core runtime and creation of the main interpreter::
/* Note: if changing anything in PyCoreConfig, also update /* Note: if changing anything in PyRuntimeConfig, also update
* PyCoreConfig_INIT */ * PyRuntimeConfig_INIT */
typedef struct { typedef struct {
int ignore_environment; /* -E switch, -I switch */ bool ignore_environment; /* -E switch, -I switch */
int use_hash_seed; /* PYTHONHASHSEED */ int use_hash_seed; /* PYTHONHASHSEED */
unsigned long hash_seed; /* PYTHONHASHSEED */ unsigned long hash_seed; /* PYTHONHASHSEED */
int _disable_importlib; /* Needed by freeze_importlib */ bool _disable_importlib; /* Needed by freeze_importlib */
} PyCoreConfig; } PyRuntimeConfig;
#define PyCoreConfig_INIT {.use_hash_seed=-1} /* Rely on the "designated initializer" feature of C99 */
/* The above assumes PEP 7 is updated to permit use of C99 features #define PyRuntimeConfig_INIT {.use_hash_seed=-1}
* Without designated initialisers, the macro would be:
* #define PyCoreConfig_INIT {0, -1, 0, 0}
*/
The core configuration settings pointer may be ``NULL``, in which case the The core configuration settings pointer may be ``NULL``, in which case the
default values are ``ignore_environment = -1`` and ``use_hash_seed = -1``. default values are as specified in ``PyRuntimeConfig_INIT``.
The ``PyCoreConfig_INIT`` macro is designed to allow easy initialization The ``PyRuntimeConfig_INIT`` macro is designed to allow easy initialization
of a struct instance with sensible defaults:: of a struct instance with sensible defaults::
PyCoreConfig core_config = PyCoreConfig_INIT; PyRuntimeConfig runtime_config = PyRuntimeConfig_INIT;
``ignore_environment`` controls the processing of all Python related ``ignore_environment`` controls the processing of all Python related
environment variables. If the flag is zero, then environment variables are environment variables. If the flag is false, then environment variables are
processed normally. Otherwise, all Python-specific environment variables processed normally. Otherwise, all Python-specific environment variables
are considered undefined (exceptions may be made for some OS specific are considered undefined (exceptions may be made for some OS specific
environment variables, such as those used on Mac OS X to communicate environment variables, such as those used on Mac OS X to communicate
@ -480,30 +489,32 @@ value indicates an error (most likely in the conversion to an integer).
The ``_disable_importlib`` setting is used as part of the CPython build The ``_disable_importlib`` setting is used as part of the CPython build
process to create an interpreter with no import capability at all. It is process to create an interpreter with no import capability at all. It is
considered private to the CPython development team (hence the leading considered private to the CPython development team (hence the leading
underscore), as the only known use case is to permit compiler changes underscore), as the only currently supported use case is to permit compiler
that invalidate the previously frozen bytecode for ``importlib._bootstrap`` changes that invalidate the previously frozen bytecode for
without breaking the build process. ``importlib._bootstrap`` without breaking the build process.
The aim is to keep this initial level of configuration as small as possible The aim is to keep this initial level of configuration as small as possible
in order to keep the bootstrapping environment consistent across in order to keep the bootstrapping environment consistent across
different embedding applications. If we can create a valid interpreter state different embedding applications. If we can create a valid interpreter state
without the setting, then the setting should go in the configuration passed without the setting, then the setting should go in the configuration passed
to ``Py_InitializeMainInterpreter()`` rather than in the core configuration. to ``Py_ConfigureMainInterpreter()`` rather than in the core runtime
configuration.
A new query API will allow code to determine if the interpreter is in the A new query API will allow code to determine if the interpreter is in the
bootstrapping state between the creation of the interpreter state and the bootstrapping state between the core runtime initialization and the creation of
completion of the bulk of the initialization process:: the main interpreter state and the completion of the bulk of the main
interpreter initialization process::
int Py_IsCoreInitialized(); int Py_IsRuntimeInitialized();
Attempting to call ``Py_InitializeCore()`` again when Attempting to call ``Py_InitializeRuntime()`` again when
``Py_IsCoreInitialized()`` is already true is a fatal error. ``Py_IsRuntimeInitialized()`` is already true is a fatal error.
As frozen bytecode may now be legitimately run in an interpreter which is not As frozen bytecode may now be legitimately run in an interpreter which is not
yet fully initialized, ``sys.flags`` will gain a new ``initialized`` flag. yet fully initialized, ``sys.flags`` will gain a new ``initialized`` flag.
With the core runtime initialised, the interpreter should be fully functional With the core runtime initialised, the main interpreter and most of the CPython
except that: C API should be fully functional except that:
* compilation is not allowed (as the parser and compiler are not yet * compilation is not allowed (as the parser and compiler are not yet
configured properly) configured properly)
@ -542,12 +553,12 @@ to be used safely for all of the remaining configuration steps (unlike the
status quo). status quo).
In addition, the current thread will possess a valid Python thread state, In addition, the current thread will possess a valid Python thread state,
allowing any further configuration data to be stored on the interpreter allowing any further configuration data to be stored on the main interpreter
object rather than in C process globals. object rather than in C process globals.
Any call to ``Py_InitializeCore()`` must have a matching call to Any call to ``Py_InitializeRuntime()`` must have a matching call to
``Py_Finalize()``. It is acceptable to skip calling ``Py_Finalize()``. It is acceptable to skip calling
``Py_InitializeMainInterpreter()`` in between (e.g. if attempting to read the ``Py_ConfigureMainInterpreter()`` in between (e.g. if attempting to read the
main interpreter configuration settings fails). main interpreter configuration settings fails).
@ -562,8 +573,8 @@ interpreter state at this point. The core API for this step is::
The config argument should be a pointer to a config struct (which may be The config argument should be a pointer to a config struct (which may be
a temporary one stored on the C stack). For any already configured value a temporary one stored on the C stack). For any already configured value
(i.e. non-NULL pointer or non-negative numeric value), CPython will sanity (i.e. any non-NULL pointer), CPython will sanity check the supplied value,
check the supplied value, but otherwise accept it as correct. but otherwise accept it as correct.
A struct is used rather than a Python dictionary as the struct is easier A struct is used rather than a Python dictionary as the struct is easier
to work with from C, the list of supported fields is fixed for a given to work with from C, the list of supported fields is fixed for a given
@ -571,7 +582,7 @@ CPython version and only a read-only view needs to be exposed to Python
code (which is relatively straightforward, thanks to the infrastructure code (which is relatively straightforward, thanks to the infrastructure
already put in place to expose ``sys.implementation``). already put in place to expose ``sys.implementation``).
Unlike ``Py_Initialize`` and ``Py_InitializeCore``, this call will raise Unlike ``Py_Initialize`` and ``Py_InitializeRuntime``, this call will raise
an exception and report an error return rather than exhibiting fatal errors an exception and report an error return rather than exhibiting fatal errors
if a problem is found with the config data. if a problem is found with the config data.
@ -588,7 +599,7 @@ such as removing ``sys.path[0]``).
Merely reading the configuration has no effect on the interpreter state: it Merely reading the configuration has no effect on the interpreter state: it
only modifies the passed in configuration struct. The settings are not only modifies the passed in configuration struct. The settings are not
applied to the running interpreter until the ``Py_InitializeMainInterpreter`` applied to the running interpreter until the ``Py_ConfigureMainInterpreter``
call (see below). call (see below).
@ -610,8 +621,6 @@ complete the main interpreter configuration. These settings are also all
passed through unmodified to subinterpreters. Fields are always pointers to passed through unmodified to subinterpreters. Fields are always pointers to
Python data types, with unset values indicated by ``NULL``:: Python data types, with unset values indicated by ``NULL``::
/* Note: if changing anything in PyMainInterpreterConfig, also update
* PyMainInterpreterConfig_INIT */
typedef struct { typedef struct {
/* Argument processing */ /* Argument processing */
PyListObject *raw_argv; PyListObject *raw_argv;
@ -686,20 +695,20 @@ Python data types, with unset values indicated by ``NULL``::
} PyMainInterpreterConfig; } PyMainInterpreterConfig;
/* Storing all state as Python object pointers */
The ``PyInterpreterConfig`` struct holds the settings that may vary between The ``PyInterpreterConfig`` struct holds the settings that may vary between
the main interpreter and subinterpreters. For the main interpreter, these the main interpreter and subinterpreters. For the main interpreter, these
settings are automatically populated by ``Py_InitializeMainInterpreter()``. settings are automatically populated by ``Py_ConfigureMainInterpreter()``.
:: ::
/* Note: if changing anything in PyInterpreterConfig, also update
* PyInterpreterConfig_INIT */
typedef struct { typedef struct {
PyBoolObject *is_main_interpreter; /* Easily check for subinterpreters */ PyBoolObject *is_main_interpreter; /* Easily check for subinterpreters */
} PyInterpreterConfig; } PyInterpreterConfig;
As these structs consist solely of object pointers, no explicit initializer
definitions are needed - C99's default initialization of struct memory to zero
is sufficient.
<TBD: did I miss anything?> <TBD: did I miss anything?>
@ -710,7 +719,7 @@ The final step in the initialization process is to actually put the
configuration settings into effect and finish bootstrapping the main configuration settings into effect and finish bootstrapping the main
interpreter up to full operation:: interpreter up to full operation::
int Py_InitializeMainInterpreter(const PyMainInterpreterConfig *config); int Py_ConfigureMainInterpreter(const PyMainInterpreterConfig *config);
Like ``Py_ReadMainInterpreterConfig``, this call will raise an exception and Like ``Py_ReadMainInterpreterConfig``, this call will raise an exception and
report an error return rather than exhibiting fatal errors if a problem is report an error return rather than exhibiting fatal errors if a problem is
@ -724,7 +733,7 @@ After a successful call ``Py_IsInitialized()`` will become true. The caveats
described above for the interpreter during the phase where only the core described above for the interpreter during the phase where only the core
runtime is initialized will no longer hold. runtime is initialized will no longer hold.
Attempting to call ``Py_InitializeMainInterpreter()`` again when Attempting to call ``Py_ConfigureMainInterpreter()`` again when
``Py_IsInitialized()`` is true is an error. ``Py_IsInitialized()`` is true is an error.
However, some metadata related to the ``__main__`` module may still be However, some metadata related to the ``__main__`` module may still be
@ -746,10 +755,10 @@ incomplete:
builtin module builtin module
This function will normally implicitly import site as its final operation This function will normally implicitly import site as its final operation
(after ``Py_IsInitialized()`` is already set). Clearing the (after ``Py_IsInitialized()`` is already set). Setting the
"enable_site_config" flag in the configuration settings will disable this "enable_site_config" flag to ``Py_False`` in the configuration settings will
behaviour, as well as eliminating any side effects on global state if disable this behaviour, as well as eliminating any side effects on global
``import site`` is later explicitly executed in the process. state if ``import site`` is later explicitly executed in the process.
Preparing the main module Preparing the main module
@ -840,16 +849,16 @@ If both ``main_stream`` and ``main_code`` are set, ``RuntimeError`` will
be reported. be reported.
If ``main_stream`` and ``prompt_stream`` are both set, main execution will If ``main_stream`` and ``prompt_stream`` are both set, main execution will
be delegated to a new API:: be delegated to a new internal API::
int _PyRun_InteractiveMain(PyObject *input, PyObject* output); int _PyRun_InteractiveMain(PyObject *input, PyObject* output);
If ``main_stream`` is set and ``prompt_stream`` is NULL, main execution will If ``main_stream`` is set and ``prompt_stream`` is NULL, main execution will
be delegated to a new API:: be delegated to a new internal API::
int _PyRun_StreamInMain(PyObject *input); int _PyRun_StreamInMain(PyObject *input);
If ``main_code`` is set, main execution will be delegated to a new If ``main_code`` is set, main execution will be delegated to a new internal
API:: API::
int _PyRun_CodeInMain(PyCodeObject *code); int _PyRun_CodeInMain(PyCodeObject *code);
@ -865,17 +874,24 @@ Internal Storage of Configuration Data
The interpreter state will be updated to include details of the configuration The interpreter state will be updated to include details of the configuration
settings supplied during initialization by extending the interpreter state settings supplied during initialization by extending the interpreter state
object with an embedded copy of the ``PyCoreConfig``, object with an embedded copy of the ``PyRuntimeConfig``,
``PyMainInterpreterConfig`` and ``PyInterpreterConfig`` structs. ``PyMainInterpreterConfig`` and ``PyInterpreterConfig`` structs.
For debugging purposes, the configuration settings will be exposed as For debugging purposes, the configuration settings will be exposed as
a ``sys._configuration`` simple namespace (similar to ``sys.flags`` and a ``sys._configuration`` simple namespace (similar to ``sys.flags`` and
``sys.implementation``. Field names will match those in the configuration ``sys.implementation``. The attributes will be themselves by simple namespaces
structs, except for ``hash_seed``, which will be deliberately excluded. corresponding to the three levels of configurations setting:
* ``runtime``
* ``main_interpreter``
* ``interpreter``
Field names will match those in the configuration structs, except for
``hash_seed``, which will be deliberately excluded.
An underscored attribute is chosen deliberately, as these configuration An underscored attribute is chosen deliberately, as these configuration
settings are part of the CPython implementation, rather than part of the settings are part of the CPython implementation, rather than part of the
Python language definition. If settings are needed to support Python language definition. If new settings are needed to support
cross-implementation compatibility in the standard library, then those cross-implementation compatibility in the standard library, then those
should be agreed with the other implementations and exposed as new required should be agreed with the other implementations and exposed as new required
attributes on ``sys.implementation``, as described in PEP 421. attributes on ``sys.implementation``, as described in PEP 421.
@ -889,7 +905,7 @@ Creating and Configuring Subinterpreters
As the new configuration settings are stored in the interpreter state, they As the new configuration settings are stored in the interpreter state, they
need to be initialised when a new subinterpreter is created. This turns out need to be initialised when a new subinterpreter is created. This turns out
to be trickier than one might think due to ``PyThreadState_Swap(NULL);`` to be trickier than one might expect due to ``PyThreadState_Swap(NULL);``
(which is fortunately exercised by CPython's own embedding tests, allowing (which is fortunately exercised by CPython's own embedding tests, allowing
this problem to be detected during development). this problem to be detected during development).
@ -898,7 +914,7 @@ add a new API::
Py_InterpreterState *Py_InterpreterState_Main(); Py_InterpreterState *Py_InterpreterState_Main();
This will be a counterpart to Py_InterpreterState_Head(), reporting the This will be a counterpart to ``Py_InterpreterState_Head()``, only reporting the
oldest currently existing interpreter rather than the newest. If oldest currently existing interpreter rather than the newest. If
``Py_NewInterpreter()`` is called from a thread with an existing thread ``Py_NewInterpreter()`` is called from a thread with an existing thread
state, then the interpreter configuration for that thread will be state, then the interpreter configuration for that thread will be
@ -909,7 +925,7 @@ will be used.
While the existing ``Py_InterpreterState_Head()`` API could be used instead, While the existing ``Py_InterpreterState_Head()`` API could be used instead,
that reference changes as subinterpreters are created and destroyed, while that reference changes as subinterpreters are created and destroyed, while
``PyInterpreterState_Main()`` will always refer to the initial interpreter ``PyInterpreterState_Main()`` will always refer to the initial interpreter
state created in ``Py_InitializeCore()``. state created in ``Py_InitializeRuntime()``.
A new constraint is also added to the embedding API: attempting to delete A new constraint is also added to the embedding API: attempting to delete
the main interpreter while subinterpreters still exist will now be a fatal the main interpreter while subinterpreters still exist will now be a fatal
@ -921,10 +937,10 @@ Stable ABI
Most of the APIs proposed in this PEP are excluded from the stable ABI, as Most of the APIs proposed in this PEP are excluded from the stable ABI, as
embedding a Python interpreter involves a much higher degree of coupling embedding a Python interpreter involves a much higher degree of coupling
than merely writing an extension. than merely writing an extension module.
The only newly exposed API that will be part of the stable ABI is the The only newly exposed API that will be part of the stable ABI is the
``Py_IsCoreInitialized()`` query. ``Py_IsRuntimeInitialized()`` query.
Build time configuration Build time configuration
@ -941,14 +957,14 @@ Backwards Compatibility
Backwards compatibility will be preserved primarily by ensuring that Backwards compatibility will be preserved primarily by ensuring that
``Py_ReadMainInterpreterConfig()`` interrogates all the previously defined ``Py_ReadMainInterpreterConfig()`` interrogates all the previously defined
configuration settings stored in global variables and environment variables, configuration settings stored in global variables and environment variables,
and that ``Py_InitializeMainInterpreter()`` writes affected settings back to and that ``Py_ConfigureMainInterpreter()`` writes affected settings back to
the relevant locations. the relevant locations.
One acknowledged incompatibility is that some environment variables which One acknowledged incompatibility is that some environment variables which
are currently read lazily may instead be read once during interpreter are currently read lazily may instead be read once during interpreter
initialization. As the PEP matures, these will be discussed in more detail initialization. As the reference implementation matures, these will be
on a case by case basis. The environment variables which are currently discussed in more detail on a case by case basis. The environment variables
known to be looked up dynamically are: which are currently known to be looked up dynamically are:
* ``PYTHONCASEOK``: writing to ``os.environ['PYTHONCASEOK']`` will no longer * ``PYTHONCASEOK``: writing to ``os.environ['PYTHONCASEOK']`` will no longer
dynamically alter the interpreter's handling of filename case differences dynamically alter the interpreter's handling of filename case differences
@ -960,10 +976,10 @@ The ``Py_Initialize()`` style of initialization will continue to be
supported. It will use (at least some elements of) the new API supported. It will use (at least some elements of) the new API
internally, but will continue to exhibit the same behaviour as it internally, but will continue to exhibit the same behaviour as it
does today, ensuring that ``sys.argv`` is not populated until a subsequent does today, ensuring that ``sys.argv`` is not populated until a subsequent
``PySys_SetArgv`` call. All APIs that currently support being called ``PySys_SetArgv`` call (TBC). All APIs that currently support being called
prior to ``Py_Initialize()`` will prior to ``Py_Initialize()`` will
continue to do so, and will also support being called prior to continue to do so, and will also support being called prior to
``Py_InitializeCore()``. ``Py_InitializeRuntime()``.
To minimise unnecessary code churn, and to ensure the backwards compatibility To minimise unnecessary code churn, and to ensure the backwards compatibility
is well tested, the main CPython executable may continue to use some elements is well tested, the main CPython executable may continue to use some elements
@ -1012,43 +1028,30 @@ Open Questions
============== ==============
* Error details for ``Py_ReadMainInterpreterConfig`` and * Error details for ``Py_ReadMainInterpreterConfig`` and
``Py_InitializeMainInterpreter`` (these should become clearer as the ``Py_ConfigureMainInterpreter`` (these should become clearer as the
implementation progresses) implementation progresses)
* Is initialisation of the ``PyMainInterpreterConfig`` struct too unwieldy to
be maintainable? Would a Python dictionary be a better choice, despite
being harder to work with from C code? Can we upgrade to requiring a C99
compatible compiler?
* Would it be better to manage the flag variables in ``PyMainInterpreterConfig``
as Python integers or as "negative means false, positive means true, zero
means not set" so the struct can be initialized with a simple
``memset(&config, 0, sizeof(*config))``, eliminating the need to update
both PyMainInterpreterConfig and PyMainInterpreterConfig_INIT when adding
new fields?
* The name of the new system Python executable is a bikeshed waiting to be
painted. The 4 options considered so far are ``spython``, ``pysystem``,
``python-minimal`` and `system-python``. The PEP text reflects my current
preferred choice (``system-python``).
Implementation Implementation
============== ==============
A reference implementation is available as a feature branch in my BitBucket A reference implementation is available as a feature branch in Nick Coghlan's
sandbox [2_]. CPython sandbox on BitBucket [2_].
It implements the ``_Py_InitializeCore`` and ``_Py_InitializeMainInterpreter`` It implements the ``_Py_InitializeRuntime`` and ``_Py_ConfigureMainInterpreter``
aspects of the refactoring as a private API. All ``_PyCoreConfig`` settings aspects of the refactoring as a private API. All ``_PyRuntimeConfig`` settings
are included, but currently only the "install signal handlers" setting is are included, but currently only the "install signal handlers" setting is
implemented for the main interpreter configuration. implemented for the main interpreter configuration.
The currently available interfaces in the reference implementation: The feature branch is a couple of iterations behind the API design in the PEP
so the exact interfaces currently available are:
* ``_Py_IsCoreInitialized`` * ``_Py_IsCoreInitialized`` (rename pending: ``_Py_IsRuntimeInitialized``)
* ``_Py_InitializeCore`` * ``_Py_InitializeCore`` (rename pending: ``_Py_InitializeRuntime``)
* ``_PyCoreConfig`` * ``_PyCoreConfig`` (rename pending: ``_Py_RuntimeConfig``)
* ``_PyCoreConfig_INIT`` * ``_PyCoreConfig_INIT`` (rename pending: ``_Py_RuntimeConfig_INIT``)
* ``_Py_ReadHashSeed`` * ``_Py_ReadHashSeed``
* ``_Py_InitializeMainInterpreter`` * ``_Py_InitializeMainInterpreter`` (rename pending: ``_Py_ConfigureMainInterpreter``)
* ``_PyMainInterpreterConfig`` * ``_PyMainInterpreterConfig``
* ``_PyMainInterpreterConfig_INIT`` * ``_PyMainInterpreterConfig_INIT``
* ``_Py_ReadMainInterpreterConfig`` * ``_Py_ReadMainInterpreterConfig``
@ -1062,7 +1065,7 @@ a fairly ad hoc fashion over the past 20+ years, leading to a rather
inconsistent interface with varying levels of documentation. inconsistent interface with varying levels of documentation.
(Note: some of the info below could probably be cleaned up and added to the (Note: some of the info below could probably be cleaned up and added to the
C API documentation for at least 3.3. - it's all CPython specific, so it C API documentation for 3.6 - it's all CPython specific, so it
doesn't belong in the language reference) doesn't belong in the language reference)