From 1694af081cff98b6abe9f3bc243303faca83ce55 Mon Sep 17 00:00:00 2001 From: Nick Coghlan Date: Sun, 3 Mar 2019 12:17:22 +1000 Subject: [PATCH] PEP 432: Update to better reflect status quo (GH-904) --- pep-0432.txt | 123 ++++++++++++++++++++++++--------------------------- 1 file changed, 57 insertions(+), 66 deletions(-) diff --git a/pep-0432.txt b/pep-0432.txt index 1c1d6cd53..af4661e86 100644 --- a/pep-0432.txt +++ b/pep-0432.txt @@ -21,11 +21,24 @@ CPython's startup behaviour when creating an alternate executable or embedding it as a Python execution engine inside a larger application. When implementation of this proposal is completed, interpreter startup will -consist of two clearly distinct and independently configurable phases: +consist of three clearly distinct and independently configurable phases: + +* Python core runtime preconfiguration + + * setting up memory management + * determining the encodings used for system interfaces (including settings + passed in for later configuration phase) + +* Python core runtime initialization + + * ensuring C API is ready for use + * ensuring builtin and frozen modules are accessible -* Python runtime initialization * Main interpreter configuration + * ensuring external modules are accessible + * (Note: the name of this phase is quite likely to change) + Changes are also proposed that impact main module execution and subinterpreter initialization. @@ -38,11 +51,16 @@ Proposal ======== This PEP proposes that initialization of the CPython runtime be split into -two clearly distinct phases: +three clearly distinct phases: +* core runtime preconfiguration * core runtime initialization * main interpreter configuration +(Earlier versions proposed only two phases, but experience with attempting to +implement the PEP as an internal CPython refactoring showed that at least 3 +phases are needed to get clear separation of concerns) + The proposed design also has significant implications for: * main module execution @@ -157,14 +175,14 @@ dominated by IO operations. However, to monitor the impact of any changes, a simple benchmark can be used to check how long it takes to start and then tear down the interpreter:: - python3 -m timeit -s "from subprocess import call" "call(['./python', '-c', 'pass'])" + python3 -m timeit -s "from subprocess import call" "call(['./python', '-Sc', 'pass'])" -Current numbers on my system for Python 3.7 (using the 3.5 -subprocess and timeit modules to execute the check, all with non-debug -builds):: +Current numbers on my system for Python 3.7 (as built by the Fedora project):: - $ python3 -m timeit -s "from subprocess import call" "call(['./python', '-c', 'pass'])" - 100 loops, best of 3: 13.9 msec per loop + $ python3 -m timeit -s "from subprocess import call" "call(['python3', '-Sc', 'pass'])" + 50 loops, best of 5: 6.48 msec per loop + +(TODO: run this microbenchmark with perf rather than the stdlib timeit) This PEP is not expected to have any significant effect on the startup time, as it is aimed primarily at *reordering* the existing initialization @@ -278,47 +296,37 @@ implementation of the import system in Python 3.3, there is no clear way to structure a draft implementation that won't be prone to the kinds of merge conflicts that afflicted the original attempt. -Accordingly, the implementation strategy now being proposed is to implement this -refactoring as a private API for CPython 3.7, before exposing the new functions -and structures as public API elements in CPython 3.8. +Accordingly, the implementation strategy was revised to instead first implement +this refactoring as a private API for CPython 3.7, and then review the viability +of exposing the new functions and structures as public API elements in CPython +3.8. -The affected APIs with a leading underscore, as they would be named in CPython -3.7: +After the initial merge, Victor Stinner then proceeded to actually migrate +settings to the new structure in order to successfully implement the PEP 540 +UTF-8 mode changes (which required the ability to track all settings that had +previously been decoded with the locale encoding, and decode them again using +UTF-8 instead). Eric Snow also migrated a number of internal subsystems over as +part of making the subinterpreter feature more robust. -* ``_Py_IsRuntimeInitialized`` -* ``_Py_InitializeRuntime`` -* ``_PyRuntimeConfig`` -* ``_PyRuntimeConfig_INIT`` -* ``_Py_ReadHashSeed`` -* ``_Py_ConfigureMainInterpreter`` -* ``_PyMainInterpreterConfig`` -* ``_PyInterpreterConfig`` -* ``_Py_ReadMainInterpreterConfig`` -* ``_PyRun_PrepareMain`` -* ``_PyRun_ExecMain`` -* ``_Py_InterpreterState_Main`` - -New APIs described in the rest of the PEP with a leading underscore are -intended to be retained permanently as private CPython implementation details. - -The principle benefit of this approach is allowing the refactoring to adopt the -new configuration structures to be handled on a setting by setting basis -over the course of the 3.7 and 3.8 development cycles, rather than having to -migrate them in a single monolithic change. It also means any new settings -introduced for these releases can be handled using the new approach from the -start, even if some existing settings have yet to be migrated. - -If all existing settings are successfully migrated to the new initialization -model in time for the 3.7.0a4 release in December 2017, then the proposal would -be to make the APIs public for the 3.7.0b1 release in January 2018, rather -than waiting for 3.8. +That work showed that the detailed design currently proposed in this PEP has a +range of practical issues, so it's currently expected to remain a private API +for CPython 3.8, with the possibility of making it public and stable in CPython +3.9. Design Details ============== -(Note: details here are still very much in flux, but preliminary feedback -is appreciated anyway) +.. note:: + + The API details here are still very much in flux, as the private refactoring + work has shown that these specific API designs aren't really viable in + practice. The header files that show the current state of the private API + are mainly: + + * https://github.com/python/cpython/blob/master/Include/cpython/coreconfig.h + * https://github.com/python/cpython/blob/master/Include/cpython/pystate.h + * https://github.com/python/cpython/blob/master/Include/cpython/pylifecycle.h The main theme of this proposal is to initialize the core language runtime and create a partially initialized interpreter state for the main interpreter @@ -1035,37 +1043,20 @@ Open Questions Implementation ============== -A reference implementation is available as a feature branch in Nick Coghlan's -CPython sandbox on BitBucket [2_]. - -It implements the ``_Py_InitializeRuntime`` and ``_Py_ConfigureMainInterpreter`` -aspects of the refactoring as a private API. All ``_PyRuntimeConfig`` settings -are included, but currently only the "install signal handlers" setting is -implemented for the main interpreter configuration. - -The feature branch is a couple of iterations behind the API design in the PEP -so the exact interfaces currently available are: - -* ``_Py_IsCoreInitialized`` (rename pending: ``_Py_IsRuntimeInitialized``) -* ``_Py_InitializeCore`` (rename pending: ``_Py_InitializeRuntime``) -* ``_PyCoreConfig`` (rename pending: ``_Py_RuntimeConfig``) -* ``_PyCoreConfig_INIT`` (rename pending: ``_Py_RuntimeConfig_INIT``) -* ``_Py_ReadHashSeed`` -* ``_Py_InitializeMainInterpreter`` (rename pending: ``_Py_ConfigureMainInterpreter``) -* ``_PyMainInterpreterConfig`` -* ``_PyMainInterpreterConfig_INIT`` -* ``_Py_ReadMainInterpreterConfig`` +The reference implementation is being developed as a private API refactoring +within the CPython reference interpreter (as attempting to maintain it as an +independent project proved impractical). -The Status Quo -============== +The Status Quo (as of Python 3.6) +================================= The current mechanisms for configuring the interpreter have accumulated in a fairly ad hoc fashion over the past 20+ years, leading to a rather inconsistent interface with varying levels of documentation. (Note: some of the info below could probably be cleaned up and added to the -C API documentation for 3.6 - it's all CPython specific, so it +C API documentation for 3.x - it's all CPython specific, so it doesn't belong in the language reference)