PEP 432: Update to better reflect status quo (GH-904)

pep-0432.txt

@@ -21,11 +21,24 @@ CPython's startup behaviour when creating an alternate executable or
 embedding it as a Python execution engine inside a larger application.
 
 When implementation of this proposal is completed, interpreter startup will
-consist of two clearly distinct and independently configurable phases:
+consist of three clearly distinct and independently configurable phases:
+
+* Python core runtime preconfiguration
+
+  * setting up memory management
+  * determining the encodings used for system interfaces (including settings
+    passed in for later configuration phase)
+
+* Python core runtime initialization
+
+  * ensuring C API is ready for use
+  * ensuring builtin and frozen modules are accessible
 
-* Python runtime initialization
 * Main interpreter configuration
 
+  * ensuring external modules are accessible
+  * (Note: the name of this phase is quite likely to change)
+
 Changes are also proposed that impact main module execution and subinterpreter
 initialization.
 
@@ -38,11 +51,16 @@ Proposal
 ========
 
 This PEP proposes that initialization of the CPython runtime be split into
-two clearly distinct phases:
+three clearly distinct phases:
 
+* core runtime preconfiguration
 * core runtime initialization
 * main interpreter configuration
 
+(Earlier versions proposed only two phases, but experience with attempting to
+implement the PEP as an internal CPython refactoring showed that at least 3
+phases are needed to get clear separation of concerns)
+
 The proposed design also has significant implications for:
 
 * main module execution
@@ -157,14 +175,14 @@ dominated by IO operations. However, to monitor the impact of any changes,
 a simple benchmark can be used to check how long it takes to start and then
 tear down the interpreter::
 
-   python3 -m timeit -s "from subprocess import call" "call(['./python', '-c', 'pass'])"
+   python3 -m timeit -s "from subprocess import call" "call(['./python', '-Sc', 'pass'])"
 
-Current numbers on my system for Python 3.7 (using the 3.5
+Current numbers on my system for Python 3.7 (as built by the Fedora project)::
-subprocess and timeit modules to execute the check, all with non-debug
-builds)::
 
-    $ python3 -m timeit -s "from subprocess import call" "call(['./python', '-c', 'pass'])"
+    $ python3 -m timeit -s "from subprocess import call" "call(['python3', '-Sc', 'pass'])"
-    100 loops, best of 3: 13.9 msec per loop
+    50 loops, best of 5: 6.48 msec per loop
+
+(TODO: run this microbenchmark with perf rather than the stdlib timeit)
 
 This PEP is not expected to have any significant effect on the startup time,
 as it is aimed primarily at *reordering* the existing initialization
@@ -278,47 +296,37 @@ implementation of the import system in Python 3.3, there is no clear way to
 structure a draft implementation that won't be prone to the kinds of merge
 conflicts that afflicted the original attempt.
 
-Accordingly, the implementation strategy now being proposed is to implement this
+Accordingly, the implementation strategy was revised to instead first implement
-refactoring as a private API for CPython 3.7, before exposing the new functions
+this refactoring as a private API for CPython 3.7, and then review the viability
-and structures as public API elements in CPython 3.8.
+of exposing the new functions and structures as public API elements in CPython
+3.8.
 
-The affected APIs with a leading underscore, as they would be named in CPython
+After the initial merge, Victor Stinner then proceeded to actually migrate
-3.7:
+settings to the new structure in order to successfully implement the PEP 540
+UTF-8 mode changes (which required the ability to track all settings that had
+previously been decoded with the locale encoding, and decode them again using
+UTF-8 instead). Eric Snow also migrated a number of internal subsystems over as
+part of making the subinterpreter feature more robust.
 
-* ``_Py_IsRuntimeInitialized``
+That work showed that the detailed design currently proposed in this PEP has a
-* ``_Py_InitializeRuntime``
+range of practical issues, so it's currently expected to remain a private API
-* ``_PyRuntimeConfig``
+for CPython 3.8, with the possibility of making it public and stable in CPython
-* ``_PyRuntimeConfig_INIT``
+3.9.
-* ``_Py_ReadHashSeed``
-* ``_Py_ConfigureMainInterpreter``
-* ``_PyMainInterpreterConfig``
-* ``_PyInterpreterConfig``
-* ``_Py_ReadMainInterpreterConfig``
-* ``_PyRun_PrepareMain``
-* ``_PyRun_ExecMain``
-* ``_Py_InterpreterState_Main``
-
-New APIs described in the rest of the PEP with a leading underscore are
-intended to be retained permanently as private CPython implementation details.
-
-The principle benefit of this approach is allowing the refactoring to adopt the
-new configuration structures to be handled on a setting by setting basis
-over the course of the 3.7 and 3.8 development cycles, rather than having to
-migrate them in a single monolithic change. It also means any new settings
-introduced for these releases can be handled using the new approach from the
-start, even if some existing settings have yet to be migrated.
-
-If all existing settings are successfully migrated to the new initialization
-model in time for the 3.7.0a4 release in December 2017, then the proposal would
-be to make the APIs public for the 3.7.0b1 release in January 2018, rather
-than waiting for 3.8.
 
 
 Design Details
 ==============
 
-(Note: details here are still very much in flux, but preliminary feedback
+.. note::
-is appreciated anyway)
+
+    The API details here are still very much in flux, as the private refactoring
+    work has shown that these specific API designs aren't really viable in
+    practice. The header files that show the current state of the private API
+    are mainly:
+
+    * https://github.com/python/cpython/blob/master/Include/cpython/coreconfig.h
+    * https://github.com/python/cpython/blob/master/Include/cpython/pystate.h
+    * https://github.com/python/cpython/blob/master/Include/cpython/pylifecycle.h
 
 The main theme of this proposal is to initialize the core language runtime
 and create a partially initialized interpreter state for the main interpreter
@@ -1035,37 +1043,20 @@ Open Questions
 Implementation
 ==============
 
-A reference implementation is available as a feature branch in Nick Coghlan's
+The reference implementation is being developed as a private API refactoring
-CPython sandbox on BitBucket [2_].
+within the CPython reference interpreter (as attempting to maintain it as an
-
+independent project proved impractical).
-It implements the ``_Py_InitializeRuntime`` and ``_Py_ConfigureMainInterpreter``
-aspects of the refactoring as a private API. All ``_PyRuntimeConfig`` settings
-are included, but currently only the "install signal handlers" setting is
-implemented for the main interpreter configuration.
-
-The feature branch is a couple of iterations behind the API design in the PEP
-so the exact interfaces currently available are:
-
-* ``_Py_IsCoreInitialized`` (rename pending: ``_Py_IsRuntimeInitialized``)
-* ``_Py_InitializeCore`` (rename pending: ``_Py_InitializeRuntime``)
-* ``_PyCoreConfig`` (rename pending: ``_Py_RuntimeConfig``)
-* ``_PyCoreConfig_INIT`` (rename pending: ``_Py_RuntimeConfig_INIT``)
-* ``_Py_ReadHashSeed``
-* ``_Py_InitializeMainInterpreter`` (rename pending: ``_Py_ConfigureMainInterpreter``)
-* ``_PyMainInterpreterConfig``
-* ``_PyMainInterpreterConfig_INIT``
-* ``_Py_ReadMainInterpreterConfig``
 
 
-The Status Quo
+The Status Quo (as of Python 3.6)
-==============
+=================================
 
 The current mechanisms for configuring the interpreter have accumulated in
 a fairly ad hoc fashion over the past 20+ years, leading to a rather
 inconsistent interface with varying levels of documentation.
 
 (Note: some of the info below could probably be cleaned up and added to the
-C API documentation for 3.6 - it's all CPython specific, so it
+C API documentation for 3.x - it's all CPython specific, so it
 doesn't belong in the language reference)