PEP 489 updates from Petr Viktorin.

- Clarify that not all problems with PEP 3121 are solved
- Add a pseudo-code overview
- Clarify that PyModuleDef_Init does very little
- Say that the execution phase isn't done for non-module objects
- Link to docs on support for multiple interpreters
- Reword text about lack of finder for multi-module extensions
- Mention changes in imp and _imp modules
- Remove stale footnote
This commit is contained in:
Berker Peksag 2015-05-20 14:27:55 +03:00
parent 9456a3b7c6
commit 5b952e1713
1 changed files with 187 additions and 20 deletions

View File

@ -20,10 +20,10 @@ Abstract
This PEP proposes a redesign of the way in which built-in and extension modules This PEP proposes a redesign of the way in which built-in and extension modules
interact with the import machinery. This was last revised for Python 3.0 in PEP interact with the import machinery. This was last revised for Python 3.0 in PEP
3121, but did not solve all problems at the time. The goal is to solve them 3121, but did not solve all problems at the time. The goal is to solve
by bringing extension modules closer to the way Python modules behave; import-related problems by bringing extension modules closer to the way Python
specifically to hook into the ModuleSpec-based loading mechanism modules behave; specifically to hook into the ModuleSpec-based loading
introduced in PEP 451. mechanism introduced in PEP 451.
This proposal draws inspiration from PyType_Spec of PEP 384 to allow extension This proposal draws inspiration from PyType_Spec of PEP 384 to allow extension
authors to only define features they need, and to allow future additions authors to only define features they need, and to allow future additions
@ -40,6 +40,10 @@ when using the new API.
The proposal also allows extension modules with non-ASCII names. The proposal also allows extension modules with non-ASCII names.
Not all problems tackled in PEP 3121 are solved in this proposal.
In particular, problems with run-time module lookup (PyState_FindModule)
are left to a future PEP.
Motivation Motivation
========== ==========
@ -161,7 +165,9 @@ not be used during importing; the module name will be taken from the ModuleSpec.
To prevent crashes when the module is loaded in older versions of Python, To prevent crashes when the module is loaded in older versions of Python,
the PyModuleDef object must be initialized using the newly added the PyModuleDef object must be initialized using the newly added
PyModuleDef_Init function. PyModuleDef_Init function. This sets the object type (which cannot be done
statically on certain compilers), refcount, and internal bookkeeping data
(m_index).
For example, an extension module "example" would be exported as:: For example, an extension module "example" would be exported as::
static PyModuleDef example_def = {...} static PyModuleDef example_def = {...}
@ -175,6 +181,149 @@ For example, an extension module "example" would be exported as::
The PyModuleDef object must be available for the lifetime of the module created The PyModuleDef object must be available for the lifetime of the module created
from it usually, it will be declared statically. from it usually, it will be declared statically.
Pseudo-code Overview
--------------------
Here is an overview of how the modified importers will operate.
Details such as logging or handling of errors and invalid states
are left out, and C code is presented with a concise Python-like syntax.
The framework that calls the importers is explained in PEP 451
[#pep-0451-loading]_.
::
importlib/_bootstrap.py:
class BuiltinImporter:
def create_module(self, spec):
module = _imp.create_builtin(spec)
def exec_module(self, module):
_imp.exec_dynamic(module)
def load_module(self, name):
# use a backwards compatibility shim
_load_module_shim(self, name)
importlib/_bootstrap_external.py:
class ExtensionFileLoader:
def create_module(self, spec):
module = _imp.create_dynamic(spec)
def exec_module(self, module):
_imp.exec_dynamic(module)
def load_module(self, name):
# use a backwards compatibility shim
_load_module_shim(self, name)
Python/import.c (the _imp module):
def create_dynamic(spec):
name = spec.name
path = spec.origin
# Find an already loaded module that used single-phase init.
# For multi-phase initialization, mod is NULL, so a new module
# is always created.
mod = _PyImport_FindExtensionObject(name, name)
if mod:
return mod
return _PyImport_LoadDynamicModuleWithSpec(spec)
def exec_dynamic(module):
if not isinstance(module, types.ModuleType):
# non-modules are skipped -- PyModule_GetDef fails on them
return
def = PyModule_GetDef(module)
state = PyModule_GetState(module)
if state is NULL:
PyModule_ExecDef(module, def)
def create_builtin(spec):
name = spec.name
# Find an already loaded module that used single-phase init.
# For multi-phase initialization, mod is NULL, so a new module
# is always created.
mod = _PyImport_FindExtensionObject(name, name)
if mod:
return mod
for initname, initfunc in PyImport_Inittab:
if name == initname:
m = initfunc()
if isinstance(m, PyModuleDef):
def = m
return PyModule_FromDefAndSpec(def, spec)
else:
# fall back to single-phase initialization
module = m
_PyImport_FixupExtensionObject(module, name, name)
return module
Python/importdl.c:
def _PyImport_LoadDynamicModuleWithSpec(spec):
path = spec.origin
package, dot, name = spec.name.rpartition('.')
# see the "Non-ASCII module names" section for export_hook_name
hook_name = export_hook_name(name)
# call platform-specific function for loading exported function
# from shared library
exportfunc = _find_shared_funcptr(hook_name, path)
m = exportfunc()
if isinstance(m, PyModuleDef):
def = m
return PyModule_FromDefAndSpec(def, spec)
module = m
# fall back to single-phase initialization
....
Objects/moduleobject.c:
def PyModule_FromDefAndSpec(def, spec):
name = spec.name
create = None
for slot, value in def.m_slots:
if slot == Py_mod_create:
create = value
if create:
m = create(spec, def)
else:
m = PyModule_New(name)
if isinstance(m, types.ModuleType):
m.md_state = None
m.md_def = def
if def.m_methods:
PyModule_AddFunctions(m, def.m_methods)
if def.m_doc:
PyModule_SetDocString(m, def.m_doc)
def PyModule_ExecDef(module, def):
if isinstance(module, types.module_type):
if module.md_state is NULL:
# allocate a block of zeroed-out memory
module.md_state = _alloc(module.md_size)
if def.m_slots is NULL:
return
for slot, value in def.m_slots:
if slot == Py_mod_exec:
value(module)
Module Creation Phase Module Creation Phase
--------------------- ---------------------
@ -223,8 +372,10 @@ Post-creation steps
If the Py_mod_create function returns an instance of types.ModuleType If the Py_mod_create function returns an instance of types.ModuleType
or a subclass (or if a Py_mod_create slot is not present), the import or a subclass (or if a Py_mod_create slot is not present), the import
machinery will associate the PyModuleDef with the module, making it accessible machinery will associate the PyModuleDef with the module.
to PyModule_GetDef, and enabling the m_traverse, m_clear and m_free hooks. This also makes the PyModuleDef accessible to execution phase, the
PyModule_GetDef function, and garbage collection routines (traverse,
clear, free).
If the Py_mod_create function does not return a module subclass, then m_size If the Py_mod_create function does not return a module subclass, then m_size
must be 0, and m_traverse, m_clear and m_free must all be NULL. must be 0, and m_traverse, m_clear and m_free must all be NULL.
@ -244,6 +395,10 @@ Module execution -- that is, the implementation of
ExecutionLoader.exec_module -- is governed by "execution slots". ExecutionLoader.exec_module -- is governed by "execution slots".
This PEP only adds one, Py_mod_exec, but others may be added in the future. This PEP only adds one, Py_mod_exec, but others may be added in the future.
The execution phase is done on the PyModuleDef associated with the module
object. For objects that are not a subclass of PyModule_Type (for which
PyModule_GetDef whoud fail), the execution phase is skipped.
Execution slots may be specified multiple times, and are processed in the order Execution slots may be specified multiple times, and are processed in the order
they appear in the slots array. they appear in the slots array.
When using the default import machinery, they are processed after When using the default import machinery, they are processed after
@ -289,12 +444,13 @@ than a PyModuleDef object.
In this case, the PyInit hook implements the creation phase, and the execution In this case, the PyInit hook implements the creation phase, and the execution
phase is a no-op. phase is a no-op.
Modules that need to work unchanged on older versions of Python should not Modules that need to work unchanged on older versions of Python should stick to
use multi-phase initialization, because the benefits it brings can't be single-phase initialization, because the benefits it brings can't be
back-ported. back-ported.
Nevertheless, here is an example of a module that supports multi-phase Here is an example of a module that supports multi-phase initialization,
initialization, and falls back to single-phase when compiled for an older and falls back to single-phase when compiled for an older version of CPython.
version of CPython:: It is included mainly as an illustration of the changes needed to enable
multi-phase init::
#include <Python.h> #include <Python.h>
@ -359,7 +515,8 @@ Subinterpreters and Interpreter Reloading
----------------------------------------- -----------------------------------------
Extensions using the new initialization scheme are expected to support Extensions using the new initialization scheme are expected to support
subinterpreters and multiple Py_Initialize/Py_Finalize cycles correctly. subinterpreters and multiple Py_Initialize/Py_Finalize cycles correctly,
avoiding the issues mentioned in Python documentation [#subinterpreter-docs]_.
The mechanism is designed to make this easy, but care is still required The mechanism is designed to make this easy, but care is still required
on the part of the extension author. on the part of the extension author.
No user-defined functions, methods, or instances may leak to different No user-defined functions, methods, or instances may leak to different
@ -503,10 +660,10 @@ export additional PyInit* symbols besides the one that corresponds
to the library's filename. to the library's filename.
Note that this mechanism can currently only be used to *load* extra modules, Note that this mechanism can currently only be used to *load* extra modules,
but not to *find* them. but not to *find* them. (This is a limitation of the loader mechanism,
which this PEP does not try to modify.)
Given the filesystem location of a shared library and a module name, To work around the lack of a suitable finder, code like the following
a module may be loaded with:: can be used::
import importlib.machinery import importlib.machinery
import importlib.util import importlib.util
@ -562,6 +719,13 @@ New structures:
PyModuleDef.m_reload changes to PyModuleDef.m_slots. PyModuleDef.m_reload changes to PyModuleDef.m_slots.
The internal ``_imp`` module will have backwards incompatible changes:
``create_builtin``, ``create_dynamic``, and ``exec_dynamic`` will be added;
``init_builtin``, ``load_dynamic`` will be removed.
The undocumented functions ``imp.load_dynamic`` and ``imp.init_builtin`` will
be replaced by backwards-compatible shims.
Possible Future Extensions Possible Future Extensions
========================== ==========================
@ -632,9 +796,6 @@ exporting a definition, yielded a much simpler solution.
References References
========== ==========
.. [#lazy_import_concerns]
https://mail.python.org/pipermail/python-dev/2013-August/128129.html
.. [#pep-0451-attributes] .. [#pep-0451-attributes]
https://www.python.org/dev/peps/pep-0451/#attributes https://www.python.org/dev/peps/pep-0451/#attributes
@ -656,6 +817,12 @@ References
.. [#findmodule-discussion] .. [#findmodule-discussion]
https://mail.python.org/pipermail/import-sig/2015-April/000959.html https://mail.python.org/pipermail/import-sig/2015-April/000959.html
.. [#pep-0451-loading]
https://www.python.org/dev/peps/pep-0451/#how-loading-will-work]
.. [#subinterpreter-docs]
https://docs.python.org/3/c-api/init.html#sub-interpreter-support
Copyright Copyright
========= =========